Compress, Serve, and Preserve Behavior

Every serving change creates a candidate

MD-15 contains a workload plan, not a benchmark result. The package remains blocked because no adapter exists and the training/serving templates conflict. Chapter 16 therefore performs a deterministic bake-off with synthetic measurements only. modelExecuted, benchmarkExecuted, and artifactCreated remain false.

Freeze the reference before changing anything: base, tokenizer, template, adapter/merge state, schema, generation defaults, precision, quantization, runtime, kernels, hardware class, scheduler, workload, evaluator versions, and raw protected cases. A change to any field creates a new candidate identity.

Candidates may vary weight or cache quantization, calibration data, kernel/runtime, batching, cache policy, speculative decoding, or routing. Change one surface at a time when causal attribution matters. A compound variant needs an ablation plan or it stays an opaque bundle.

Quantization can reduce memory or compute cost, but quality and speed effects depend on method, model, calibration, hardware, and runtime. [CLM-046] Nominal bit width does not determine loaded bytes, kernel support, latency, or task behavior. Scales, metadata, unquantized layers, adapters, cache precision, conversion format, and runtime copies remain in the ledger.

AWQ, GPTQ, QLoRA, bitsandbytes, and TensorRT-LLM describe bounded methods or implementations. Their published perplexity, memory, or speed results do not establish Mosaic behavior. Calibration examples must be authorized, representative of the intended activation distribution, separate from evaluation, and versioned.

LLME-CASE-012 contributes only the lesson that quantization needs method-specific calibration and behavior replay. Its reported model, benchmark, hardware, and runtime results are non-portable. LLME-CASE-011 contributes only the serving lesson that KV allocation and scheduling affect a measured workload; its throughput is not a Mosaic capacity estimate.

Long description

A colorful realistic three-dimensional frontier table plots several candidate capsules against behavior, tail, throughput, memory, and cost gauges. Two fast points sit behind protected-regression barriers.

Verify compatibility before comparing speed

Each candidate must pass the MD-14 identity gate: base and configuration digests, tokenizer vocabulary and special tokens, template golden serialization, adapter binding, dtype and quantization format, runtime support, schema, stops, and generation defaults. Unsupported kernels do not silently fall back; the observed kernel path and fallback state are recorded.

The first failure candidate reports lower weight-memory units but uses an unsupported kernel and therefore gains no synthetic speed. It is not promoted on memory alone. The second doubles synthetic throughput but increases citation-schema failures on long Gujarati-English inputs. Aggregate target score looks stable because short English cases dominate. The protected slice rejects it.

Replay target shorthand, citation retention, neutral instruction, English, Gujarati, Gujarati-English, abstention, conflict, untrusted instruction, unauthorized source, no external effect, schema validity, and long-context cases. Preserve raw outputs and pass-to-fail transitions. Perplexity and generic benchmark scores remain diagnostic instruments, not acceptance evidence.

Paged KV management, batching, caching, and speculative decoding can improve serving efficiency under bounded workloads. [CLM-047] Their benefits may reverse with model pairing, acceptance rate, sequence distribution, concurrency, hardware, or implementation. Speculative decoding must preserve the intended distribution under the exact algorithmic assumptions and still pass application behavior.

Measure prompt/output throughput, request throughput, TTFT, inter-token latency, end-to-end p50/p95/p99, errors, memory high-water mark, cache occupancy, batch composition, and cost units. Separate offered load from completed work. Report warm/cold state and sample counts.

Build the frontier after hard gates

Apply compatibility and protected-behavior gates before resource ranking. A candidate with a forbidden language, citation, abstention, schema, or control transition is not on the feasible frontier. Among survivors, remove dominated candidates: another candidate is at least as good on all decision dimensions and better on one.

Serving configuration must be selected on a measured behavior-resource frontier including protected segments and latency tails. [CLM-048] A frontier expires when package, workload, hardware, runtime, traffic, evaluator, or behavior contract changes. It is evidence for one decision, not a permanent leaderboard.

The deterministic companion compares reference-full, quant-fast-unsafe, quant-memory-only, and alternate-runtime-balanced. Reference-full remains the fallback. The unsafe fast point fails Gujarati citation/schema. The memory-only point is dominated because its unsupported kernel provides no latency or throughput gain. The balanced point is only a simulated frontier candidate; the blocked package prevents release.

Degrade deliberately under pressure

Graceful degradation is a predeclared state machine, not arbitrary quality loss:

1. use the verified full reference while budgets pass;
2. use a behavior-qualified alternate candidate only if its identity matches;
3. reduce optional context within contract and expose that state;
4. queue or rate-limit with an honest delay;
5. abstain when evidence, schema, or latency validity cannot be preserved;
6. fall back to the prior managed/no-tune path;
7. stop when no qualified path remains.

Never silently drop authoritative evidence, truncate required citations, switch language handling, disable controls, or route to an unqualified model. A fallback response states its limitations without leaking system details. Recovery requires replaying the affected cases before leaving degradation.

Long description

A vivid three-dimensional staircase descends through explicit service states. Guard rails prevent a shortcut from full service to an unsafe fast model.

Platform selects deployment, capacity, autoscaling, and operational routing. LLM engineering supplies the behavior-resource evidence and degradation contract. Product/domain/language, privacy, security, and release authorities retain their decisions. Managed optimization exposes only provider features and measured external behavior; hidden kernels and hardware are not invented.

MD-15 closes with a deterministic frontier, rejection ledger, and degradation plan. No candidate is released.

Practice

Reconstruct the frontier, reject the fast slice-regressing candidate and the dominated memory-only candidate, then walk a burst through the degradation ladder. Pass when compatibility, target, retention, language, controls, service tails, and fallback evidence determine every transition.