Adapt Efficiently With PEFT
Evaluate low-rank adapters as traceable behavior-and-resource candidates whose base, tokenizer, template, runtime, quantization, and merge state must remain compatible.
An adapter is a dependency, not a tiny model
Chapter 10 forwarded a simulated SFT comparator, not a trained checkpoint. The real MD-12 run remains blocked. Chapter 11 adds equally explicit PEFT simulations so we can ask the engineering question: if two candidates meet the behavior contract, what resource and lifecycle evidence distinguishes them?
LoRA freezes the base weights and learns low-rank updates on selected transformations, reducing the trainable-parameter and delta-storage burden in the settings studied by its authors. [CLM-031] LLME-CASE-009 bounds that adapter evidence to its evaluated settings. The formulation does not promise equal quality, safety, latency, or total cost for a new model and task. Small trainable state still depends on a large base, tokenizer, template, runtime, evaluators, and serving plan.
For a transformed weight matrix, the adapter represents an update through lower-dimensional factors. Rank limits the update subspace; scale controls its contribution. Target modules decide where updates enter. Dropout and initialization shape the experiment. These are hypotheses, not magic constants.
Pin the entire compatibility tuple
The Mosaic adapter manifest binds:
- base model ID, revision, configuration digest, and weight digest;
- tokenizer ID, revision, vocabulary, special-token map, and digest;
- training and serving template IDs plus golden serialization;
- runtime, library, dependency lock, precision, quantization, and hardware class;
- adapter method, target modules, rank, scale, dropout, seed, and artifact digest;
- training-data, mask, experiment, evaluator, and checkpoint lineage;
- merged or unmerged state and the exact merge procedure;
- license/provenance facts, limits, owner, and disposition.
The teaching fixture uses the exact base revision rev-synthetic-a1, tokenizer tok-synthetic-a1, and runtime 1.0.0-teaching. Because the serving template mismatch remains unresolved, every PEFT row has executable: false. Resource and behavior values are deterministic fixtures for comparison, not measurements.
CLM-031; it makes no quality claim.Long description
A crisp colorful three-dimensional cutaway shows tall frozen layer blocks with small luminous adapter modules inserted beside them. A physical key joins each module to a plate carrying the exact base revision.
Design bounded adapter candidates
Start with a reason for each variable. If the residual concerns a narrow shorthand mapping, attention-projection targets with two bounded ranks may be reasonable hypotheses. If evidence suggests broader representation change, do not silently expand modules; create a new experiment.
The simulation declares two candidates: peft-r8-attn and peft-r16-attn-mlp. It reports illustrative trainable-parameter counts, peak-memory units, adapter-storage units, training-time units, load time, and merged/unmerged inference consequences. Numbers are comparable only inside this synthetic fixture.
PEFT method support differs across layers, quantized bases, merging, hotswap, and runtime behavior. [CLM-032] Maintained PEFT documentation is the implementation authority for its current interfaces; QLoRA reports resource results for specified hardware, models, quantization, optimizer, and tasks. Recheck library and runtime support at execution time. Never import a paper’s memory ratio as a capacity promise.
Quantized-base adaptation adds precision surfaces: quantization format and calibration, compute dtype, dequantization path, optimizer state, kernels, and merge support. A base that loads is not necessarily a base that can train, merge, hotswap, or serve under the intended stack.
Resource evidence needs definitions. Peak allocated memory differs from reserved memory. Adapter bytes omit the base, tokenizer, runtime, and replicas. Training duration depends on sequence distribution, batching, hardware, topology, checkpointing, and logging. Measure under the exact candidate identity and publish raw definitions.
Test wrong-base and wrong-template failures first
The failure injection pairs an adapter for rev-synthetic-a1 with rev-synthetic-a2. Shapes happen to match, so a permissive loader could accept it. The compatibility validator rejects before load because the base digest differs. Another candidate uses the right base but the wrong tokenizer digest. A third serves without the training template. All fail closed.
Merged and unmerged forms are distinct artifacts. If merging is supported, compare logits or behavior within a declared tolerance under the same runtime and precision. Record the merge algorithm, source adapter, destination base, dtype, digest, and irreversibility assumptions. “Merged successfully” means the operation completed; it does not mean behavior is equivalent.
Hotswapping requires its own support and state-isolation checks. Concurrent adapters must not leak between requests. Caches and batching must carry adapter identity. These serving-platform mechanisms remain platform/MLOps responsibility, while the LLM engineer specifies the behavior and compatibility evidence the platform must expose.
CLM-032 and CLM-033; values are illustrative.Long description
A polished three-dimensional testing rig holds four candidate capsules. Separate gauges show target, retention, language, memory, storage, and runtime compatibility. One small adapter capsule is blocked by an identity lock.
Compare behavior and resources on one frontier
An efficient adaptation must still pass the same target, retention, control, and inference-compatibility evidence as every other candidate. [CLM-033] Evaluate no-tune, SFT, and PEFT with identical held-out cases, generation policy, raw-output retention, uncertainty, and hard gates.
Order the decision:
- reject incompatible artifact tuples;
- reject forbidden target, retention, language, abstention, citation, untrusted-instruction, or no-effect transitions;
- identify candidates that add target value beyond uncertainty;
- compare resource, storage, inference, merge, rollback, and operational complexity;
- retain a Pareto frontier rather than hiding tradeoffs in one score;
- ask named owners whether any surviving candidate warrants a bounded next step.
The companion’s peft-r8-attn fixture passes the illustrative protected gates and uses fewer synthetic training-memory and delta-storage units than the SFT fixture. peft-r16-attn-mlp raises the aggregate target fixture but fails Gujarati citation retention. It is rejected, regardless of its trainable-parameter efficiency. None is a trained artifact.
The managed model remains the no-weight-access comparator. A managed fine-tuning service, if later considered, must expose enough versioned data, job, checkpoint, evaluation, and rollback identity to satisfy the common behavior contract. We do not invent hidden weight or optimizer details.
Close MD-12 as a simulated decision, not an execution
mosaic-peft-candidate-simulation.json records the frozen tuple, LoRA hypotheses, resource definitions, wrong-base failure, four-way matrix, hard gates, and a simulated frontier. trainingExecuted and artifactsCreated remain false. The actual MD-12 disposition remains blocked; only the teaching comparison marks peft-r8-attn as the candidate carried into the Chapter 12 simulation.
Practice
Detect the wrong-base adapter, compare the two PEFT rows with no-tune and SFT, and justify a disposition without using parameter count as quality. Pass when compatibility, target, retention, language, controls, resources, merge state, and authority are all explicit.