Establish the Training Experiment
Specify an inspectable, resumable training experiment whose state, resources, behavioral hooks, and stop rules are frozen before execution.
A training command is not an experiment
MD-11 now contains synthetic instruction evidence, bounded preference evidence, and protected retention/control lanes. The training/serving template mismatch remains unresolved. Chapter 9 therefore establishes MD-12 as a plan and smoke-test protocol; it does not execute optimization.
A training run manifest needs base/tokenizer, data/splits, objective, optimizer/schedule, batch/sequence, precision, seeds, code, hardware class, checkpoints, and evaluators. [CLM-025] The manifest makes a run inspectable within declared software, hardware, and numerical limits. It cannot guarantee bitwise portability.
Bind the experiment identity
The manifest links:
- MD-09 hypothesis, baseline, intervention, disconfirmation, and stop rules;
- exact base weights/config/tokenizer/template/runtime tuple;
- MD-10 recipe, pipeline, and five vault manifests;
- MD-11 instruction/preference/retention artifacts;
- renderer, masks, sequence/packing rules, and batch sampler;
- objective and which tokens contribute;
- optimizer, weight decay, scheduler, warmup, clipping, and update count;
- per-device microbatch, devices, accumulation, and effective global batch;
- storage/compute/accumulation precision and sharding strategy;
- code, dependencies, container/environment, seeds, hardware class, and topology;
- checkpoint cadence, contents, retention, lineage, and recovery procedure;
- resource logs and target/retention/control evaluation hooks;
- owners, authority gates, budget, and automatic stop conditions.
Every change creates a new manifest identity. Human-friendly run names are labels, not identity.
The experiment card also names the question and alternative explanations. The candidate hypothesis is that a supervised PEFT intervention changes the bounded shorthand mapping. Alternative explanations include sampling, template mismatch, evaluator drift, data leakage, retrieval/config change, and effective-batch differences. Every artifact and hook exists to distinguish these causes.
Do not place secrets, raw protected examples, or unrestricted paths in the manifest. Use content identities and approved references. Access control, secret management, scheduling, durable storage, and cluster reliability remain platform/MLOps responsibilities; the behavioral experiment still records the environment it depends on.
CLM-025; it does not show a successful run.Long description
A colorful realistic three-dimensional cockpit links sealed data and configuration modules to seed, code, hardware, checkpoint, log, and evaluation instruments. A central manifest plate carries lineage arrows; the start lever remains locked.
Calculate effective batch explicitly
For simple data parallelism:
effective batch = microbatch per device x data-parallel devices x gradient accumulation steps
Record sequences and supervised tokens too, because variable lengths make row counts poor measures of optimization work. If packing or token-based batching changes, record the distribution.
The failure injection changes accumulation from 4 to 8 while logging only microbatch 2 on 2 devices. The effective batch changes from 16 to 32. A schedule expressed in steps now sees a different example/token exposure. The validator rejects the manifest until batch semantics and schedule are reconciled.
Optimizer papers describe mechanisms, not universally correct hyperparameters. Tülu 3 (LLME-CASE-008) is one staged recipe, not Mosaic’s configuration. Chapters 10-11 will compare bounded candidates only after this frame is accepted.
Sequence policy records maximum length, truncation side, packing, padding, sampling buckets, and supervised-token distribution. A batch can contain the same row count but twice the supervised tokens. Log examples, total tokens, and supervised tokens per update so resource and learning curves remain interpretable.
Gradient clipping, regularization, warmup, and schedule units are explicit. 10% warmup must name ten percent of what planned update count; a resumed or shortened run must not silently recalculate it.
Treat precision and sharding as experiment state
Mixed precision and sharding change memory/communication behavior and checkpoint procedures. [CLM-026] Record parameter, gradient, optimizer, activation, reduction, and accumulation dtypes; loss scaling; sharded objects; offload; communication; and state-dict format.
PyTorch AMP and FSDP documentation are living, version-specific interfaces. Hardware, operators, kernels, topology, and library versions affect numerical and recovery behavior. Platform/MLOps owns reliable shared infrastructure and capacity; LLM engineering owns the model-experiment specification and behavior linkage.
CLM-026; it reports no hardware capacity.Long description
A realistic three-dimensional memory cabinet holds distinct shelves for weights, activations, gradients, optimizer state, and checkpoints. Batch and sequence blocks enter from one side while a finite-memory gauge and sharding links remain visible.
Require preflight gates before scale
The experiment state machine is:
- manifest/schema validation;
- data lineage and protected-vault access validation;
- template/tokenizer/mask compatibility validation;
- one-batch forward/loss smoke test;
- tiny bounded overfit diagnostic;
- save/load and exact checkpoint-content inspection;
- interrupted-run resume equivalence test;
- resource-envelope probe;
- target/retention/control evaluator-hook dry run;
- named experimental-authority decision.
Because the inherited template blocker remains, Mosaic stops at step 3. Later steps are specified but not fabricated. A tiny overfit test would diagnose whether data, masks, objective, and optimizer can fit a tiny authorized sample; it would not establish generalization.
The one-batch smoke test, once allowed, checks shapes, finite loss, intended supervised-token count, gradient presence only on intended parameters, and absence of protected-vault access. It produces no quality claim. The tiny overfit diagnostic uses a disposable subset separate from final evaluation and asks whether the implementation can reduce loss on exactly those examples. Failure indicates a bug or incompatible setup; success only clears a mechanics gate.
The resource probe uses a bounded sequence/batch grid under the declared hardware class. It logs peak allocated/reserved memory, step time components, communication, checkpoint time, and failure state. It does not extrapolate a production capacity promise.
Make resume stateful
A resumable checkpoint includes model/base/adapter state as applicable, optimizer, scheduler, scaler, random-number states, data sampler/cursor, global update and token counts, accumulation position, manifest/data/code identity, and distributed/sharding metadata required by the exact stack.
The second failure injection restores only model weights. Learning-rate schedule restarts, optimizer moments disappear, sampler returns to the beginning, and update count is ambiguous. The validator marks resume-invalid-missing-state. Calling that a resume would corrupt comparison and possibly repeat data.
Test interruption at a declared boundary, restore into the same compatible environment, and compare the next bounded state with an uninterrupted control. Exact equality may be unavailable across some stacks; define tolerances and limitations before testing.
Checkpoint retention and deletion rules belong in the manifest. A checkpoint can encode training data and sensitive patterns; access and lifecycle require competent authority. The experiment cannot keep every intermediate forever merely for convenience. Conversely, deletion that makes a claimed comparison irreproducible becomes a stated limitation.
Attach behavior hooks and stop rules
Training loss and resource completion do not establish target behavior improvement. [CLM-027] Loss can fall while protected behaviors regress or examples leak. At checkpoint cadence, run frozen target, retention, segment, control, and no-change comparisons with unchanged evaluators.
Stop before or during a run for template/identity mismatch, protected-vault access, unsupported target, non-finite loss, unexpected effective batch, resource budget breach, invalid checkpoint, resume divergence, target futility under the preregistered rule, or any forbidden regression. Named authorities can impose additional stops; engineering cannot waive them.
Resource logs include timestamps, updates, examples/tokens, duration components, memory observations, failures, retries, checkpoint size/time, and environment identity. They are experimental evidence, not production capacity.
Evaluation hooks run outside the optimizer step and bind checkpoint identity to raw case results. Never select a checkpoint by training loss and evaluate only the winner. Apply the preregistered cadence or selection rule and retain rejected-checkpoint evidence. Protected hard gates can stop the run even while target loss improves.
The experiment authority packet names platform readiness, data/privacy/legal status, security review, product/domain value, resource budget, and experiment owner. Technical completeness cannot imply approval. The fixture remains blocked on template compatibility even if every authority were otherwise present.
Open MD-12 without inventing a run
mosaic-training-experiment.json stores the full manifest, calculated effective batch, preflight state machine, checkpoint contract, evaluation hooks, budgets, and stops. It includes invalid batch and resume candidates as teaching failures. Overall status: md12-experiment-specified-blocked-before-training.
The start lever remains locked by template compatibility and experimental authority. No loss curve, adapter, checkpoint, compute consumption, or behavior outcome is reported.
Practice
Repair the effective-batch manifest, enumerate checkpoint state, diagnose the bad resume, trace every data/evaluator artifact, and explain why the template gate stops before smoke execution.
Pass when another engineer can reconstruct intended state, costs, recovery, hooks, and stops and no full run begins before all gates. Fail when seed means portable, weights-only means resume, loss means quality, or platform work and behavior authority collapse into one role.