Supervise the Model

Loss is an instrument, not the decision

Mosaic Desk enters Chapter 10 with an unusually valuable result: MD-12 refuses to start. The frozen open-weight candidate uses base revision rev-synthetic-a1, tokenizer revision tok-synthetic-a1, and runtime 1.0.0-teaching, but its training and serving templates disagree. That incompatibility still blocks every real batch. We will not manufacture a checkpoint to make the story convenient.

Instead, the companion creates a deterministic teaching simulation. Its numbers are fixtures, not measured model outcomes. The simulation lets us practise checkpoint selection while retaining the actual experiment status blocked-before-training.

Supervised fine-tuning presents formatted demonstrations and increases the likelihood of selected target tokens. Which tokens count depends on the data, chat template, and loss mask. [CLM-028] If boilerplate, copied evidence, or user text is accidentally supervised, a falling objective can reward the wrong pattern. The objective reports how a candidate fits its supervised surface; it does not certify the language-task contract.

Inspect a rendered batch before optimization

For every sampled record, inspect four aligned views:

1. the structured source record and its authorization;
2. the exact serialized sequence after the pinned chat template;
3. token IDs, boundaries, truncation, padding, and packing;
4. the target mask showing which tokens contribute to loss.

Count total tokens and supervised tokens by record type and language. The Mosaic failure injection gives repetitive English answer boilerplate far more supervised tokens than Gujarati evidence links. A row-balanced mixture is therefore not token-balanced. Training loss may fall mostly by memorizing the common wrapper while the rare behavior remains unchanged.

Mask checks must fail closed. User requests, untrusted retrieved instructions, padding, and system controls are not assistant targets. Truncation must not remove the citation while retaining a fluent answer. Packed records need explicit boundaries so one example cannot supervise another. A tiny-overfit diagnostic, once authorized, can show that the implementation fits a disposable sample; it cannot show held-out improvement.

Long description

A colorful realistic three-dimensional workshop shows a sealed base-model block receiving supervised example cards. Three checkpoint capsules leave the workshop, pass through a separate evaluation scanner, and arrive at a gate labelled select or reject.

Read curves as competing hypotheses

Plot training and development loss against updates and supervised-token exposure, but also plot frozen behavioral measurements at preregistered checkpoints. When training loss declines and development loss rises, memorization is plausible. When both decline but citation correctness stalls, the objective may emphasize an easier correlated feature. When target behavior improves while abstention falls, the candidate violates a protected boundary.

The companion simulates no-tune, sft-early, and sft-late. sft-early improves the target fixture without crossing protected gates. sft-late has the lowest simulated training loss and highest aggregate target score, yet loses abstention and Gujarati citation behavior. It is rejected. This is a deterministic decision exercise, not a claim that such runs occurred.

Checkpoint selection should use held-out target, retention, segment, control, and resource evidence rather than minimum training loss. [CLM-029] Compare every checkpoint with the unchanged no-tune baseline on identical cases, generation settings, retrieval snapshot, schema, and evaluators. Preserve raw outputs and case-level transitions. An average cannot cancel a forbidden pass-to-fail change.

Use a matrix with at least these columns:

• exact candidate and checkpoint identity;
• target contract clauses and error categories;
• citation and neutral-instruction retention;
• Gujarati, English, and Gujarati-English segments;
• abstention, conflict, untrusted-instruction, unauthorized-source, and no-effect controls;
• training and evaluation resource observations;
• uncertainty, missingness, and raw-example links;
• hard-gate result and named disposition owner.

The no-tune row is not an obsolete control. It is the counterfactual that reveals whether added complexity earned value. A candidate that ties it within uncertainty but costs more and introduces new lifecycle surfaces has not earned promotion.

Long description

A vivid three-dimensional plotting table carries a descending train-loss ribbon, a flattening held-out ribbon, and a retention ribbon that bends into a red guarded zone. A raised stop marker sits before the late checkpoint.

Never let an aggregate erase a language

Synthetic and multilingual training gains require independent held-out evaluation. [CLM-030] Aya (LLME-CASE-007) demonstrates that multilingual coverage can be made explicit, but its reported model and language results do not transfer to Mosaic Desk. The staged-recipe evidence in LLME-CASE-008 and the Self-Instruct synthetic-data route remain bounded to their own data and evaluations; generated data can reproduce generator errors. Neither case licenses an aggregate-only conclusion.

Report each declared language and mixed-language segment separately. Preserve comparable task intent where translation is appropriate, but do not assume translation preserves shorthand, cultural context, evidence conventions, or error severity. A language authority reviews ambiguous cases; an LLM engineer cannot declare linguistic adequacy from fluent output.

The failure rule is deliberately asymmetric: improvement in English cannot compensate for a forbidden Gujarati citation regression. Low-volume segments remain visible with their sample sizes and uncertainty. If a segment lacks adequate evidence, record insufficient-evidence; do not merge it into an average until it disappears.

Decide with raw behavior and authority intact

Selection begins with hard gates, not rankings. Reject identity mismatch, invalid masking, leakage, unauthorized data access, non-finite optimization, invalid resume, evaluator drift, or a forbidden regression. For survivors, compare target value, uncertainty, retention, language variation, resources, reversibility, and operational consequences.

Data and domain owners validate examples and meaning. Language reviewers own linguistic adequacy. Security and safety authorities own their controls. Platform/MLOps owns reliable training infrastructure and resource accounting. LLM engineering binds the experiment to behavior evidence and recommends retain, stop, or reject. It does not waive another authority’s gate.

The managed model remains a dated comparator under the same external contract. Its internal objective and weights are opaque, so compare observable behavior and declared service properties without pretending it was trained under the open-weight recipe.

Complete the simulated SFT decision

mosaic-sft-checkpoint-simulation.json retains the real block, exact synthetic identities, illustrative curves, candidate matrices, language slices, hard gates, and decision logic. Its trainingExecuted field is false. sft-early is only the simulated SFT comparator forwarded to Chapter 11; it is not an artifact and cannot be deployed.

Practice

Audit the token-dominance failure, compare all three rows, and write a checkpoint disposition. Pass only if you reject sft-late despite its lower simulated loss and higher aggregate target score, preserve the Gujarati and abstention failures, and state why the exercise produces no training claim.