Run Experiments That Isolate Change

A better demo can be an uninterpretable experiment

Mosaic Desk enters Chapter 15 with a reproducible baseline, a frozen retrieval/context path, a segmented MD-06 case set, a consequence-aware error taxonomy, and calibrated judgment methods. The team proposes four improvements at once:

• revise the instruction message;
• switch the reranker;
• change the generator;
• upgrade the model grader.

The aggregate score rises in the synthetic rehearsal. What changed behavior?

There is no defensible answer. The prompt may improve evidence use while the reranker harms exact identifiers. The new generator may change abstention. The new grader may simply prefer longer answers. Even if the combined candidate is useful, the experiment cannot attribute the result to one factor.

Engineering progress requires a decision claim narrower than the demo.

Write the decision before the hypothesis

Start with the decision the evidence will inform:

Decide whether instruction bundle MSG-0.2.0 should replace MSG-0.1.0 for the current Mosaic proposal path while every other behavior-facing factor remains frozen.

Then write a falsifiable hypothesis:

On evaluation set MD06-SET-0.1.0, the candidate reduces qualifier-loss errors without increasing unsupported claims, required-abstention misses, authorization failures, or designated language-segment regressions beyond predeclared gates.

The hypothesis names a variable, population, expected direction, protected outcomes, and evidence that could reject it. “Make responses better” cannot fail and therefore cannot guide an experiment.

Predeclare disposition options: retain the baseline, revise and rerun, reject the candidate, scope it to supported segments, or advance it to a separately authorized release gate. Score collection without a decision is unfinished work.

Freeze controls as a comparison identity

For an instruction experiment, bind:

• model/provider or checkpoint and access posture;
• endpoint/runtime, tokenizer, chat template, and hardware where applicable;
• task contract and case-set version;
• corpus, ingestion, chunking, retriever, filters, reranker, and assembly policy;
• system/developer messages except the named instruction module;
• decoding, seed/repeat policy, maximum output, schema, and validators;
• citation behavior and effect isolation;
• evaluator taxonomy, rubrics, graders, humans, and adjudication state;
• environment, concurrency, cache state, date, and trial ordering;
• cost and latency collection method.

The candidate differs in one declared variable family. A “prompt change” can include several coordinated lines within one versioned instruction module, but it cannot quietly include a new model and retrieval path. If two factors must change together for compatibility, name the bundle and narrow attribution to that bundle.

A causal model-system claim needs a baseline, explicit hypothesis, controlled variable family, repeated trials where relevant, and decision criteria. Provider evaluation guidance supports iterative, versioned comparison, but attribution remains bounded by the design and some production changes cannot be perfectly isolated. [CLM-043]

Long description

Two colorful three-dimensional experiment rigs sit side by side. Identical model, retrieval, context, schema, case, and judge modules are locked. Only one instruction cartridge differs. Paired case cards pass through repeated trial loops, then results enter language, evidence, risk, and latency lenses.

Preregister the experiment packet

Before inspecting candidate results, store:

• experiment ID, owner, date, and decision deadline;
• hypothesis and rationale;
• baseline and candidate identities;
• controlled variable and frozen factors;
• included/excluded cases and segments;
• pairing, ordering, and repeat policy;
• primary and diagnostic metrics;
• hard gates and segment gates;
• latency, cost, capacity, and failure-state measures;
• uncertainty summaries appropriate to the data;
• stopping, invalidation, and rerun conditions;
• confounds already known;
• disposition vocabulary and authority.

Hash the packet before results. A preregistered plan reduces the temptation to choose a favorable metric, segment, or stopping point afterward. It does not make a weak design strong; it makes changes to the design visible.

If exploratory analysis discovers a useful pattern, label it exploratory and design a new confirmation run on unexposed cases. Do not rewrite the original hypothesis.

Pair cases and preserve raw outcomes

Run baseline and candidate on the same case versions and source snapshots. Pairing removes some case-mix variation and enables direct inspection of transitions:

• pass to pass;
• fail to pass;
• pass to fail;
• error category changed;
• answer to abstain;
• abstain to unsupported answer;
• no material change;
• judge disagreement or invalid trial.

For stochastic paths, execute the predeclared repeat policy and preserve every trial. A single seed or temperature zero can still be affected by service changes, parallelism, kernels, or hidden implementation details. Repeats measure variability under the observed setup; they do not prove a universal distribution.

Keep raw request/result/trace identities, component evidence, error records, and grader decisions. Summary tables are indexes into evidence, not replacements for it.

Select uncertainty methods that match the design

There is no universal test for LLM-system experiments. Choose summaries based on what was sampled and paired.

Report counts and denominators for sparse segments. For paired binary outcomes, show the direction and number of discordant pairs. For repeated numeric measures, show distribution and tails rather than only mean. Use resampling or interval methods only when their dependence and sampling assumptions are plausible. If cases share templates or source families, do not pretend they are independent rows.

Distinguish measurement uncertainty, run variability, label disagreement, environmental variability, and unsupported population coverage. A narrow interval around a biased case set is still a weak claim. A visible difference may be operationally irrelevant, while an uncertain rare authorization failure can still block progression under a hard gate.

Inspect segments before believing the aggregate

Report the frozen MD-06 slices:

• language and code switching;
• appliance family and ambiguity;
• evidence state;
• answerability and expected behavior;
• ordinary versus designated consequence;
• exact identifier versus paraphrase;
• thread length and attachment state;
• nominal, edge, adversarial, and out-of-scope.

Aggregate gains should not erase segment regressions or uncertainty. Holistic evaluation and NIST’s pilot report reinforce scenario- and limitation-aware measurement, while statistical treatment depends on the sample and process. A tiny segment can reveal a serious regression without supporting a stable rate estimate. [CLM-044]

Define hard gates separately from optimization metrics. Any another-tenant exposure, prohibited effect, fabricated citation, or required-abstention miss in a protected case can stop the candidate regardless of aggregate improvement. Product and risk owners define consequence priorities; the engineer prevents aggregation from hiding the evidence.

Measure resources without inventing production capacity

Collect latency components, tail summaries, token/teaching-unit use, retries, failures, and estimated or observed cost under a versioned environment. A local or synthetic run supports only that environment.

Managed-path records include endpoint, region, provider-exposed model/version, concurrency, caching, request options, rate-limit behavior, and unavailable internals. Open-weight records additionally include checkpoint digest, tokenizer/template, runtime, kernels, quantization, hardware, batching, scheduler, and capacity state.

Compare like with like where possible. If environments differ materially, state that access posture is part of the variable bundle. Platform/SRE owners supply runtime and capacity facts. The LLM engineer integrates behavior and resource evidence without declaring operational readiness.

Diagnose confounds before disposition

Common confounds include:

• case-set exposure during prompt development;
• changed source snapshot or retrieval index;
• model or provider alias drift;
• grader version or prompt change;
• baseline and candidate run at different load or cache state;
• retry policy hiding failures;
• different truncation or tokenization;
• order effects in pairwise judgment;
• multiple testing and favorable metric selection;
• manual exclusion of candidate regressions;
• transformed duplicates treated as independent.

Classify each as controlled, measured, bounded, unresolved, or experiment-invalidating. Do not explain it away after seeing the outcome. An unresolved confound narrows the claim or forces a rerun.

LLME-CASE-004 contributes the bounded lesson that retrieval stages can be isolated. It does not establish which retriever or reranker wins for Mosaic. LLME-CASE-005 contributes the bounded lesson that judge order and style need control. It does not supply a universal grader.

Stage experiments from cheap evidence to expensive evidence

Not every candidate deserves a full protected-set run. Use progressive gates without reusing evidence dishonestly.

1. Contract check: validate configuration, no-effect boundary, schemas, and deterministic invariants.
2. Development diagnosis: inspect known cases to determine whether the mechanism behaves as intended.
3. Calibration replay: verify judges and segment instrumentation remain credible.
4. Frozen comparison: run the preregistered paired experiment on authorized unexposed cases.
5. Protected review: inspect designated consequential cases under controlled access.
6. Release handoff: provide the packet to a separate bounded release process.

Failure at an early gate stops expensive execution. Passing an early gate does not predict later quality. Record which evidence was exposed at each stage so a future run does not call a reused case holdout.

If exploration repeatedly consults a protected set, retire its holdout status and create a new version from unexposed authorized families. Do not solve overfitting with a label change.

Compare environment variance honestly

An API candidate and a self-hosted candidate may not be runnable under identical infrastructure. Define the comparison claim carefully.

For behavior, keep cases, source snapshots, criteria, and semantic contracts identical. For resources, report each environment and avoid attributing hardware or provider scheduling differences to the model alone. If the decision is specifically about access posture, the environment bundle is the controlled variable and the claim is about the measured bundle.

Interleave baseline and candidate trials when time drift is plausible. Randomize order within authorization and caching constraints. Record cold/warm state, concurrency, retries, throttling, and failures. Do not discard rate-limited or out-of-memory trials without a predeclared invalidation rule; operational failures are evidence for the configured path.

If a managed alias changes mid-run or an open-weight server is rebuilt, stop or split the experiment by identity. Combining runs under one label creates false precision.

Write the decision memo from the evidence packet

The memo contains:

• decision requested and responsible authority;
• hypothesis and whether evidence supported it;
• frozen identities and scope;
• primary, diagnostic, segment, and resource results;
• pass-to-fail and fail-to-pass cases;
• hard-gate outcomes;
• uncertainty, label disagreement, and confounds;
• negative findings;
• recommended disposition and alternatives;
• residual risks and owner handoffs;
• trigger for rerun, rollback, or scope review.

Use causal wording no stronger than the design. “The instruction-bundle candidate was associated with fewer qualifier losses under this frozen paired fixture” is narrower than “the prompt fixed grounding.” Preserve that precision through presentations and issue summaries.

Keep negative findings and visible regressions

An experiment should produce a useful record even when the candidate loses.

Retain:

• hypotheses not supported;
• segments that regress;
• raw cases behind each regression;
• cost/latency tradeoffs;
• grader disagreement;
• confounds and invalid trials;
• rejected mechanisms and why;
• trigger that could justify a new experiment.

Negative evidence prevents the same weak change from being rediscovered. It also keeps future teams from seeing only the surviving candidate.

An experiment should end in retain, revise, reject, scope, or release disposition rather than score collection alone. The evaluate-improve loop and current evaluation guidance motivate decision-oriented iteration; the five-word vocabulary is this book’s engineering pattern, and formal release authority remains external. [CLM-045]

Use dispositions precisely

Retain

Keep the baseline. The candidate did not support the hypothesis, violated a gate, or introduced unresolved risk.

Revise

The mechanism remains plausible but the implementation, rubric, or design needs a new preregistered experiment. Do not revise in place and reuse the same protected evidence as confirmation.

Reject

Evidence contradicts the mechanism or its tradeoff is unacceptable for the scoped decision. Preserve a reopening trigger.

Scope

The candidate may support a narrower population, language, source state, or access path. Scope reduction requires an honest contract and owner approval; it cannot hide excluded users.

Release

The experiment supplies evidence to a separate release gate. It does not itself approve deployment, risk, privacy, safety, domain policy, capacity, or external effects.

Long description

A realistic three-dimensional decision hall receives an experiment folder containing paired results, segment regressions, uncertainty bands, resource evidence, and confound notes. Five distinct gates route the folder to retain, revise, reject, scope, or a formal release-review checkpoint. No gate connects directly to production.

Failure injection: the four-change improvement

Construct a synthetic combined candidate with a new prompt, reranker, generator, and judge. Its aggregate teaching score rises. Gujarati citation support falls, required abstention misses appear in conflict cases, and synthetic p99 duration increases.

The correct diagnosis is not “the new model traded quality for latency.” Four changes and a changed judge make that attribution unsupported.

Repair the experiment:

1. restore the baseline judge and all other controls;
2. select one variable family, such as the prompt module;
3. freeze MD06-SET-0.1.0 and group related cases;
4. preregister the qualifier-loss hypothesis and hard gates;
5. run paired ordered trials and required repeats;
6. inspect Gujarati, conflict, abstention, and tail-latency slices;
7. preserve every pass-to-fail transition;
8. classify uncertainty and confounds;
9. issue a retain/revise/reject/scope/release recommendation;
10. route release and tradeoff authority to named owners.

The fixture uses constructed results to prove disposition logic. It claims no real model, provider, language, latency, or quality outcome.

Practice: defend an experiment packet

Take the confounded four-change experiment and produce:

1. the decision and falsifiable hypothesis;
2. baseline, candidate, and controlled variable identities;
3. frozen cases, slices, judges, and environment;
4. pairing, repeat, ordering, and stopping policy;
5. primary, diagnostic, hard-gate, resource, and uncertainty measures;
6. raw transition table and segment regressions;
7. confound classification;
8. negative findings and reopening triggers;
9. one disposition with rationale and residual uncertainty;
10. authority and non-scope handoff.

Pass when one claim maps to one controlled change, case-level and segment evidence remain visible, statistical language matches the design, and no aggregate or experiment packet self-authorizes release.

The reviewer should be able to reconstruct the candidate, reproduce the comparison mechanics, locate every excluded trial, and explain why the recommended disposition is no broader than the observed evidence.

The MD-07 experiment packet

Mosaic opens MD-07 with preregistered decision/hypothesis records, frozen baseline and candidate identities, one controlled variable family, paired cases and repeats, component/segment/resource results, raw regressions, uncertainty and confound records, negative findings, hard gates, and an explicit disposition. Chapter 16 will turn accepted and rejected evidence into behavior-facing release, rollback, monitoring, and change-replay gates without weakening the task contract.