Establish a Reproducible Baseline

“We used the same model” is not a baseline

The MD-02 decision gives Mosaic Desk a provisional access posture, rejected alternatives, and a migration seam. The next experiment appears simple: send the seed cases to the selected path and save the responses.

An engineer runs the washer case on Monday and captures a promising proposal. On Thursday, another engineer copies the visible prompt and receives a different structure. The experiment log says both runs used candidate-primary. It does not record the provider revision, message adapter, chat template, evidence order, decoding settings, output limit, schema mode, evaluator version, or code revision. The first response was copied into a document; the raw response envelope was discarded.

The team cannot tell whether it observed expected sampling variation, a silent alias change, a template change, an application change, or an evaluation change. It has two outputs and no controlled comparison.

A baseline is not the first good response. It is a timestamped package that binds configuration, cases, raw observations, evaluators, environment, and known gaps tightly enough for another engineer to reconstruct the intended experiment and explain where exact replay is impossible.

Reproducible does not always mean bit-for-bit identical. A managed service may hide runtime details. Sampling can yield different sequences. Hardware and kernels can affect open-weight numerical behavior. The engineering requirement is stronger than “try again” and more honest than “guaranteed deterministic”: identify everything you can, preserve what happened, repeat where variation matters, and label unavailable surfaces.

Define run identity before running

A useful baseline has several nested identities.

Task identity

Record the language-task contract revision, behavior clauses, case-set revision, segment labels, evidence bundle, and prohibited effects. A run against changed cases is not a repeat of the same evaluation.

Request identity

Record semantic messages, role mapping, prompt or instruction revision, evidence order, retrieval/tool state, output contract, and application-side transformations. Store both the provider-neutral request and the adapter-rendered representation where observable and allowed.

Model-system identity

Record the managed model identifier or open-weight checkpoint; tokenizer and chat template when observable; decoding strategy and fields; context and output limits; schema or grammar mode; safety or provider features that can affect output; client and runtime versions; and the date.

Execution identity

Record code revision, adapter revision, environment, region, concurrency, retry policy, timeout, cache state where relevant, and hardware/runtime details for self-hosted paths. Secrets never belong in the manifest.

Evidence identity

Record raw response bytes or a privacy-approved representation, provider metadata, normalized result, validation disposition, evaluator/rubric version, reviewer judgment when used, latency observation, usage observation, and cost inputs with their date.

A reproducible behavior baseline identifies model, prompt/messages, context, decoding, schema, evaluator, and runtime configuration. Official optimization and evaluation documentation supports versioned evaluation loops, while generation and template documentation shows that inference settings and serialization are behavior-facing. These are current interfaces, not universal or permanent field lists. Hardware and provider nondeterminism can still prevent bitwise reproduction. [CLM-013]

Long description

A crisp three-dimensional lockbox contains interlocking, labeled version blocks. Four large blocks read model, prompt, context, and runtime. Smaller blocks for decoding, schema, evaluator, cases, and adapter fill the remaining space. A hash tag seals the box, while a side compartment labeled not observable remains visibly empty but documented.

Use a manifest that can say “unknown”

The manifest should be machine-readable enough to validate and human-readable enough to review. A minimal structure includes:

{
  "runId": "MD03-BL-001",
  "fictional": true,
  "taskContractVersion": "0.2.0",
  "caseSetVersion": "seed-0.1.0",
  "accessPosture": "managed",
  "model": {"identifier": "synthetic-managed-a", "revision": "not-observable"},
  "messages": {"contractVersion": "0.1.0", "adapterVersion": "managed-0.1.0"},
  "generation": {"temperature": 0, "topP": 1, "maxOutputUnits": 256},
  "runtime": {"clientVersion": "fixture-0.1.0", "serverVersion": "not-observable"},
  "evaluatorVersion": "contract-checks-0.1.0",
  "knownNondeterminism": ["provider runtime not observable"]
}

The values above are synthetic teaching values. They are not a real provider recommendation or model run.

The manifest distinguishes three states that incomplete logs often collapse:

• known and pinned: a value and revision are recorded;
• known but changeable: a provider alias or dated policy is recorded with a trigger;
• not observable: the access path does not expose the surface.

Not observable is evidence about the boundary. It is not a defect to conceal and not permission to invent a value.

Managed-path manifest

Record the public model identifier, request fields, adapter version, returned metadata, region or service configuration that the provider exposes, and explicit unknowns. Store the provider documentation version or access date for volatile behaviors. Do not claim an underlying checkpoint, tokenizer, template, or server runtime unless the service exposes it.

Open-weight manifest

Add checkpoint and artifact integrity, tokenizer files, chat template, generation configuration, precision, runtime and kernel versions, hardware type, driver stack, and serving configuration. More observable fields can improve diagnosis, but the team has also accepted more compatibility and operational duties.

Both paths emit the same common run/result/error schema. Their adapter envelopes retain meaningful differences.

Preserve raw evidence without creating a data leak

A baseline that stores only an evaluator score cannot be re-examined when the rubric changes. Preserve the input reference, raw provider result, normalized result, validation events, and evaluator output together.

Raw does not mean unrestricted. A production trace could contain customer data, technician notes, or sensitive evidence. Privacy and security owners define allowed fields, access, minimization, retention, deletion, and audit. The LLM engineer records the policy reference and designs the evidence package to satisfy it. For this book, all companion inputs are synthetic and contain no customer records.

Useful separation includes:

• immutable synthetic fixture or approved case reference;
• raw response stored under the applicable policy;
• privacy-minimized comparison record;
• deterministic validation events;
• evaluator judgment and version;
• limitation and authority fields.

Never embed API keys, bearer tokens, private endpoints, or raw production content in a run manifest or repository.

Repeat when variation can change the decision

One output can show that a path executed. It cannot characterize a response distribution.

Repeated trials are needed when stochastic behavior can affect the decision. Official evaluation guidance recommends multiple trials for variable systems, but it does not specify one universally sufficient count. The required design depends on the observed variance, segment, consequence, evaluator, and decision. Agent-oriented evaluation guidance can contribute transferable advice about trials, but agent results or examples do not become evidence for Mosaic. [CLM-014]

Start by declaring why repetition matters. Examples:

• an abstention sometimes becomes an unsupported proposal;
• a required source identifier is intermittently omitted;
• Gujarati-English inputs vary more than English inputs;
• a timeout/retry path changes which response reaches the reviewer;
• cost or latency tails could invalidate an operating assumption.

Then record the trial count, order, seed if supported, concurrency, cache state, and stopping rule. A seed may reduce one source of variation in a particular stack. It cannot guarantee identical results across provider revisions, devices, kernels, library releases, or every sampling implementation.

Long description

A single locked configuration cartridge enters a circular trial rig. Several numbered result capsules emerge at different positions inside a translucent band labeled variation. A separate cartridge with a changed template sits outside the band and is labeled different configuration.

The deterministic test double has a smaller job

Live model runs are inappropriate for many software tests. They may cost money, require secrets, depend on a network, vary, and change outside the repository. Use a deterministic test double for the contracts that deterministic software owns.

The Mosaic companion turns a frozen synthetic request and manifest into predictable result fixtures. It can verify that:

• the common request has required identity fields;
• secrets are absent;
• a success fixture retains a trace and evidence reference;
• an invalid-output fixture reaches the invalid state;
• timeout, provider/auth failure, abstention, and escalation remain explicit;
• hashing is stable for the same canonical manifest;
• changing a behavior-facing field changes the run identity.

It cannot verify language quality, factuality, multilingual capability, live latency, provider behavior, or production safety. A deterministic test double verifies software contracts but cannot establish live-model quality. This is a bounded engineering synthesis from evaluation principles: HELM and official eval guidance demand task evidence, but neither says that the particular Mosaic fixture is sufficient. [CLM-015]

This division prevents two opposite mistakes. Do not make unit tests flaky by hiding live model calls inside them. Do not report green deterministic fixtures as proof that a live candidate satisfies the task contract.

Failure injection: variation mislabeled as improvement

Run the same synthetic case three times against a future authorized live path. Suppose one output is shorter and receives a higher reviewer score. A report says the new prompt improved the system.

The manifest reveals that the prompt did not change. The three results came from one stochastic configuration. The correct statement is that the trial outputs varied and one received a higher judgment under evaluator version E. More trials or a paired experiment may be needed before making a change claim.

Now reverse the failure. Two runs return very similar prose, and the report calls them repeats. The second run used a new adapter version and different evidence ordering. Similar output does not make the configurations identical. The baseline identity changed even if the observed response did not.

Failure injection: the silent alias or template change

The team reruns the Monday cases using the same friendly model alias. The provider has changed the backing behavior or the local template library has changed serialization. The Thursday run looks different.

Diagnose in layers:

1. Compare task and case-set hashes.
2. Compare semantic messages and evidence ordering.
3. Compare adapter and rendered-template identity.
4. Compare model/provider identifiers and returned metadata.
5. Compare decoding and output constraints.
6. Compare code, client, runtime, hardware, concurrency, cache, and retry fields.
7. Compare evaluator version before comparing scores.
8. Record surfaces that remain unobservable.

If several layers changed, report a system change. Do not credit or blame the prompt, model, or provider without isolation.

Benchmarks are discovery evidence, not this baseline

Public benchmarks can suggest candidates, methods, and failure hypotheses. Their tasks, populations, prompts, evaluators, model versions, and operating assumptions differ from Mosaic. The baseline must run the actual language-task contract on named case and consequence segments.

This does not make public benchmarks useless. It assigns them a valid evidence role:

• discover candidate capabilities;
• identify likely weak segments;
• compare research under its published setup;
• inspire evaluation dimensions;
• detect when a claim exceeds its source.

They supplement the intended-use packet. They do not replace it.

Use a failure taxonomy before interpreting differences

When a replay differs, classify the difference before changing anything.

Identity failure

A required behavior-facing field is missing, changed, or ambiguous. Examples include a model alias without a resolved revision, an unknown template upgrade, or an evaluator whose rubric changed without a new version. The remedy is to repair identity or report that comparison is impossible.

Execution failure

The intended configuration did not complete as designed: authentication failed, a timeout occurred, a retry changed request order, the wrong region was used, or a cache affected the observation. Preserve the failure as evidence. Do not replace it with a successful rerun and erase the original state.

Contract failure

The common request/result/error interface was violated. A required case reference disappeared, an invalid result was treated as success, or an error lacked its class. Deterministic tests should catch these defects without a live model.

Behavior variation

The complete recorded configuration is the same within observable limits, yet outputs or judgments differ. This is the state that may require repeated trials and distribution-aware evaluation. It is not automatically a defect and not evidence of a configuration improvement.

Evidence failure

Raw output, evaluator detail, case identity, or privacy-approved trace cannot be located. Even if a score remains, the result is not reviewable enough for a causal claim.

Attribution failure

Several configuration families changed, but the report assigns the effect to one of them. Restore an isolated comparison or narrow the statement to “the configured system changed.”

The taxonomy turns “not reproducible” into an actionable diagnosis. It also preserves negative results: timeout and auth failures remain part of the baseline interface, while live quality remains unsupported until authorized runs exist.

Sign the baseline decision, not the model’s quality

At the end of baseline construction, record one of three engineering dispositions:

• usable for controlled comparison: identity, fixtures, evidence paths, and known gaps are sufficient for the next bounded experiment;
• repair before comparison: missing fields or contract defects would confound the next change;
• blocked by authority or policy: non-synthetic trace, data use, or access cannot proceed without an owner decision.

This disposition is intentionally narrower than model acceptance. A baseline can be perfectly documented and behaviorally poor. It can also be promising yet unusable because evidence retention is unauthorized. The sign-off says whether Chapter 6 can compare a message change responsibly, not whether Mosaic may ship.

Build and review the Mosaic baseline

Use the companion artifacts under baseline/.

1. Inspect the manifest and locate every behavior-facing field.
2. Run the deterministic baseline fixture.
3. Confirm that the same canonical manifest produces the same identity.
4. Change the message-contract revision and observe a new identity.
5. Inspect success, invalid, timeout, auth failure, abstain, and escalate fixtures.
6. Identify which tests prove software behavior and which live claims remain unsupported.
7. Add an unavailable managed field as not observable; do not invent its value.
8. Write the privacy/retention owner required before any non-synthetic raw trace is stored.

The baseline decision record should say usable for controlled Chapter 6 comparison, not production ready.

Practice: repair an incomplete run

You receive this log:

model=candidate-primary; temperature=0; prompt=v2; result looked better

Repair it.

• Identify every missing task, request, model-system, execution, and evidence field.
• State which missing fields can be recovered and which must remain unknown.
• Define three repeated trials and the decision they inform.
• Preserve raw and normalized results under a named synthetic or authorized trace policy.
• Attach evaluator identity and limitations.
• Explain why temperature zero does not prove determinism.
• Explain why a deterministic fixture does not prove live quality.

Pass when another engineer can reconstruct the intended configuration, locate raw observations and evaluators, distinguish the three trials from configuration changes, and name unavoidable gaps. Fail when a model alias, screenshot, seed, or final score stands in for the package.

The MD-03 baseline state

Mosaic Desk enters MD-03 with:

• provider-neutral request, result, and error interfaces;
• a selected access posture and adapter identity;
• task-contract and case-set revisions;
• complete behavior-facing configuration fields;
• managed-path unknowns or open-weight artifact/runtime identity;
• a canonical run hash and replay instructions;
• raw synthetic outputs and deterministic validation events;
• explicit trial and nondeterminism notes;
• evaluator versions and evidence limitations;
• privacy/retention authority required for future non-synthetic traces;
• a deterministic test double whose claims stop at software contracts.

The baseline is a comparison point, not ground truth, a quality guarantee, or production approval. Chapter 6 changes the instruction/message interface one component at a time. If that chapter cannot preserve this baseline, it cannot attribute an observed difference to the instruction change.