Curate, Deduplicate, and Separate

Curation is a claim-changing operation

Chapter 4 froze mosaic-data-recipe.json with an explicit negative decision: a convenient production export is unauthorized, contains personal data, and overlaps holdout families. Chapter 5 does not search for replacement data. It executes the accepted recipe against a small fabricated inventory and asks whether the resulting partitions can support an interpretable experiment.

Every normalization, filter, duplicate decision, and split assignment changes what the future training result could mean. The pipeline must therefore make accepted, rejected, quarantined, and unresolved rows equally traceable. A clean output directory without a decision ledger is not clean evidence.

The companion contains synthetic strings and hashes. Similarity scores and thresholds are chosen to demonstrate decisions; they are not recommended production values, privacy metrics, or contamination guarantees.

Preserve the raw identity before transformation

Canonicalize only the record envelope needed to compute an immutable source digest. Keep content and metadata roles distinct so an ordering change in JSON does not invent a new example while a content change does. Bind every child transformation to:

• raw record digest;
• source-family and parent IDs;
• recipe and transform versions;
• authoring or generation batch;
• authority and retention state;
• exact before/after fields;
• decision, reason code, and reviewer;
• timestamps or batch sequence where required by the audit.

Do not retain raw content beyond an approved rule merely because hashing is useful. If an authority requires deletion, retain only the permissible deletion evidence and derived identifiers. A digest can remain personal or linkable data; hashing is not anonymization.

The fabricated pipeline uses content-addressed IDs to support replay. It explicitly refuses to claim that its records are anonymous, private, or licensed.

Normalize without erasing the task

Normalization should remove representational noise only when the behavior contract permits it. Mosaic records Unicode normalization form, line-ending and whitespace rules, header handling, quoted-thread segmentation, and approved signature detection. Appliance codes, evidence identifiers, language switches, message roles, punctuation used by the structured parser, and source revisions must survive.

Run invariants after each step:

1. authorized fields remain authorized;
2. task-bearing codes round-trip;
3. message and evidence boundaries remain visible;
4. language/script labels are not silently replaced;
5. source-family lineage remains connected;
6. target/protected clauses remain mapped;
7. rejected content cannot re-enter through a derivative.

If a normalization rule makes two semantically distinct records identical, the rule is too destructive for this task. Version and revise it rather than manually restoring selected examples.

Separate quality, authority, and duplicate decisions

A record can be well written and unauthorized. It can be authorized and malformed. It can be high quality and duplicate a frozen evaluation family. Use independent decision fields.

Mosaic’s filter stages are:

• source authority and allowed-purpose gate;
• prohibited content and retention gate;
• structural/schema validation;
• evidence-link and target-state validation;
• language/domain/slice validation;
• exact duplicate detection;
• family and approximate overlap detection;
• split eligibility and group assignment;
• post-split contamination audit.

Each rejection has a stable reason such as unauthorized-purpose, personal-data-unresolved, schema-invalid, unsupported-target, exact-duplicate, or holdout-family-overlap. A quarantine has an owner and next decision. Silent dropping makes acceptance rates, mixture changes, and bias impossible to audit.

Filtering and normalization choices alter the learned distribution and must be recorded as data decisions. [CLM-015] A language detector, quality heuristic, or signature remover can change which dialects, short messages, or refusal cases remain. The effect on behavior requires target experiments; a higher filter score is not a better dataset by definition.

Detect exact duplicates first

Exact duplication is representation-dependent. Compute at least a raw digest and a declared normalized-content digest. For structured episodes, consider per-field and whole-record identities. Two records with different source IDs but identical authorized input, context, and target may be duplicate learning units. Two records with identical text but different source revisions may need separate evidence treatment rather than automatic merging.

Choose a canonical survivor using a versioned rule: stronger provenance, clearer authorization, primary origin, earlier approved version, or higher reviewed completeness. Record all aliases. Never delete the duplicate relationship because a row was removed from training.

The Dolma and deduplication studies in LLME-CASE-006 demonstrate large-scale documented filtering and duplication work in their own pretraining setting. Their thresholds, corpus rights, model results, and scale do not transfer to Mosaic’s post-training dataset.

Treat near duplication as a reviewable hypothesis

Deduplication can reduce memorization and evaluation contamination but thresholds and scope create tradeoffs. [CLM-013] Approximate methods may compare character n-grams, tokens, normalized fields, embeddings, templates, metadata, or known transformation graphs. Each representation misses some overlap and invents some similarity.

Mosaic deliberately creates two traps.

False duplicate removal. Two unrelated service messages end with the same long templated signature. A naive whole-message similarity rule marks them duplicates. The repaired pipeline removes the known signature only for similarity comparison, preserves original content under its retention rule, and compares task-bearing fields. The records remain distinct.

Indirect evaluation leakage. A training candidate paraphrases a frozen evaluation message and changes whitespace, so exact hashes differ. Both records retain the same synthetic source-family ID, and a family-aware plus approximate-content audit detects the crossing. The training candidate is rejected as holdout-family-overlap.

No threshold eliminates semantic uncertainty. Sample matches above and below the boundary, stratify by language and record type, record false-removal and missed-overlap examples, and keep residual uncertainty in the dataset card.

Calibrate a threshold without calling it truth

Build a small reviewed calibration set of known same-family transformations, known distinct records that share templates, and ambiguous pairs. The set itself must be versioned and separated from final evaluation. Compare candidate representations and thresholds against the reviewed relationship labels. Report false merges, missed relationships, and uncertain pairs by language and record type.

Select a threshold for a declared use. A conservative train-evaluation firewall may route ambiguous pairs to rejection or review, while within-train deduplication may retain ambiguous repetition to avoid erasing legitimate variants. One number need not govern every boundary.

Recalibrate when normalization, tokenizer, embedding model, signature rules, language mix, or source type changes. Even then, the result estimates behavior on the calibration pairs; it does not establish a universal semantic-duplicate boundary.

Long description

A colorful realistic three-dimensional map shows record cards moving toward train, development, and evaluation barriers. Two unrelated cards share a detachable signature strip and are released from a false alarm. A paraphrased child card remains linked by a family cord to an evaluation card and is stopped at the overlap gate.

Assign groups before assigning rows

Random row splitting is invalid when records share a thread, source document, appliance incident, template seed, translation, paraphrase, generator prompt, or synthetic parent. Construct a group key that captures every known dependency, then assign the group to one purpose.

Train, development, evaluation, retention, and control sets need explicit separation and identity. [CLM-014]

• Train supplies supervised learning records after all gates.
• Development supports bounded iteration and method choices; repeated use can overfit decisions.
• Evaluation measures the frozen target claim and remains inaccessible to data authors and training procedures except through approved evaluation execution.
• Retention protects existing capabilities and forbidden-regression segments.
• Control diagnoses no-tune, prompt/context, routing, retrieval, evaluator, and deterministic behavior without becoming a training reservoir.

Historical overlap with a base model’s pretraining may be unknowable, especially for managed services or opaque corpora. Record that limitation. The local firewall can prove only what it tested across known artifacts.

Physical and logical separation should agree. Give evaluation and retention vaults distinct access roles, storage prefixes, encryption/key policies where competent owners require them, and pipeline credentials. The training authoring process receives only train IDs and aggregate coverage gaps. A path check should fail if it attempts to resolve protected target content.

This is an engineering control design, not a security certification. Security and privacy owners decide the actual storage, access, monitoring, and retention implementation. The companion demonstrates only an allowlisted reference boundary between synthetic JSON objects.

Long description

Five colorful three-dimensional vaults sit on separate platforms. Each carries a digest seal and manifest. Family cords join records only inside one vault; a crossing cord is cut at an audit checkpoint and routed to a rejection ledger.

Audit every boundary after the split

Pre-split deduplication is not enough. Compare train against development, evaluation, retention, and control; development against every protected set; and all derived families against their parents. Run exact, normalized, family, template-seed, and declared approximate checks. Include prompt examples, retrieval fixtures, manuscript examples, and evaluator references where they could leak expected targets.

For each pair, publish:

• comparison representations and versions;
• thresholds and why they were chosen;
• match counts and sampled decisions;
• accepted aliases, rejected crossings, and quarantines;
• language/domain/slice distribution of matches;
• known blind spots and inaccessible corpora;
• reviewer and disposition.

After any repair, recompute manifests and rerun every boundary. A moved row changes both vault identities. Do not patch a report while leaving stale hashes.

Freeze manifests and the rejected ledger

Each vault manifest contains recipe ID, curation pipeline ID, split rule, group assignments, ordered record IDs and digests, source/slice counts, transform and filter summaries, and creation state. The combined dataset card links all five identities, overlap reports, rejection ledger, known gaps, authorities, and prohibited uses.

Audit artifacts expose, not eliminate, data limitations. LLME-CASE-008 shows the value of preserving stage-level recipe artifacts in one bounded post-training project. It cannot prove Mosaic’s local data are adequate or clean. A perfect manifest can faithfully document an unrepresentative dataset.

The rejected ledger stores minimal permissible evidence: derived record ID, source family, decision stage, reason code, rule version, reviewer state, deletion status, and links allowed by retention policy. It does not become a shadow training dataset. Rejected personal content must not persist merely for debugging.

Reconcile counts as conservation of evidence

At each stage, publish a count equation:

input = accepted + rejected + quarantined + pending + deleted-under-authority

Then break it down by source, language, behavior clause, and family. A row can change state, but the transition must have a reason and version. If the totals do not reconcile, stop the freeze. Missing rows may indicate a pipeline bug; duplicated rows may indicate a retry or join error.

Counts alone can still mislead. Ten paraphrases from one parent are one family for leakage analysis. A thousand easy positive records cannot erase the absence of conflict or abstention families. Report both row and family counts plus the reviewed mixture change from recipe to curated output.

The dataset card should answer: what was intended, what entered, what changed, what was rejected, what remains, who reviewed each decision, which sets are protected, which overlaps were checked, which could not be checked, and which future events invalidate the version. It is a map of evidence and limitations, not a badge.

Complete MD-10 at a bounded gate

mosaic-curation-manifest.json executes the recipe on fabricated records. It preserves the signature false-positive repair, rejects the paraphrased evaluation relative, assigns known families to one of five purpose vaults, records all hashes, and reports residual unknown overlap.

The gate returns md10-data-gate-complete-synthetic-only. That means another engineer can reproduce the synthetic partitions and their decisions. It does not authorize real data, approve retention, clear Chapter 2 compatibility blockers, or start training.

Chapter 6 receives only the train-eligible synthetic records plus read-only coverage summaries for protected vaults. It may author evidence-linked instruction records and render template fixtures. It must not read evaluation targets into training or convert rejected rows into demonstrations.

Practice: repair a contaminated split

Using the Chapter 5 companion:

1. reproduce raw and normalized digests;
2. verify every transform and filter decision;
3. inspect the signature-driven false duplicate and retain both valid records;
4. detect the paraphrased holdout relative through family and approximate evidence;
5. assign whole groups to purpose-specific vaults;
6. rerun every cross-vault audit after repair;
7. reconcile record counts with accepted, rejected, quarantined, and deleted states;
8. state what the final manifests cannot prove.

Pass when hashes, transformations, rejection reasons, group rules, split identities, overlaps, and residual uncertainty reproduce. Fail when a universal similarity threshold is claimed, deduplication is treated as privacy, rejected rows disappear, evaluation relatives cross into training, or manifest completeness becomes a data-fitness claim.