Design the Retrieval Question

“Add RAG” is not a requirement

Mosaic Desk leaves MD-04 with a finite context contract. Every candidate record has a source class, trust state, permission and freshness decision, token allocation, and omission behavior. The typed output requires valid evidence references and cannot create an effect.

The current evidence inventory is still assembled manually. Someone proposes the obvious next step: “Put all company documents in a vector database and add RAG.”

That sentence chooses a family of implementation before defining the evidence problem.

It does not say which Mosaic claims require external evidence, which records are authoritative, who owns them, how quickly they change, which tenant may see them, what unit counts as relevant, how absence is represented, or how retrieval and generated behavior will be evaluated independently. It treats similarity as if it implied permission and support.

Retrieval engineering begins with a question:

For which task claims does Mosaic need external, updateable, attributable evidence, and what authorized evidence unit would allow the system to support, withhold, or escalate each claim?

The answer may be retrieval-augmented generation. It may also be a deterministic lookup, a search interface for a reviewer, a fixed policy bundle, a parametric response for a low-consequence task, or abstention. Technology follows the evidence contract.

Enumerate claims that need evidence

Start from the language-task clauses and typed proposal fields.

For each possible claim, ask:

• Can the application answer from deterministic state?
• Must the information be current or updateable without changing model weights?
• Must a reviewer inspect the supporting source?
• Does the answer depend on tenant, appliance family, date, jurisdiction, warranty program, or case-specific history?
• Is absence meaningful?
• Can conflicting sources exist?
• Who designates source authority?

Examples for fictional Mosaic:

Claim need	Candidate path	Reason
Case identifier and authorized tenant	Deterministic lookup	Exact application state; model inference is inappropriate.
Current bulletin stop condition	Retrieval or deterministic policy lookup	Updateable and attributable external evidence is required.
Exact part compatibility	Deterministic catalog lookup where available	Structured identity and compatibility rules may be stronger than prose retrieval.
Summarize supplied technician note	Direct context	The authorized record is already in the request bundle.
Explain a general low-consequence concept	Parametric candidate plus evaluation	External evidence may not materially improve the defined task.
Warranty approval	Abstain/escalate	Mosaic has no authority regardless of retrieval.

Retrieval is justified only when external evidence ownership, freshness, or coverage materially improves the task. The original RAG research established one architecture for combining non-parametric retrieval with generation on knowledge-intensive benchmarks, while NIST guidance reinforces provenance, risk, and governance concerns. The decision rule here is a task-specific engineering synthesis, not a mandate to use RAG. [CLM-025]

Long description

A colorful three-dimensional decision tree begins with current, attributable, authorized evidence need. Exact application facts route to a lookup drawer, reviewer exploration routes to search, updateable evidence for generated proposals routes to retrieval, bounded general language can route to a parametric branch, and unsupported or unauthorized needs end at abstain.

Separate five evidence paths

Parametric response

The configured model generates without application-supplied external evidence. This may be appropriate only where the task contract permits it and evaluation supports the behavior. It cannot cite a current policy merely because the prose sounds familiar.

Deterministic lookup

Software queries a structured authoritative system by exact or constrained keys. Use it when the task is fundamentally identity, status, permission, or compatibility lookup. Do not convert a database fact into fuzzy retrieval without reason.

Search for a human

The system returns candidate records for an authorized reviewer without generating a synthesized answer. This can be the better boundary when judgment is complex, source reading is essential, or generation adds little value.

Retrieval plus generation

An authorized retrieval path supplies evidence units to the context assembler; the model produces a bounded proposal and provenance fields. This adds at least two behavior systems: retrieval selection and generated use.

Abstain or escalate

No allowed path can supply required evidence, sources conflict beyond deterministic resolution, or the decision lies outside Mosaic’s authority. The correct output is an explicit state, not a best-effort search across unauthorized sources.

Hybrid tasks can use several paths, but each claim needs one declared evidence route.

Build the source and authority map

Do not begin by ingesting every available document. Build a map.

For each source family record:

• source owner and domain authority;
• system of record and stable identifier;
• tenant, case, role, region, and purpose access rules;
• creation, effective, expiry, and supersession timestamps;
• allowed claim types;
• evidence unit and parent record;
• update and deletion process;
• provenance fields that must survive indexing and context assembly;
• known conflicts and adjudication owner;
• consequences of absence, staleness, or unauthorized status.

Mosaic’s manuals, service bulletins, warranty policies, parts catalog, technician notes, and case messages do not have equal authority.

A manual may describe general repair procedure but not warranty eligibility. A current bulletin may override an older instruction for a specific appliance family. A technician note is an observation, not policy. A parts record may be authoritative for compatibility within its revision. A customer message is evidence of reported experience, not proof of cause.

Long description

A crisp library-workshop contains separate shelves for manuals, bulletins, warranty, parts, and notes. Every document has an owner tab, freshness clock, and permission lock. A bright semantically similar card is blocked because its tenant lock does not match, while a less flashy authorized record remains eligible.

Relevance is only one predicate

A candidate can be semantically similar and still be unusable.

Define eligibility before ranking:

eligible(record, request) =
  permitted principal and tenant
  AND allowed purpose
  AND current or explicitly historical state
  AND applicable appliance/model scope
  AND accepted source family for the claim
  AND intact provenance

Only eligible records enter relevance comparison. This order matters. Filtering after generation can expose unauthorized content to the model and allow it to influence output even if the citation is removed.

Relevance does not establish source authority, permission, currency, or answerability. NIST risk guidance, prompt-injection research, and official citation interfaces all support preserving provenance and trust boundaries, but metadata and authority rules remain organization-specific. A provider citation feature can format source links; it cannot decide that another tenant’s bulletin is allowed or authoritative for Mosaic. [CLM-026]

Define the evidence unit

“Document” may be too large, while an arbitrary token chunk may be too small.

An evidence unit should preserve enough meaning and provenance to judge a claim. It may be:

• a bulletin section with title, effective date, appliance scope, and supersession status;
• one warranty clause with program and jurisdiction metadata;
• a parts compatibility row plus catalog revision;
• a technician-note event with author role, timestamp, and case scope;
• a message turn with quoted-history boundaries;
• a structured policy object rather than text.

Do not choose unit size solely for embedding convenience. A fragment that loses a negation, condition, table header, or supersession link can rank well while becoming misleading.

Write the unit contract before chunking:

• stable unit ID and parent source ID;
• exact content boundaries;
• claim types it may support;
• required metadata;
• permission inheritance;
• freshness/supersession behavior;
• display/citation target;
• invalidation and reindex trigger.

Chapter 10 will implement candidate generation, chunking, representation, and reranking. Chapter 9 freezes what those mechanisms must preserve.

Write query intent without pretending it is the answer

A retrieval query represents an evidence need. It is not ground truth.

For the washer case, a request such as latch problem is underspecified. A contract-level query object might contain:

• appliance family and confirmed model, when authorized;
• symptom or error code from named case evidence;
• required claim type, such as service-stop-condition;
• applicable date;
• tenant/case scope;
• allowed source families;
• language and exact identifiers;
• freshness rule;
• maximum eligible evidence units;
• required no-evidence behavior.

The query generator can be deterministic, model-assisted, or mixed. If a model rewrites the query, record both forms and test whether identifiers, negation, scope, and permission constraints survive. The model cannot expand allowed source scope.

Separate retrieval claims from generated-behavior claims

LLME-CASE-003 describes the original RAG method boundary: a dense retriever supplied non-parametric evidence to a generator, and the paper reported results on its selected research tasks. It does not provide a modern production recipe, Mosaic benchmark, or universal improvement claim.

The durable lesson is separation.

Retrieval claims ask:

• Did an eligible supporting unit appear in the candidate set?
• Did unauthorized, stale, or inapplicable units remain excluded?
• Was the relevant unit ranked within the context budget?
• Did provenance and parent identity survive?
• What happened when no eligible unit existed?

Generated-behavior claims ask:

• Did the proposal use the supplied evidence correctly?
• Did each evidence-bearing field cite an eligible unit?
• Did the system preserve conflicts and limitations?
• Did it abstain or escalate when evidence was absent?
• Did it avoid unsupported or unauthorized claims?

A retrieval-grounded system needs independent retrieval and generated-behavior claims. The original RAG and RAGAS research provide method and evaluation ideas under their conditions; metric selection and automated evaluator credibility remain bounded. A good retrieval result can still be ignored or misused, and a fluent answer can hide retrieval failure. [CLM-027]

Define answerability states

Retrieval does not return only results or no results.

Supported

At least one eligible current evidence unit supports the bounded claim, and the typed proposal passes provenance/semantic validation.

Partially supported

Evidence supports some fields but required elements remain missing. Narrow the proposal and expose the gap.

Conflicting

Eligible sources disagree and no deterministic authority rule resolves them. Preserve both and escalate or present bounded uncertainty.

Stale only

Relevant units exist but fail currency requirements. Do not present them as current support.

Unauthorized only

Relevant records exist outside allowed scope. Treat the request as having no eligible evidence; do not reveal the existence or content beyond authorized policy.

No evidence

No eligible unit supports the claim. Abstain or ask for permitted missing information. Parametric continuation cannot silently replace required evidence.

Not retrieval-appropriate

The claim belongs to deterministic lookup, human judgment, or an authority outside Mosaic. Route it accordingly.

These states feed the Chapter 7 typed result and Chapter 8 context trace.

Failure injection: relevant but forbidden

The synthetic corpus contains two bulletins with similar latch language. SB-TENANT-B-4 is a strong semantic match but belongs to another tenant. SB-A-7 is authorized for Mosaic’s case and includes the applicable date and appliance scope.

A naive system ranks the other-tenant bulletin first, includes it in context, and later removes the citation because permission validation fails. That is too late. Unauthorized content already influenced the generation path.

The correct sequence is:

1. authenticate the principal and bind case/tenant scope;
2. select allowed source families and purpose;
3. exclude unauthorized, revoked, stale-for-current-use, and inapplicable units;
4. rank only eligible candidates;
5. preserve unit and parent provenance;
6. pack within the MD-04 budget;
7. generate a typed proposal;
8. validate evidence references and behavior state;
9. expose no effect.

Now revoke the authorized bulletin. The system enters stale-only or no-evidence state according to remaining records. It does not fall back to the other tenant’s bulletin.

Use the deterministic retrieval-question contract

The companion adds a source map and decision fixture under retrieval/.

It validates that:

• exact application state selects deterministic lookup;
• updateable, attributable bulletin claims can justify retrieval;
• warranty approval routes to abstain/escalate rather than retrieval;
• eligibility precedes relevance;
• other-tenant and revoked units never become candidates;
• every evidence unit retains source, owner, freshness, permission, and parent identity;
• retrieval and generated-behavior claims remain distinct;
• no-evidence and conflict states are explicit;
• no path exposes an external effect.

The fixture does not implement embeddings, a vector store, or a model call. It tests the question and boundary that later implementation must satisfy.

Reject tempting retrieval projects early

Two rejection patterns save substantial work.

The exact-record problem

A case has a confirmed part number, and the parts system exposes an authorized compatibility relation keyed by appliance model and revision. Embedding descriptions and asking a generator to infer compatibility adds ambiguity to an exact governed lookup. Use the deterministic path, retain its source revision, and let Mosaic summarize the result only if that serves the task.

Retrieval may still help a reviewer find explanatory manuals, but it does not replace the compatibility fact.

The missing-authority problem

A stakeholder wants retrieval over warranty documents so Mosaic can approve claims. Better evidence does not grant decision rights. Retrieval may assemble the clauses for an authorized warranty reviewer, but the output remains a proposal or escalation. Reject the autonomous approval requirement and preserve the human authority boundary.

A third rejection is useful: if the input already contains the complete authorized evidence unit and no freshness gap exists, adding an index may not materially improve the task. Evaluate direct context first.

For every rejected RAG case, record the requirement, simpler path, evidence, limitation, owner, and trigger that could reopen retrieval. “RAG is unnecessary” should be as traceable as “RAG is justified.”

Define the contract that Chapter 10 must earn

The retrieval implementation will be acceptable only if it can answer the questions frozen here:

• Which eligible unit should be found for each query fixture?
• Which unauthorized, stale, revoked, or inapplicable unit must never enter candidates?
• What candidate-set depth is compatible with the MD-04 budget?
• Which identifiers and provenance fields must survive representation and reranking?
• How are exact identifiers, rare terms, language variation, and semantic paraphrases tested?
• What result represents empty eligible evidence?
• Which metrics judge retrieval, and which judge generated use?
• Who owns corpus updates, index rebuilds, access rules, and domain authority?

These are acceptance questions, not predetermined numeric thresholds. Chapter 10 will generate evidence and preserve negative findings rather than inventing a target now.

Practice: turn “search manuals” into a contract

Choose one Mosaic claim and produce:

1. the claim and why external evidence is or is not needed;
2. the selected path: parametric, lookup, search, retrieve, or abstain;
3. authoritative source families and owners;
4. tenant/purpose/role permissions;
5. freshness and supersession rules;
6. evidence-unit boundaries and required metadata;
7. query object and identifiers;
8. retrieval acceptance questions;
9. generated-behavior acceptance questions;
10. supported, partial, conflict, stale, unauthorized, and no-evidence behavior;
11. trace fields and reindex triggers;
12. named authorities and non-scope.

Then reject two unjustified RAG proposals: one where an exact deterministic lookup is stronger, and one where the requested decision is outside Mosaic’s authority.

Pass when relevance cannot override eligibility, absence is explicit, every source has an owner and currency rule, retrieval and answer claims are independent, and no implementation product has been selected prematurely.

The MD-05 retrieval problem statement

Mosaic Desk opens MD-05 with:

• a claim-to-evidence-path decision tree;
• authoritative source and owner map;
• permission, tenant, purpose, freshness, and supersession contract;
• evidence-unit specification with parent provenance;
• structured query intent;
• eligibility-before-relevance rule;
• supported, partial, conflicting, stale-only, unauthorized-only, no-evidence, and not-retrieval-appropriate states;
• independent retrieval and generated-behavior claims;
• context-budget handoff;
• no-effect and authority boundaries;
• a relevant-but-forbidden failure fixture.

No vector database, embedding model, chunking strategy, or reranker has been selected. Chapter 10 will build and compare retrieval paths against this contract. A candidate earns use by retrieving eligible evidence within the budget, not by producing a plausible similarity score.