Evaluate Retrieval and Generation Together

A good answer can hide a failed evidence path

Mosaic Desk enters Chapter 12 with a complete MD-05 path: a frozen synthetic corpus, ingestion and chunk identities, authorized candidate trace, selected evidence envelopes, context assembly trace, stable citation handles, explicit source states, and a typed proposal interface.

Now imagine two cases.

In the first, retrieval places the correct current bulletin first and assembly includes its qualifying span. The generated proposal ignores the condition and recommends a next step unsupported by the evidence.

In the second, retrieval returns no eligible supporting unit because the corpus lacks the required bulletin. The generated behavior abstains and explains the missing evidence state.

An answer-only evaluator may mark both cases “unhelpful.” A retrieval-only dashboard may mark the first successful and the second failed. Neither view explains the system behavior. The first is a generation-use failure after retrieval success. The second is a retrieval/corpus failure with correct bounded generation behavior.

Joint evaluation preserves both component evidence and the end-to-end consequence.

Start with answerability and expected behavior

Every evaluation case needs more than a question and reference answer. Record:

• case and segment identity;
• authenticated scope and allowed source families;
• applicable source snapshot and label version;
• answerability state under that snapshot;
• eligible supporting, conflicting, stale, and forbidden unit IDs;
• claim-level qualifying spans and source authority notes;
• expected proposal state: answer, partial, abstain, escalate, or fail closed;
• prohibited claims and effects;
• consequence and required judge type;
• uncertainty or annotator disagreement.

This prevents a correct abstention from being scored as an empty answer when no eligible evidence exists. It also prevents a fluent answer from passing when the source is forbidden or the requested decision belongs to a human authority.

The case record is versioned. A source update can change answerability without any model or retriever change. Re-adjudicate affected cases instead of treating old labels as timeless ground truth.

Evaluate each layer with the evidence it can support

End-to-end RAG quality can fail at corpus, query, retrieval, ranking, context, generation, citation, or evaluation layers. Foundational RAG, dense retrieval, late interaction, and RAG evaluation research motivate the linked stages, but this eight-layer taxonomy is the book’s diagnostic synthesis. It is not a universal standard, and a local system may split layers further. [CLM-034]

Corpus

Ask whether the required source exists in the frozen snapshot, is ingested completely, carries the correct revision and permissions, and is represented by an evidence unit that preserves its qualifiers. A missing source cannot be repaired by reranking.

Query

Ask whether the evidence need, exact identifiers, negation, appliance scope, language, date, and allowed claim type survived query construction or rewriting. A query can faithfully retrieve the wrong question.

Candidate retrieval

Ask whether at least one eligible supporting unit appears within a declared candidate depth and whether forbidden units remain absent. Candidate recall is about opportunity, not final use.

Ranking and reranking

Ask where labeled supporting and conflicting units appear relative to the assembly cutoff. Rank-oriented metrics can summarize many cases, but case-level traces reveal whether a critical unit was displaced by duplicates or noise.

Context assembly

Ask whether the selected unit, qualifying span, provenance, trust label, conflict, and reserved output budget survive packing. A retrieved unit that is truncated before its exception is not available in the form the evaluation assumes.

Generation

Ask whether the proposal uses included evidence, preserves uncertainty and conflict, cites allowed handles, avoids unsupported claims, and enters the expected bounded state.

Citation

Ask whether each citation resolves to the intended source revision and span and whether that span supports the associated claim. Correct formatting and correct support are different checks.

Evaluation

Ask whether labels, metrics, graders, adjudicators, and aggregation rules are credible for the claim being made. An evaluator can be the failing component.

Long description

A colorful three-dimensional evaluation bench has two independent stations. The retrieve station inspects eligible candidates, rank, and source state. The generate station inspects evidence use, citations, and bounded behavior. Their signed evidence cards enter a joint case folder that preserves both results, including a branch where retrieval fails but abstention passes.

Choose metrics from diagnostic questions

Do not begin with the metrics a library happens to expose. Begin with the failure question.

For retrieval:

• candidate recall at depth asks whether any labeled eligible unit was available;
• precision at depth asks how concentrated labeled units were among retrieved candidates;
• reciprocal rank emphasizes the position of the first labeled unit;
• discounted cumulative gain can represent graded relevance across positions;
• forbidden-candidate rate checks authorization invariants and should also be inspected as a hard case-level failure;
• freshness/applicability pass rate checks whether selected units meet source-state rules;
• stage latency and operation counts show the resource path under a frozen workload.

For generated behavior:

• required-state accuracy asks whether answer, partial, abstain, escalation, or fail closed was selected;
• claim support asks whether each evidence-bearing item is supported by an included span;
• citation resolution and citation entailment ask different questions;
• unsupported-claim and prohibited-effect rates expose boundary violations;
• conflict preservation asks whether disagreement remained visible;
• completeness asks whether all required supported fields were addressed without rewarding invented content.

For the joint system:

• answerability-conditioned behavior separates supported from no-evidence cases;
• retrieval-success/generation-failure counts reveal ignored or misused evidence;
• retrieval-failure/correct-abstention counts preserve safe behavior rather than scoring it as total failure;
• end-to-end task criteria connect the technical path to the Chapter 2 contract without granting approval authority.

Retrieval metrics and generated-answer metrics answer different questions and should not be collapsed prematurely. BEIR and RAGAS illustrate retrieval and generated-output evaluation constructs under their own assumptions. Metric values depend on relevance granularity, label quality, cutoff, evaluator configuration, and case mix; automated generation metrics require calibration. [CLM-035]

A composite score may eventually support a bounded release rule, but only after its components, weights, hard gates, segments, and failure consequences are explicit. Never allow a strong average to cancel an authorization violation or a failure on a designated critical segment.

Build the joint failure matrix

For each case, preserve retrieval state, assembly state, generation state, expected behavior, observed consequence, and likely responsible layer. Useful rows include:

Retrieval	Assembly	Generation	Interpretation
supporting unit found	included intact	supported proposal	candidate for end-to-end pass
supporting unit found	included intact	unsupported claim	generation-use failure
supporting unit found	qualifier truncated	wrong claim	assembly failure before generation diagnosis
supporting unit absent	no-evidence state	abstain	retrieval/corpus failure; generation behavior passes
conflicting units found	conflict preserved	escalate	bounded behavior passes; domain adjudication remains
forbidden unit appears	must fail closed	any answer	authorization boundary failure
wrong source marked relevant	included intact	source-faithful claim	label/source-authority or retrieval failure; not correctness

The matrix does not prove causality by itself. It narrows hypotheses using recorded interventions and traces. To strengthen attribution, hold later stages constant while substituting known-good evidence, or replay generation with the same assembled bundle. Change one component at a time when practical.

Treat label quality as part of measurement

Retrieval metrics require judgments about which evidence units support which claims. Generated-behavior metrics require judgments about answerability, support, completeness, and acceptable abstention. Those labels can be incomplete or disputed.

Create an annotation record with case version, source snapshot, unit IDs, qualifying spans, rubric version, annotator roles, independent judgments, disagreement reason, adjudicator, and final status. Do not force adjudication when the correct state is unresolved. A disputed label can remain outside a headline aggregate while still appearing in the coverage report.

Granularity matters. A document-level relevance label can award credit even when the retrieved chunk omits the needed exception. A binary label can hide that one unit is supporting and another is merely background. A reference answer can become stale when the source revision changes. Match label granularity to the claim and keep the source span available for review.

Measure agreement only for the judgment process it describes. High agreement among general reviewers does not confer domain authority. Low agreement may reveal an ambiguous rubric, insufficient source context, a genuinely contested policy, or an unsuitable automated grader. Record the cause before changing thresholds.

Aggregate only after segment inspection

An overall score mixes answerable and unanswerable cases, languages, evidence conditions, appliance families, query types, and consequence levels. Report the case distribution and named segments first.

At minimum, separate exact identifiers from paraphrases, single-source support from conflict, current evidence from stale-only, English from the multilingual/code-switched segments in the task contract, and ordinary guidance from designated high-consequence cases. Preserve counts and uncertainty; a segment with two synthetic cases cannot support a broad population claim.

Hard invariants remain case-level gates. An unauthorized candidate, prohibited effect, fabricated citation, or failure to abstain when required cannot be averaged away by many easy successes. The accountable release owner may later define a bounded decision rule, but the engineer supplies the decomposed evidence and limitations.

Run paired interventions

Three paired tests are especially useful.

Oracle-context replay

Replace retrieved evidence with the adjudicated eligible bundle while preserving the generation configuration. If the failure remains, retrieval was not sufficient to explain it. If behavior recovers, inspect corpus, query, candidate, rank, and assembly layers.

An oracle bundle is an evaluation instrument, not a production path. It may be expensive or require domain judgment unavailable online.

Frozen-context generation replay

Keep the exact assembled bundle and vary only model, message, decoding, or output-validator configuration. This isolates generation-facing changes while preserving evidence supply.

Frozen-output evaluator replay

Keep the exact candidate, context, and proposal traces while varying metric or grader versions. Disagreement exposes evaluator sensitivity without changing system behavior.

All replays bind case, corpus, query, retriever, assembler, model/runtime, message, decoding, schema, validator, and evaluator identities. Otherwise a changed result cannot be attributed.

Calibrate automated indicators against credible judgment

Automated evaluation can scale coverage and make regressions visible. It can also reproduce evaluator-model preferences, miss domain qualifiers, or reward a source-faithful answer grounded in an incorrect policy.

Groundedness or faithfulness metrics do not prove source truth or organizational authority. NIST guidance, RAGAS research, and provider citation documentation support evaluation and provenance practices while preserving their limits. Terminology varies, automated evaluators require calibration, citation support requires task-specific verification, and domain owners decide whether a source is correct and authoritative for the case. [CLM-036]

Use an evaluator ladder:

1. deterministic checks for IDs, permissions, revisions, schema, citation resolution, and prohibited effects;
2. reference-based or rule-based checks where a stable answer exists;
3. calibrated model indicators for bounded language judgments;
4. trained human review for ambiguous support and behavior;
5. qualified domain review for source correctness and policy meaning;
6. named accountable authority for release, risk, or consequential decisions.

Record disagreements. Do not overwrite the original label because an automated grader was decisive. A disputed-case queue should contain raw evidence references, competing judgments, rationales, versions, authority needed, and disposition state.

Preserve managed and open-weight evaluator differences

Both generator paths consume the same case records, retrieval traces, assembled bundles, and intended-use criteria. This keeps the comparison about behavior rather than mismatched evidence.

A managed path may expose native citation locations, token usage, or provider evaluation services. Record feature versions, hidden behavior, data terms, and unavailable logits or internal representations. Do not treat provider-native metrics as portable definitions.

An open-weight path may allow direct tokenizer traces, constrained evaluator deployments, log probabilities, or hardware profiling. It also requires ownership of evaluator artifacts, templates, runtimes, quantization, serving configuration, and capacity. Extra observability does not make a metric correct.

The common output is a provider-neutral case result with component evidence, limitations, disputed judgments, and no effect.

Failure injection: two answers, opposite diagnoses

Run two synthetic Mosaic cases.

Case A: evidence found, condition ignored

EV-A7-SEC3 is eligible, ranked first, and included intact. The current bulletin says inspection applies only when a documented precondition holds. The proposal recommends inspection without checking the condition.

• corpus: pass;
• query: pass;
• retrieval/rank: pass;
• assembly: pass;
• generated support: fail;
• citation resolution: may pass;
• citation entailment: fail for the unconditional claim;
• expected disposition: invalid or escalate, no effect.

Case B: retrieval wrong, abstention correct

No eligible supporting unit exists in the snapshot. A noisy candidate is correctly excluded. The proposal returns abstain with missingEvidence and cites nothing.

• corpus/coverage: gap recorded;
• retrieval opportunity: fail for support, pass for forbidden exclusion;
• assembly: correct no-evidence state;
• generation behavior: pass;
• end-to-end usefulness: bounded by the missing source;
• expected disposition: abstain, no effect.

LLME-CASE-003 contributes the limited lesson that retrieval and generation remain linked failure surfaces in the original RAG method. LLME-CASE-004 contributes the limited lesson that retrieval stages are separable experiment variables. Their published benchmark results do not establish Mosaic quality, current model rankings, or production fitness.

Long description

A realistic three-dimensional diagnostic tree begins with a failed proposal card and branches backward through eight labeled stations. Each branch contains a trace artifact and a question mark rather than a blame symbol. One path ends at missing corpus evidence, another at truncated context, another at unsupported model use, and another at evaluator disagreement.

Practice: defend a causal diagnosis

Build eight paired synthetic cases covering corpus absence, query drift, candidate miss, rerank displacement, qualifier truncation, ignored evidence, citation mismatch, and evaluator disagreement.

For each case:

1. declare answerability, eligible sources, expected state, and prohibited behavior;
2. score retrieval, assembly, generation, and citation separately;
3. select metrics by diagnostic question and state their limitations;
4. preserve case-level traces beneath every aggregate;
5. run an oracle-context, frozen-context, or frozen-output intervention;
6. classify the responsible or still-uncertain layer;
7. route domain-sensitive disputes to the qualified judge;
8. retain safe abstention as positive bounded behavior;
9. identify any hard failure that an average cannot offset;
10. emit a joint result with no release or external-effect authority.

Pass when another reviewer can reproduce the classification, distinguish absence from misuse, see the raw evidence behind automated indicators, and identify questions that no metric can settle.

The first MD-06 evidence

Mosaic opens MD-06 with:

• versioned answerability and relevance labels;
• independent corpus, query, candidate, rank, assembly, generation, citation, and evaluator evidence;
• metric-to-question rationale and limitations;
• paired component and end-to-end case results;
• oracle-context and frozen-component intervention records;
• a joint failure-attribution matrix;
• preserved correct abstentions and hard authorization failures;
• calibrated automated-indicator comparisons;
• a disputed-case queue with judge and authority assignments;
• no universal threshold, production claim, or automated-judge authority.

This evidence is diagnostic, not yet representative. Chapter 13 must build the case population, segmentation, annotation, split, contamination, and refresh process needed to make broader evaluation claims credible. Chapter 14 will then assign error severity and judgment methods without confusing technical evidence with accountable decision authority.