Name the Errors That Matter

An average cannot tell you what went wrong

Patchwork compares two synthetic ranking configurations. The challenger responds to more relevant cases. Its aggregate recall improves. If the team reports one number, the challenger wins.

The case trace tells a different story.

The challenger presents a known incompatible item in the long-tail segment and permits a denied source to cross the response threshold. It retrieves more of the ordinary items the team wants and fails two controls the product cannot trade away. The right decision is not ship the higher score. It is reject the challenger under the current policy, diagnose the mechanism, and preserve the negative evidence.

Evaluation becomes decision-grade only when errors have names connected to consequences. An error taxonomy says what failed. A severity model says why it matters. A segment report says whom or what context it affects. Detectability and reversibility say how likely the system is to catch and repair it. A control maps the failure to containment. A release disposition says what the evidence permits next.

Metrics are instruments inside that structure. Different task types and error meanings require different measurements; no library function chooses product consequence for you. [CLM-048]

Separate error, cause, and consequence

These three levels are often collapsed.

An error is an observable mismatch against a defined criterion: an incompatible candidate was displayed, a relevant candidate was omitted, a stale attribute was presented as current, or the system abstained when enough evidence existed.

A cause is the responsible mechanism or condition: query normalization removed a unit, the index lacked permission metadata, ranker weights overvalued copied seller text, a policy version was stale, or the interface hid an unresolved field.

A consequence is what the error can do: waste money, delay repair, expose restricted information, increase user effort, mislead a decision, or block useful service.

One error can have several causes. One cause can create several errors. One error can have different consequences by segment. Keep the layers separate so teams do not tune the nearest model whenever the product fails.

Suppose an irrelevant result appears first. The cause may be weak candidate generation, a missing structured filter, ranking, stale catalog data, or presentation that overemphasizes the first card. The consequence may be minor browsing effort for a common item and severe delay for a scarce long-tail repair. The taxonomy names the observed failure before diagnosis assigns cause.

Build a consequence-oriented taxonomy

PF-07 v0.2 defines seven error categories and one correct non-response outcome.

Irrelevant candidate

The candidate is permitted and not known incompatible but does not satisfy the task criterion. Consequences include wasted review effort and reduced discovery value. Detect through reference evidence or calibrated review.

Known incompatible candidate presented

A deterministic rule or authoritative evidence identifies a conflict, yet the candidate reaches assisted presentation. This is critical because a score or explanation has overridden policy. The release gate allows zero.

Unsupported compatibility implication

The evidence may support discovery but not fit. The interface or explanation implies more certainty than the evidence state permits. This can occur even when the candidate is relevant.

Stale evidence used as current

The system presents an attribute under a stronger freshness promise than its source supports. Staleness is semantic: one old descriptive field may be acceptable while a withdrawn compatibility rule is not.

Unauthorized evidence exposure

A denied or unknown-permission source influences the result or becomes visible. This is a purpose/access control failure, not a ranking-quality tradeoff. The release gate allows zero.

Certainty exceeds evidence

Language, badges, scores, or ordering cause users to infer a probability or guarantee the system has not established. A calibrated model score can still be semantically unrelated to user-facing fit confidence.

Unnecessary abstention

The system withholds a response even though sufficient permitted evidence exists under the contract. The consequence is lost coverage and added user effort. This is a real error, but its severity may differ from a false match.

Correct abstention

The system withholds a claim because a unit is missing, evidence conflicts, permission is unavailable, or no supported candidate exists. Correct abstention is a successful outcome, not a failed prediction.

The last category prevents teams from maximizing response rate by treating every abstention as failure.

Give every error record enough context

An error record contains:

• stable error ID and taxonomy version;
• behavior-contract clause;
• evaluation case and trace;
• observed error category;
• consequence and affected actor;
• segment and intersection;
• severity rationale;
• frequency evidence and denominator;
• detectability before consequence;
• reversibility after consequence;
• responsible layer hypothesis;
• existing control and control result;
• residual uncertainty;
• evaluator and rubric version;
• release implication;
• correction owner and authority.

Do not write severity: high without rationale. State whether the item can cause a wrong purchase, whether the interface exposes uncertainty, whether the user can correct before acting, and whether the failure is detected automatically.

Severity is not frequency

A rare permission exposure can block release. A frequent low-cost wording variation may not.

Use separate dimensions:

• frequency evidence: observed rate or constructed presence under a defined set;
• severity: magnitude and type of consequence;
• detectability: likelihood and timing of detection;
• reversibility: ability to restore the affected state;
• exposure: how many actors or requests can encounter the path;
• uncertainty: confidence in each assessment.

A product owner, domain authority, privacy/security authority, and release authority may have different roles in setting tolerances. The Applied AI Engineer supplies evidence and implementation options. Severity is not discovered by a metric alone.

For the fictional Patchwork policy:

• incompatible presented: critical, zero tolerated;
• permission exposure: critical, zero tolerated;
• stale evidence used as current: high, blocks the stronger assisted claim;
• unsupported implication: high, reduce scope or revise presentation;
• irrelevant candidate: moderate, optimize after controls pass;
• unnecessary abstention: moderate, improve coverage without weakening controls.

These are teaching choices, not universal marketplace thresholds.

Map category to control and disposition

The topology has four edges for every category.

1. Consequence: what can happen?
2. Segment: where does it concentrate?
3. Control: what detects or contains it?
4. Disposition: what may the team do next?

An incompatible candidate connects to wrong purchase or unsafe use, long-tail/critical categories, deterministic policy, and block/reject. Unnecessary abstention connects to lost utility, sparse or ambiguous segments, clarification/retrieval improvement, and revise. Correct abstention connects to avoided unsupported consequence and retain.

This structure stops quality issue from becoming a bucket that no owner can act on.

Choose metrics after criteria

A criterion says what counts as acceptable behavior. A metric summarizes observations under that criterion.

For candidate relevance, the team might use precision at a selected display depth, recall of eligible reference candidates, reciprocal rank, or pairwise preference. For exclusion policy, use violation count. For evidence fidelity, use clause-level rubric outcomes. For abstention, use coverage and error among responded cases. For calibration, compare score ranges with observed outcomes under a defined label.

The scikit-learn evaluation documentation usefully distinguishes task and metric families. It cannot tell Patchwork whether false matches or missed candidates are more costly. [CLM-048]

Avoid defaulting to F1 because it combines precision and recall. The equal weighting and harmonic form may not reflect consequence. Avoid AUC as a release command because it averages ranking across operating points the product never uses. Avoid accuracy where class balance hides severe errors. Avoid an LLM judge score where deterministic policy can decide.

Confusion matrices need a threshold and population

For a binary respond/abstain decision:

• true positive: respond when relevant supported evidence exists;
• false positive: respond when it does not;
• false negative: abstain when it does;
• true negative: abstain when it does not.

The terms are shorthand. They do not express permission or severity. A false positive caused by an irrelevant candidate differs from one caused by unauthorized evidence. Split them by taxonomy.

The counts depend on a threshold and case population. Change either and the matrix changes. Always record score semantics, threshold, suite version, segment, and denominator.

For retrieval/ranking, candidate-level and task-level matrices answer different questions. One query can present several candidates. A task can have multiple acceptable results. Define the unit.

Confidence is not correctness

A model score may be a logit, similarity, rank score, token probability, or post-processed estimate. It is not automatically the probability that a product claim is correct.

Calibration compares predicted probability-like values with observed outcome frequency under a defined population and label process. Research shows modern neural networks can be miscalibrated and that calibration methods are context-dependent. [CLM-049] Even well-calibrated values can be wrong on an individual case and can shift across segments or deployment conditions.

Patchwork’s synthetic scores are not probabilities. The reliability data groups score ranges and compares mean score with observed constructed relevance. A mismatch demonstrates why the interface cannot say 82 percent compatible.

Calibration must be checked after relevant change and by critical segment where evidence allows. A global curve can hide local miscalibration. Do not use a tiny bin as precise evidence.

Abstention creates a risk-coverage decision

A higher threshold typically reduces coverage and false responses while increasing abstention. Selective prediction research formalizes risk-coverage tradeoffs, but the application must define which risk and what users do after abstention. [CLM-049]

Patchwork’s state is richer than accept/reject:

• respond with structured evidence;
• clarify a missing fact;
• abstain from compatibility language;
• degrade to a weaker mode;
• prohibit excluded behavior;
• escalate a resolvable domain conflict.

A threshold should not replace these deterministic states. Permission, prohibited scope, known incompatibility, and required-input rules apply before a learned score. Among eligible cases, the score may help choose response or abstention.

Measure:

• coverage;
• error among responses;
• critical error count;
• correct and unnecessary abstention;
• user effort after abstention;
• fallback success;
• segment differences.

High abstention can be safe and unusable. High coverage can be useful and reckless. The contract chooses an operating point with evidence.

Sweep thresholds before selecting one

The companion evaluates the challenger at 0.50, 0.70, and 0.85. For each point it records precision, recall, coverage, false-match rate, abstention rate, and segment report.

Do not select the threshold after reading the release set and still call the result independent. Use development/evaluator-development evidence to choose a candidate policy, freeze it, then evaluate on held-out cases.

When the curve has no acceptable point, reject or change the mechanism. Do not hide the impossibility by averaging.

Preserve critical segments

Aggregate performance can hide disparities and localized failure. Specific official and peer-reviewed evaluations demonstrate demographic and intersectional differences in particular face-analysis systems and datasets; these results do not apply to Patchwork, but they demonstrate why aggregate accuracy is insufficient. [CLM-051]

Patchwork’s current segments concern task/data states, not human demographics:

• ordinary common family;
• long-tail family;
• denied permission;
• missing unit;
• stale source.

The challenger gains ordinary relevant responses and fails long-tail plus permission controls. The segment report prevents ordinary volume from dominating the decision.

For human populations, choose segments with purpose, affected-party input, privacy/legal review, measurement validity, and sufficient evidence. Fairness is not a label attached to a model after one dashboard. Different error directions and allocation/representation harms may matter. This book does not certify fairness.

Report intersections with judgment

Single dimensions can hide a failure at an intersection: long-tail family plus image-only input plus missing unit, or new catalog region plus stale source.

Do not enumerate every possible intersection and fish for stories. Predeclare critical ones from consequence and system structure. Ensure enough evidence or report uncertainty. Exploratory slices can generate hypotheses for a new set.

Protect privacy. A small group can be re-identifiable, and unstable rates can mislead. Suppress or restrict output where necessary without declaring the segment passed.

Multiple dimensions conflict

Holistic evaluation research demonstrates multi-scenario and multi-metric reporting. It also shows why one scalar cannot preserve every dimension. [CLM-050]

Patchwork decisions can conflict across:

• relevance and critical false match;
• coverage and abstention;
• explanation usefulness and evidence fidelity;
• latency and deeper retrieval;
• cost and repeated evaluation;
• common-family gain and long-tail regression;
• user preference and authorized behavior;
• provider portability and behavior stability.

Use a decision table with guardrails and optimization objectives. Guardrails are not terms to trade casually. Optimizations happen only inside the acceptable region.

For v0.2, zero permission and incompatible presentations are guardrails. Aggregate recall is an optimization objective. The challenger violates guardrails; its recall gain is irrelevant to adoption until fixed.

Product proxies can reward the wrong behavior

Clicks, dwell time, acceptance, conversion, repeat use, and explicit preference can be useful signals. They can also reward confident overstatement, novelty, friction, or short-term satisfaction.

First-party search engineering reports illustrate how engagement and relevance signals enter ranking work. Current incident reporting also shows that positive qualitative, offline, and online signals can miss an important behavioral category. These cases are bounded evidence, not universal proof. [CLM-053]

Patchwork separates:

• candidate relevance;
• evidence fidelity;
• policy/control success;
• user comprehension;
• correction;
• downstream task outcome;
• engagement/business proxies.

A user clicking the first candidate does not establish compatibility. A user preferring the confident explanation does not authorize certainty. A conversion improvement cannot override a permission exposure.

Evaluator error belongs in the record

An evaluation can be wrong because the evaluator is biased, inconsistent, unqualified, or used outside calibration. Model-based judge studies report position, verbosity, and self-related biases in studied settings, while vendor grader interfaces require application testing. [CLM-052]

For every metric or judgment, record evaluator type and version. Deterministic policy can classify known incompatibility. A trained rater can assess whether wording preserves uncertainty. A catalog specialist can resolve selected attribute evidence. Named authority decides release.

If evaluator limitations are large enough to change the decision, the correct result is unresolved. Chapter 11 builds the judgment stack.

Distinguish severity from root cause priority

A critical error may have a simple cause. A moderate error may require a new model. Prioritize by consequence and evidence, not technical novelty.

For each error cluster:

1. verify the criterion and label;
2. locate the first incorrect transition in the trace;
3. reproduce with a deterministic fixture where possible;
4. identify affected versions and segments;
5. test containment;
6. propose the smallest mechanism change;
7. add a development regression case;
8. preserve fresh release evidence;
9. route policy or authority changes correctly.

Do not change the threshold if the cause is a permission bypass. Do not retrain a ranker if the interface mislabels a score. Do not blame a model for stale catalog semantics.

Use severity before results

If the team assigns severity after seeing which configuration wins, it can redefine an inconvenient failure as minor.

Predeclare:

• taxonomy version;
• severity criteria;
• critical segments;
• threshold candidate;
• zero-tolerance categories;
• escalation and release authority;
• how ambiguous cases are handled.

Then freeze the policy with the experiment plan. Changes are allowed when evidence shows the policy was wrong, but the change must be explicit and the old/new conclusions shown.

The critical regression gate

The companion gate takes rows, score field, threshold policy, critical segments, and allowed budgets. It reports:

• pass/fail;
• critical error cases;
• critical-segment recall failures;
• named reasons;
• required authority.

The challenger fails for:

• critical-error budget exceeded;
• permission exposure;
• incompatible candidate presented;
• critical-segment recall below floor.

The gate is intentionally stricter than the aggregate metric. Passing it would mean only that the declared synthetic conditions did not violate the policy. It would not authorize release.

Issue a disposition, not a score report

Use five dispositions:

• go: evidence supports the bounded next step and authority approves;
• conditional go: proceed only with named controls, scope, monitoring, or evidence conditions;
• reduce scope: retain a safer subset or weaker promise;
• delay/revise: correct implementation or acquire evidence;
• stop/reject: evidence contradicts the value or control requirement.

For the challenger, disposition is reject. A revision might keep new weights for ordinary candidates but restore deterministic restrictions and rebuild long-tail evidence. That becomes a new configuration and experiment, not a reinterpretation of the failed result.

A decision review under conflicting metrics

Suppose a meeting shows:

• aggregate recall increases;
• aggregate coverage increases;
• ordinary-segment precision stays acceptable;
• long-tail relevant coverage decreases;
• one incompatible long-tail item responds;
• one denied-permission item responds;
• user preference for the challenger explanation increases.

Ask in order:

1. Did any non-tradeable control fail? Yes.
2. Is the error real under current taxonomy/evidence? Yes, in constructed cases.
3. Can the aggregate gain compensate? No.
4. Can scope remove the failing segment and permission path safely? Permission must be fixed globally; long-tail scope might be reduced.
5. What evidence is missing? Representative data, corrected control tests, new holdout, user comprehension.
6. Who decides? Product, catalog-domain, privacy/security as applicable, and release authority.

The meeting ends with actions and owners, not a prettier dashboard.

Calibration can shift

If Patchwork later uses a probability-like score, calibration must be scoped to component/model version, data distribution, segment, and time. A provider update, catalog change, selection policy, or user population can shift the relationship.

Monitor calibration only where labels arrive credibly and with acceptable delay. Avoid turning implicit feedback into truth. If labels arrive selectively for confident responses, the observed set is biased.

Use confidence intervals or uncertainty summaries appropriate to the sampling design. Do not draw a smooth reliability curve from a handful of synthetic points. The companion data is pedagogical.

Error budgets require meaning

An error budget can support operating decisions if it names error, population, window, consequence, and authority. One percent error budget is meaningless when permission exposure and irrelevant results share a denominator.

Use separate budgets:

• permission exposure: zero under verified cases;
• known incompatible presentation: zero;
• stale-current presentation: zero for strong assisted state;
• irrelevant top result: bounded by segment and task evidence;
• unnecessary abstention: improvement objective;
• trace incompleteness: threshold defined by audit policy.

Budgets do not legalize harm. Formal authorities set tolerances under context.

Write criterion records before scoring

A criterion record makes one judgment reproducible enough to challenge.

For known incompatibility is absent, record:

• criterion ID and version;
• behavior clauses PF-B02 and PF-B04;
• unit: displayed candidate within one request;
• positive condition: no current deterministic incompatibility under domain-rule version;
• failure condition: a conflicting candidate reaches assisted presentation;
• required evidence: normalized task constraints, candidate attributes, rule version, trace;
• evaluator: deterministic policy check;
• severity: critical;
• ambiguity: unresolved semantics route to abstention/domain review;
• authority: catalog-domain owner;
• change triggers: attribute schema, rule, unit/tolerance, source authority.

For explanation preserves evidence, the unit is one explanation tied to one frozen packet. The criterion checks that matched, unresolved, conflict, source, and limitation states remain intact. The evaluator is a calibrated rubric stack, not string equality. A permission violation or invented fact is a stop condition.

For clarification is understandable, the unit is an interaction. Evidence includes whether intended users identify the missing fact, provide it correctly, and understand the weaker fallback. A rater guessing that the copy is clear is not enough for a strong user claim.

Criterion records prevent metric drift. If the team changes what counts as relevant after reading the challenger output, the criterion version changes and clean evidence must be rebuilt.

Measure ranking at the product depth

Patchwork does not display an unbounded ranked list. If the interface shows three candidates, evaluate the top three and the behavior around them.

Possible measures:

• eligible candidate recall within generated set;
• precision among displayed candidates;
• top-position critical error count;
• reciprocal rank of the first acceptable candidate;
• normalized discounted gain when graded relevance is credible;
• deduplication correctness;
• evidence completeness for each displayed candidate;
• empty/abstain behavior when none qualifies.

Do not optimize a top-100 metric when users see three. Do not treat a relevance grade as permission. Apply eligibility before ranking evaluation and report any control violation separately.

Where multiple candidates are acceptable, one exact answer reference is insufficient. Use a reference set or criterion. Where domain evidence is unresolved, evaluate abstention or escalation rather than ranking the disputed item as correct.

Compare two operating points with exact cases

At threshold 0.70, the companion’s baseline responds to six constructed relevant cases and abstains on four additional relevant ordinary cases. Its critical segments stay inside the synthetic policy. The challenger responds to the four extra ordinary cases, raising recall.

It also responds to:

• D-008, labeled incompatible in long-tail;
• D-010, labeled denied permission.

The same threshold therefore has different meanings across configurations. Scores are not necessarily calibrated or comparable just because both fall in [0,1].

At 0.85, challenger coverage drops. Some critical errors may disappear because their scores fall below threshold, but relevant long-tail items also remain below it. Raising threshold can mask a mechanism that cannot separate valid and invalid cases. The team may still reject it because no useful operating point satisfies guardrails and coverage.

At 0.50, coverage rises and more nonrelevant cases respond. That point can be useful for diagnosis but is outside release policy.

The sweep is evidence about the decision surface. The frozen threshold is a policy choice that must survive held-out evaluation.

Investigate false match and abstention together

Threshold teams often optimize false matches first and then discover the product abstains for everyone difficult.

Build a two-sided report:

Responded cases

• relevant supported;
• irrelevant;
• incompatible;
• unsupported implication;
• stale;
• unauthorized;
• overconfident;
• segment and consequence.

Non-responded cases

• correct abstention;
• unnecessary abstention;
• clarification available;
• ordinary fallback available;
• prohibited scope;
• unresolved review;
• dependency failure;
• segment and user effort.

This makes safe but useless visible. It also prevents coverage pressure from weakening permission or incompatibility rules.

Analyze detection timing

An error caught before presentation differs from the same error reported after purchase.

Use detection stages:

• construction/development;
• offline evaluation;
• pre-release integration;
• shadow/canary;
• before user presentation;
• after presentation but before effect;
• user correction;
• support/incident;
• delayed outcome review.

Record which control can detect each category and at what latency. Permission and known incompatibility should be pre-presentation controls. Relevance may require reference or user outcome. Overconfidence can be caught by rubric plus comprehension testing. Long-term value may require later signals.

Detectability influences release. A moderate error that is invisible until harm may deserve stronger treatment than a similar error caught automatically.

Analyze reversibility

Reversibility is consequence-specific.

• Reordering a candidate before display is fully reversible.
• Correcting a label after display may not undo a user’s belief.
• Canceling an unconfirmed draft prevents an effect.
• Refunding a purchase does not recover lost time or safety.
• Deleting exposed information does not undo disclosure.

Do not call an effect reversible because software can issue compensation. Record residual consequence and affected party.

Patchwork v0.1 avoids external effects, which improves reversibility. The chapter still treats misleading presentation as consequential.

Build an error triage view

Triage ranks work without flattening evidence.

For each open cluster, show:

• highest consequence;
• affected critical segment;
• number and denominator under scoped evidence;
• detection stage;
• reversibility;
• control status;
• suspected layer;
• recurrence/change exposure;
• owner;
• release disposition.

Use unknown rather than zero where evidence is absent. Sort zero-tolerance control failures first, then high consequence/difficult detection, then coverage/value debt.

Do not prioritize by raw ticket count. One duplicated incident can create many reports; one silent failure can create none.

Connect errors to traces

An error report should link the first incorrect transition, not only final output.

For D-008:

1. query valid;
2. candidate retrieved with high challenger score;
3. deterministic incompatibility should exclude it;
4. constructed challenger fixture models presentation above threshold;
5. gate records ERR-INCOMPATIBLE;
6. disposition rejects challenger.

In a real implementation, determine whether policy failed, was bypassed, or the fixture represents a ranker-only comparison before policy. The root layer matters for correction, while the category matters for release.

Keep error taxonomy stable across causes. If the same incompatible presentation later comes from stale rules, the category remains while cause changes.

Test metric implementation

Metrics are code and can be wrong.

Test:

• empty denominators;
• all-positive/all-negative cases;
• missing labels;
• abstention representation;
• duplicate/group weighting;
• segment membership;
• threshold equality;
• non-finite scores;
• excluded/prohibited cases;
• paired version identity;
• rounding and report serialization.

The companion returns null when a denominator is absent instead of pretending zero. A segment with no relevant cases has undefined recall. The release gate should not interpret undefined as pass.

Review whether micro, macro, weighted, or per-query averaging matches the decision. Show raw counts.

Treat calibration as an evaluation project

If a score is intended as probability-like, specify:

• outcome label and time horizon;
• population and selection process;
• score version;
• binning or calibration method;
• train/tune/evaluation split;
• sample size and uncertainty;
• segment analysis;
• shift monitoring;
• user-facing mapping;
• fallback when calibration is unsupported.

Calibration error summaries can hide where the curve is wrong. Show reliability data and decision impact. A well-calibrated 0.7 score still means about three of ten fail under the defined outcome frequency, which may be unacceptable for the claim.

Do not calibrate similarity to compatibility without valid compatibility labels. Patchwork’s score stays internal.

Distinguish model uncertainty from evidence state

Model uncertainty is one component estimate. Evidence state includes permission, freshness, missing facts, conflicts, domain rules, and scope.

A model can be confident on an unauthorized record. A calibrated rank score can order two candidates while both lack required dimensions. A low score can coexist with an authoritative exact identifier.

Policy precedence:

1. prohibited scope;
2. identity/permission;
3. required deterministic exclusion;
4. evidence conflict/missingness;
5. learned score threshold;
6. presentation state.

This prevents uncertainty math from becoming authority.

Review error equity without slogans

When behavior affects human groups, ask:

• Which groups are affected or excluded?
• Are labels/criteria meaningful across groups?
• Are error directions and consequences symmetric?
• Does data collection create privacy or burden?
• Does abstention shift work unfairly?
• Does one group receive a weaker fallback?
• Are intersections visible?
• Who participates in defining acceptable outcomes?
• Which authority governs the decision?

Do not call a system fair because rates match on one dataset. Do not assume equal rates are always the right criterion. Preserve context, affected-party evidence, and limitations.

Patchwork’s fictional task-state segments avoid human fairness claims; the method prepares the reader to route real concerns correctly.

Audit the user-facing language

Error taxonomy must reach presentation.

Check labels such as:

• best match;
• compatible;
• recommended;
• confidence;
• verified;
• safe;
• approved.

Each word implies a criterion and authority. If the system only supports candidate discovery, use evidence-specific language. Show why a candidate appeared, what is unresolved, and what the user should verify.

Test iconography and ordering. A green badge can overrule cautious text. Accessibility readers may hear elements in a different order. The interface is an evaluator of meaning.

Separate evaluation from monitoring

Evaluation applies criteria to defined cases to support a decision. Monitoring watches signals during operation to detect change or failure. They connect but are not interchangeable.

A monitoring alert on rising abstention does not say the abstentions are unnecessary. It triggers evaluation. A frozen case suite can pass while production distribution shifts. An incident can reveal a missing category and update both taxonomy and cases.

Chapter 15 will build signals. For now, each error category names observable precursors and a production investigation path.

Govern taxonomy changes

Add or change a category when:

• a failure does not fit existing definitions;
• two categories lead to different consequence/control/disposition;
• a criterion or product promise changes;
• evaluator evidence shows ambiguity;
• an incident reveals a hidden affected party;
• categories overlap so much that reports are unstable.

Record old/new definitions, affected cases/results, migration rules, authority, and version. Do not silently relabel historical errors to improve trend lines.

Keep cause taxonomies separate. A model, data, interface, or infrastructure cause can underlie any product error.

Build the release evidence table

Evidence	Baseline	Challenger	Decision use
Aggregate recall	lower	higher	optimization evidence
Ordinary coverage	lower	higher	product-value hypothesis
Long-tail recall	passes constructed floor	below floor	critical segment block
Incompatible presentation	zero	one	zero-tolerance block
Permission exposure	zero	one	zero-tolerance block
Calibration	unsupported	unsupported	no probability claim
User comprehension	unmeasured	unmeasured	no explanation claim
Authority	not requested	not requested	no release authorization

The table makes the decision auditable. A reader can see why recall does not win.

Write a correction experiment

After rejecting challenger, propose a discriminating correction:

• enforce permission and known incompatibility as pre-score gates;
• retain challenger weights only among eligible candidates;
• keep long-tail excluded from the strong state;
• add development cases for the two failures;
• create fresh release groups;
• freeze threshold and evaluator;
• rerun paired comparison.

This tests whether the gain can survive controls. It does not lower the guardrails.

Apply the taxonomy to a high-consequence router

Consider the fictional care-operations message router. It classifies administrative messages into work queues and must not make clinical claims.

Its taxonomy might include:

• urgent message routed to ordinary queue;
• ordinary administrative message over-escalated;
• clinical interpretation presented as system fact;
• message exposed to an unauthorized queue;
• language/format segment unsupported;
• stale patient/workflow context used;
• correct escalation;
• unnecessary escalation.

Severity depends on delay and consequence, not classifier confidence. Detectability depends on whether a trained reviewer sees the case before routing affects service. Reversibility depends on whether the message can be recovered within the necessary time.

Aggregate accuracy could hide one urgent segment. A zero-tolerance clinical-claim or access control remains separate from routing optimization. Domain and privacy authorities define criteria and release policy. The Applied AI Engineer implements and measures them.

This satellite shows the taxonomy transfers while Patchwork thresholds do not.

Apply the taxonomy to a forecast

A predictive replenishment system has no retrieval or generation. Errors include underforecast, overforecast, unsupported cold-start estimate, stale-feature forecast, missing uncertainty, incorrect abstention, and unnecessary fallback.

Consequences vary by item: stockout, waste, planner workload, or service delay. Thresholds may be item/segment-specific under valid evidence. Calibration concerns forecast intervals and outcome coverage, not language confidence.

The same method works: define criterion, severity, segment, detectability, reversibility, control, and disposition. This keeps the chapter from becoming an LLM-evaluation recipe.

Review evidence freshness

An error policy can age.

Review after:

• task or user population change;
• new product scope;
• incident/new failure category;
• source/label semantics change;
• model/provider change;
• evaluator drift;
• interface wording change;
• regulation/policy/authority change;
• new affected-party evidence.

Record review date and owner. A threshold supported last year may not survive changed scores or population. Do not reuse it because the numeric field still exists.

Design the dashboard around decisions

A useful view leads with:

• current release disposition;
• critical gate status;
• zero-tolerance violations;
• critical segment results;
• coverage/abstention;
• taxonomy by consequence;
• evaluator/gap status;
• versions and time;
• owner and next action.

Drill down to raw cases/traces. Avoid a single quality dial. Use color plus text/icon and accessible tables. Show undefined denominators and missing evidence.

Do not display probability-like scores without semantics. Do not make correct abstention red by default. Visual hierarchy should reflect consequence, not volume.

Challenge the taxonomy with counterexamples

For each category, find cases at its border.

• A relevant candidate with unresolved dimension: relevant but unsupported for compatibility.
• A stale title with current dimensions: field-level freshness differs.
• A denied candidate deduplicated with an allowed source: permission must be provenance-specific.
• A correct result explained with a guarantee: candidate correct, presentation overconfident.
• An abstention with ordinary search available: correct assisted abstention, product fallback may still fail.
• A user correction after display: earlier error can persist in belief even after state changes.

If reviewers classify borders inconsistently, refine definitions and examples. Keep multi-label records when one outcome violates several criteria; do not force exclusivity merely for charts.

Keep authority visible in thresholds

The engineer can compute a sweep and recommend an operating point. The catalog-domain owner validates incompatibility semantics. Product authority sets the promise and user fallback. Privacy/security authorities govern exposure controls. Release authority approves the bounded next step.

Record signatures or decisions according to organizational process. Do not encode threshold = 0.70 and omit who accepted the consequence. A configuration value is not governance evidence.

If authorities disagree, preserve the conflict and maintain the safer state. A launch deadline is not an adjudicator.

The Chapter 10 acceptance packet

The completed packet includes taxonomy/version; criterion records; severity/detectability/reversibility rationales; raw cases; aggregate/segment/intersection results; score semantics; reliability data; threshold sweep; correct/unnecessary abstention; evaluator limitations; critical gate; correction experiment; disposition; and authority handoff.

Reviewers should be able to reconstruct why the aggregate winner lost and which evidence would change that decision. If the packet only shows a score, the chapter’s job is unfinished.

Ask what the metric cannot see

For every headline measure, write a blind-spot line.

• Recall cannot see permission, unsupported certainty, or user comprehension.
• Precision cannot show missed useful cases or fallback effort.
• Coverage cannot distinguish useful response from harmful response.
• Calibration cannot establish semantic correctness or authority.
• Agreement cannot establish truth.
• Clicks cannot establish compatibility or long-term value.
• Latency cannot establish quality.
• Zero observed incidents cannot establish absence when detection is weak.

Then pair the metric with the necessary controls or evidence. This habit is more durable than memorizing a preferred metric.

Reconcile duplicate and multi-label failures

One request can produce several error records. A denied candidate may also be incompatible and overconfidently described. Preserve all violated criteria, then choose a primary category for operational ownership if useful. The release gate must see every zero-tolerance violation.

Group repeated manifestations by root trace or canonical case so denominator counts are not inflated. Report both affected requests and error events. A burst retry can produce many log events from one user consequence.

Document grouping logic and test it. Changing grouping can alter trends; version the report rather than silently rewriting history.

Communicate uncertainty plainly

Say one constructed critical case failed rather than the failure rate is 6.25 percent when the suite was purposive. Say recall is undefined for this segment because it has no relevant reference cases rather than zero. Say calibration unsupported rather than low confidence.

Precise language keeps decision makers from treating synthetic mechanics as population estimates. It also tells the team exactly what evidence to acquire next.

Completion test

Hand the packet to someone who did not build the challenger. They should identify critical failures, affected segments, threshold, score limitation, evaluator, and authority without asking which configuration the team prefers. They should reproduce the failing gate and propose a correction that does not weaken permission or incompatibility controls.

If they instead debate whether recall matters more than safety in the abstract, the taxonomy has not connected evidence to decision. Rewrite criteria and disposition until the boundary is operational.

Archive the exact report, policy version, raw counts, cases, and decision. A later team should be able to see whether a new threshold, segment, or evaluator genuinely changed evidence or merely changed presentation. This history prevents the same critical failure from returning as an undocumented tradeoff.

The chapter is complete only when every headline gain can be challenged by named consequence evidence and every critical failure can stop the next step.

That standard keeps optimization subordinate to the product promise and its named authorities.

It keeps error evidence actionable.

It also keeps critical segments visible.

PF-07 v0.2

The artifact now contains:

• taxonomy with eight outcome categories;
• severity/detectability/disposition fields;
• critical segments;
• threshold and abstention policy;
• aggregate and segment report functions;
• reliability data;
• threshold sweep;
• failing critical regression gate;
• limitations and named authority.

Its constructed data has sixteen cases. The challenger improves recall because it retrieves additional ordinary relevant items. It also responds to incompatible and denied-permission cases and misses relevant long-tail cases. The preserved conflict is the chapter’s evidence.

Practice: reject the aggregate winner

Use two configurations on one frozen case set. Predeclare taxonomy, severity, threshold candidates, critical segments, and guardrails.

Required output:

• error records;
• aggregate metrics;
• segment/intersection report;
• reliability data if scores have probability semantics;
• threshold sweep;
• correct/unnecessary abstention counts;
• critical regression gate;
• release disposition;
• uncertainty and authority handoff.

Inject a configuration that improves the average and harms a critical segment. Pass only when the decision rejects, revises, or reduces scope despite the aggregate win.

Do not use a default F1/AUC, call a model fair/safe from one score, display uncalibrated probability, select severity after results, or let average quality mask a critical error.

Handoff to credible judgment

The taxonomy tells us what needs judgment. It does not say who or what can judge every criterion credibly.

Schema and effect count can be deterministic. Relevance may use reference evidence and trained raters. Explanation fidelity may use a bounded model grader plus human calibration. Compatibility under conflict belongs to specialist evidence and domain authority. Release belongs to named authority.

Chapter 11 will assign each claim to the cheapest credible evaluator, expose evaluator bias, measure limited agreement, preserve disagreement, and refuse to turn majority vote into truth.