Release to Learn Safely

Release answers one uncertainty, not every hope

Patchwork has encouraging offline evidence. The text discovery path passes its frozen evaluation and operational envelope. Six controls pass synthetic tests. Image-heavy queries still have tail and evidence gaps. Ambiguous legacy variants remain outside deterministic compatibility coverage.

A deadline does not erase those gaps. A broad launch would ask users to discover whether unsupported segments work. A bounded release asks a narrower question:

Does the evidenced text discovery behavior hold for a small, attributable cohort under real-use conditions while image-heavy and ambiguous legacy cases remain on the existing deterministic path?

This question defines population, exposure, versions, signals, duration, stop conditions, rollback, and authority. It also identifies what the release cannot learn.

This chapter creates PF-11 v0.1, a fictional readiness and rollout packet. The companion implements shadow mode, deterministic cohort routing, restricted segments, a stop gate, rollback replay, and five decision dispositions. It performs no production exposure and no external effects.

Production is not a substitute for missing evidence

Some uncertainties require real use: user comprehension, traffic interactions, operational variability, adoption, or long-term correction behavior. Others should be answered before exposure: schema validity, permission enforcement, prohibited effects, deterministic compatibility, rollback mechanics, and known critical cases.

The fact that production data is more realistic does not make every production experiment justified. Exposure creates consequence. A small cohort can still include a person for whom the error is severe. A reversible feature can still show a misleading result that cannot be unseen.

Start with an uncertainty register:

• what is unknown;
• why offline, simulation, shadow, or internal evidence is insufficient;
• which exposure can discriminate the hypotheses;
• which population is necessary;
• potential consequence per exposed unit;
• maximum blast radius;
• signal and expected delay;
• stop and rollback;
• authority;
• what remains unknown afterward.

If a question can be answered without user-visible exposure, prefer that route.

Choose an exposure mode by consequence and evidence

Offline replay runs frozen or sampled cases with no production traffic. It is strong for reproducibility and weak for live interactions.

Shadow observes live-shaped inputs or traffic while hiding new output and preventing effects. It can reveal distribution, latency, and disagreement, but not user response. Privacy and purpose still apply.

Internal exposure shows the behavior to trained or authorized staff. It helps test workflow and comprehension, but staff are rarely representative users.

Canary sends a bounded percentage or unit of traffic to the change and compares it with a control. Canarying is partial, time-limited exposure whose value depends on attribution, representative-enough signals, and an executable response. [CLM-089]

Named cohort restricts by justified eligibility, such as supported text behavior. It can reduce one risk while reducing generalizability.

Regional or tenant exposure follows operational or contractual boundaries. It is not safe merely because geography is small.

Broad exposure requires evidence and authority appropriate to the expanded claim.

Patchwork uses shadow, then internal, then a bounded text cohort. Image-heavy and ambiguous legacy segments stay on control. The stages are not a maturity ritual. Any stage can delay, reduce, or stop.

Readiness is a packet, not a score

AI readiness spans data, model, application code, infrastructure, interfaces, monitoring, ownership, and recovery. A cross-layer rubric organizes questions; it does not authorize release. [CLM-090]

The release packet includes:

• intended behavior and prohibited states;
• supported and restricted population;
• data, evaluation, and segment evidence;
• error thresholds and negative cases;
• latency, capacity, cost, and failure envelope;
• control implementation and residuals;
• privacy and security review;
• human-review capability;
• feature, config, model, data, schema, and control versions;
• unresolved gaps and owners;
• exposure mode, cohort, duration, and control;
• signals, attribution, and missingness;
• stop, rollback, roll-forward, and reconciliation;
• communications and support;
• named authority and disposition;
• evidence expiry.

Do not collapse these into one readiness percentage. A critical permission gap cannot be averaged with strong latency. A missing rollback owner cannot be offset by high recall.

Version the thing being released

A behavior release is more than a code commit. Record:

• feature and route version;
• configuration and thresholds;
• learned component or provider adapter;
• prompt or policy where applicable;
• data and index snapshot;
• schema and interface;
• control packet;
• evaluation suite;
• cohort router and salt;
• observability definitions;
• rollback target.

Semantic versioning and registries help identify artifacts, but behavior compatibility needs replay and operational evidence. Secure release practices protect artifact integrity. [CLM-094]

If one version cannot be pinned, record access time, provider identifier, observed behavior, and replay limitation. A release that cannot identify its configuration cannot attribute a regression.

Patchwork binds feature=patchwork-find, config 0.1.0, local deterministic adapter, synthetic data 0.1.0, schema 0.1.0, and controls PF-10-0.1.0.

Define blast radius in more than percentage

Ten percent of traffic can still be large, concentrated, or consequential. Blast radius includes:

• number of users or tasks;
• duration and repeat exposure;
• supported segments;
• consequence severity;
• effect capability;
• data and permission scope;
• geography or tenant;
• dependency load;
• reversibility;
• detectability and signal delay;
• user support and repair capacity.

Patchwork’s cohort allows no effects. It restricts image-heavy and ambiguous legacy cases. It uses deterministic routing so the same subject remains attributable. The control population receives current deterministic search.

Eligibility is not a hidden convenience. It is a claim boundary. Report that text evidence says nothing about restricted image or ambiguity behavior.

Cohort routing must be stable and inspectable

Random routing without stable assignment can expose the same user to changing behavior and contaminate attribution. A deterministic hash of a non-sensitive stable subject identifier and versioned salt can produce a repeatable bucket.

Before bucket selection, enforce eligibility. Restricted segments always route to control. Unknown segments route to control by default. The hash must not override scope.

Record:

• routing version and salt;
• eligibility and restriction rules;
• unit of assignment;
• exposure percentage;
• control route;
• cross-device or session behavior;
• privacy treatment of identifiers;
• exclusion and missingness;
• emergency override.

Determinism does not create representativeness. A ten-percent hash cohort may still differ through missing telemetry, eligibility, geography, or product behavior.

Shadow mode has value and limits

Shadow mode runs the candidate path without showing output or allowing effects. It can compare candidate sets, failures, latency, cost, and control decisions against the current path.

It cannot measure user comprehension, trust, correction, or downstream outcome. It can still consume sensitive data, provider quota, and capacity. It may create logs or vendor processing. Apply the same purpose and minimization controls.

Patchwork’s shadow result explicitly says outputVisible=false and effects=[]. A hidden tool effect is not shadow.

Use shadow to close implementation and distribution questions. Do not present shadow success as evidence of user value.

Predeclare stop triggers

Stop conditions must exist before exposure and connect to automatic or human action.

Patchwork stops for:

• any confirmed critical error;
• any permission bypass;
• any autonomous effect attempt;
• text p95 outside its budget;
• audit-control loss;
• unknown effect state.

Each trigger names population, window, confirmation rule, owner, action, and authority. A zero-tolerance invariant can stop on one confirmed case. A noisy rate needs a suitable denominator and delay.

Qualitative and behavior-specific signals matter. A first-party rollback case shows that positive offline and user signals can coexist with harmful behavior that evaluation missed. [CLM-093]

Do not wait for statistical significance when a critical invariant fails. Significance estimates uncertainty in a defined comparison; it does not authorize continued consequence.

Rollback is a tested state transition

A feature flag can route traffic away from a change. It does not undo output users already saw, reverse external effects, restore corrupted data, invalidate caches, close tickets, or repair trust.

Define:

• trigger and decision authority;
• exact config or artifact to disable;
• control route and capacity;
• in-flight request handling;
• state and effect reconciliation;
• data or schema compatibility;
• cache and index handling;
• observability continuity;
• user and support communication;
• replay and verification;
• roll-forward conditions.

Patchwork has no effects in the bounded cohort. Rollback disables the feature configuration, routes to deterministic search, reconciles in-flight traces, and replays text plus restricted segments. The packet explicitly notes that user-visible output cannot be unseen.

Rehearse rollback before release. If the control route lacks capacity or cannot read the new schema, the flag is not a rollback guarantee.

Preserve operating-point and segment evidence

Calibration and selective prediction create risk-coverage tradeoffs. Segment results can justify a narrower release even when aggregate behavior passes. [CLM-092]

Patchwork keeps image-heavy and ambiguous legacy cases restricted. This is not unfair exclusion disguised as safety. It is an explicit evidence gap with owners and a plan to close it. Product and domain authorities decide whether the reduced scope provides legitimate value.

Track both error directions and correct abstention. A release that suppresses every answer may avoid false compatibility but fail the task. A release that maximizes coverage may violate critical controls.

Avoid segment mining after results. Predeclare mechanism-linked cohorts, then inspect new categories as exploratory evidence for future cases.

Assign release roles and decision rights

At minimum:

• implementation owner verifies artifact identity;
• behavior/evaluation owner reports quality and negative evidence;
• operations owner verifies budgets and rollback;
• data/domain owner reviews source and compatibility scope;
• privacy/security reviewers address relevant controls;
• product owner defines value and user consequence;
• release authority chooses disposition;
• incident authority can stop exposure.

One person may hold several roles, but evidence and authority remain distinct. Risk controls and release dispositions depend on context and named governance roles. [CLM-091]

No engineer should accept absent stakeholder risk because a deadline arrived. The correct technical response is reduce scope, delay, or stop and make the missing decision visible.

Use five dispositions

Go: required evidence and authority support the declared scope.

Conditional go: a bounded scope proceeds under explicit condition, owner, expiry, and stop rule.

Reduce scope: supported segments proceed while unresolved segments remain on control or unavailable.

Delay: a decision or evidence can be acquired and exposure is not justified yet.

Stop: consequence, control, ownership, or recovery makes even bounded exposure unjustified.

Patchwork’s packet says reduce-scope as an artifact recommendation. Without named authority, the executable decision function returns delay. With authority, it can return reduce-scope. Deadline pressure does not turn either into go.

Communicate facts, unknowns, and triggers

Support and operators need:

• current scope and versions;
• who is exposed and who is restricted;
• expected behavior and limitations;
• known unknowns;
• signals and stop state;
• current disposition and authority;
• user repair path;
• next decision time or evidence event.

Avoid speculative assurances. Say No critical error observed in N covered cases rather than safe. Say Image-heavy behavior remains restricted rather than coming soon if no date is authorized.

Communication is part of containment. If support cannot identify the cohort or route reports, the blast radius is larger than the router percentage suggests.

Exercise the conflicting packet

The Patchwork evidence is intentionally mixed:

• supported text evaluation passes;
• image and ambiguity gaps remain;
• text tail budget passes;
• six controls pass synthetic tests;
• a residual blocks ambiguous legacy scope;
• local rollback replay passes;
• no production privacy or outcome evidence exists;
• a deadline requests broader release.

The justified disposition is not broad go. Reduce scope after authority, or delay without it. Image and ambiguous traffic remain on control. The cohort answers only the text real-use question.

If any critical signal fires, stop and rollback. The learning value of continued exposure never overrides a zero-tolerance permission or agency boundary.

Build readiness from blocking questions

For behavior:

• Which clauses are supported?
• Which states clarify, abstain, degrade, review, or stop?
• Which prohibited transitions have negative tests?
• Which user consequence remains unresolved?

For evaluation:

• Does the suite cover the release population?
• Are critical segments and both error directions visible?
• Are thresholds and dispositions predeclared?
• Which cases are synthetic, selected, or representative?
• Has contamination been checked?

For operations:

• Do end-to-end tails hold by segment and cold/warm state?
• Is capacity sustainable with bounded retry and observability overhead?
• Which fallback preserves evidence and authority?
• Has rollback run against the current artifact?

For controls:

• Are policy objectives implemented and tested?
• Which residual blocks scope?
• Are privacy and security reviewers current?
• Can a tool effect occur?
• Who may accept remaining risk?

For users and organization:

• Is limitation visible?
• Can support identify and repair exposure?
• Are correction and feedback routes ready?
• Who stops, communicates, and decides continuation?

Record answer as met, not met, unknown, not applicable with justification, or conditionally met with expiry. A missing answer is not a pass.

Match each stage to a learning claim

The stages in PF-11 have different questions.

Shadow

Question: Does the candidate path run on live-shaped text inputs within resource and control envelopes, and how does it disagree with current search?

Evidence: paired candidate state, latency, cost, permission and compatibility decisions, abstention, and trace completeness.

Cannot learn: user comprehension, interaction, or outcome.

Stop: critical invariant, privacy collection gap, unbounded load, or attribution failure.

Internal

Question: Can authorized staff understand evidence, limitations, correction, and cohort state in the actual interface?

Evidence: task walkthrough, correction, support flow, accessibility, and operator runbook.

Cannot learn: representative user value or long-term behavior.

Stop: misleading presentation, unavailable correction, or unsupported effect path.

Bounded text cohort

Question: Does supported text behavior hold under limited real-use conditions against the control route?

Evidence: critical error, behavior state, text tail, correction, selected qualitative reports, and control signals.

Cannot learn: image-heavy, ambiguous legacy, broad population, or long-term value.

This mapping prevents stage completion from becoming automatic progression.

Define the control population

A control is not merely users outside the canary. It should represent the current allowed alternative under comparable conditions.

Record:

• assignment unit and eligibility;
• current behavior and versions;
• traffic and time comparability;
• instrumentation equivalence;
• contamination and spillover;
• fallback or crossover;
• missing observations;
• user interaction changes caused by the experiment.

If the candidate interface changes layout, engagement differences may reflect presentation rather than ranking. If support staff know cohort status, responses may differ. These are confounds to document.

For high-consequence invariants, the control does not justify one failure in the candidate. Zero-tolerance policy remains separate from comparative lift.

Choose duration from evidence delay

Run for one week is not a design. Duration follows:

• minimum cases for the declared comparison;
• weekday, region, or season variation;
• feedback and outcome delay;
• repeat-user effects;
• operational peak;
• maximum acceptable exposure;
• reviewer and support capacity;
• stop-signal latency.

Some signals arrive quickly, such as permission bypass. Others arrive after a purchase, correction, or support report. A short canary can miss delayed consequence. A long canary can expose too much while waiting.

If the necessary outcome delay exceeds acceptable blast radius, choose a different study, smaller effect capability, stronger offline proxy, or no release.

The companion’s 100 synthetic cases, 20 synthetic users, and 200 synthetic requests are teaching bounds, not recommendations.

Make stop signals operational

For each stop trigger, specify:

Trigger	Confirmation	Action	Owner	Authority	Communication
critical error	adjudicated case or deterministic invariant	disable cohort	release operator	incident/release authority	product and support
permission bypass	validated control event	disable immediately	security/data owner	security authority	restricted incident channel
effect attempt	any tool state beyond allowed draft	kill tool route	tool owner	product/security authority	affected-user route
tail breach	defined window and count	pause ramp or reduce	operations owner	release authority	operations update
audit loss	confirmed missing required records	stop consequential path	observability owner	privacy/security authority	incident channel

Test alerts before release. Verify the owner receives them, can execute the action, and sees enough redacted evidence. A trigger no one can observe or act upon is documentation.

Separate pause from rollback. Pause holds stage while evidence is checked. Rollback restores the control path. A critical permission or effect violation should not wait in pause for convenience.

Plan ramps as decisions, not percentages

A ramp record includes:

• current and proposed stage;
• evidence accumulated since last decision;
• negative and missing evidence;
• segment distribution;
• signal health and delay;
• stop-trigger rehearsal status;
• control and rollback version;
• proposed blast-radius increase;
• reviewer dispositions;
• authority and expiry.

Do not preapprove the whole sequence 1 -> 5 -> 25 -> 100. Each expansion changes exposure and may require a new claim. A low-volume stage can lack rare critical cases; a high-volume stage can create capacity behavior not previously observed.

Keep stages stable long enough for their question, but never wait out a critical failure. Do not change model, prompt, data, and cohort percentage simultaneously. If configuration changes, reset attribution and version the plan.

Rehearse rollback under realistic constraints

Run a local or staging rehearsal that includes:

• feature disable;
• route propagation delay;
• in-flight requests;
• old-path capacity;
• cache and schema compatibility;
• data written by new path;
• trace continuity;
• support visibility;
• frozen replay;
• authority handoff.

Inject partial rollback. One region or instance may retain the flag. Verify detection and reconciliation. Inject control-path overload. Decide whether to shed traffic, reduce capability, or stop.

For tool effects, rollback may be impossible. Use roll-forward, compensation, or user repair. This is why Patchwork’s bounded cohort allows no effects.

Record rehearsal limits. A local toggle proves mechanics in the fixture, not production propagation time or capacity.

Handle stateful release changes

Model and ranking changes can write caches, embeddings, feedback labels, or user preferences. Schema and data migrations can make old code unable to read new state.

Before release, classify state:

• ephemeral and safe to discard;
• versioned and readable by both paths;
• reversible through migration;
• compensatable but not reversible;
• externally visible and not retractable.

Define dual-read or dual-write only with clear consistency and cleanup. A feature flag around output does not undo state created by a background job.

Patchwork’s fixture avoids effects, but it still records traces and candidate comparisons. Rollback reconciles in-flight traces and keeps restricted segments on control.

Treat qualitative reports as stop evidence

Provide a fast route from support or feedback to adjudication. Include cohort and version identity without exposing raw content broadly.

One report can reveal a novel critical category. It cannot estimate prevalence. Add a case, search bounded related evidence, and decide whether the category violates a stop condition.

Do not let positive engagement outrank a confirmed consequential error. Metric hierarchy belongs in the preregistered plan.

Communicate to support which reports require immediate escalation: unsupported compatibility, permission exposure, autonomous action, unknown effect, and inability to correct.

Check representativeness without pretending completeness

Compare cohort and eligible population on justified dimensions:

• segment and modality;
• device and region;
• new and returning use;
• long-tail frequency;
• cold and warm state;
• permission state;
• input size;
• correction and support access.

If telemetry is sampled, account for inclusion. If privacy prevents a dimension, state the gap or use an approved study.

A representative cohort for common text behavior may still contain too few critical cases. Maintain deterministic release gates and targeted evaluation alongside online comparison.

Never broaden from text cohort passed to Patchwork passed. The claim must follow eligibility.

Manage deadline conflict explicitly

When a deadline arrives, list options:

• keep control behavior;
• release supported text only;
• internal or shadow only;
• delay for domain decision;
• remove the risky feature;
• stop the initiative.

Estimate what each preserves and sacrifices: learning, user value, consequence, operational cost, stakeholder commitment, and reversibility.

Do not represent absent privacy, security, domain, or business approval as a technical waiver. Escalate with a clear decision record. If authority does not respond, default to the previously authorized state.

Patchwork’s deterministic disposition illustrates this: unresolved gaps plus no authority returns delay, even under deadline pressure. Authority can choose reduced scope; it still cannot turn missing evidence into broad readiness.

Create the go/no-go record

Record:

• decision and timestamp;
• scope and exclusions;
• evidence versions and hashes;
• facts, unknowns, and negative evidence;
• unresolved risks and owners;
• chosen disposition;
• conditions and expiry;
• signals and stop triggers;
• rollback and communication;
• reviewer positions;
• named authority;
• next decision.

If a condition expires, exposure does not silently continue. Renew evidence or stop according to policy.

The record should be understandable after team turnover. Avoid undocumented links and private chat decisions.

Work the Patchwork disposition under five variations

Variation 1: current mixed evidence

Text behavior, operations, and six controls pass. Image-heavy and ambiguous legacy gaps remain. Named authority is present.

Disposition: reduce scope. Release only bounded eligible text. Keep restricted segments on control.

Variation 2: authority absent

Evidence is unchanged, but domain and release authority have not responded.

Disposition: delay. The engineer cannot accept the residual. Shadow may continue only if already authorized and privacy/capacity remain bounded.

Variation 3: one permission bypass

Everything else passes. One confirmed candidate exposes denied evidence.

Disposition: stop. Do not average or restrict only after exposure until containment and control evidence are rebuilt.

Variation 4: text p95 exceeds budget

Quality passes, but tail grows under cohort load. No critical semantic error occurs.

Disposition: pause or reduce stage, then diagnose capacity and fallback. If the stop policy declares tail breach, execute it. Do not silently raise the budget after results.

Variation 5: deadline demands broad launch

No new evidence closes image or ambiguity gaps.

Disposition: unchanged. Deadline pressure can influence which authorized option is chosen, not transform missing evidence into readiness.

The variations show why a single checklist score cannot decide.

Design attribution for a combined system

When behavior changes, identify what changed:

• assignment and cohort;
• feature/config;
• model or provider;
• prompt/policy;
• data/index;
• schema/interface;
• control;
• UI presentation;
• support or feedback path;
• unrelated concurrent release.

Freeze unrelated changes where practical. Annotate unavoidable ones. Preserve a control population. If a provider changes during the stage, pause interpretation and replay.

Use trace version IDs to correlate, but association is not cause. Chapter 18 will test hypotheses through paired evidence.

Do not change threshold after seeing results without versioning a new decision. A post-hoc threshold can make a failing stage look planned.

Avoid cross-unit contamination

One user’s exposure can affect another through seller responses, shared caches, marketplace behavior, team collaboration, or public output. Assignment independence may fail.

Patchwork allows no seller contact in the cohort, reducing one spillover. Shared ranking cache still needs cohort/config keys so candidate behavior does not leak between treatment and control.

If spillover is likely, choose tenant, region, or time-based assignment with appropriate analysis, or avoid the experiment. Record the larger blast radius.

Support agents can contaminate behavior if they give cohort-specific help. Train and instrument support without scripting away real usability evidence.

Bound privacy in experimentation

The release purpose does not justify collecting every interaction. Reuse the Chapter 15 signal catalog. Add only fields needed for the declared uncertainty and attribution.

Record:

• assignment identifier transformation;
• cohort and version;
• behavior and critical errors;
• minimal context/evidence IDs;
• correction and selected feedback;
• latency/capacity;
• retention and deletion;
• access and export.

Do not log raw inputs or outputs by default. Shadow traffic is still processing. Internal users still have privacy interests. Experiment analysis can create derived sensitive cohorts.

If the needed outcome requires new data purpose or longer retention, obtain review before exposure.

Define user-visible behavior for each stage

Shadow: no visible change.

Internal: interface labels the experimental state, evidence, limitations, correction, and no-effect boundary to trained participants.

Bounded cohort: eligible users receive the supported text path. Restricted inputs route to existing behavior or explicit unsupported state. Users can inspect evidence and correct input. Seller contact remains a draft-only workflow outside the cohort’s release question.

Rollback: users see current deterministic search. If they already observed an incorrect candidate, support and repair routes remain.

Do not hide experimental limitations in internal metadata. Interface behavior is part of release evidence.

Check accessibility and support readiness

A release can technically work while making evidence, correction, or abstention inaccessible. Test keyboard, screen reader, contrast, mobile layout, plain-language states, and assistive workflows appropriate to the product.

Support readiness includes:

• cohort lookup under privacy controls;
• known limitations and prohibited assurances;
• critical escalation categories;
• correction and user repair;
• current stop and rollback state;
• owner and communication cadence.

Internal staff should not tell users a result is guaranteed because the interface looks confident. Provide approved bounded language.

Accessibility gaps can reduce scope or stop release; they are not cosmetic follow-up when they block correction or evidence access.

Model the blast radius numerically and semantically

Create a table:

Dimension	Bound	Evidence	Stop
assignment	10 percent eligible text	deterministic router	routing mismatch
requests	200 synthetic in fixture	trace count	count exceeded
duration	bounded evidence window	start/end record	expiry
segments	text warm/cold only	segment gate	restricted entry
effects	none	tool state	any effect attempt
critical errors	zero	adjudicated/control event	any confirmed case
tail	text budget	distribution	threshold breach
privacy	allowlisted traces	schema monitor	prohibited field or loss

Percentages alone cannot express this envelope. The semantic bounds are often more important.

Decide what happens when signals disagree

Predeclare priority:

1. permission, agency, and critical behavior controls;
2. user consequence and adjudicated critical reports;
3. behavior and segment quality;
4. operational tails and saturation;
5. comprehension and correction;
6. engagement and business proxies.

This is not a universal hierarchy. It reflects Patchwork’s fictional contract. Authorities approve the actual hierarchy.

If engagement rises and compatibility errors appear, stop on the critical behavior. Do not wait for engagement analysis. If latency improves and evidence coverage falls, reject the optimization.

Record disagreements rather than use a blended score.

Preserve negative release evidence

Keep:

• restricted segments and reasons;
• failed ramp attempts;
• triggered stops;
• rollback duration and gaps;
• reports that exposed new categories;
• unrepresentative cohort findings;
• instrumentation loss;
• reviewer dissent;
• conditions that expired.

Future teams need to know why broad exposure was rejected. Deleting failed canaries creates repeated incidents.

Version the packet after correction. Link old and new evidence. A passing later run does not erase prior affected-user decisions.

Release review agenda

1. State the one uncertainty.
2. Show why less exposure cannot answer it.
3. Define population, exclusions, and consequence.
4. Walk evidence by class and preserve blocking gaps.
5. Verify versions and artifact identity.
6. Rehearse routing, stop, rollback, and communication.
7. Review signal delay and missingness.
8. Ask each reviewer for scoped disposition.
9. Ask named authority for go, conditional go, reduce, delay, or stop.
10. Record expiry and next decision.

Do not ask Are we comfortable? Ask which claim is supported and which authority accepts which residual.

Simulate stop and rollback

Route three fictional subjects: eligible text, image-heavy, and ambiguous legacy. Confirm the latter two stay control. Let the eligible subject enter the cohort under deterministic bucket assignment.

Inject a critical error. The stop gate returns disable-and-rollback. Disable the feature. Route all three to control. Confirm no effects exist, in-flight state reconciles, restricted segments remain restricted, and frozen replay runs.

Then inspect user-visible limitation: a candidate shown before rollback cannot be unseen. Add it to affected-output review. This prevents the exercise from claiming flag reversal equals full recovery.

Validate the deterministic router as a control

Routing bugs can expand exposure silently. Test:

• every restricted segment routes control;
• unknown segment routes control;
• eligible subject receives stable bucket across requests;
• salt or version change is intentional and recorded;
• assignment identifier does not leak into analytics exports;
• control and cohort paths emit comparable version events;
• emergency disable overrides bucket;
• cache keys separate behavior versions;
• support lookup cannot change assignment.

Monitor cohort size against expected eligible traffic, restricted-segment leakage, duplicate assignment, and missing version. A ten-percent setting with twenty-percent observed cohort is a stop signal until explained.

The companion hashes routerSalt:subjectId and applies eligibility first. This proves deterministic local mechanics only. A production router also needs authentication, tenant boundary, privacy review, distributed config propagation, and resilience.

Treat feature flags as configuration with failure modes

A flag can be stale in one process, cached in a client, mis-scoped, or unavailable. Default behavior matters. Record propagation and consistency expectations.

Test:

• flag off before launch;
• partial propagation;
• control-path capacity;
• old client behavior;
• background jobs ignoring the flag;
• region or tenant mismatch;
• flag service outage;
• rollback under load;
• audit of who changed it.

Do not grant broad flag access as a shortcut. Release and emergency roles need least privilege and logged changes.

A flag is one rollback mechanism. It does not reconcile data or observed output. Keep the broader rollback packet.

Design a roll-forward option

Rollback may be unsafe after state or schema change. A roll-forward plan defines a corrected version, migration, compatibility, and verification.

Choose between rollback and roll-forward based on:

• reversibility of state;
• old-path compatibility;
• time to safe correction;
• current consequence;
• capacity;
• user repair;
• authority.

Patchwork’s no-effect cohort favors rollback. A future tool-enabled release might require compensation and roll-forward for completed actions.

Rehearse the decision before exposure. Do not discover that rollback destroys new state during incident.

Check stop-trigger latency

A stop signal that arrives after the cohort ends cannot contain it. For each trigger, measure or estimate:

• event generation delay;
• collection and processing;
• threshold window;
• adjudication;
• alert delivery;
• human decision;
• config propagation;
• in-flight completion.

The sum defines potential additional exposure. Reduce cohort or effect capability when signal delay is long.

Critical deterministic controls should stop synchronously at the boundary where possible. Qualitative reports are slower and need a smaller blast radius plus support escalation.

Do not write an invented production stop time from a synthetic exercise. Report the tested local sequence and the unverified production components.

Decide whether to continue after a noncritical regression

Suppose text p95 rises modestly but remains below hard stop. The team can continue, pause, or reduce based on the declared decision rule. Consider user consequence, trend, saturation, expected learning, and remaining headroom.

Record the decision before changing thresholds. If the stage continues, name the monitor and maximum bound. If it pauses, preserve assignment and data for attribution.

Suppose abstention rises while critical errors remain zero. Determine whether correct controls encountered a different traffic mix or retrieval degraded. Do not optimize coverage until diagnosis.

The release packet should make these gray decisions inspectable without weakening zero-tolerance gates.

Close or advance a stage

At stage end, produce:

• planned and actual exposure;
• population and segment counts;
• version integrity;
• signal health and missingness;
• quality, behavior, operations, control, and qualitative evidence;
• stop or pause events;
• negative cases;
• unresolved gaps;
• user repair status;
• disposition recommendation;
• reviewer and authority decision.

Advancement requires a new decision. Closing without advancement can still produce learning. A failed stage is valuable if it prevents broader consequence.

Do not reuse a stage report after material version change. Link it as historical evidence and run the relevant comparison again.

Final release questions

• What exact uncertainty requires exposure?
• Why cannot shadow or internal evidence answer it?
• Who and what can be affected?
• Which segments are explicitly excluded?
• Are assignment and attribution stable?
• Which evidence classes pass and which block?
• Are stop triggers faster than acceptable consequence growth?
• Is rollback or roll-forward rehearsed under current versions?
• Can support identify, correct, and escalate?
• Does privacy purpose cover the experiment?
• Who can stop immediately?
• Who can authorize this scope?
• What remains unknown after success?

A release that cannot answer these is not a learning plan. It is exposure with hope.

Before the decision meeting ends, rehearse the first five minutes of failure. Who sees the trigger? Who disables the route? Which exact version becomes control? How are in-flight requests handled? Which audience receives the first factual update? Where is the affected-output review recorded? If participants answer with a team name instead of a person or delegated role, the packet is unfinished.

Then rehearse success. Success does not mean automatic expansion. It means the stage produced evidence about its declared uncertainty without violating stops. Review missingness, delayed outcomes, restricted segments, qualitative disagreement, and evidence expiry. Only a new authority decision may expand the claim and blast radius.

Keep the control route observable during the entire stage. If its behavior, capacity, or instrumentation changes, the comparison may lose meaning and rollback may lose safety. Pause attribution, identify the changed version, and restore comparability before interpreting candidate results.

Every stage should end with one of the five dispositions, even when the evidence is inconclusive. Acquire more evidence must specify which uncertainty, method, owner, exposure bound, and deadline; it cannot become an indefinite canary.

The safest learning plan is sometimes no new exposure. That is a valid evidence-based decision when consequence, signal delay, reversibility, or missing authority makes the proposed question unjustified.

What the companion proves

Run:

node --test content/publications/applied-ai-engineering/companion/tests/chapter-17.test.mjs

The tests establish that PF-11 v0.1 is complete and effect-free; restricted segments cannot enter; eligible routing is deterministic; shadow output is invisible; unresolved gaps delay without authority and reduce scope with authority; a critical error stops; and rollback restores control while preserving restrictions.

They do not establish production readiness, representativeness, real user value, statistical power, privacy approval, rollback of observed output, or authorization.

From bounded release to real-use diagnosis

Bounded release does not guarantee benign behavior. It makes exposure attributable, consequence limited, and response executable.

Patchwork now has a precise route, versions, signals, critical stop triggers, rollback, communications, and authority boundary. Chapter 18 introduces a fictional incident in which aggregate engagement rises while compatibility complaints also rise. The incident will test whether the team contains first, rejects model blame without evidence, protects trace privacy, and changes durable artifacts rather than only patching code.