Engineer Release and Recovery

A release is a decision to expose a particular system state to a particular cohort under particular evidence and recovery conditions.

A pipeline can repeat that decision. It cannot decide that the evidence is sufficient, that a migration is reversible, that an exception is authorized, or that a backup will restore.

Release engineering makes change reproducible. Recovery engineering makes failure state explicit and rehearsed.

Bind the release as one versioned record

Record together:

• source commit and immutable artifact hash/provenance;
• build/toolchain/dependency evidence;
• code, API, configuration, schema, migration, data, model, prompt, index, policy, and grader versions;
• feature-control states and cohort;
• verification/control/security evidence bundle;
• known limitations, exceptions, authority, and expiry;
• compatibility window and migration state;
• health/quality/adoption/cost gates;
• owner, approver, schedule, support/on-call;
• stop/abort condition and recovery path.

The companion validates this binding and computes a deterministic SHA-256 hash over the synthetic artifact record. A hash identifies bytes/record content; it does not prove provenance, security, or correctness by itself. [CLM-117]

Semantic Versioning can signal compatibility expectations for a declared public API. It does not solve configuration, schema, data, deployment, or behavior compatibility. [CLM-120]

Review the complete release identity

Imagine commit abc123 produced artifact hash H1, but the canary uses policy bundle P4, prompt M7, schema S3, configuration C9, and feature state on for a named cohort. A rollback that changes only code to the prior artifact does not recreate the prior system. Schema/data, policy, index, configuration, and external effects may remain new.

The release record should let an operator reconstruct the exact bundle and see which parts can change independently. Store links and hashes where appropriate, not secrets or sensitive content. If a material artifact cannot be identified, the release evidence is incomplete.

Review provenance separately from identity. A hash can confirm equality with a recorded byte sequence; it does not prove who built it, which source/dependencies produced it, whether review occurred, or whether it is safe. Bind build and approval evidence explicitly.

Design the release path as evidence gates

A provider-neutral path can be:

1. source review and required checks;
2. reproducible artifact/build provenance;
3. unit/contract/security/static companion evidence;
4. migration compatibility and backup/restore prerequisites;
5. representative environment and failure/recovery tests;
6. owner-approved release record and change window;
7. internal/synthetic or least-exposed cohort;
8. observation window using absolute and comparative gates;
9. widen, hold, restrict, or recover;
10. close evidence, remove/expire controls, and update operations.

Automation and consistency reduce manual variance, but can consistently deploy a bad decision. Evidence/authority remain necessary. [CLM-118]

Walk one change through the gates

For Orchid, suppose the change improves evidence retrieval and requires a new index plus backward-compatible evidence metadata.

1. Contract tests prove old/new readers handle the declared metadata window.
2. The index is built from approved synthetic/controlled evidence and receives a content/version record.
3. Critical-segment evaluation binds model, prompt, index, policy, and grader versions.
4. Migration capacity is checked before any persistent record changes.
5. A representative environment injects stale index, missing evidence, partial migration, and unavailable identity recovery.
6. The release owner reviews limitations and selects an internal cohort.
7. Absolute wrong-binding/prohibited-action gates and comparative latency/fallback gates are watched.
8. The team widens, holds, or recovers according to the precommitted record.

The sequence is valuable because each gate can stop the change and names the evidence needed to continue. CI passed is only one input.

Make each gate actionable

For every gate define:

• required evidence and version;
• pass/fail/inconclusive interpretation;
• owner and decision authority;
• affected cohort and exposure duration;
• signal freshness/coverage requirements;
• stop condition;
• recovery action and access;
• exception rule and expiry.

Do not let a missing signal appear green. Do not silently ignore failed critical-segment verification because infrastructure is healthy.

Use progressive exposure with both relative and absolute checks

Small, time-limited changes can reduce blast radius and reveal environmental behavior. Compare control and changed cohorts where meaningful, but also apply absolute guardrails.

A canary can look “better than control” while both violate an acceptable latency or error boundary. Cohorts can differ in site, workload, evidence quality, user role, or connectivity. Low volume can hide rare/high-consequence failure. Preproduction parity is incomplete.

Record:

• why the cohort is representative for the intended claim;
• baseline/control and possible confounders;
• minimum observation/volume/critical-case coverage;
• service, workflow, data, AI/policy, security, support, capacity, and cost gates;
• authority to widen/hold/stop;
• rollback/roll-forward/isolation feasibility.

Progressive exposure is useful only with meaningful cohorts, absolute criteria, and precommitted recovery. [CLM-119]

Version configuration and feature controls

Configuration changes system behavior and needs review, provenance, validation, promotion, rollback/roll-forward analysis, access control, audit, and environment-difference evidence.

A feature-control record includes owner, purpose, default, environment/tenant/region/cohort scope, prerequisites/interactions, signal, decision authority, expiry/removal, and recovery behavior. [CLM-127]

Avoid long-lived flags that create multiple untested systems. Test on/off and material combinations. Remove the flag and dead path after the decision, unless the control is an intentional operational switch with continuing ownership.

Plan migrations around compatibility and externally visible state

For every persistent/API/event change define:

• old/new readers and writers;
• expand/contract or other compatibility sequence;
• backfill, dual-read/write, verification, and reconciliation;
• load, connection, locking, duration, pause/resume;
• partial completion and idempotent migration intent;
• feature/config coordination;
• destructive step and point of no return;
• backup/restore scope;
• rollback, roll-forward, isolation, or stop;
• data owner/authority and evidence.

Create a compatibility table across old/new readers, writers, schema, and externally visible effects. State which combinations are allowed and for how long. An expand/contract path might add nullable/new fields, deploy tolerant readers, deploy new writers, verify/backfill, then remove old behavior only after consumers and recovery paths are ready.

Do not call a destructive step reversible because the migration script has a down function. Test whether information, external effects, and consumer interpretation can actually return. When old and new states must coexist, expose that window in monitoring and support.

Capacity-gate before mutation

The companion migration checks requested connections against a capacity limit before changing state. When requested capacity exceeds the limit, it stops with zero applied records.

This demonstrates an important order: validate prerequisites before irreversible work. A production migration also needs realistic workload, connection-pool, lock, replica, storage, backup, and maintenance-window evidence.

Expose partial/divergent state

The synthetic failure after one of two records produces schema versions [2, 1] and the state divergent. It does not relabel the migration failed and assume old state.

Readers/writers must tolerate the compatibility window or exposure must remain isolated. Recovery then chooses based on actual state, not desired version.

Preserve intent during recovery

Migration/recovery retries need stable intent and semantic equivalence. An accepted write with missing final response is unknown until reconciled. Reissuing can duplicate or conflict. HTTP method-level idempotency is insufficient without application state. [CLM-121]

Choose recovery from state and consequence

Ask in order:

1. What user/external state exists now?
2. Is immediate isolation/disable/fallback required to contain consequence?
3. Are dependencies and recovery access healthy?
4. Is the change actually reversible without losing or misinterpreting state?
5. Did destructive/external writes occur?
6. Can old/new code coexist with current schema/data?
7. Is a small roll-forward repair safer/faster?
8. Is a tested, scoped, integrity-checked restore available?
9. What authority and communication are required?
10. How will the recovered user-visible state be verified?

The answer can be:

• rollback: compatible reversible code/config change with preserved state;
• roll-forward: current state cannot safely return, and a bounded repair exists;
• isolate: unhealthy dependency/cohort or containment first;
• restore: verified backup/restoration path fits scope/consequence;
• stop/escalate: no verified safe path.

Rollback is not universally safest after destructive or external change. [CLM-122]

This tree is original synthesis, not a formal standard. [CLM-124]

Compare recovery options for one partial migration

State: one of two records is schema v2, one is v1; old/new readers are compatible, no irreversible external action occurred, and a bounded v2 repair is tested.

• rollback code only: may leave v2 data under old assumptions; reject unless compatibility and state restoration are proven.
• reverse migration: risks losing new fields and requires an equally tested path; reject for this rehearsal.
• roll forward: complete the second record, reconcile both, verify readers and user state; selected.
• isolate: useful if consumers cannot tolerate divergence while repair runs.
• restore: unnecessary here and could lose newer valid state; retain only if integrity/compatibility evidence supports it.
• stop/escalate: required if authority, access, or state evidence is insufficient.

The selection is not a preference for roll-forward. It follows the actual state and verified path.

Treat backup and restore as different evidence

A backup record needs creation time/version/scope/retention/encryption/access/region/dependencies. Recovery evidence additionally needs:

• restore environment/procedure/access;
• integrity and semantic validation;
• compatibility with code/config/schema;
• missing interval and external-effect reconciliation;
• observed time and bottlenecks;
• owner and unresolved gaps.

The companion deliberately rejects a backup that lacks restoreTestedAt and integrityVerified. A backup is not recovery evidence by existence. [CLM-123]

Test stale/corrupted/missing backup, permission failure, wrong region/key, capacity, partial restore, and service verification. Protect restored sensitive data.

Exercise break-glass under the failed dependency

Emergency recovery access must define eligible actors, trigger, scope, duration, authentication, notification, audit, actions, review, and rotation.

Exercise it when the target identity/control plane/network/secret dependency is unavailable. A runbook that depends on the failed service is fictional. Do not weaken routine access merely to simplify the exercise. [CLM-128]

The FDE can execute/record within delegated scope; customer security/operations authorities own emergency access and risk decisions.

Record a recovery rehearsal

For each scenario capture:

• scenario/initial externally visible state;
• artifact/config/schema/migration/dependency versions;
• preconditions/access/authority;
• action selected and why alternatives were rejected;
• expected state and target;
• actual steps/state/timing;
• signals/evidence/integrity/user verification;
• communication/escalation;
• gap, owner, correction, and retest;
• environment and production limitation.

The local companion:

1. creates two schema-v1 synthetic records;
2. fails after migrating one, producing divergent state;
3. roll-forwards both records to v2;
4. verifies completed state;
5. records 42 milliseconds against a 100-millisecond local target;
6. states explicitly that this is not a production RTO claim.

A local observation cannot become a production commitment. [CLM-126]

Conduct the release and recovery exercise

Ask a second operator to reproduce the bundle from the release record, then run the synthetic migration. Inject capacity pressure before mutation and verify zero records change. Rerun within the limit, fail after one record, and confirm the state is labeled divergent. The operator must choose among rollback, roll-forward, isolation, restore, and stop using current evidence, then execute and verify the selected path.

The rehearsal record must include rejected alternatives, access/authority, observed state, integrity/user verification, timing, assistance, and production limitations. Break one dependency used by recovery and prove the emergency path does not secretly depend on it.

Pass when another operator can repeat the result and explain why the action is safe for the tested state. Fail if success depends on the original author narrating hidden steps, if only the infrastructure process is verified, or if local timing becomes a production RTO promise.

Manage feature controls as temporary release state

A feature flag can reduce exposure, but it also creates multiple system configurations. Record the default, scope, owner, creation decision, prerequisites, interacting controls, on/off behavior, signals, stop/recovery use, expiry, removal criteria, and data/schema compatibility.

Test at least the material combinations. Turning off the interface may not stop background work. Turning off the model may still leave stale suggestions in a cache. A cohort flag may not reverse external inventory effects. A recovery switch that was never exercised can fail when needed.

At each release review classify a control as:

• active experiment/exposure control with current decision window;
• intentional long-lived operational control with permanent owner/tests;
• expired/dead state awaiting removal;
• unsafe/unowned divergence that blocks widening.

Remove code/config/data paths when the decision closes. If removal itself is risky, plan and verify it as a release.

Design a restore test that proves usable state

Choose a representative backup version and record what it contains and omits: data, schema, configuration, encryption/key dependency, index/evidence versions, audit, and external effects. Restore into an approved isolated environment using the access path expected during incident conditions.

Verify:

• cryptographic or storage integrity where appropriate;
• schema/application/config compatibility;
• record counts plus semantic invariants;
• tenant/region and access boundaries;
• missing interval and external-effect reconciliation;
• user-visible workflow state;
• observed time, bottlenecks, and assistance;
• cleanup/protection of restored sensitive data.

The test can reveal that a backup is valid bytes but unusable with the current application, or that inventory reservations after the backup cannot be reconstructed. Those are recovery gaps, not minor documentation issues.

Exercise break-glass without normalizing it

Disable the normal identity dependency in the rehearsal. The authorized operator must invoke the approved emergency path, authenticate independently enough to survive the failure, receive only the required scope/duration, perform named recovery actions, notify designated owners, generate audit evidence, and exit/rotate access afterward.

Record every dependency. If emergency access relies on DNS, secrets, approval service, device posture, or network control that shares the incident failure, repair the design or state the limitation. Do not respond by leaving permanent broad credentials available.

The FDE can help implement and exercise the mechanism. Customer security/operations authorities decide eligibility, conditions, and residual risk. [CLM-128]

Transfer the recovery tree across changes

For a stateless UI defect with no external effects, rollback may be simple. For an append-only event schema, roll-forward with compatible readers may be safer. For a corrupted tenant-specific index, isolate and rebuild may preserve other cohorts. For a regional data loss with verified backup, restore plus reconciliation may be necessary. For unknown authority or state, stop and escalate.

Using several change types prevents the team from memorizing one preferred action. The tree is a decision method: actual external state, reversibility, compatibility, access, evidence, and consequence determine the path.

Failure modes and repairs

“Pipeline means safe”

Repair: evidence/authority gates, artifact binding, cohort/stop/recovery.

“One-click rollback”

Repair: inspect external/data state, compatibility, point of no return, and rehearse the actual path.

“Backup exists”

Repair: restore/integrity/compatibility/timing/access evidence.

Hidden destructive migration

Repair: explicit destructive step, approval, backup, compatibility, roll-forward/restore plan, capacity gate.

Flag as permanent architecture

Repair: owner/default/scope/tests/signals/expiry/removal.

Recovery credential untested

Repair: exercise under dependency failure, minimize scope, review/rotate.

Health gate without user meaning

Repair: Chapter 13 promises/cohorts/quality/support, absolute guardrails, signal freshness.

Complete OA-08

The operating artifact now includes:

• version-bound release record and artifact hash;
• evidence-gated progressive path;
• configuration/feature lifecycle;
• migration compatibility/capacity/partial-state plan;
• recovery decision tree;
• backup/restore and break-glass evidence;
• executed synthetic roll-forward rehearsal;
• owner/authority/support/runbook/improvement path.

Operational readiness joins change, release, recovery, support, ownership, and improvement with executed evidence rather than documents alone. [CLM-125]

The Chapter 14 gate

Before Chapter 15 evaluates production crossing, require:

• bound code/artifact/config/schema/migration/model/policy/evidence/cohort versions;
• quality/authority/exception gates;
• compatibility window and destructive point;
• migration load/partial/idempotent/reconciliation evidence;
• progressive cohorts plus absolute criteria;
• tested feature-control stop/removal;
• rollback/roll-forward/isolate/restore/stop choice by actual state;
• verified restoration and break-glass access;
• executed recovery rehearsal and user-visible verification;
• explicit local-versus-production limitations;
• passing companion tests and reproducible command.

The gate passes when a reviewer can reproduce the intended release, force a representative failure, choose recovery from state/consequence, and verify the recovered result without inventing rollback or recovery claims.

Chapter 15, Cross the Production Threshold, will decide whether the combined outcome, verification, operating, release, and recovery evidence justifies exposing a real cohort.