Operate, Release, and Recover

Course boundary: Optional Komal learning material. Not required by Abhyaas. NOT FOR LIVE CERTIFICATION BANK. Timings are deterministic local fixtures and are not production SLO, RTO, or RPO evidence.

A production path is not ready because it can return a successful response. It must expose whether user promises are being met, diagnose meaningful failures, bind releases to evidence, and recover from state changes under controlled conditions. This module joins observability, release, migration, and recovery into one operating decision system.

Demonstration: healthy service, failing workflow

Run:

npm run course:fde -- --module 04

The first output shows healthy aggregate availability. The low-connectivity cohort nevertheless exceeds its evidence-review window and falls back more often. The second output simulates a partial schema migration. One record reaches version 2 while another remains version 1, so the state is labeled divergent; blind rollback is unavailable because state has changed.

These are different failure classes:

• service health asks whether the component answers;
• workflow health asks whether eligible users can complete the promised work;
• data health asks whether identity, semantics, freshness, and lineage remain usable;
• model/decision health asks whether evidence-bound behavior stays inside policy;
• operating health asks whether owners can diagnose, support, release, and recover.

An aggregate green signal cannot substitute for the others.

Scenario walkthrough: design signals from promises

Start with the bounded promise:

Qualified technicians in the included cohort can obtain a reviewable, evidence-bound inspection suggestion or a safe visible fallback inside the operating window.

Derive signals by breaking the promise:

Promise fragment	Signal	Segment	First action	Owner
qualified technician	authorization/qualification denials	role, region	inspect identity/scope/expiry	identity owner
included cohort	eligibility and exposure counts	cohort/site	reconcile cohort config	release owner
evidence-bound	missing/stale/conflicting evidence rate	source/version	inspect retrieval and provenance	evidence owner
suggestion inside policy	prohibited or schema-invalid output	equipment/risk class	block and inspect policy/model bundle	product/control owner
safe fallback	fallback available and completed	connectivity/dependency	verify fallback path	operations owner
operating window	end-to-end latency and abandonment	site/connectivity	follow diagnostic tree	operations owner

For every alert candidate, specify privacy treatment, correlation fields, threshold rationale, segment, owner, and first action. A signal that cannot change an action is a dashboard decoration, not an alert.

Guided lab: release and recovery record

Step 1: build the diagnostic tree

The symptom is evidence review exceeds the operating window. Design discriminating checks for:

• site connectivity;
• retry amplification;
• stale or mismatched configuration;
• evidence service latency;
• model budget/latency;
• approval queue delay;
• fallback availability;
• telemetry loss.

Order checks by information gain, consequence, and cost. Do not begin with a restart. Preserve correlation across the request, dependencies, approval, response, and user fallback.

Step 2: bind the release artifact

Create a release record that names exact versions for code, configuration, schema, migration, model/prompt if used, policy, and evidence corpus. Link the accepted verification records, selected cohort, monitoring conditions, decision owner, stop triggers, rollback/roll-forward choice, and communication plan.

Deploy latest is not a version. Tests passed is not a traceable evidence set.

Step 3: choose recovery by state consequence

Classify each change:

• stateless code with a compatible prior artifact;
• additive compatible schema;
• destructive or externally visible state;
• partially applied migration;
• changed external side effect with ambiguous completion;
• provider/version retirement that removes the prior artifact.

Choose rollback, roll-forward, reconciliation, restore, or reduced-scope containment. Explain why the selected operation is actually available and compatible.

Step 4: execute the local rehearsal

Run the companion recovery rehearsal:

npm run rehearse:companion

Inspect both failed and recovered states. Record scenario, action, target, observed local duration, integrity check, gaps, and limitation. The fixture’s 42 ms observation can show deterministic comparison with a 100 ms local target. It cannot support a production RTO claim.

Step 5: make the release decision

Use the module’s low-connectivity failure and recovered migration evidence. Select one:

• full go;
• conditional go with expiry and owner;
• reduced cohort;
• delay;
• stop.

Name the evidence that permits the selected scope, the unresolved gap, and the exact trigger that reverses or halts exposure.

Debugging drill: recovery theater

Review this statement:

We have backups and can roll back if anything goes wrong.

Find at least six gaps. Consider restore execution, integrity, credentials, network path, schema compatibility, external side effects, migration partiality, prior artifact availability, time measurement, decision authority, and user communication. Then rewrite the statement as an evidence record with only what the local rehearsal demonstrated.

The drill also injects migration capacity pressure before a state change. The safe result is to stop before changing state, not to start and hope the system remains recoverable.

Project checkpoint: OA-08 and OA-09

Produce:

• promise-to-signal map;
• diagnostic tree;
• exact release record;
• recovery option analysis;
• executed rehearsal record;
• readiness disposition with cohort, expiry, owner, stop conditions, and communication.

The packet fails if the rollback field is a slogan, if the monitoring plan reports only aggregate uptime, or if a backup is treated as verified restoration.

Completion evidence

Submit the six records above plus the raw deterministic module output and rehearsal output. A reviewer must be able to trace a degraded cohort from user consequence through signal, diagnosis, release decision, recovery action, and next verification.

Prohibited overclaims

Local in-memory execution does not establish production performance, capacity, network behavior, dependency failure distribution, alert quality, support readiness, RTO, RPO, restore integrity, or release authority. It demonstrates the structure and deterministic behavior of the exercise only.