Make Execution Durable and Effects Recoverable

FieldOps has selected one agent and one accountable owner. Process loss must not erase either. A transcript is insufficient: it cannot prove which state was committed, who owned the lease, whether an approval remained live, or whether an external reservation happened before the response disappeared.

AR-08 v0.1.0 couples a versioned checkpoint to an effect ledger and recovery matrix. Persistence alone is not durability. Durable agent execution also needs ownership, replay and idempotency, cancellation, compatibility, and external-effect semantics; no cited runtime or protocol supplies all of them universally. [CLM-020]

Classify before retrying

A validation failure stops. A transient read may retry within a bound. Throttling uses coordinated backoff and seeded jitter. A timeout definitely before send can resume after revalidation. A timeout after possible effect is ambiguous and enters reconciliation. Known completion enters verification. Lease loss stops the old owner. Stale or incompatible checkpoints are rejected or migrated. Cancellation stops new work and reconciles anything in flight.

Retry policy belongs to one layer. If SDK, client, proxy, queue, and worker all retry, a small failure becomes a storm. Record retryable codes, maximum attempts, deadline, backoff, jitter, and the owner allowed to schedule the retry.

A timeout after an effect is not failure evidence. Blind retry can duplicate a non-idempotent action. The exact reconciliation path depends on the service’s stable request identifiers and read-after-write behavior. [CLM-019]

The distinction is operational, not linguistic. TIMEOUT describes what the caller observed, not what the remote service did. Before the request left the process, retry may be safe. After a connection accepted bytes, the same timeout may conceal a committed reservation. After the service returned but before FieldOps recorded the response, the effect may be complete even though the run appears unfinished. The recovery record stores observation and knowledge separately: transport outcome, send boundary, stable request identity, remote lookup support, and the last state the ledger can prove.

A useful matrix has one row per failure class and explicit columns for evidence needed, retry owner, attempt ceiling, reconciliation read, corrective action, escalation, and terminal disposition. “Try again” is not a disposition. SUCCEEDED, FAILED_NO_EFFECT, FAILED_EFFECT_REMAINS, CANCELLED_NO_EFFECT, and CANCELLED_EFFECT_REMAINS communicate materially different truths.

Build a failure envelope before a retry policy

A retry policy starts too late. First define the failure envelope: the interval in which the system may have changed even though the caller cannot observe the change. For a local parser, the envelope is narrow. Either bytes can be parsed or they cannot, and no external state changes. For a networked reservation, the envelope begins when the request might reach the service and ends only when an authoritative observation establishes the effect. Between those points, caller knowledge and world state can diverge.

Describe each operation along four axes. The effect axis says whether it is read-only, locally reversible, compensatable, irreversible, or prohibited. The observation axis says whether the caller can prove pre-send failure, receive an accepted response, query a stable record, or remain unable to know. The identity axis says whether repeated requests share a stable semantic key and whether the service validates changed intent. The time axis says how long keys, leases, approvals, and reconciliation reads remain meaningful. These axes determine recovery more reliably than an error string.

Consider a manual search. A timeout loses an answer but does not mutate inventory. A bounded retry may be reasonable after checking the remaining deadline. Consider an inventory hold. The same transport timeout may hide a committed hold. The next step is reconciliation, not repetition. Consider sending a customer notification. The remote service may accept the message but expose no read-by-key API. The only honest state may be DELIVERY_UNKNOWN; a second send needs an explicit duplicate-message policy, not generic retry logic.

Errors also differ by who can resolve them. A malformed argument is a design or validation error. Retrying it consumes capacity without changing the premise. A permission denial is an authority result. Searching for another token would be privilege escalation, not recovery. A rate limit is a capacity signal; bounded backoff may help. A policy conflict needs domain review. A stale checkpoint needs compatibility handling. A lost lease is a coordination event that disqualifies the old worker. A cancelled run is an instruction to stop new work, not an exception to swallow.

Build the matrix with concrete evidence requirements. For TIMEOUT_BEFORE_SEND, the transport must prove that no bytes could have been accepted. For AMBIGUOUS_EFFECT, the record names the semantic key and reconciliation method. For KNOWN_COMMITTED, it names the stable remote reference and verification fields. For LEASE_LOST, it names the newer owner or lease epoch. For INCOMPATIBLE_CHECKPOINT, it names the old schema, current schema, and allowed migration. If evidence is unavailable, the matrix must say so.

The terminal column matters because downstream consumers need to decide what to do. FAILED_NO_EFFECT permits a clean new request. FAILED_EFFECT_REMAINS demands operational attention. FAILED_EFFECT_UNKNOWN forbids assuming either absence or completion. CANCELLED_EFFECT_REMAINS warns that user intent changed after a commit. A single ERROR bucket destroys this information and encourages unsafe automation.

Keep retry budgets end to end

Suppose an agent loop allows three attempts, the HTTP client retries twice, a service mesh retries twice, and the downstream worker redelivers three times. The apparent three-attempt policy can create dozens of calls. Each layer sees its own local budget while the dependency experiences the product. Coordinated retry means one logical owner carries attempt number, elapsed deadline, and remaining capacity through every layer that can do so.

The budget should include model calls, tool calls, human waits, backoff, and reconciliation. Backoff is not free: while the run sleeps, state ages, approvals approach expiry, leases may lapse, and users wait. A retry that starts inside the network deadline but completes after the business deadline is still unacceptable. Use both an attempt ceiling and an absolute deadline.

Jitter reduces synchronized retries but does not make an unsafe effect retriable. Seeded jitter in the companion makes tests repeatable; production jitter is an operational control whose distribution and bounds belong to the platform owner. Record the chosen delay in the event log so incident review can distinguish intended backoff from scheduler delay.

A retry consumes the same semantic intent only while all meaning-bearing fields remain equal. Before every attempt, compare proposal hash, principal, tenant, capability version, authority, target, quantity, and policy version. If intent changed, close the old attempt and create a new proposal. A retry counter must never become a bridge from an expired approval to a modified action.

When a dependency asks clients to retry on a broad status class, translate that advice into the local contract. Is the operation read-only? Does the provider retain keys long enough? Can the client query outcome? Does the error prove rejection before commit? Which statuses are terminal under FieldOps policy? Vendor guidance informs the adapter; it does not override the effect and authority model.

Evidence interpretation drill

For each observation below, decide what it proves and what it does not:

• A socket connection was refused. It may prove that this endpoint accepted nothing, but it does not prove that an earlier attempt had no effect.
• The client deadline expired. It proves only that the client stopped waiting.
• The service returned 409 for a repeated key. It may indicate a safe duplicate or a changed-intent conflict; inspect documented semantics and stored request hash.
• An inventory read shows quantity reduced by one. Without a stable reference, it may not identify which actor caused the reduction.
• The local ledger says SENT. It proves dispatch was recorded, not that the remote service committed.
• A compensation request returned success. It proves the corrective API responded; verify the resulting state and preserve the original record.
• A workflow resumed after restart. It proves some state was recoverable, not that external effects, authority, or code compatibility were correct.

This discipline prevents a common reasoning error: treating any convenient signal as proof of the desired world state. Recovery decisions should cite the observation and the inference rule. If the inference depends on service documentation or local policy, record that dependency.

Checkpoint identity and ownership

The durable checkpoint records run, attempt, state, state version, owner lease, lease expiry, contract versions, event sequence, remaining budgets, proposal hash, approval reference, and effect reference. A worker may advance it only while holding the current lease and expected version.

On recovery, read the latest durable checkpoint, confirm version compatibility, acquire ownership, inspect cancellation, then query the effect ledger before emitting another action. Replaying model context is never the first recovery step.

The lease prevents concurrent writers; the version prevents a stale writer from overwriting newer truth. They solve different problems. A worker that loses the lease stops issuing effects even if local work continues. A worker with a lease but a stale expected version rereads and reclassifies. The checkpoint freezes the goal, proposal, delegation, capabilities, and policy versions used to reach the state. Recovery revalidates volatile inputs instead of treating an old checkpoint as permission.

Checkpoint frequency is a design decision. Too sparse and recovery repeats expensive work or loses the evidence behind a proposal. Too frequent and the system pays serialization, storage, coordination, and migration costs. FieldOps checkpoints at responsibility boundaries: validated intake, frozen proposal, before effect, reconciled effect, durable wait, and terminal disposition. It does not persist hidden reasoning as evidence.

Design the checkpoint as a recovery contract

A checkpoint is not a memory dump. It is the minimum durable contract from which a qualified worker can determine the next permitted transition. That definition excludes process addresses, open connections, transient callbacks, and provider-specific objects that cannot be reconstructed. It includes stable identifiers, contract versions, evidence references, budgets, ownership, cancellation, and unresolved external effects.

Separate the record into five groups. Identity contains run, task, tenant, principal, and parent identifiers. Control contains state, state version, event sequence, owner, lease epoch, cancellation, deadline, and remaining budgets. Contract contains goal, capability, policy, delegation, approval, and adapter versions. Evidence contains validated source references, proposal hash, uncertainty, and artifact versions. Effects contains semantic keys, dispatch observations, remote references, reconciliation status, and corrective actions.

These groups age differently. Run identity is stable. Ownership changes on failover. Delegation and approval expire. Evidence becomes stale. Adapter and schema versions change on deployment. Effects accumulate and must not be overwritten. A recovery implementation that deserializes one opaque blob cannot apply distinct rules to each group.

Checkpoint writes need concurrency semantics. FieldOps uses expected state version plus lease epoch. The writer says, in effect, “advance version 17 to 18 only if I still hold lease epoch 7 and version 17 is current.” A failed conditional write is not a transient storage annoyance. It signals that another actor or event changed the run, so the worker rereads before doing anything else.

Do not hold a lease forever. A lease has a short renewal interval, an expiry, and a monotonic epoch. The epoch distinguishes a new owner from an old owner whose clock or network view is stale. Even if the old process wakes and sees its original lease identifier, the higher epoch fences it from writes and effects. External services that support fencing tokens can enforce the epoch; otherwise the adapter must minimize the interval and reconcile any in-flight ambiguity.

Clocks introduce another boundary. Wall-clock timestamps support audit and human understanding, but lease correctness should use a trusted time source and conservative skew assumptions. The application cannot infer safety from its local clock alone. Platform/SRE owns the actual clock, storage, and failover guarantees; the manuscript contract states what the application requires and how it fails closed when those requirements are unavailable.

Choose checkpoint boundaries intentionally

Checkpoint after validation so a restart does not repeat untrusted parsing as if it were accepted evidence. Checkpoint after proposal freeze so approval binds an immutable artifact. Checkpoint before effect so recovery can identify a prepared intent. Checkpoint after reconciliation so repeated recovery does not query and interpret the same effect inconsistently. Checkpoint before human wait so a process crash cannot erase the pending decision. Checkpoint terminal state so late events cannot reopen the run.

Some operations need a prepare record before dispatch and a result record after observation. There is inevitably a gap between local write and remote commit unless both systems share a transaction, which these services do not. The effect ledger exists to bridge that gap through identity and reconciliation. Pretending that checkpoint timing eliminates the gap merely hides the distributed systems problem.

Avoid checkpointing after every model token or internal thought. Such data is expensive, difficult to migrate, may contain sensitive content, and is not the evidence a new worker needs. Persist explicit intermediate artifacts: selected record IDs, validated claims, proposed action, policy decision, remaining alternatives, and why execution is blocked. A replacement should be able to continue without reproducing hidden reasoning.

Retention follows purpose. Active checkpoints remain while the run can resume. Terminal checkpoints may retain a bounded audit projection, not every transient context item. Effect ledgers may require longer retention than model context because duplicate suppression and incident analysis depend on them. Deletion requests must preserve lawful audit obligations while removing content no longer needed; privacy and records specialists define those policies.

Recovery bootstrap procedure

1. Load the run by stable identifier from authoritative storage.
2. Verify schema and contract versions before acquiring effect authority.
3. Read terminal and cancellation state. A terminal run does not restart.
4. Acquire a new lease epoch with a conditional write.
5. Recheck that the loaded version remains current.
6. Restore budgets, deadline, and retry counters.
7. Validate identity, delegation, capability, and approval references.
8. Revalidate volatile evidence required for the next transition.
9. Inspect unresolved effect entries before planning or dispatch.
10. Choose one declared transition and append its event.

The order prevents recovery from becoming a privilege escalation. Loading serialized state cannot restore an expired token. Acquiring a lease cannot revive cancelled work. Replaying an event cannot bypass current policy. A worker that cannot verify a required contract stops with a named reason.

Deployment compatibility

Long-running work crosses deployments. Every checkpoint names the schema and behavioral contract versions needed to interpret it. Backward compatibility means more than successful JSON parsing: the new code must preserve transition meaning, proposal hashing, capability semantics, authority checks, and effect identity. A renamed state that changes when dispatch is allowed is a behavioral migration.

Use one of four dispositions. Replay compatible means the new worker can continue unchanged under a tested compatibility promise. Migrate means a deterministic, audited transformation produces a new checkpoint and explains every changed field. Pin means old code drains old runs under bounded operational ownership. Stop means the run cannot safely continue and moves to human resolution. “Best effort deserialize” is not a fifth option.

Migration itself cannot issue external effects. Test it against frozen checkpoints representing each state, including waiting, cancelled, ambiguous effect, and partially compensated. Preserve the old record and migration version. If the transformation cannot decide, quarantine rather than inventing missing semantics.

Effect ledger and semantic identity

The ledger binds one idempotency key to principal, task, run, proposal, part, slot, and quantity. Same key plus same intent returns the original effect. Same key plus different intent is an integrity error. Another random key does not make a duplicate business intent safe.

Crash after commit but before local record is the central case. Recovery queries the ledger. If it finds the matching effect, record and verify it. If absence is authoritative, revalidate and possibly redispatch with the same key. If unknown, escalate. Never claim exactly-once external execution.

Compensation is another effect. Releasing stock may fail, be delayed, or leave consequences. It needs authority, key, ledger, verification, and final disposition. It never erases the original reservation.

Ledger entries move through explicit knowledge states such as PROPOSED, SENT_UNKNOWN, CONFIRMED, REJECTED, COMPENSATION_PROPOSED, and COMPENSATED. These are local assertions backed by evidence references. The ledger never upgrades SENT_UNKNOWN to failure because a deadline elapsed. It queries the remote system by semantic key or provider reference. If the service cannot answer, the run escalates with the ambiguity intact.

Semantic identity matters more than a random retry token. reserve(slot-19, technician-4, incident-82, proposal-hash) represents one intended reservation. A corrected proposal for another slot is a new intent and gets a new key. Reusing the old key could suppress a valid effect; minting a new key for an unchanged retry could duplicate it. Caller and service owner must agree on key scope, retention, collision behavior, and repeated-request meaning.

Cancellation follows the same honesty rule. It prevents new work after observation but cannot retroactively cancel an effect already committed. The worker records cancellation time, reconciles in-flight effects, and returns a terminal state saying whether effects remain.

Specify semantic idempotency, not merely a key header

Idempotency is a relationship among intent, key, service behavior, and retention. A client-generated string alone offers no guarantee. The adapter contract answers: which fields define one intent, where uniqueness is scoped, how long the service remembers a key, what response a duplicate receives, what happens when the same key carries changed parameters, and how clients query the recorded result.

For the FieldOps reservation, intent includes tenant, incident, part, slot, technician, quantity, requesting principal, and proposal hash. Omitting tenant risks cross-tenant collision. Omitting quantity may suppress a legitimate changed request. Including a volatile timestamp would make identical retries appear different. The canonicalization function needs explicit field order, normalization, and version. A change to canonicalization is a compatibility event.

Generate the key before first dispatch and persist it with the prepared effect. Never derive it from an attempt number. Attempt numbers distinguish transport tries under one intent; keys distinguish intended business effects. The ledger records both. An operator looking at attempt three should see that it still targets effect E-82, not mistake three attempts for three authorized reservations.

The server should store request hash and result atomically with effect commit when its design permits. A repeated matching hash can return the recorded result. A repeated key with a different hash returns an integrity conflict. If the server only deduplicates key strings without semantic validation, the client must treat changed-intent reuse as a critical local error and acknowledge the remaining provider limitation.

Retention must cover the maximum retry and reconciliation horizon. If a service forgets keys after twenty-four hours while a workflow can resume after seven days, the adapter cannot promise safe redispatch on day seven. It can query a durable business record, require human reconciliation, or stop. Extending local ledger retention does not extend provider deduplication.

Late arrivals complicate the horizon. A timed-out request may arrive after the caller starts reconciliation. A service that supports a stable key can still collapse it. Without that support, a negative read may be only temporarily negative. Define how long to wait, which consistency model applies, and when the result becomes operationally unresolved rather than falsely absent.

Reconciliation is a first-class capability

Treat reconciliation reads as typed capabilities with their own permissions, errors, freshness, and evidence. A generic search is not automatically authoritative. The ideal query accepts the semantic key and returns effect state plus immutable parameters and stable reference. A weaker query may search by incident and slot, requiring disambiguation. A read that returns eventual-consistency data must expose that limitation.

The reconciliation algorithm compares all intent fields. A record with the same slot but different incident is not the target. A record with the same key but different quantity is a conflict. A matching record in PENDING may require another bounded observation rather than dispatch. A matching terminal failure may permit retry only if the service proves no effect. An unknown record preserves ambiguity.

Record the evidence used: query capability version, time, returned reference, parameter hash, service status, and consistency caveat. This turns reconciliation from a hidden branch into an auditable decision. It also supports incident analysis when service behavior violates the documented contract.

Reconciliation can be expensive. Queries may be rate limited or require elevated read scope. Budget them and define escalation. Do not compensate merely because reconciliation is inconvenient. Corrective action without knowing the original effect can worsen state.

Compensation as forward recovery

Compensation is often described as undo, but external systems rarely rewind history. Releasing a reservation creates a second event. A message already delivered may require a correction, not deletion. A technician already dispatched may need reassignment and communication. Money transferred may require refund workflows with separate timing and fees. The original consequence may persist.

Define compensation eligibility, authority, parameters, deadline, evidence, and expected residual. Some effects are not compensatable. Others allow partial correction. A compensation request can itself time out after commit, creating another ambiguous effect. Therefore it uses the same prepare, semantic key, ledger, reconciliation, verification, and disposition pattern.

FieldOps names the relationship: compensation C-82 addresses effect E-82. It does not mutate E-82 from committed to nonexistent. If C-82 succeeds, the ledger records both and the current business state. If it fails, operators can see the outstanding reservation and failed release. This history is essential for audit and user communication.

Compensation authority may be narrower or broader than original authority. A user authorized to reserve may be allowed to release their own hold, while a cancellation after dispatch may require a coordinator. Do not infer corrective authority from original permission. The AR-05 hook validates it separately.

Cancellation races

Cancellation can arrive before proposal, after proposal, during approval, immediately before dispatch, while the request is in flight, after commit, or during compensation. Each position has a different result. Before dispatch, stop cleanly and close pending approvals. During wait, revoke the checkpoint so late approval cannot resume. In flight, mark cancellation and reconcile. After commit, block new work and consider authorized compensation. During compensation, reconcile both effects.

Event ordering needs sequence numbers or conditional state transitions. If cancel and dispatch race on version 17, only one conditional write should prepare the next state. If dispatch wins and then cancellation advances version 18, the ledger shows an in-flight effect requiring reconciliation. If cancellation wins, the dispatching worker’s write fails and its lease/epoch guard blocks sending. Where a remote request may already have left, the system keeps ambiguity rather than assuming the local race prevented it.

User-facing status should distinguish “cancellation requested,” “new work stopped,” “effect reconciliation pending,” and “cancelled with effect remaining.” A spinner that jumps directly to “cancelled” may be comforting but operationally false.

Crash laboratory

Crash before effect resumes only after lease, cancellation, authority, freshness, and budget checks. Crash after commit before local record reconciles by key. Crash after record before response replays the recorded result. Crash while waiting preserves the durable human checkpoint. Crash during deployment requires compatible code or explicit migration.

A duplicate event is ignored through sequence and semantic identity. A late callback cannot revive an expired lease or terminal run. Failed compensation leaves both effects and escalates; it does not report restoration.

Temporal is one current durable-runtime example, not the canonical architecture. MCP 2025 Tasks remains experimental, and the July 2026 work is a release candidate. A2A supplies task-state vocabulary but not local effect recovery or authority. Adapters must preserve AR-08 semantics and their status labels.

Map runtimes and protocols without outsourcing semantics

A durable runtime can provide persisted workflow state, timers, retries, activity boundaries, and replay behavior. Those are valuable primitives. The application still decides which activity is an external effect, how semantic identity is formed, what authority must be current, and how ambiguous outcomes are reconciled. A runtime may replay application code deterministically while a remote reservation remains outside its transaction.

Map AR-08 onto any runtime through an adapter table. For each local state, name the runtime state or event, what data is durable, who owns the lease, how cancellation propagates, how versions migrate, and which application checks occur before an effect. If the runtime retries activities automatically, coordinate that feature with the single retry owner. If it records completion only after an activity returns, handle commit-before-return ambiguity inside the activity adapter.

Protocol task states can help another component report submitted, working, waiting, completed, failed, cancelled, or rejected. They do not prove the truth of the remote state. A remote completed message is evidence to validate against the agreed artifact and effect contract. A remote cancelled state does not prove no effect remains. Local authority, tenant, data policy, and audit requirements continue to apply.

Status matters. Experimental MCP Tasks may change semantics and should not become an invisible production dependency. A release candidate can support exploration while retaining compatibility gates and pinned versions. A2A task vocabulary can shape interoperability tests without granting a remote agent trust. Every adapter record names spec version, feature status, unsupported local semantics, and fallback.

Provider neutrality does not mean lowest-common-denominator design. It means the book’s invariants are expressed independently, then each implementation is tested for how completely it realizes them. One provider may offer durable timers but weak effect reconciliation. Another may supply server-side keys but no workflow migration. The gap stays visible.

Six crash points in one reservation

Walk a single FieldOps intent through six crash locations.

Before prepare. No effect entry exists. Recovery revalidates the request and may construct the proposal again, but it does not assume prior authorization or evidence remains current.

After prepare, before send. The ledger contains a stable key in PROPOSED. Recovery verifies that transport could not have sent, reacquires ownership, checks cancellation and authority, and may dispatch under the same intent.

After send, before remote commit is known. The ledger becomes SENT_UNKNOWN. Recovery queries. It does not retry until absence is authoritative and redispatch is allowed.

After remote commit, before local record. This is observationally similar to the previous point. The query finds the effect, and local state converges without redispatch.

After local record, before checkpoint advance or response. Recovery finds CONFIRMED, verifies the effect, advances the checkpoint, and replays the result. It never repeats the effect to recreate a response.

After checkpoint, before user notification. The run is complete but communication may be missing. Notification is a separate effect with its own identity. Do not rerun the reservation to regenerate a message.

The crash locations demonstrate why a transcript cannot drive recovery. The same last user-visible message can correspond to no prepared effect, an ambiguous commit, a confirmed effect, or a completed run with missing notification.

Late, duplicate, and reordered events

Queues and callbacks may deliver more than once or out of order. Give every event a stable identifier, run identifier, expected state/version, producer identity, and semantic type. The run stores the highest accepted sequence where ordering exists and a bounded deduplication record where it does not. Duplicate receipt should return the prior processing disposition rather than silently execute again.

A late approval for an expired checkpoint is recorded as late and denied. A late tool callback for an old lease can contribute reconciliation evidence but cannot advance state directly. A completion arriving after cancellation may establish that an effect remains. A cancellation arriving after terminal success records user intent but does not rewrite history.

Out-of-order evidence may still be useful. Separate admission to the audit trail from permission to mutate active state. This distinction preserves forensic information without allowing stale actors to seize control.

Recovery when dependencies disagree

Suppose the ledger says CONFIRMED, the service query says missing, and inventory quantity suggests a change. Do not choose the most convenient source. Classify each source by authority, freshness, and identity precision. A local ledger can prove what FieldOps recorded, not current remote state. A service query may be eventually consistent. Quantity may aggregate unrelated effects. The conflict becomes a new evidence state and escalates under a named owner.

If two remote endpoints disagree during a regional incident, the platform and service owners define which is authoritative. The agent should not majority-vote system-of-record state unless that rule is explicitly part of the contract. If authority cannot be resolved within the deadline, stop with the conflict documented.

This is an important operational skill: recovery is not always computation. Sometimes the correct action is to preserve evidence, prevent further mutation, and transfer a precise unresolved state to a human or service owner.

Eight traces, not one happy path

1. Crash before dispatch: reacquire the lease, revalidate authority and inputs, then dispatch once under the original semantic intent.
2. Commit before local record: query by key; confirm and record the existing effect.
3. Record before response: replay the recorded result without redispatch.
4. Crash during approval wait: restore WAITING_HUMAN; do not invent approval.
5. Incompatible deployment: migrate explicitly or stop; never guess how old state maps.
6. Duplicate input event: reject by event sequence and semantic identity.
7. Late callback: ignore if lease, run, or checkpoint is terminal or expired.
8. Compensation failure: preserve original and corrective records, mark effects remaining, and escalate.

For every trace, write what was known before failure, what survived, which actor owns recovery, what external query is authoritative, and which terminal state follows. The answer is sometimes “stop with unresolved effect.” That is safer and more truthful than manufacturing success.

Skill procedure

Start from the frozen AR-07 v1.0.0 topology. Add a versioned checkpoint and one lease owner. Define semantic effect identity before writing retry code. Assign exactly one retry layer. Give each retriable class an attempt ceiling, deadline, seeded jitter, and budget. Define authoritative reconciliation reads and separately authorized compensation. Inject all eight traces. Accept the design only when every trace ends with one owner, a non-contradictory ledger, and an honest disposition.

The platform team still owns storage, queues, scheduling, clocks, and runtime availability. Service owners define idempotency and reconciliation behavior. Security and domain owners define authorization. The Agentic AI Engineer owns how the run uses those primitives without widening authority or hiding ambiguity.

Worked FieldOps recovery

FieldOps proposes reserving part P-204, slot S-19, quantity one for incident I-82. The proposal has passed capability and authority checks, and the semantic key binds that exact tuple to its proposal hash. The worker writes EFFECT_PROPOSED, obtains checkpoint version 17 under lease L-6, and dispatches. The connection drops before the response arrives.

The worker does not label the attempt failed. It writes SENT_UNKNOWN if it still owns the lease, then stops dispatch. Suppose the process crashes first. A replacement worker reads version 17, acquires L-7, observes the unresolved ledger entry, and calls the service’s authoritative lookup with the same semantic key. The service returns an existing reservation and stable reference. FieldOps records CONFIRMED, verifies part, slot, incident, quantity, and principal, advances the checkpoint with a compare-and-set, and completes without a second reservation.

Now change one fact: the service has no key lookup and its inventory read cannot distinguish this reservation from another actor’s reservation. The effect remains ambiguous. The replacement cannot treat a missing response as absence, cannot mint a new key, and cannot compensate an effect it cannot identify. It records FAILED_EFFECT_UNKNOWN, names the operator escalation, and stops. Durability succeeded because the system preserved an uncomfortable truth.

In another trace, reconciliation confirms the reservation but the user cancelled while the process was down. Cancellation blocks new work. Policy may allow an authorized release as compensation. That release receives its own proposal, key, capability check, authority check, response, and verification. If it fails, the final disposition says the reservation remains. “Cancelled” alone would be false.

Common durability traps

Transcript recovery: replaying messages can regenerate a different action and cannot establish what the service committed. Recover contracts and ledgers first.

Retry multiplication: libraries and workers each retry independently. Assign one owner and include downstream behavior in the attempt budget.

Key without intent: a UUID prevents nothing if each attempt receives a new UUID. Bind stable business semantics and reject changed intent under the old key.

Lease as permission: a lease coordinates writers; it does not grant domain authority. Both must pass.

Checkpoint forever: old serialized state may not match new code or policy. Declare compatibility, migration, retention, and terminal handling.

Compensation optimism: corrective effects can be partial, irreversible, costly, or disallowed. Report their consequences and failure states.

Cancellation theater: cancelling local compute while leaving a remote effect unresolved is not complete cancellation.

The design review should ask for evidence from every trace, not a screenshot of a successful resume. A durable system is recognizable by the precision of its failed states.

Operate the recovery system

Design is incomplete until an operator can understand and control a failed run. The recovery surface should answer: what is the run trying to do, who owns it, what state is durable, which effects are known, which are ambiguous, what authority is live, what deadline remains, and what actions are permitted now? Raw workflow history may be available as drill-down, but the first view should be a decision packet.

A recovery decision packet

The header names run, tenant, goal, consequence class, current state, state version, owner lease, and cancellation. The effect panel lists each semantic intent, attempts, last transport observation, remote reference, reconciliation evidence, and current knowledge state. The authority panel shows principal, delegation scope, approval binding, expiry, and revocation. The compatibility panel shows checkpoint, code, adapter, and policy versions. The budget panel shows time, calls, retries, and human escalation remaining.

The packet recommends a disposition but does not conceal alternatives. RECONCILE should name the authoritative query and expected result classes. COMPENSATE should name the corrective proposal and required authority. STOP should say whether an effect remains or is unknown. The operator should never need to infer that a green badge means no external consequence.

Allowlisted controls reduce accidental authority. An operator may acquire takeover, run a specific reconciliation read, approve a specific corrective proposal, mark an external reference after independent verification, cancel new work, or close with an unresolved disposition. A generic “edit state” control bypasses the contract and should be reserved for a separate audited break-glass process owned by the platform and security teams.

Every manual action appends an event with principal, authority basis, before/after version, evidence, reason, and time. The audit record does not need sensitive payloads or hidden reasoning. It needs enough to reconstruct who changed which decision-bearing state and why.

Alerts that lead to action

Alert on states that require a defined owner response: lease repeatedly stolen, reconciliation budget exhausted, unknown effect beyond deadline, key collision, incompatible checkpoint without migration, compensation failure, late callbacks after terminal state, and retry amplification. Do not page merely because a workflow remains legitimately waiting for a human within SLA.

Each alert names first action. A key collision stops dispatch and asks the service owner to inspect canonicalization. An unknown effect asks the operator to use the named reconciliation route. An incompatible checkpoint points to the migration or pin runbook. A retry storm disables the local retry owner or dependency route according to incident policy. Alerts without a first action create noise rather than control.

Aggregate metrics should preserve consequence segments. Track ambiguous-effect rate, reconciliation success, duplicate suppression, compensation attempts and failures, lease conflicts, recovery latency, terminal unresolved effects, and migration stops. A low average recovery time can hide a small number of high-consequence unknown effects. Report counts and denominators, not just percentages.

These metrics are operational signals, not proof of safety or correctness. A falling ambiguity rate may reflect fewer effects, changed traffic, or missing instrumentation. Incident review and trace sampling remain necessary.

Runbook: ambiguous reservation

1. Freeze new dispatch for the semantic intent.
2. Confirm the current run, state version, lease epoch, and cancellation state.
3. Validate the key and proposal hash against the prepared effect.
4. Use the adapter’s authoritative read; record query version and response.
5. If a complete matching effect exists, record and verify it.
6. If authoritative absence exists and authority remains live, decide whether same-key redispatch is permitted.
7. If parameters conflict, stop as integrity error.
8. If outcome remains unknown, exhaust only the declared observation budget.
9. Escalate to the service owner or operator with the evidence packet.
10. Close with a precise terminal disposition when the business deadline ends.

The runbook never asks the operator to “just retry.” It distinguishes redispatch under proven absence from a blind repeat. It also prevents a human from minting a new key merely to make the interface proceed.

Runbook: lost owner

First establish that the lease expired according to the authoritative store. Acquire a higher epoch conditionally. Fence or disable the old worker where the platform supports it. Read all events and effects after the checkpoint version the new worker loaded. Apply cancellation and terminal state before other transitions. Reconcile in-flight effects. Only then resume planned work.

If the old worker later returns, its writes fail version or epoch checks. Its callback can be stored as evidence but not applied directly. If it may have dispatched after losing the lease, the corresponding intent becomes ambiguous and follows reconciliation. The platform team investigates why fencing failed; the application preserves effect truth meanwhile.

Runbook: incompatible deployment

Pause new ownership acquisition for affected checkpoint versions. Identify whether the incompatibility is serialization, state transition, proposal hashing, capability semantics, or effect identity. Run the tested migration on a copy. Compare stable identities, authority bindings, unresolved effects, and terminal guards. If every invariant holds, write the migrated checkpoint with its transformation record. Otherwise pin compatible code or transfer to manual resolution.

Do not roll back application code blindly when external effects may have occurred under the newer version. Code rollback and business-effect recovery are separate decisions. Reconcile both versions’ effects and confirm which adapters generated them.

Design review through counterexamples

Counterexamples expose whether a design relies on optimistic assumptions.

“The queue guarantees exactly once”

Even if a queue delivers one message once under its documented scope, a worker may crash after the remote service commits and before message acknowledgement. Redelivery can repeat the effect. Another producer may emit a semantic duplicate. The remote service may process a request twice after its own failover. The correct claim is narrower: the queue provides a delivery property, while the application and service cooperate on semantic deduplication and reconciliation. [CLM-019]

“The workflow history is the source of truth”

It is authoritative for recorded workflow events under the runtime’s guarantees. It is not automatically authoritative for inventory, payments, messages, or remote tasks. External systems retain their own truth. Recovery compares them through stable identities instead of declaring one log universally correct.

“All actions have compensations”

Some effects are irreversible; others have only partial remedies. A notification cannot be unread. A technician who travelled cannot have the trip erased. A release may free inventory but not undo missed availability for another user. The design must state residual consequence and authority.

“We can ask the model what it already did”

A generated account is not durable evidence. The model may reconstruct, omit, or contradict events. Use event records, capability receipts, remote references, and authoritative queries. Model output may summarize those records for an operator, but it cannot replace them.

“Cancellation means the user no longer cares”

Cancellation changes permission for new work. It does not resolve external state. A cancelled reservation can still block inventory. The terminal state must expose the effect and corrective options.

“If reconciliation is hard, retry is practical”

Practicality does not change semantics. Repeating an ambiguous non-idempotent effect converts uncertainty into possible duplication. If the service offers no usable identity or read, narrow autonomy, add a human process, redesign the integration, or reject the capability.

“A human can fix the ledger”

A human can provide evidence or exercise authorized override, but direct ledger editing destroys traceability. Record corrective events and preserve original observations. If break-glass mutation is unavoidable, isolate it with dual control, reason, before/after values, and later review.

Failure-injection workshop

The workshop uses deterministic synthetic services so readers can observe state, not just final messages. Each trace begins from the same frozen proposal and authority. The harness varies one failure at a time, then combines selected failures after individual behavior is understood.

For the pre-send trace, force the transport to reject before bytes leave. Expect one prepared ledger entry, zero remote effects, and a bounded redispatch using the same key. For the ambiguous commit trace, commit remotely and drop the response. Expect SENT_UNKNOWN, a reconciliation read, one confirmed effect, and no second dispatch. For the post-record crash, persist the result then terminate before response. Expect result replay.

For the waiting trace, pause on approval and destroy the worker. Expect the checkpoint to remain WAITING_HUMAN, with no effect capability active. For deployment, change proposal hash semantics while a checkpoint waits. Expect migration or stop. For duplicate input, deliver the same event twice. Expect one transition and an auditable duplicate disposition.

For late callback, expire the lease and let a previous worker return success. Expect the callback to enter evidence but fail direct transition. Reconciliation decides whether the effect exists. For compensation failure, confirm original reservation, authorize release, commit release ambiguously, and make the verification read unavailable. Expect both records and an unresolved corrective state.

After the isolated traces, combine cancellation with ambiguous commit and deployment with expired approval. Combined faults test ordering. A design that passes each single fault can still fail when cancellation is applied after a stale worker resumes or when migration accidentally refreshes old authority.

Readers should draw three timelines for each trace: local run state, remote effect state, and knowledge state. The timelines will not always align. That mismatch is the lesson. Recovery converges them where evidence allows and preserves disagreement where it does not.

Construct AR-08 v0.1.0

The chapter artifact is a reviewable durability packet, not a framework configuration. It has a checkpoint contract, effect-ledger contract, recovery matrix, compatibility policy, eight trace fixtures, operator decision packet, and boundary statement. A second engineer should be able to inspect the packet and predict every transition without relying on the original author’s intuition.

Checkpoint contract checklist

Name the stable run and task identifiers. Record tenant and verified principal without embedding credentials. Define every durable state and legal transition. Add expected state version and lease epoch to mutation. Carry cancellation, deadline, attempt, tool, time, and human-wait budgets. Pin goal, capability, delegation, policy, approval, adapter, and schema versions. Reference evidence and artifacts by immutable identifiers. Include every unresolved effect and its semantic key. Define terminal states and whether each may retain an effect.

For each field, answer who writes it, when it changes, which source is authoritative, and how old versions migrate. A field named status with no transition owner is not a contract. A field named context containing an arbitrary blob is a hidden compatibility and privacy problem.

The packet includes a sample at every difficult state: before effect, ambiguous effect, waiting for human, cancelled with in-flight request, incompatible version, compensation pending, and terminal with effect remaining. Schema validation alone cannot prove meaning, so tests assert invariants across these samples.

Effect-ledger checklist

Define the intent fields and canonicalization version. Define key scope and creation point. Store request hash, attempts, transport observations, remote reference, reconciliation observations, and verification. Make knowledge states monotonic except through explicit correction events. Link compensations without overwriting originals. Record retention and provider deduplication horizon separately.

Test same key and same intent, same key and changed intent, new key and same business intent, late arrival after negative read, remote match with mismatched parameters, duplicate callback, and compensation ambiguity. The most dangerous test is often new key with same intent because many systems treat it as unrelated while the business sees a duplicate.

Recovery-matrix checklist

Every row names trigger, proof, owner, permitted transitions, prohibited actions, attempts, deadline, reconciliation, compensation, escalation, and terminal state. Include validation failure, transient read, throttling, pre-send timeout, ambiguous effect, known completion, non-compensatable effect, lease loss, stale checkpoint, incompatible version, cancellation, duplicate event, late callback, and contradictory evidence.

Review the negative space. What happens when the reconciliation service is down? When key retention expired? When both old and new workers claim ownership? When the approval is live but the capability version changed? When compensation requires unavailable authority? A matrix containing only recoverable cases is a success-path table, not a recovery design.

Compatibility checklist

State which checkpoint versions current code can read and execute, not merely parse. Pin protocol and adapter statuses. Define migration inputs, outputs, invariants, rollback of the migration record, and failure quarantine. Test cancellation and unresolved effects through migration. Define how pinned old workers are monitored and retired.

Version proposal hashing and intent canonicalization explicitly. A silent change can make the same action appear different, defeating deduplication, or make different actions collide. Compatibility review therefore includes business semantics, not only storage schemas.

Assess the eight traces

The twenty-point rubric gives five points to classification. Full credit requires distinguishing caller observation from effect truth and naming the evidence needed. A learner loses points for treating every timeout alike, but automatic failure is reserved for blind retry of an ambiguous non-idempotent effect.

Four ownership points require one current lease holder, fenced stale workers, a single retry owner, and an explicit accountable operator for unresolved states. A design with two helpful recovery workers receives zero ownership points because the duplicate-effect risk is structural.

Six effect-correctness points cover stable semantic identity, same-intent deduplication, changed-intent rejection, authoritative reconciliation, separate compensation, and preservation of effect history. Saying “idempotency key” without defining intent earns at most one of these points.

Five disposition points require each trace to end in a truthful terminal or durable waiting state, with cancellation and residual effects visible. ERROR, DONE, and CANCELLED alone are insufficient. The learner must state what effect exists or remains unknown and who acts next.

Trace questions

For crash before effect, which evidence proves pre-send, and what must be revalidated? For commit before local record, how does the new worker find the effect without repeating it? For record before response, which stored result can be replayed? During approval wait, what survives and which authority is still absent? During deployment, which behavioral invariant decides compatibility? For duplicate event, where is deduplication stored? For late callback, how can it inform reconciliation without reviving stale ownership? For failed compensation, which effects and residual consequences appear in the final state?

Answers must identify source and inference, not merely choose a verb. “Reconcile” earns credit only with an authoritative query, identity comparison, possible outcomes, and escalation when unknown.

Extension exercise: redesign a weak service

Assume the inventory service accepts reservation requests but provides no idempotency keys and no read by client reference. It returns a generic timeout and exposes only total remaining quantity. Ask learners to propose three bounded options.

One option is to remove autonomous reservation and return a human-executed proposal. Another is to build a service-owned reservation command with stable client intent and query support. A third may serialize requests through a trusted local adapter that owns a durable outbox and receives service acknowledgements, while acknowledging that remote commit ambiguity still needs cooperation. “Store the request locally and retry” is not sufficient.

Compare the options by effect correctness, authority, latency, operational burden, and remaining ambiguity. The exercise teaches that architecture sometimes changes the capability boundary instead of adding clever recovery code.

Extension exercise: cancellation under pressure

Give learners a timeline: approval at 10:00, dispatch at 10:02, client timeout at 10:03, user cancellation at 10:04, lease expiry at 10:05, old worker callback at 10:06, remote query unavailable until 10:20. The business deadline is 10:15.

Require a state/event table. The answer should block new work at 10:04, transfer ownership after lease expiry, accept the callback only as evidence, preserve ambiguity while the authoritative read is unavailable, and reach a deadline disposition stating cancellation with effect unknown. If later evidence confirms the reservation, an authorized corrective workflow begins; the earlier terminal record remains historically accurate for what was known at 10:15.

Know the ownership boundary

The Agentic AI Engineer owns run-level semantics: when the agent may advance, how intent maps to effects, how authority is rechecked, how evidence determines recovery, and which terminal states are reported. That role designs the adapter requirements and tests the end-to-end behavior.

Platform and SRE teams own the durability substrate: storage availability, queue delivery, scheduling, lease primitives, clocks, backup, deployment, and runtime observability. The agent engineer cannot promise those properties from application code. Instead, the contract names required guarantees and behavior when they fail.

Service owners own actual command semantics, key retention, atomicity, consistency, reconciliation APIs, and compensation possibilities. An adapter document cannot create server-side idempotency. Security and identity specialists own authentication, token protection, and policy enforcement. Domain owners decide whether an effect or compensation is permissible. Human operations own incident command and break-glass processes.

Boundaries do not remove collaboration. A durability review brings these owners together around the same traces. The agent engineer supplies semantic intent and failure cases. The platform team explains leases and failover. The service owner demonstrates reconciliation behavior. Security verifies authority. Operations confirms the decision packet and escalation path. Each claim remains attributed to the system that can make it true.

Durable and volatile knowledge

The principles in this chapter are durable: distinguish observation from effect truth; persist versioned state; maintain one owner; bind retries to semantic intent; reconcile ambiguous effects; treat compensation as a new effect; preserve cancellation and terminal truth; test crashes at state boundaries.

Implementation APIs are volatile. Runtime replay rules, SDK pause/resume calls, protocol task schemas, release-candidate features, and vendor retry defaults change. Keep them in adapters and source notes. Before publication or implementation, verify current status and pin versions. Do not let a current product example become the definition of durable execution.

This separation also guides maintenance. A runtime migration should leave AR-08 invariants unchanged. If an implementation cannot express one, record the gap and narrow the capability rather than weakening the book’s contract silently.

Final FieldOps review

Walk the dossier from AR-07 v1.0.0. The selected topology remains one agent and one final owner. AR-08 v0.1.0 does not add agents, capabilities, or authority. It adds state and evidence that survive interruption: checkpoint versions, lease epochs, semantic effect records, recovery decisions, compatibility status, and terminal dispositions.

The reservation fault proves the central claim. A response can disappear after commit. FieldOps neither repeats blindly nor invents exactly once. It queries by semantic identity, verifies parameters, and converges when evidence permits. When evidence does not permit, it stops with ambiguity visible. [CLM-019]

The crash lab proves the second claim at the architecture level. Persisted state alone would still permit stale ownership, replay under expired authority, duplicate effects, incompatible deployment, and false cancellation. The complete design combines persistence with ownership, idempotency, reconciliation, cancellation, and compatibility. [CLM-020]

The next chapter receives a durable waiting state but no human decision yet. That distinction is deliberate. The system can survive a pause; Chapter 11 must make the pause useful by binding an exact decision to evidence, authority, time, rejection, and takeover.

Handoff rehearsal

End the chapter by handing AR-08 v0.1.0 to a reviewer who did not build it. Give them only the checkpoint schema, effect ledger, recovery matrix, compatibility table, and eight traces. Ask them to recover three randomly selected runs without consulting implementation authors.

In the first run, the checkpoint is version 17, the lease expired, a reservation is SENT_UNKNOWN, and cancellation arrived later. The reviewer should acquire a new epoch, preserve cancellation, query the exact semantic key, and finish with either cancelled effect confirmed, cancelled no effect, or cancelled effect unknown. They must not dispatch.

In the second run, the ledger is CONFIRMED, the user never received a response, and current code changed checkpoint schema. The reviewer should separate workflow migration from the already completed business effect. If migration is safe, replay the recorded result; if not, communicate the confirmed effect through a bounded manual path. Repeating the reservation is never required to reconstruct the response.

In the third run, compensation is SENT_UNKNOWN after the original effect was confirmed. The reviewer reconciles the corrective effect separately and preserves both histories. If release cannot be determined, the terminal record states original effect and corrective effect uncertainty.

Observe where the reviewer asks for missing information. Each question reveals an implicit assumption that belongs in the artifact. Common gaps include which read is authoritative, how long the provider retains keys, whether a late callback may update evidence, which authority permits compensation, and who owns an unresolved effect after the business deadline.

The rehearsal passes when independent reviewers reach the same permitted disposition from the same evidence. They need not use identical words, but they must preserve ownership, authority, intent, effect truth, and uncertainty. Disagreement signals an underspecified contract or a genuine policy decision that needs an owner.

Record disagreements explicitly. One reviewer may interpret a negative inventory read as authoritative absence while another knows the service is eventually consistent. Resolve the underlying contract with the service owner; do not average the decisions. Another may assume compensation follows original approval while policy requires a new coordinator decision. Resolve authority with the domain owner. The durability packet improves when these hidden premises become named rules, evidence requirements, or stop conditions.

Repeat the rehearsal after changing runtime or adapter. Equal dispositions across implementations are stronger evidence of provider-neutral semantics than equal API shapes.

Archive the completed traces with their contract versions and expected terminal states. They become regression evidence for future migrations, retry changes, service integrations, and incident corrections. A new implementation earns compatibility by reproducing the invariants, including the honest unresolved outcomes.

This exercise also tests operational usability. A design can be logically sound yet too opaque to recover under pressure. Improve field names, evidence links, alerts, and runbook steps without weakening invariants. The target is not automation at any cost; it is a system whose state allows safe machine or human action after interruption.

A complete recovery review conversation

Use the following review sequence when a team proposes to make an agent run durable. It keeps the meeting centered on semantics rather than product demos.

Begin with one effectful task and ask the designer to state the business intent in one sentence. Which fields make this request the same as an earlier request? Which change would make it a new request? If the answer is “the prompt,” intent is not yet stable. Convert it into typed fields and a versioned canonical form.

Next ask where the first durable record is written. The designer should point to a prepared effect with key and proposal hash before dispatch. Ask what happens if that write succeeds and the process dies. Then ask what happens if dispatch succeeds and the following write dies. The answer should move from safe resume to ambiguity and reconciliation without changing identifiers.

Ask who owns the run at every moment. Look for lease epoch, conditional state version, expiry, and stale-worker fencing. Ask whether the queue, runtime, scheduler, and application can each retry. If more than one layer can initiate the same effect, require one coordination rule and an end-to-end budget.

Ask which system can prove the remote effect. A local log of send() is not enough. The designer should name a read, its identity precision, consistency, permission, and negative-result meaning. Then remove that read in a fault scenario. The system should stop or escalate, not quietly switch to retry.

Ask how cancellation interleaves with dispatch. Put cancel one millisecond before the conditional prepare, one millisecond after prepare, and after possible commit. A correct design yields different evidence states while consistently blocking unauthorized new work. Ask how late approval or callback is handled after cancellation and lease expiry.

Ask whether compensation is permitted, who authorizes it, and what remains afterward. Demand a second effect record. If the design edits the first record to ROLLED_BACK, inspect whether the external world and audit history truly support that statement. Usually they do not.

Ask how a seven-day-old checkpoint resumes under new code. The answer should identify schema, behavior, adapter, proposal hash, authority, and key-retention compatibility. Parsing old JSON is only the first check. Remove compatibility and require a safe disposition.

Finally ask what the user and operator see when recovery cannot know. If the interface has only success and failure, the design is not ready. It needs a state such as effect unknown, an accountable next owner, and a communication that does not promise absence or completion.

Review evidence, not confidence

For each answer, request an executable trace or durable artifact. A diagram can explain the design but does not prove it. A unit test of the key function does not prove the service honors the key. A runtime restart demo does not prove ambiguous external effects. Match evidence to claim.

Structural evidence includes schema validation, required fields, and legal transitions. Deterministic behavior evidence includes crash fixtures and ledger assertions. Integration evidence includes an actual service sandbox demonstrating duplicate response and read-by-key behavior. Operational evidence includes alert routing and a runbook rehearsal. Production evidence would require observed behavior under declared real conditions and remains outside this chapter’s synthetic companion.

When evidence is absent, narrow the conclusion. “The fixture prevents duplicate ledger entries” is supportable from the companion. “The production reservation service will never duplicate” is not. The review record should preserve both the observed result and its limitation.

Communicate unresolved effects

Users need actionable language. Instead of “booking failed,” say that the confirmation response was lost, the system is checking whether the reservation occurred, and another attempt is temporarily blocked to avoid duplication. If the deadline expires, say whether support or an operator owns the check and what the user should not do.

Operators need more detail: run and effect identifiers, exact intended parameters, last known state, reconciliation attempts, cancellation, authority, and next safe action. Service owners need request timing, key, adapter version, transport evidence, and observed inconsistency. Tailor altitude without changing facts.

Do not expose secrets, raw private context, or speculative model narratives. Evidence references and typed parameters are usually sufficient. Communication is part of recovery because user retries outside the system can create duplicates. Clear status reduces that risk without pretending uncertainty has vanished.

Exit criteria

Approve the durability design only when every effect has semantic identity, every ambiguous outcome has a reconciliation or stop path, every run has one current owner, every retry is bounded and coordinated, every checkpoint has compatibility semantics, every cancellation position has a disposition, every compensation remains visible, and every trace ends in a truthful state.

The artifact should also state what is unsupported: external exactly once, universal rollback, guaranteed service availability, perpetual key retention, automatic migration, and authority renewal. Those exclusions are signs of an engineered boundary, not defects to hide.

Review and handoff

Complete eight recovery traces. Score classification 5, one owner 4, effect correctness 6, final disposition 5. Automatic failure follows an exactly-once claim, blind ambiguous retry, compensation-as-rollback, checkpoint without version/owner, or two retry owners.

AR-08 v0.1.0 ends with pause/resume compatibility and the AR-05 authority hook. Chapter 11 turns that durable wait into an executable human checkpoint.