Put AI Inside a Bounded Workflow

AI should enter the workflow only after the workflow can say no to it.

If a deployment cannot define the task, error consequence, permitted data/tools, deterministic rules, human authority, evaluation evidence, fallback, latency/cost budget, version, and stop condition, it is not ready to add uncertain model behavior.

The question is not “Where can we use AI?” It is:

Which bounded decision or evidence task benefits from uncertainty/flexibility enough to justify its measurement and control burden?

NIST AI RMF 1.0 and its Generative AI Profile support lifecycle/use-case risk framing. Version matters: the framework was under active revision at the research cutoff. Current security/risk taxonomies can prompt discovery but do not provide complete coverage or certification. [CLM-057] [CLM-064] [CLM-065]

Climb the mechanism ladder only as needed

Consider five mechanisms:

1. deterministic rule/calculation;
2. structured search/retrieval;
3. model suggestion/classification/summarization;
4. constrained model-selected tool/action under policy;
5. bounded agentic planning/execution.

Choose the lowest mechanism that satisfies the outcome. First-party agent engineering guidance similarly recommends simple composable workflows when sufficient and emphasizes ground truth, checkpoints, and stopping conditions for agents. It is vendor practice, not a universal standard. [CLM-058] [CLM-060]

For Orchid:

• equipment eligibility and safety rules remain deterministic;
• manual/evidence retrieval is structured and provenance-visible;
• a model may summarize/rank evidence or propose next inspection steps;
• safety-relevant action requires qualified approval;
• automatic parts ordering and open-ended agents remain out of scope.

Decompose the task

“Assist technicians” is not evaluable. Decompose:

• identify/confirm equipment: deterministic/data reconciliation plus human confirmation;
• retrieve approved evidence: search/retrieval with access/provenance;
• summarize evidence: bounded model task;
• rank possible next inspection steps: bounded model task plus deterministic eligibility;
• decide safety-relevant action: qualified human/domain decision;
• check inventory: deterministic interface read;
• order part: external action, deferred;
• record decision: deterministic workflow/audit.

For each subtask record input, output, success criteria, errors, consequence, fallback, owner, data/tools, latency/cost, and evaluation.

Write an AI decision record

Include:

• workflow task and outcome/guardrail served;
• non-AI alternatives and why insufficient/sufficient;
• input data/provenance/sensitivity;
• model purpose and prohibited use;
• output meaning/non-meaning;
• tools/actions and permissions;
• deterministic pre/post controls;
• human role, qualification, evidence, authority, workload;
• error taxonomy and consequence;
• evaluation data/graders/segments/threshold owners;
• latency/cost/capacity budget;
• fallback/abstain/escalate/stop;
• model/prompt/retrieval/tool/policy versions;
• monitoring/change/release/rollback;
• residual limits and formal owners.

The model is one component in a broader production system of data, interfaces, controls, evaluation, monitoring, operations, and users. [CLM-061]

Build an error taxonomy from workflow consequence

For Orchid:

• wrong evidence cited;
• relevant evidence omitted;
• unsupported statement;
• wrong equipment context;
• outdated manual/source;
• unsafe/prohibited action suggested;
• required escalation omitted;
• overconfident language;
• refusal/abstention when help was possible;
• excessive latency/cost;
• tool request outside permission;
• privacy/sensitive-data disclosure;
• inconsistent output across equivalent cases.

Record consequence, detectability, control, evaluation, threshold owner, and release action.

Design evaluation as a system

Useful distinctions include task, trial, dataset/case, grader, transcript/trace, outcome, harness, suite, capability evaluation, and regression evaluation. [CLM-059]

Task and case

Define the exact workflow behavior and case context. Include common, boundary, exception, high-consequence, adversarial/misuse, denied, and fallback cases.

Data provenance

Record source, permission, construction, version, split, segment, leakage risk, correction, and representativeness. The companion uses synthetic cases only.

Grader

Use deterministic checks where possible: required evidence ID, prohibited action, policy state, schema, latency/cost. Human/domain graders handle contextual quality under rubric/qualification. Model graders may scale judgment but require calibration and can share failure modes.

Harness and version

Freeze model interface, model/version, parameters, prompt/system instructions, retrieval/index, tools, policy, environment, and grader versions. A score without system version is weak evidence.

Segments

Report by workflow consequence, equipment family, evidence quality, language/environment, user cohort, or other legitimate critical segment. Do not rely on one average.

Decision

Precommit pass/guardrail/stop/waiver authority. Evaluation is useful only when it changes release or scope.

The companion evaluation preserves common and safety-critical segments, demonstrating how an aggregate could conceal a critical failure.

Separate capability from regression

Capability evaluation asks whether the approach can meet the task under defined conditions. Regression asks whether a changed system preserves accepted behavior.

Early exploratory cases should not become the only release suite. Curate validated cases, maintain held-out/rotating/adversarial sets, and guard against tuning directly to every known example.

Production traces can reveal new cases, but they require permission, minimization, de-identification/handling, provenance, review, and separation from evaluation leakage.

Build the evaluation dataset from workflow states

Do not begin by collecting “interesting prompts.” Derive cases from OA-02 decisions, states, exceptions, and OA-03 guardrails.

For Orchid include:

• confirmed equipment with complete current evidence;
• multiple/absent equipment candidate;
• stale/missing/conflicting manual or inventory evidence;
• normal versus safety-relevant decision;
• required escalation;
• low-connectivity/timeout/fallback;
• prohibited action;
• adversarial/untrusted manual content;
• irrelevant or sensitive data request;
• ambiguous user instruction;
• no answer supported by evidence;
• equivalent cases with formatting/order changes;
• high-consequence equipment subfamily;
• reviewer disagreement.

Each case record contains:

• case ID/version/segment;
• fictional/synthetic or approved-source provenance;
• workflow state/task and user goal;
• input evidence/permissions;
• expected acceptable behaviors, prohibited behaviors, and abstention conditions;
• grader/rubric and qualified owner;
• consequence/severity/detectability;
• control expected to catch the error;
• capability/regression/monitoring use;
• known limitation and change history.

Avoid one gold string where several answers are acceptable. Grade properties: cites required evidence, does not invent missing facts, stays within task, triggers required escalation, avoids prohibited action, uses calibrated uncertainty, and meets format/budget.

Prevent leakage

Separate development examples from held-out acceptance. Track whether production traces were seen during tuning. Rotate or refresh cases when the system memorizes a small suite. Keep high-consequence regression cases stable while adding new variants.

Leakage does not make a case useless; it changes the claim. A memorized case can be a regression check and weak capability evidence.

Calibrate graders

Deterministic graders are ideal for exact invariants:

• required evidence ID present;
• prohibited action absent;
• JSON/schema valid;
• policy state/approval correct;
• latency/cost within test budget;
• tool call/argument allowed;
• no cross-region reference.

Human/domain graders evaluate contextual usefulness, completeness, uncertainty, and professional judgment. Provide rubric, examples, qualification, blinding/order where feasible, disagreement handling, and calibration cases.

Model graders can assist with scale. Compare their decisions against qualified human labels across critical segments. Investigate shared bias, prompt sensitivity, position/order effects, verbosity preference, and inability to verify external evidence.

Report grader uncertainty and disagreement. Do not hide it inside the system score.

Evaluate the whole decision path

A good model output can fail inside a bad system. Evaluate:

1. correct equipment/context binding;
2. permitted evidence retrieval;
3. evidence quality/provenance/freshness;
4. model input construction;
5. output behavior;
6. deterministic policy;
7. human review/decision;
8. tool/action execution where allowed;
9. audit/state/user message;
10. outcome/fallback/support behavior.

This prevents model-only evaluation from masking retrieval, identity, UI, policy, or approval failures.

For example, a summary may accurately reflect the wrong equipment manual. Model scoring alone can pass it. End-to-end evaluation must fail at equipment binding/provenance.

Threat-model model input and tools

Treat customer documents, retrieved content, user text, and tool results as untrusted inputs according to context.

Ask:

• Can content instruct the model to ignore policy or expose data?
• Can one tenant/region influence another through shared index/cache/context?
• Can output cause unauthorized tool selection or arguments?
• Can sensitive data appear in prompt, output, logs, traces, graders, or vendor retention?
• Can repeated requests exhaust cost/capacity or probe policy?
• Can a user rely on fabricated citations or hide evidence?
• Can tool output be forged/stale/misbound?
• Can an attacker manipulate evaluation data or production feedback?

Mitigations can include data/tenant isolation, input labeling, retrieval allowlists, output schema, deterministic authorization/policy, tool allowlists, argument validation, least workload identity, evidence verification, rate/budget limits, redaction/minimization, audit, monitoring, and human approval.

OWASP’s current community guide can prompt categories, but it is not a complete system threat model or certification. [CLM-064]

Design abstention and fallback

Abstention is a product state, not an error string.

Define when the system must abstain:

• equipment/context unconfirmed;
• evidence absent/stale/conflicting;
• task outside approved scope;
• policy requires qualified decision;
• output cannot meet confidence/evidence rule;
• model/tool unavailable or budget exceeded;
• authorization/region/tenant denied;
• detected attack/unsafe input;
• evaluation/monitoring stop state.

For each abstention, tell the user what is known, what is missing, what action is permitted, and how to escalate. Do not expose sensitive policy internals unnecessarily.

Fallback can be evidence-only search, deterministic checklist, manual workflow, qualified escalation, deferred queue, or stop. The fallback itself needs access, usability, support, and recovery evidence.

Bound agent loops

If a later deployment introduces an agent, define:

• goal/task boundary and termination state;
• maximum steps/time/cost/tool calls;
• tool allowlist and per-call authorization;
• state/memory/data retention;
• checkpoints and human approval;
• ground truth/status verification;
• duplicate/retry/idempotency/reconciliation;
• loop/stall detection;
• interruption/resume/cancel;
• audit/trace and privacy;
• compensation/recovery;
• eval suite for multi-step behavior.

An agent should not continue because a model says it is not done. Termination and budget enforcement live outside model output.

Orchid’s first scope does not require an agent. Keeping this explicit prevents “agentic” capability from entering through a library default.

Govern model and prompt change

Record a system version bundle:

• model/provider/version;
• prompt/system instructions;
• retrieval/index/corpus versions;
• tool definitions/permissions;
• deterministic policy/control version;
• dataset/grader/harness versions;
• application/config/release version.

A model alias can change behavior without code change. A retrieval corpus update can change answers without model change. A prompt fix can interact with tool schema. Evaluate and release the bundle.

Classify changes by risk and required evidence. Run targeted plus regression suites. Canary with quality and workflow signals. Preserve rollback to an actually available/compatible bundle. If provider version retirement prevents rollback, plan a validated alternative before migration.

Decide when not to use AI

Reject AI when:

• deterministic rules solve the task with less uncertainty;
• required data cannot be used legitimately;
• errors cannot be detected/contained relative to consequence;
• no qualified owner can define acceptable behavior;
• no representative evaluation can be built;
• latency/cost/availability break the workflow;
• tool permission would be too broad;
• fallback/support/recovery is unavailable;
• expected value does not justify control/operation burden.

The decision can be “retrieval only,” “human workflow with better evidence,” or “stop.” That is an engineering outcome, not a failure to innovate.

Build a control ladder

Controls should match consequence:

• remove AI from deterministic task;
• constrain input/data/retrieval scope;
• constrain output format;
• block prohibited tool/action;
• require evidence/provenance;
• validate against deterministic policy;
• abstain/fallback on insufficient evidence;
• require qualified human approval;
• limit cohort/rate/action;
• monitor quality/incident/support;
• disable/rollback/stop.

The companion applyControlLadder blocks ambiguous equipment, prohibited suggestions, and missing qualified approval before presenting the suggestion.

Human review is a designed control

Human review does not automatically create safety, attentiveness, scale, qualification, or accountability. [CLM-063]

Define:

• which cases require review;
• reviewer qualification/authority;
• evidence and uncertainty shown;
• time and workload;
• independent decision versus rubber stamp;
• approve/reject/edit/escalate actions;
• disagreement/audit;
• reviewer error evaluation;
• coverage when unavailable;
• consequence of automation bias.

For Orchid, the model never defines the safety rule. The qualified approver receives evidence, rule state, suggestion, limitations, and identity/audit context.

Bound tools and actions

If a model can call tools, define:

• allowlisted tool/operation;
• read/write/irreversibility;
• argument schema and semantic validation;
• identity/tenant/region/data scope;
• deterministic policy before execution;
• user/human approval;
• intent/idempotency/reconciliation;
• result/error disclosure;
• rate/cost/time limits;
• audit/monitoring/recovery;
• stop/kill mechanism.

Do not give a model a broad credential and rely on instructions for access control. Permissions and policy live outside the model.

Budget latency, cost, and capacity

Define budgets per eligible case and segment. Include retrieval, model calls, tool calls, retries, review, and fallback.

The companion runWithBudget returns visible fallback on cost or timeout. It uses synthetic “cost units,” not vendor prices. Real pricing and performance are volatile and require current evidence.

If the model times out, the user should see an approved evidence-only/manual path, not a silent spinner or repeated expensive calls.

Use deployment-like simulation carefully

Static evaluation can miss multi-turn, tool, timing, environment, or behavioral context. A first-party research article reports that deployment-like simulation can expose some missed behavior while retaining production-tail limitations. [CLM-062]

Simulate:

• changing/contradictory evidence;
• tool unavailable/denied/slow;
• low connectivity;
• prompt injection/untrusted content;
• user correction/pressure;
• long context/state;
• reviewer unavailable;
• model/version change;
• cost/latency limits;
• repeated/adversarial attempts.

Simulation does not authorize live exposure. It is one evidence layer.

Read an evaluation report without being fooled

Suppose a synthetic suite contains 900 common cases and 100 safety-critical cases. The system passes 880 common and 40 safety-critical cases. Aggregate pass rate is 92 percent. The safety-critical pass rate is 40 percent.

The aggregate is mathematically correct and operationally misleading if the critical segment governs exposure. The report should show:

• case counts/provenance by segment;
• pass/fail/error/abstain categories;
• consequence-weighted review without hiding raw counts;
• confidence/uncertainty and grader disagreement;
• failure taxonomy distribution;
• controls that caught or missed each failure;
• comparison to previous system bundle;
• known coverage gaps;
• release disposition and owner.

The book does not claim these synthetic numbers for Orchid. They illustrate why the companion test preserves segment failure.

Abstention can be correct

If evidence is missing, abstention may pass while an unsupported confident answer fails. But excessive abstention can make the workflow useless. Measure coverage and appropriate abstention separately from correctness on answered cases.

Control performance matters

Record whether the raw suggestion was unsafe and whether deterministic policy/approval blocked it. A safe final system can contain a weak model under strong controls for a limited task, but control burden, user experience, and residual failure remain visible. Do not publish only the post-control score if the model change affects workload/cost.

Connect AI errors to production decisions

Use a matrix:

Error class	Consequence	Required control/evidence	Release action
wrong equipment context	wrong evidence/action	match confirmation; provenance; end-to-end tests	block exposure
unsupported statement	mistrust/wrong decision	evidence-required output; grader; UAT	repair or restrict
unsafe/prohibited suggestion	harm	deterministic policy; critical suite; qualified approval	zero accepted violations under owner rule; block
appropriate abstention	delay/manual work	fallback and support capacity	monitor burden
excessive abstention	low value/adoption	segmented coverage eval	reduce scope/change
latency/cost breach	abandonment/capacity	budgets/fallback/cohort	hold ramp
tool permission violation	unauthorized effect	external authorization/allowlist/audit	block/incident

Threshold language belongs to the owner and evaluation plan. “Zero accepted violations” can apply to observed test/release criteria but is not proof that no production violation can occur.

Design the review interface as part of the control

The reviewer should see:

• task/request and confirmed equipment context;
• source/evidence references, versions, freshness, and gaps;
• model suggestion separated from verified facts;
• deterministic policy result and prohibited state;
• uncertainty/abstention reason;
• approve/reject/edit/escalate choices appropriate to authority;
• consequence and next workflow state;
• audit/privacy notice where required.

Avoid interfaces that place the suggestion prominently and evidence behind extra navigation, or default to approve under time pressure. Measure review time, evidence inspection, disagreement, override, and support burden.

If the qualified reviewer cannot understand why evidence was selected, the system may shift verification work rather than reduce it.

Run a pre-mortem

Before rollout, ask: “The bounded AI path caused a serious incident or failed adoption. What happened?”

Candidate conditions:

• wrong equipment binding bypassed evaluation;
• retrieved manual was obsolete;
• untrusted content manipulated output;
• reviewer approved by habit under workload;
• low-connectivity users saw stale partial evidence;
• model alias changed behavior;
• fallback was unavailable;
• traces logged sensitive data;
• tool permission was broader than intended;
• aggregate monitoring hid the critical segment;
• support could not identify the system bundle.

For each, identify prevention/detection/containment/recovery/owner and add missing cases or controls. A pre-mortem is not evidence of control effectiveness; it improves the test plan.

Keep current tools volatile

Vendor evaluation APIs and agent frameworks evolve. The OpenAI Evals platform documented at the research cutoff was scheduled for late-2026 deprecation. Teach portable evaluation artifacts and interfaces rather than durable UI/API promises. [CLM-066]

The companion’s deterministic model interface can be replaced by a provider adapter later; tests/control behavior remain stable.

Orchid OA-05 AI section

Decision. Use AI conditionally for bounded summary/ranking; deterministic retrieval/policy/approval remain separate; non-AI fallback exists.

Inputs. Confirmed equipment, approved evidence with provenance, selected workflow context; no unrestricted ticket archive.

Outputs. Suggestion with evidence references and uncertainty; not authority or fact by itself.

Errors/evals. Workflow consequence taxonomy, critical segments, synthetic suite, deterministic/domain graders, capability/regression separation.

Controls. Match confirmation, data/tool scope, evidence requirement, deterministic eligibility, abstention/fallback, qualified approval, cohort/budget, monitoring/disable.

Versions. Model interface, model/test double, prompt, retrieval/index, tool, policy, grader, dataset.

Companion. Deterministic model, budget fallback, control ladder, segmented evaluator; 4 additional tests.

Conduct the bounded-AI design exercise

Start with the stakeholder request use AI to recommend the fix. Decompose the workflow into equipment confirmation, approved evidence retrieval, inventory state, candidate generation, eligibility policy, qualified review, and recorded action.

For each step decide deterministic logic, search/retrieval, model behavior, human judgment, or rejection. Justify why AI is needed, what non-AI baseline exists, which errors matter, and which action remains prohibited.

Build a small versioned evaluation set covering clear, ambiguous, missing, conflicting, stale, adversarial, low-connectivity, and critical equipment states. Define graders and segment dispositions before results. Inject an aggregate pass with one critical prohibited suggestion and require a blocking decision. Calibrate a model grader against qualified human labels and preserve disagreement.

Then run the companion under missing evidence, exhausted budget, policy denial, expired qualification, and feature-disabled conditions. Confirm abstention/fallback is visible, dependencies/tools remain scoped, and the model never becomes the authorization or approval authority.

Pass when a reviewer can see the task, evidence, system bundle, errors, controls, owners, residual limits, and stop/disable path. Fail if the proposal relies on prompt wording, an aggregate benchmark, or human in the loop without qualification, evidence, workload, authority, and interface design.

AI failure modes

AI-first decomposition

Repair: define workflow task and compare deterministic/retrieval alternatives.

Aggregate benchmark

Repair: task cases, consequence segments, graders, system version, release decision.

Prompt-only control

Repair: enforce permissions, policy, approval, budgets, and audit outside model.

Human-loop theater

Repair: qualification, evidence, authority, workload, evaluation, fallback.

Vendor interface as architecture

Repair: stable provider-neutral model/tool/eval interfaces and versioned adapter.

Production traces without governance

Repair: purpose, permission, minimization, provenance, handling, leakage control.

Silent fallback

Repair: explicit state, user path, signal, support, decision.

The Chapter 9 gate

Before governance/control evidence is finalized, OA-05 must contain:

• AI decision record and non-AI alternative;
• task decomposition and prohibited uses;
• data/tool/identity/action boundary;
• error/consequence/detectability taxonomy;
• evaluation cases/provenance/graders/segments/versions;
• deterministic/human/system control ladder;
• latency/cost/capacity/fallback;
• monitoring/change/release/disable;
• authority and residual limitations;
• passing deterministic companion tests.

The gate passes when the system can reject AI, abstain safely, explain its evidence boundary, and stop under precommitted conditions.

Chapter 10, Build Security and Governance Into Delivery, connects all system/AI risks to control objectives, implementation, verification evidence, formal approval, and escalation without turning framework names into compliance claims.