Applied AI Engineering
Applied AI Engineering
From Model Capability to Dependable Product Behavior
Komal Nakrani
First edition/Applied AI Engineering · Volume 1
Applied AI Engineering
From Model Capability to Dependable Product Behavior
A practical field book for turning uncertain model capability into useful, measurable, reliable, and governable product behavior through a continuous evidence loop.
- Edition
- First edition
- Version
- 1.0.0
- Published
Table of contents
01Behavior, Not MagicEstablish product behavior as the unit of accountability and build a responsibility charter around a recursive evidence loop.45 min02Start With the User TaskTurn a request for AI into a falsifiable task, consequence, baseline, value hypothesis, evidence log, and valid stop path.55 min03Write the Behavior ContractSpecify required, uncertain, abstaining, escalating, degraded, and prohibited behavior with explicit evidence and authority.60 min04Choose the Simplest Adequate MechanismCompare deterministic, retrieval, predictive, multimodal, generative, agentic, and hybrid mechanisms against the behavior contract and select the least complex adequate path.65 min05Make Task Data RepresentativeBuild a permitted, versioned task-data substrate whose population, provenance, labels, segments, leakage, missingness, and limitations can support or reduce the behavior claim.70 min06Engineer Context and RetrievalDesign an authorized context path that separates query transformation, candidate generation, ranking, permission, freshness, assembly, provenance, and empty or conflicting evidence behavior.75 min07Draw the Combined System BoundarySeparate learned behavior, deterministic policy, application state, trust, failure, ownership, and authority so every consequential claim crosses validation before use.80 min08Build an Inspectable Vertical SliceImplement one production-shaped behavior path that succeeds, abstains, fails safely, exposes evidence, and runs locally without provider secrets.85 min09Turn Consequences Into Evaluation CasesBuild a versioned evaluation set whose cases, segments, provenance, ambiguity, coverage, splits, leakage controls, and limits map to the behavior contract.80 min10Name the Errors That MatterClassify errors by consequence, segment, severity, detectability, reversibility, and control, then choose thresholds and release dispositions without hiding critical failures behind an average.80 min11Combine Machine, Human, and Domain JudgmentAssign each system claim to deterministic checks, references, calibrated graders, trained raters, specialists, or accountable authorities without laundering agreement into truth.80 min12Run Experiments That Change DecisionsPreregister reproducible comparisons that isolate a change, preserve paired cases, segments, guardrails, confounds, and negative evidence, and end in an explicit disposition.85 min13Design for Probabilistic FailureModel failure by layer, consequence, propagation, and state, then contain it with bounded retries, idempotency, circuits, degradation, reconciliation, and explicit recovery evidence.85 min14Budget Latency, Capacity, and CostTurn user time, tail latency, concurrency, capacity, retries, cache semantics, quality, and total cost into a segment-preserving product envelope.80 min15Observe Behavior Without Betraying UsersBuild decision-bearing signals, redacted traces, feedback paths, retention controls, and diagnostic views that reveal product behavior without routine raw-content surveillance.85 min16Implement Controls and Preserve AuthorityTranslate threats and harms into implemented, tested, monitored controls with explicit residual limits, qualified review, and named authority.90 min17Release to Learn SafelyMatch exposure to the next justified uncertainty with versioned readiness evidence, restricted cohorts, explicit stop triggers, and rehearsed rollback authority.85 min18Diagnose Real Use and IncidentsContain consequence, reconstruct layered evidence, test and disconfirm hypotheses, protect sensitive traces, verify recovery, and change durable artifacts.90 min19Change Models Without Losing the ProductInventory, replay, compare, release, and roll back model or provider change against an unchanged product behavior contract.90 min20Earn Reuse From Repeated EvidenceSeparate reusable mechanisms from local evidence and authority, validate stable seams across consumers, and reject abstractions that have not earned their blast radius.80 min21Lead Applied AI DecisionsDirect attention across unlike systems, communicate at the right altitude, preserve evidence and authority, and grow by improving the decisions and people around the work.80 min