NEWProduction Web Themes & Turnkey ArchitecturesGet Lifetime Pass ($199) →
KNKomal Nakrani
Get All Access
ThemesDocsAll-Access PassGet All Access ($199)
Book overview
08/Forward Deployed Engineering

Fit the Customer Environment

Translate the logical design into explicit deployment topology, identity, secrets, configuration, tenancy, regional, network, capacity, and environment responsibilities.

A logical design does not run in a customer environment.

Code runs in a region, cluster, machine, process, network path, tenant, and release configuration. It acts through identities whose permissions expire or drift. It depends on DNS, certificates, secrets, rate limits, capacity, clocks, storage, egress, and customer-operated systems. A staging environment may share the API shape and differ in the behavior that matters.

The FDE’s job is to translate the logical boundary into deployable responsibility without hiding customer work behind a box labeled “infrastructure.”

Provider architecture guidance offers question sets for reliability, security, performance, operations, cost, and capacity. NIST zero trust guidance rejects implicit trust based only on network location. Use these principles in context; do not call a design safe because it is inside a VPC or follows a provider diagram. [CLM-050] [CLM-051] [CLM-055]

The environment-difference matrix and topology, identity, and configuration records in this chapter are original synthesis tools rather than formal external standards. [CLM-056]

Draw the deployment topology

Start from Chapter 6 containers and place each into its actual or proposed runtime.

For Orchid, the topology must show:

  • technician device/browser and low-connectivity path;
  • customer ingress/gateway or approved access path;
  • Orchid Assist web/API/workflow runtime;
  • adapters and outbound paths to ticketing, registry, ERP, manuals, identity, audit, telemetry;
  • state/cache/index stores;
  • Region West and tenant/data-residency boundaries;
  • control/configuration/release plane;
  • operational signals/support/recovery access;
  • external network/DNS/certificate dependencies;
  • approved egress and denied paths.
Topology from technician device through network and ingress to customer runtime, workflow services, adapters, stores, customer systems, control plane, telemetry, and support with region and tenant boundaries.
F08.1 - Region West deployment topology. Connectivity, tenancy, regional, capacity, and operating assumptions are visible before launch.

Do not imply a runtime choice before it is decided. Use capability labels first; record selected provider/product as a versioned alternative.

Create an environment matrix

Perfect parity is often impossible. The professional task is to expose differences that can change behavior.

For local, test, staging, and production, record:

  • runtime/topology/scale;
  • identity/token/authorization source;
  • tenant/region/data category;
  • dependencies: stub, sandbox, or live;
  • data distribution/volume/quality;
  • network, latency, rate, timeout, and failure behavior;
  • secrets/certificates/DNS;
  • configuration/feature/cohort controls;
  • telemetry/audit/support access;
  • release/migration/recovery mechanism;
  • policy/control/approval evidence;
  • known difference, consequence, test/monitor/control, and owner.

Environment parity is incomplete when consequential differences remain. Those differences need explicit tests, controls, monitoring, or accepted gaps. [CLM-053]

Example:

Difference Consequence Compensating evidence
local adapter uses deterministic stub real timeout/rate/auth semantics unproven contract tests plus customer sandbox/live bounded verification
staging uses test identity production role/tenant/region mapping unproven production authorization dry run and denied-case evidence
staging dataset is small/clean capacity and data-quality tails hidden synthetic scale/failure fixtures plus production cohort signals
recovery uses local script customer permissions/control-plane path unproven customer-environment recovery rehearsal

The companion compareEnvironments function makes differences explicit. It does not judge them safe; the record adds consequence/owner/evidence.

Design human and workload identity separately

Human identity answers who the user is and which assigned context/role applies. Workload identity answers which software process may call which resource operation. Do not pass unrestricted human authority through the application.

For a technician request:

  1. authenticate the human under customer-approved identity;
  2. resolve tenant, region, assignment, and role context;
  3. authorize the user action on the Orchid resource;
  4. let the bounded workload identity call selected dependencies;
  5. enforce downstream resource/operation/field constraints;
  6. record correlation, actor, workload, policy decision, and result;
  7. deny safely when context is missing or mismatched.

For safety approval, require a separately authorized qualified identity. The FDE does not define qualification.

Identity flow from technician and approver sessions through gateway policy, bounded workload identity, downstream authorization, secret retrieval, audit, expiry and revocation, with denied region, tenant, and role paths.
F08.2 - Identity is bounded at every hop. Human context informs policy but never becomes unrestricted service authority.

The companion authorization model checks principal, resource, operation, tenant, region, permission, and expiry. It is a teaching model, not a production identity provider.

Write an identity and access record

For each principal-to-resource relationship, record:

  • principal type/identity source;
  • authentication method and assurance required by the owner;
  • tenant, region, assignment, device/session, or workload context;
  • resource and operations;
  • permission source/policy version;
  • grant/approval/provisioning owner;
  • token/credential audience, scope, lifetime, rotation/revocation;
  • service-to-service delegation or impersonation behavior;
  • sensitive fields/data purpose;
  • audit/correlation;
  • denied/expired/mismatch behavior;
  • support and break-glass path;
  • test evidence and review date.

Example:

IAM-OA-APPROVER

  • Principal: customer human identity mapped to qualified service-approver role for Region West.
  • Resource: Orchid safety-approval record for assigned/eligible ticket.
  • Operations: view supporting evidence; approve/reject/escalate; not edit safety policy.
  • Context: tenant Orchid, Region West, active assignment/qualification.
  • Lifetime: bounded customer session; role/qualification changes revoke future authority.
  • Audit: human identity, role/policy version, ticket/evidence version, decision, time, correlation.
  • Denial: workflow remains awaiting-qualified-approval; no fallback that bypasses approval.

The record separates identity proof, qualification, assignment, and permission. A valid login is not enough.

Confused deputy risk

Orchid Assist has workload access to customer systems. It must not use that access for a user request the user is not permitted to initiate.

Bind user intent/context to the service operation. Authorize at the application resource and again at downstream resource boundaries where required. Minimize fields and operations. Audit both human and workload identities. Do not accept arbitrary resource identifiers simply because the workload credential can access them.

Map network flows as policies

For every ingress/egress relationship, record:

  • source/destination runtime and owner;
  • protocol/port/name resolution;
  • identity/authentication and encryption;
  • operation/data category/region/tenant;
  • direction and initiation;
  • proxy/gateway/firewall/egress control;
  • timeout/rate/connection assumptions;
  • audit/telemetry;
  • failure/fallback;
  • change/review owner.

Avoid broad “allow customer network” rules. The architecture should be able to answer why Orchid workflow needs to reach manual/registry/inventory endpoints and why it cannot reach unrelated systems.

DNS and certificates are dependencies. Expired certificates, failed rotation, wrong name, or split DNS can mimic application failure. Put them in the signal/runbook/ownership plan.

Choose placement from constraints

Common placement patterns include:

  • customer-managed runtime;
  • vendor-managed regional runtime;
  • split plane with customer-side connector;
  • on-premise or edge component;
  • approved managed service inside customer account/subscription;
  • local/offline component for selected behavior.

Compare each on:

  • data/residency and customer control;
  • identity/network integration;
  • update/release velocity;
  • observability/support access;
  • capacity/reliability/recovery;
  • secrets/key ownership;
  • customer skills/operations;
  • cost and exit/migration;
  • product reuse versus customer-specific coupling.

There is no universal best placement. A customer-managed runtime may satisfy control and create upgrade/support burden. A vendor-managed runtime may improve consistency and violate a data/network constraint. A connector can reduce inbound access and become a difficult stateful dependency.

Record the decision and revisit trigger rather than presenting the provider name as architecture.

Promote configuration with evidence

Configuration should move through an evidence path:

  1. schema/semantic validation;
  2. code review and owner approval for material changes;
  3. versioned immutable artifact or record;
  4. environment-specific binding without secrets;
  5. dry run or policy evaluation where possible;
  6. bounded rollout;
  7. observed runtime/health verification;
  8. rollback/forward and reconciliation;
  9. audit and change history.

Configuration examples include region/tenant maps, feature cohorts, timeout/retry, manual/index version, policy/model selection, data freshness thresholds, and endpoints. A configuration change can have code-level consequence and must receive proportionate tests/review.

The environment record contains reference IDs to secrets, not secret values. The companion’s rejection is intentionally obvious; real systems need secret scanning, access control, and runtime mechanisms beyond it.

Treat clocks and time zones as environment dependencies

Chapter 7 distinguished event times. The environment must provide usable clock behavior.

Record:

  • authoritative time source/synchronization expectation;
  • UTC storage and local display rules;
  • tolerated skew for tokens, ordering, and correlation;
  • daylight-saving/local schedule effects;
  • behavior when device clocks are wrong/offline;
  • timestamp provenance (client versus server/source);
  • monitoring and incident diagnosis.

An expired-token error caused by clock skew, or a late event sorted by device time, can create authorization and workflow failures. Do not assume timestamps are comparable because they share a format.

Make capacity assumptions falsifiable

Create a capacity envelope:

  • eligible cases/time and peak pattern;
  • concurrent users/workflows;
  • dependency calls per case and fan-out;
  • payload/index/model context size;
  • connection/thread/process limits;
  • rate quotas and shared tenancy;
  • queue/backlog growth and drain;
  • storage/log/audit volume and retention;
  • timeout/retry amplification;
  • cost per case and budget owner;
  • load/failure test and production signal.

Use ranges tied to evidence. A pilot cohort still needs protection against loops and bursts. A small user count can generate large model/retrieval/dependency demand.

Define overload behavior: reject, shed optional work, queue, degrade to evidence-only/manual, limit cohort, or stop. Preserve high-consequence approval/audit behavior; do not degrade by skipping a control.

Review a denied case end to end

Use this synthetic scenario:

  1. An East-region technician has a valid customer session.
  2. They present a West-region ticket identifier, accidentally or maliciously.
  3. Gateway authenticates the identity but detects tenant/region/assignment mismatch.
  4. The request is denied before West data retrieval.
  5. User receives a safe message without disclosing West resource details.
  6. Audit records principal/context/resource type/policy/result/correlation, within privacy constraints.
  7. Support can distinguish legitimate assignment error from attempted misuse through authorized evidence.
  8. No workload call to registry/manual/inventory occurs.

The companion test proves a region mismatch is denied in the teaching policy. It does not prove production identity mapping, gateway enforcement, or downstream isolation. Those require customer-environment evidence.

Create the environment readiness packet

Before Chapter 9, package:

  • topology/flows and placement decision;
  • environment-difference matrix;
  • identity/access records and denied cases;
  • network/egress/DNS/certificate flows;
  • tenant/region/data/storage/telemetry boundaries;
  • secret/configuration lifecycle and promotion;
  • capacity/latency/connectivity envelope;
  • clock/time assumptions;
  • support/operating/recovery/break-glass access;
  • tests, simulations, owner reviews, and remaining production gaps.

The environment is “fit” only within the selected scope and evidence, never as a blanket property.

Apply OAuth guidance only where OAuth is used

RFC 9700 is the current OAuth 2.0 security best current practice at the research cutoff. A deployment using OAuth 2.0 should account for it within the actual flow/threat context. [CLM-052]

Do not add OAuth terminology to an architecture that uses another mechanism. If OAuth is selected, document:

  • client type and authorization flow;
  • redirect/issuer/audience validation;
  • token sender constraints where applicable;
  • scope/resource/tenant mapping;
  • refresh/rotation/revocation/expiry;
  • storage and browser/mobile considerations;
  • consent/delegation versus application permission;
  • replay/substitution/mix-up and logging risks;
  • failure/user/support behavior.

A standards reference does not replace a threat model or correct implementation.

Treat secrets as lifecycle responsibilities

Never place secrets in source, manuscript, fixtures, logs, screenshots, or static configuration records.

For each secret/credential/certificate:

  • purpose, owner, consumer, resource/operation scope;
  • creation/provisioning channel;
  • storage/injection and exposure boundary;
  • rotation/expiry/revocation;
  • audit/detection;
  • recovery/break-glass;
  • environment separation;
  • cleanup at handoff.

The companion environment validator rejects configuration records containing fields named as secret material. That is a narrow guard, not secret scanning or production security.

Separate configuration from secret and code

Version non-secret configuration, including:

  • environment/tenant/region;
  • endpoint/reference IDs;
  • timeout/retry limits;
  • feature/cohort policy;
  • schema/model/prompt/policy versions;
  • freshness/quality thresholds when approved;
  • signal/alert settings;
  • dependency modes and fallback;
  • release version.

Define validation, default, owner, change/review, rollout, and observed-state evidence. A default that broadens access or exposure is not safe merely because it is convenient.

Design tenant and regional isolation

Tenant and region must appear in identity, data keys, storage, cache/index, logs, queues, configuration, and operational access where relevant.

Ask:

  • Can one tenant/region reference another’s identifier?
  • Does cache/index key include the boundary?
  • Can support query across boundaries, under what authority/audit?
  • Where do backups/telemetry/error payloads reside?
  • What happens when dependency data belongs elsewhere?
  • Can configuration accidentally enable a cohort in another region?
  • How is deletion/retention/revocation scoped?

Private networking or IP allowlisting does not establish isolation or data-residency compliance. [CLM-055]

The companion tests reject a west resource requested by an east principal and reject west configuration with east residency.

Budget capacity and latency across the path

Start from workflow need and allocate a budget across:

  • device/network/ingress;
  • authentication/authorization;
  • workflow coordinator;
  • equipment/manual/inventory dependencies;
  • retrieval/model/policy/approval;
  • state/audit/telemetry;
  • retries/queues/fallback.

Record peak/cohort assumptions, concurrency, request size, rate limits, connection pools, queue age, cold start, cache behavior, and low-connectivity segments. The fictional case keeps numeric budgets unresolved.

If the total budget cannot be met, choices include narrower evidence, precomputation/cache with freshness/control cost, asynchronous flow, explicit wait, offline/fallback, reduced cohort, or rejection. Do not hide the miss behind an average.

Design for intermittent connectivity

For Orchid field sites:

  • distinguish no network, slow network, request timeout, and dependency failure;
  • preserve user intent and local workflow state where appropriate;
  • avoid duplicate actions after reconnect;
  • show evidence freshness and incomplete state;
  • queue only operations safe to delay/replay;
  • protect local data/secrets;
  • define conflict/reconciliation;
  • provide approved manual/escalation path;
  • segment signals by connectivity/site.

Offline support is not automatically required. The scope may exclude offline action while offering clear fallback. The decision traces to outcome/guardrail and cohort.

Plan break-glass without making it normal

Recovery access must be usable when the primary path is impaired, tightly controlled, time-bounded, logged, reviewed, and rehearsed.

A first-party Cloudflare postmortem reports that an access dependency complicated rollback and discusses break-glass access/known-good recovery identifiers. The transferable question is whether recovery shares the failed dependency, not that routine controls should be weakened. [CLM-054]

Record:

  • trigger and authorized requester/approver;
  • independent identity/path/dependencies;
  • least emergency permission;
  • duration/expiry;
  • immutable audit and communication;
  • credential custody/rotation;
  • rehearsal and after-action review;
  • return to normal access.

Define environment failure behavior

Walk through:

  • identity provider unavailable;
  • token/certificate expired;
  • DNS/certificate/network route failure;
  • region/tenant mismatch;
  • secret/config unavailable or stale;
  • configuration partially applied;
  • rate/capacity exceeded;
  • telemetry/control plane unavailable;
  • dependency accessible from staging but not production;
  • recovery operator lacks permission.

For each, identify user state, safe fallback, detection, containment, owner, and recovery.

Adapt the same design to three customer patterns

The durable contract should survive placement changes even when implementation differs.

Customer cloud account/subscription

Orchid components run under customer-managed cloud ownership. Customer networking/identity/operations teams control foundational access; product/FDE supplies artifacts and perhaps deployment automation.

Questions:

  • Who applies infrastructure/runtime changes?
  • How do product updates arrive and receive review?
  • Can the product team diagnose without broad customer access?
  • Who owns backups, patching, capacity, certificates, and incident command?
  • How are customer-specific changes prevented from forking permanently?

Vendor-managed regional service

The vendor operates the runtime and customer connects approved systems/data.

Questions:

  • Which data may cross and remain in the vendor region?
  • How are tenant isolation, customer keys, support access, export/deletion, and audit evidenced?
  • How do customer identities/roles map without over-delegation?
  • Which outage/recovery/communication obligations exist?
  • Can the customer exit and retrieve required records?

On-premise or disconnected edge

Selected components run near field sites with constrained connectivity.

Questions:

  • Which capability must work disconnected, and which must fail closed or defer?
  • How are software/configuration/policy/manual versions signed, distributed, observed, and rolled back?
  • How are identity, revocation, time, audit, and queued synchronization handled?
  • What happens to conflicting updates after reconnection?
  • Who maintains hardware/runtime and supplies support evidence?

Do not teach three separate products. Keep the logical interfaces, workflow states, controls, evidence, and ownership contract stable where possible; instantiate topology and operating decisions per environment.

Test environment assumptions before the launch gate

Evidence should include:

  • clean deployment from approved artifact/config;
  • positive and denied identity/authorization paths;
  • region/tenant isolation attempts;
  • dependency connectivity, TLS/certificates, DNS, rate/timeout;
  • representative data volume/quality and low-connectivity behavior;
  • secret injection/rotation/revocation without source exposure;
  • configuration promotion and observed application;
  • telemetry/audit/support access;
  • failure/fallback and recovery/break-glass rehearsal;
  • owner review and unresolved-gap disposition.

Production may still differ. The evidence packet states which claims are from local, test, staging, customer sandbox, or bounded production. An environment name is not evidence provenance.

Avoid test identities that are more powerful than production

A shared staging administrator can make every integration appear functional while hiding role/region/tenant denial behavior. Use representative least-privilege test roles, including denied and expired states. Where customer identity cannot exist in staging, simulate the claims locally and schedule production dry-run evidence before cohort exposure.

Avoid clean data as the only environment difference

Use synthetic missing, conflicting, stale, and cross-region fixtures. Customer-controlled test data can add realistic semantics within approved boundaries. Never copy production data casually to create “parity.”

Keep the FDE from becoming the environment

If only the FDE knows which configuration, credential reference, network exception, manual command, or dashboard makes the deployment work, the architecture is not operational.

For every environment-specific action, require:

  • versioned procedure or automation;
  • owner and backup owner;
  • least access and expiry;
  • observable result;
  • failure/recovery path;
  • handoff demonstration.

The FDE may execute early work, but the target operating model must be visible from design. Convenience access should shrink as automation/ownership evidence grows.

Orchid OA-05 environment section

Topology. Region West customer runtime, user/network path, Orchid components/stores, external systems, control/operations planes, owners/boundaries.

Identity. Human technician/approver plus bounded workload identities; resource/operation/tenant/region authorization; expiry/revocation/audit; denied paths.

Secrets/config. No source secrets; lifecycle records; versioned non-secret config with validation/observed state.

Isolation. Region/tenant in identity/data/cache/log/queue/config/support/recovery.

Environment matrix. Local/test/staging/production differences, consequences, tests/monitoring/owners.

Capacity/connectivity. Segmented budget assumptions, rate/backpressure, low-connectivity fallback/reconciliation.

Recovery access. Independent, least, time-bound, audited, rehearsed break-glass plan.

Companion. Authorization and environment/config validation with 4 additional passing tests.

Conduct the environment review exercise

Take the logical Orchid design and instantiate it for three fictional environments: a customer cloud account, a vendor-managed regional service, and a disconnected edge site.

For each, produce topology, identity flow, network policy, secret/config lifecycle, tenant/region isolation, capacity/connectivity budget, telemetry/support access, break-glass path, and owner matrix. Then inject:

  • expired technician permission;
  • workload identity scoped to the wrong tenant/region;
  • blocked egress or DNS failure;
  • stale configuration with successful deployment status;
  • secret rotation while a worker remains active;
  • clock skew affecting expiry/audit order;
  • capacity saturation plus retry demand;
  • unavailable normal identity during recovery.

Record observed state, user/fallback behavior, data/external effects, evidence, owner, and recovery. Pass when the same logical workflow contracts remain clear while environment-specific responsibilities and failure behavior are explicit. Fail if an administrator identity, copied production data, hidden FDE command, or environment name substitutes for representative evidence.

Environment failure modes

VPC as identity

Repair: per-request principal/resource/operation/context authorization.

Staging equals production

Repair: environment-difference matrix and compensating evidence.

Shared long-lived credential

Repair: workload identity, least operation, rotation/expiry/audit/revocation; route exceptions to owner.

Secret in configuration

Repair: separate secret lifecycle/injection; keep records secret-free.

Region as a label only

Repair: enforce in identity, storage, cache, telemetry, support, config, recovery.

Break-glass depends on failure

Repair: independent path/known-good access and rehearsal.

Capacity after launch

Repair: budget/assumptions/failure/load evidence and cohort control before exposure.

The Chapter 8 gate

Before bounding AI behavior, OA-05 must show:

  • deployment topology and operating ownership;
  • environment-difference matrix;
  • human/workload identity flows and authorization context;
  • secret/configuration lifecycle and observed state;
  • tenant/region/data isolation;
  • network/egress/DNS/certificate dependencies;
  • capacity/latency/rate/backpressure and connectivity assumptions;
  • break-glass/recovery access;
  • failure walkthroughs and companion tests;
  • explicit gaps/owners/approvals.

The gate passes when “works in the customer environment” is decomposed into testable runtime and ownership claims.

Chapter 9, Put AI Inside a Bounded Workflow, decides whether uncertainty-producing model behavior belongs in this environment and how task, data, tools, evaluation, controls, latency, cost, fallback, and human authority remain visible.