AI Architecture That Holds in Production: Boundaries, Evals, Failure Modes
A pilot demo once looked flawless on stage and collapsed the week after: the model answered well, the tools were real, and nobody could say which system owned identity, data egress, or the kill switch. The slide said “AI strategy.” The runtime said “chat with side effects.”
That gap is why I treat AI architecture as a set of enforceable boundaries—not a model shortlist. This post is a practical reference for engineers building the seams and for engineering managers or business owners who need one shared scoreboard: what must be true before agents, RAG, or copilots touch production data.
What AI architecture actually is
I define AI architecture as the decisions that keep probabilistic components (models, prompts, retrieval, tools) coherent with the rest of the enterprise stack: identity, data classification, integration contracts, evaluation, operations, and blast radius.
It is not “which LLM is smartest this quarter.” Model choice sits inside the shape. If the shape is wrong, a better model only fails faster.
| Layer | Job | Failure if weak |
|---|---|---|
| Control | AuthN/Z, policy, rate limits, human gates | Unowned actions; “temporary” god credentials |
| Data | What may be retrieved, logged, or trained on | Leakage via prompts, indexes, or traces |
| Intelligence | Models, prompts, routing, fallbacks | Cost spikes; silent quality cliffs |
| Orchestration | Agents, tools, workflows, MCP/APIs | Wrong tool, wrong order, real side effects |
| Evidence & ops | Evals, traces, SLOs, kill switch | You cannot prove or stop bad behaviour |
Related deep-dives on this blog: AI solution architect decisions that survive security review, evaluating AI agents in production, and enterprise MCP gateway design in .NET.
Who this is for (two audiences, one scoreboard)
| Role | Questions to force | “Good” looks like |
|---|---|---|
| Business / product | What wrong action costs us? Who stops it at 2 a.m.? | Narrow first scope; clear human path; no mystery writes |
| Engineering manager | Who owns prompts and tools? What is the SLO? | Named owner; dashboard; change control |
| Engineers | Can every failure be diagnosed from a trace? | Typed tools; redacted logs; reproducible evals |
Same system. Different questions. One dashboard beats three decks.
Boundaries before models
I refuse to argue temperature or vendor logos until four boundaries are written down:
- Identity — Which principal acts? User-delegated vs service identity? How long do elevated tokens live?
- Data — Classification of corpus and prompts; retention of traces; whether retrieval can cross tenancy.
- Tools — Read vs write; allow-lists; rate limits; which actions need a human gate.
- Stop — Kill switch, feature flag, or disconnect that a human can operate without redeploying the model.
If any of those is “we will harden later,” you are not doing AI architecture—you are scheduling an incident. I covered the security-review angle in the AI solution architect post; the point here is structural: boundaries are first-class architecture artefacts, not backlog polish.
Failure modes I design for (not hope against)
| Mode | Symptom | Architectural counter |
|---|---|---|
| Confident wrong action | Tool 200 OK; business outcome wrong | Task success metrics; shadow mode; write gates |
| Ungrounded answers | Facts tools never returned | Cite-or-refuse; retrieval contracts; honesty checks |
| Prompt / data leakage | Secrets in context or logs | Redaction; separate stores; least-privilege retrieval |
| Tool sprawl | Every API becomes an agent toy | Gateway, schemas, policy; MCP only behind control plane |
| Eval theatre | Pretty chat scores; no ops metrics | Score the loop: task, tools, cost, escalation |
| No owner | Incidents bounce between teams | Named architect + on-call for AI path |
Industry write-ups keep circling the same theme: pilots fail less from “dumb models” and more from missing workflow integration, guardrails, and measurement. Architecture is how you make those boring and testable.
Evals belong in the diagram
If evaluation is a slide after go-live, it will lose to feature pressure. I wire evals as part of the path:
- Offline — golden tasks, tool-choice cases, regression on prompt/tool changes
- Shadow — model proposes; humans or rules still execute high-impact writes
- Online — task success, wrong-tool rate, unsupported claims, cost per success, escalation reasons
I will not open high-impact write tools until those signals exist on a dashboard someone actually watches. Detail and targets live in Evaluate AI agents in production.
Decision guide: how much architecture before build?
| Situation | Minimum bar | Do not skip |
|---|---|---|
| Internal FAQ over public docs | Retrieval scope + citation + cost cap | Logging of full prompts with secrets |
| Agent with read-only tools | Typed tools, traces, task success metric | Unscoped “run any SQL” helpers |
| Agent with writes (refund, ticket, provision) | Shadow → gated writes; kill switch; owner | Shared long-lived admin identity |
| Multi-team / multi-tenant platform | Gateway, tenancy in every retrieval/tool call | Per-app copy-paste MCP servers with prod secrets |
For Sydney-enterprise framing of the role itself, see what an AI architect actually does in Sydney enterprises. This article is the technical spine those role conversations should rest on.
What I refuse to call “done”
- A chatbot UI with production credentials and no eval harness
- RAG over a junk drawer index with no tenancy or freshness rules
- “The model will be careful” as the write-path control
- Observability that stores secrets in plain prompt logs
- Ownership that ends when the vendor workshop ends
Mild opinion: most teams do not need a twelve-layer reference model on day one. They need five boring answers written where operators can find them—identity, data, tools, proof, stop—then a thin vertical slice that respects all five.
Integration reality (where AI meets the rest of the estate)
In enterprises I work with, the fragile seam is rarely the model API. It is the hand-off into systems of record: service buses, CRM writes, identity providers, batch jobs that still own the truth overnight. AI architecture has to name those contracts the same way any integration architect would—timeouts, idempotency, poison messages, compensating actions—not hand-wave them as “tooling.”
When agents become another producer/consumer on the bus, treat them like any other unreliable client: bounded retries, clear poison paths, and no shared god credentials. If you already run disciplined messaging (for example Peek-Lock settlement on Azure Service Bus), reuse that muscle. Do not invent a parallel “AI-only” reliability story.
Multi-model routing belongs here too, but as a cost and resilience control—not a science fair. Route simple classification to a cheap model; reserve expensive reasoning for tasks that pay for it; always keep a fallback when a provider blips. The architecture decision is the policy and the observability, not the router logo.
FAQ: AI architecture
What is AI architecture?
AI architecture is the set of boundaries and runtime controls that let models, retrieval, and tools operate safely inside an enterprise system—identity, data, orchestration, evaluation, and blast radius—not just model selection.
How is AI architecture different from ML architecture?
Classical ML architecture often centres on training pipelines, features, and batch or online scoring. Generative and agentic systems add tool side effects, prompt/data leakage paths, and evaluation of multi-step loops. You still need data and platform discipline; you also need action policy.
Where should evals sit in the design?
In the main path: offline suites for change control, shadow modes before write tools, and online metrics (task success, tool errors, cost, escalation) on an owned dashboard.
Do I need an “AI architect” title to do this work?
No. You need a named owner for the decisions. The title varies; the vacuum of ownership is what hurts.
What is a sensible first production slice?
One narrow intent, read-mostly tools, explicit escalation, measurable task success, and a kill switch. Expand tools only after the scoreboard is green.
Synthesis
Good AI architecture is quiet: clear principals, constrained tools, grounded answers, and a way to stop. Bad AI architecture is loud demos and silent side effects. If you only remember one sequence, use this: boundaries → thin slice → evals on the loop → then model shopping. Everything else is decoration until those hold.
