Model-Agnostic AI Architecture: Swap LLMs Safely
Model-agnostic AI architecture means the model is a replaceable part. Keep contracts, tools/MCP boundaries, retrieval with ACL, an eval harness, and observability stable. Swap the model, prompt pack, or price tier only through one adapter, and only after golden-set, regression, cost/latency, safety, and canary gates pass.
The kid picture
Think of your AI system like a toy robot.
- The body is your app: the forms it accepts, the tools it may touch, the files it may read, the tests you run, the logs you keep.
- The head is the model: the part that guesses the next words or picks a tool.
Heads get replaced often. New ones are cheaper. Old ones get worse relative to the new ones. If you glue the head to the body with secret sauce, every head change breaks the robot.
Model-agnostic means: the body stays the same. The head plugs in through a clear socket. You only change the head after a checklist says the new head still does the job.
Why binding everything to one model is fragile
Two things move under your feet:
- Price — today’s “cheap” model is next month’s “expensive,” or the reverse.
- Behaviour — same prompt, different model → different tool picks, different refusals, different wrong answers.
If your code says “call Provider X model Y with this exact prompt string” in twenty places, a swap becomes a rewrite. If your tests are “does it sound nice?”, you will ship a silent change in how money, tickets, or data get touched.
So you separate policy and proofs (body) from word prediction (head).
What stays stable
Do not rebuild these for each model.
1. Contracts (inputs and outputs)
A contract is a form. “You must send JSON shaped like this. You must return JSON shaped like that.” Example: { intent, citations[], tool_calls[], refuse: bool }. The model fills the form. Your code trusts the form, not the vibe.
2. Tools / MCP boundaries
Tools are the hands. The model may ask to press a button. Your gateway decides which buttons exist, who is allowed, and what gets logged. Swap the head; keep the same button list and the same audit trail. For how I measure agents before write tools open, see Evaluating AI Agents in Production.
3. Retrieval and ACL
Facts come from your search index with access rules. The model should not be the place you hide “do not show secret docs.” Filter by user rights before the model sees text.
4. Eval harness
A fixed set of real questions and expected outcomes (golden set). Run it the same way for every head. Score task success, wrong tools, empty citations, refusals. Broader boundary and failure-mode thinking: AI Architecture That Holds in Production.
5. Observability
For each answer: which model, which prompt pack version, which chunks, which tools, latency, token cost. Without this, you cannot compare heads.
These five are the socket. Everything else hangs off them.
What you may swap
| Piece | What it is | Why it moves |
|---|---|---|
| Model | The head | Better quality, lower cost, new limits |
| Prompt pack | Instructions + few examples, versioned | Tune behaviour without new code |
| Price tier | Which SKU / region / rate limit | Budget and capacity |
Swap through one adapter: your code talks to “the model port,” not to five SDKs scattered in features.
Validation gates before you trust a new head
Run these in order. Fail means keep the old head.
- Golden set — Same cases as production cares about. Score with pass/fail, not vibes.
- Regression evals — Compare new head vs old head on the same set. Flag drops in success, spikes in wrong tools, new refusals that block good work.
- Cost and latency budgets — p95 time and $/1k successful jobs must fit the envelope you already agreed.
- Refusal and safety — Forbidden actions stay blocked by policy code, not by hoping the new head is polite.
- Canary — Small slice of live traffic with a kill switch. Only then full cutover.
Analogy: before you put a new battery in the robot, you run the same obstacle course in the garage, check the bill for power, then let it walk one hallway before the whole house.
Simple decision rule: when to change models
Change the model when all are true:
- A measured need exists (cost too high, quality too low, or a hard limit on the old head).
- The new head beats the old one on the golden set for the metrics that matter or matches quality at clearly lower cost.
- Cost/latency budgets still hold.
- Canary is boring for an agreed window (no spike in bad tool calls or empty citations).
Do not change because a blog post said a new name is cool.
Checklist you can paste into a PR
- All model calls go through one adapter
- Outputs match a versioned schema
- Tools/MCP allowlist unchanged by the swap
- Retrieval ACL runs before the prompt
- Golden set + regression report attached
- p95 latency and $/success within budget
- Canary plan + kill switch named
- Prompt pack version recorded in logs
FAQ
What is model-agnostic AI architecture?
A design where the LLM is swappable. Stable parts: input/output contracts, tool allowlists, retrieval ACL, eval harness, and traces. Changing parts: model, prompt pack, price tier.
What should stay stable when you change models?
Contracts, tools/MCP policy and audit, retrieval + ACL, golden-set evals, and logs that record model version, prompt pack, retrieved IDs, tools, latency, and cost.
What checks do you run before swapping an LLM?
(1) Golden set (2) regression vs current model (3) cost and p95 latency budgets (4) refusal/safety still enforced in code (5) canary with a kill switch, then cutover.
When should you change models?
Only with a measured need (cost, quality, or hard limits), a win or equal quality at lower cost on the golden set, budgets intact, and a boring canary window.
Is model-agnostic the same as multi-model routing?
No. Routing picks a model per request. Model-agnostic means any chosen model still plugs into the same contracts and gates. You can do both; start with the stable socket.
Closing
Keep the body. Swap the head. Prove the swap. That is model-agnostic AI architecture in plain terms. Written from the desk of an AI Solution Architect · Sydney.
Related architecture notes: case studies. Questions about the architecture: contact.
