RAG Architecture That Fails in Production Tickets (What I Change)
The first support ticket after a RAG go-live rarely says “the embedding model is wrong.” It says the bot invented a policy clause, cited a retired PDF, or missed the answer that sat two paragraphs below the chunk boundary. The demo still looked fine on the golden set.
That is the gap this post closes. I treat RAG architecture as the seams around retrieval—not a vector database brand. Engineers get the failure modes I actually change in code and ops. Managers get one scoreboard: can we prove the system retrieved the right evidence before it spoke?
What RAG architecture is (and is not)
RAG architecture is the design of how enterprise knowledge becomes grounded answers: ingest, chunk, index, retrieve, rank, constrain generation, evaluate, and operate under change. The LLM is one component. Most production pain lives outside it.
It is not “embed the wiki and ship ChatGPT with your logo.” Naive embed → top-k → prompt works for demos. It collapses when documents version weekly, queries are multi-hop, or wrong citations become a compliance event.
| Layer | Job | Ticket if weak |
|---|---|---|
| Corpus & ACL | What may be indexed; who may see it | Cross-tenant or stale secret leakage |
| Chunk & enrich | Boundaries, metadata, parent refs | Answer exists but never retrieves |
| Retrieve & rank | Hybrid search, filters, rerank, k policy | Fluent wrong answers; noisy context |
| Generate under policy | Citations, refuse-when-empty, tools | Hallucinated claims with fake confidence |
| Eval & ops | Retrieval metrics separate from answer quality | You debug blindly after every content drop |
This sits inside broader AI architecture boundaries (control, data, intelligence, evidence). RAG is the data-and-evidence path for grounded Q&A—not a substitute for tool policy when the system must act.
Failure modes I see in production tickets
These show up across internal knowledge bots, support assist, and policy copilots. The symptoms differ; the root causes repeat.
1. Chunking by token count, not meaning
Fixed 512-token windows with a polite overlap are the quickstart default. They split procedures mid-step, orphan table headers, and bury the sentence that answers the query in a neighbouring chunk the retriever never ranks high enough.
What I change: prefer structure-aware splits (headings, sections, code fences) first; use semantic or recursive splitters when structure is weak; store parent document IDs so “small-to-big” can expand a hit into full section context. Overlap is a bandage, not a strategy.
2. End-to-end scores hide retrieval failure
Teams measure “answer looks good” on a handful of happy-path questions. Retrieval can be broken while the model still invents a plausible paragraph. When tickets arrive, nobody knows whether to fix the index, the prompt, or the model.
What I change: score retrieval alone—recall@k, precision@k, MRR, and “gold doc in top-k”—on a frozen query set before judging generation. If gold evidence is missing, do not tune the system prompt.
3. One top-k and one embedding path for every intent
FAQ lookups, multi-hop “compare these two policies,” and “find the table cell” are different jobs. A single dense embedding + k=5 treats them the same. Multi-hop queries need decomposition or iterative retrieve; numeric/table queries often need BM25 or structured filters more than cosine similarity.
What I change: hybrid dense + sparse search; metadata filters (product, region, doc type, effective date); a cheap intent router that adjusts k, filters, and whether to rerank. Rerankers cost latency—use them when precision matters more than p50.
4. Stale indexes and silent corpus drift
Policies change on SharePoint Friday afternoon. The index still ranks Tuesday’s PDF. Citations look authoritative. Legal and support both lose trust.
What I change: versioned document IDs, effective dates in metadata, delete/tombstone on unpublish, and a freshness SLO (max lag from source to searchable). Surface “as of” dates in the UI when the domain requires it.
5. Citation theatre without grounding checks
The model lists three links. None of them support the claim. Users learn to ignore sources; auditors learn to escalate.
What I change: require span-level or chunk-level citation IDs in the prompt contract; refuse or hedge when retrieval confidence is low; spot-check faithfulness on a sample (claim supported by cited text). Optional: only allow quotes from retrieved chunks, not free synthesis for high-risk domains.
6. ACL ignored at retrieve time
The index is a flattened bag of “everything the crawler saw.” User A retrieves a draft only User B’s group should see. This is not an LLM bug—it is an architecture bug.
What I change: filter by principal at query time (or maintain per-tenant indexes); never rely on the prompt to “not show” restricted text. Same rule as any search product.
That loop is the minimum shape I defend in design reviews. Fancy graph RAG or agentic multi-step retrieve comes after this loop is measurable.
Decision guide: what to change first
When a ticket lands, I pick the cheapest layer that explains the symptom—not the trendiest paper.
| Symptom | Likely layer | First move |
|---|---|---|
| Right answer in corpus, never surfaces | Chunk / hybrid / filters | Inspect gold chunk; fix boundaries; add BM25 |
| Right chunks, wrong or invented claim | Generate / prompt / model | Tighten citation rules; lower temperature; refuse-empty |
| Correct last month, wrong this week | Ingest / freshness | Reindex pipeline; effective-date filter |
| Works on FAQ, fails on “compare A vs B” | Routing / multi-retrieve | Decompose query; two retrieves + synthesis |
| Metrics green, users still complain | Eval set mismatch | Rebuild gold set from real tickets |
Shared scoreboard for eng and leadership:
- Retrieval health: % of eval queries with gold evidence in top-k (target before you argue about models).
- Grounding: sample rate of claims supported by cited chunks.
- Freshness: p95 source→index lag and % answers citing superseded docs.
- Safety: zero ACL violations in red-team retrieve tests.
- Cost/latency: p95 end-to-end and $/1k queries—so quality work does not silently burn budget.
If retrieval health is weak, swapping models is theatre. If retrieval is strong and grounding is weak, fix the generation contract. That sequencing alone cuts weeks of thrash.
When RAG is the wrong tool
Not every “chat with our data” problem wants RAG architecture. I push back when:
- The task is transactional (create ticket, move money, change charger config)—that needs tools, identity, and side-effect policy, not only retrieval. See evaluating AI agents in production and enterprise MCP gateway patterns.
- The corpus is tiny and stable—a curated FAQ or single policy pack may beat a vector index.
- Answers must be exact numbers from systems of record—query the API or warehouse; do not hope the PDF chunk still matches the ledger.
RAG shines for explanatory, policy, and document-grounded Q&A with a changing corpus. Stretching it into an action bus is how you inherit every retrieval failure plus every integration failure.
A lean production baseline
Before multi-agent orchestration theatre, I want this baseline live:
- Documented source inventory + ACL model + retention.
- Structure-aware chunking with metadata (source, version, dates, tenant, sensitivity).
- Hybrid retrieval + optional rerank; filters always available.
- Citation contract and refuse-when-empty behaviour for high-risk intents.
- Separate retrieval eval set (built from real questions) run on every index or prompt change.
- Traces that store query, filters, retrieved IDs, and final citations—not only the answer text.
Security review will still ask harder questions about data classification and egress—I covered that frame in AI solution architect decisions that survive security review. RAG does not get a free pass because “it’s only reading.”
FAQ
What is RAG architecture in one sentence?
The system design for grounding model answers in retrieved enterprise evidence—covering ingest, access control, chunking, retrieval, ranking, generation policy, evaluation, and operations—not just “add a vector database.”
Why does demo RAG fail after go-live?
Demos use clean docs and scripted questions. Production adds ACL, stale content, multi-hop queries, and users who phrase questions unlike your eval set. Failures are usually retrieval and ops, not model IQ.
Should we always use semantic chunking?
No. Start from document structure. Use semantic or recursive splitters when structure is missing or messy. Measure recall@k after each change; chunking is a hypothesis, not a religion.
Do we need a reranker on day one?
Not always. Fix hybrid search, metadata filters, and chunk quality first. Add a reranker when precision@k plateaus and latency budget allows.
How is RAG architecture different from agent architecture?
RAG optimises evidence for answers. Agents optimise tool use and side effects under policy. Many products need both: retrieve to inform, then call tools under a gateway—with separate evals for each path.
What should engineering managers track weekly?
Retrieval hit rate on the gold set, grounding sample results, index freshness, p95 latency/cost, and open incidents tagged “wrong source” vs “invented claim.” One dashboard beats slideware model comparisons.
Closing
Good RAG architecture is boring on purpose: clear corpus ownership, chunks that respect meaning, hybrid retrieve you can explain, citations you can audit, and metrics that separate “did we find it?” from “did we say it?” The teams that win production tickets are the ones who can point to a layer and change it—without swapping the model every sprint.
If you are shaping the wider control plane around this, start from AI architecture boundaries and failure modes, then hang RAG as the grounded-read path inside that shape.
