5 RAG Architecture Mistakes That Kill Production Accuracy (And How to Fix Them)

I've built RAG systems that hit 96.8% retrieval accuracy in production. I've also built ones that started at 40% and needed emergency rewrites. The difference wasn't the LLM — it was the architecture decisions made before any model was chosen. Here are the five mistakes I see most often when teams take RAG from prototype to production. 1. Treating Chunking as an Afterthought Most tutorials show you how to split documents into 512-token chunks with 50-token overlap and move on. This works for demos. It fails catastrophically on real business documents. The problem: A contract clause that spans three paragraphs gets split across two chunks. Neither chunk contains the complete clause. The LLM gets partial context and hallucinates the rest. What actually works: Use semantic chunking that respects document structure. For structured documents (contracts, legal filings, compliance reports), chunk by logical section — not by token count. A 2,000-token chunk that contains a complete clause is f