Top 5 Mistakes Engineers Make While Building RAG Systems

Story Time 😁 I was working on a project that is highly dependent on RAG. User can be able to upload any document ( pdf, Word, PPT, ppt etc ), and AI can be able to answer any type of question from it. When I started building the project, the first choice was always RAG, but the way I handled it broke a lot of things. I uploaded the document regarding a security protocol of a school, it was 100 page pdf, and I asked a question, "Tell me which protocols to follow," and it answered beautifully. But then the second question was, "Summarize the document." It's not able to consider everything in the document. Then the third question was "how many pages does this document have?" It's not able to answer that either. Retrieval-Augmented Generation (RAG) looks simple on a whiteboard: Embed → Store → Retrieve → Prompt → Generate. In production, it’s rarely that clean. After building and testing multiple RAG pipelines, I’ve noticed the same engineering mistakes repeated. Here are the top 5. 1. Usi