The Hidden Problem With Multi-Model AI Systems: Context Window Mismatch

Notes from building infrastructure for 17,000+ LLMs One of the promises of modern AI infrastructure is simple: You should be able to switch models whenever you want. Different models have different strengths. Some are faster. Some are cheaper. Some reason better. Some support large context windows. In theory, you route requests dynamically and get the best of each. In practice, something breaks almost immediately. Context windows don’t match. The Moment Everything Breaks Imagine this common scenario A conversation begins on a large context model. Maybe something like a 128k context window. The system prompt is fairly large. The user has been chatting for a while. Tools have been called. A RAG system has pulled in documents. Everything works. Then your router decides to switch to a smaller model. Maybe for latency or cost reasons. Suddenly the entire state no longer fits. The request fails or the model behaves unpredictably. This happens because the model’s context window is not just ho