Why Your Agent's Eval Suite Won't Catch Production Failures

Your eval suite passed. Your agent is degrading in production. These two facts are not contradictory - they're the expected outcome when you treat offline evaluation as a sufficient signal for production reliability. Offline evals and production outcome tracking solve different problems. Conflating them is how you end up with green CI checks and a support queue full of AI-generated nonsense. What Evals Are Actually Measuring A typical eval setup looks like this: you have a dataset of input/expected-output pairs, a harness that runs your agent against them, and a set of metrics (accuracy, BLEU score, LLM-as-judge ratings). You run this before deploying. If it passes, you ship. This is useful. It catches regressions when you change your prompt, swap models, or restructure your agent logic. It gives you a baseline for comparison across configurations. But the eval suite is measuring a fixed distribution. Your labeled dataset reflects the traffic patterns, model behaviors, and user intent