What Changed When Our Production NLU Pipeline Hit a Hard Ceiling (A Live Case Study)

Two months into a high-growth quarter, the natural language stack that powered a customer-facing pipeline stopped scaling. On 2025-08-12 the system responsible for intent classification, response drafting, and tool routing across live chat and email began dropping escalations and timing out under load. The stakes were clear: delayed responses were causing support SLAs to slip, transaction funnels stalled, and engineering cycles were consumed by firefighting instead of shipping features. As a senior solutions architect responsible for reliability and cost, the task was to diagnose why a mature pipeline - one built on established transformer models and layered retrieval - had become brittle at scale. Discovery The initial investigation showed a familiar pattern: tail-latency spikes and memory pressure during long, multi-turn dialogs. The model serving layer would report transient OOMs when conversations went beyond the 8k token checkpoint we had assumed was safe. Profiling the inference