Stop Queuing Inference Requests

Most inference backends degrade under burst. This is not specific to LLMs. It applies to any constrained compute system: • a single GPU • a local model runner • a CPU-bound worker • a tightly sized inference fleet When demand spikes, most systems do one of two things: 1. Accept everything and let requests accumulate internally. 2. Rate-limit arrival at the edge. Both approaches hide the real problem. Queues grow. Latency stretches. Retries amplify pressure. Memory usage becomes unpredictable. Overload turns opaque. You don’t see failure immediately. You see slow decay. ⸻ The Missing Boundary There’s a difference between rate limiting and execution governance. Rate limiting controls how fast requests arrive. Execution governance controls how many requests are allowed to run. Those are not the same. You can rate-limit and still build an unbounded internal queue. If you don’t enforce a hard cap on concurrent execution, the backend becomes the queue. And queues under burst are silent liabi