Every Millisecond Is a Lie: What Latency Benchmarks Won't Tell You

Here's an uncomfortable truth: that P50 latency number your team celebrates in standups is actively misleading you. It's the average experience of your luckiest users, not the bleeding-edge reality of your slowest ones. And in production LLM systems, the gap between P50 and P99 latency isn't a gentle slope — it's a cliff. I've watched teams optimize their median response time down to 180ms while their P99 quietly ballooned to 4.2 seconds. Users don't remember the fast responses. They remember the one time the chatbot froze mid-sentence during a demo with the board. The Three Latency Lies Lie #1: Tokens per second is your north star metric. Tokens per second (TPS) matters, but it's a throughput metric masquerading as a speed metric. A system pushing 120 TPS means nothing if time-to-first-token (TTFT) is 1.8 seconds. Users perceive speed through TTFT and inter-token latency, not aggregate throughput. A system streaming at 45 TPS with a 200ms TTFT will feel twice as fast as one doing 120