Inference Observability: Why You Don't See the Cost Spike Until It's Too Late

The bill arrives before the alert does. Because the system that creates the cost isn't the system you're monitoring. Inference observability isn't a tooling problem — it's a layer problem. Your APM stack tracks latency. Your infrastructure monitoring tracks GPU utilization. Neither one tracks the routing decision that sent a thousand requests to your most expensive model, or the prompt length drift that silently doubled your token consumption over three weeks. By the time your cost alert fires, the tokens are already spent. The Visibility Gap Inference cost is generated at the decision layer. Routing decisions, token consumption, model selection, retry behavior — these are the variables that determine what you pay. But most observability exists at the infrastructure layer. Here's how the layers break down: Layer What It Tracks What It Misses Infrastructure CPU, GPU, memory, latency Token usage, routing decisions, model selection Application Errors, response time, request volume Model d