Sherlock Holmes: The Case Of AI Brought Down Our Servers

There are two kinds of production incidents. The first kind gives you signals. Metrics drift slightly off baseline. Latency edges upward. Dashboards turn yellow long before anything turns red. You have time to reason about it. The second kind doesn’t negotiate. It lets you sleep peacefully and then informs you in the morning that your server died multiple times overnight. This was the second kind. The Setup We’re building a voice agent platform. Calls come in from users. Audio streams over WebSocket. We integrate with Twilio for real-time media streams. AI agents process the conversation, decide what to say next, and occasionally invoke tools. Some of those tools query our database to fetch context or perform actions. Architecturally, nothing unusual. A fairly standard real-time pipeline: streaming input, AI orchestration, tool execution, database lookups. And everything had been working fine. Then one night, one of our Kubernetes pods limited to 1GB of memory started crashing repeated