How I Turned My Worst On-Call Nightmare Into a Browser Game

It's 3:14 AM. Your phone is screaming. You silence the PagerDuty alert, sit up in the dark, and try to remember what you're on call for. Right. The analytics platform. You open your laptop, blink at the brightness, and read the alert title: "No space left on device." Oh no. You SSH in. The disk is 100% full. The database has crashed because it can't write WAL files. The API is returning 500s. You have three engineers waking up in your Slack channel asking "what's wrong?" and a support queue filling with "why is everything down?" You start digging. df -h . The /var partition is gone. du -sh /var/* . The log directory is 78GB. du -sh /var/log/* . The analytics worker has 64 gigabytes of logs. You stare at the number for a second. git log --oneline -n 10 . There it is. Eight days ago, commit 9b4d7e3 : debug: temporarily enable debug logging for analytics issue TODO comment in the body: "REMEMBER TO TURN THIS OFF AFTER DEBUGGING!!!" Narrator: She did not remember. At debug level, that anal