How backend production systems actually fail

Systems in production tend to experience incidents, though some more than others. Most of the time, when something goes wrong in production, the code is doing exactly what it was written to do. The problem is that production introduces conditions that cannot be fully simulated ahead of time. In this article, I will discuss how these failures actually happen, group them into three patterns, mention why these patterns are dangerous, and touch on lessons that can be learned. Production systems don't fail because code is bad; they fail because reality isn't always consistent. Prerequisites Before I proceed, please note that this article is for: Backend Engineers People running production systems Anyone who has dashboards that say "green" while users complain Failure Patterns Failure Pattern #1: Cascading Failures Cascading failures occur when one service in a system becomes slow or fails, which in turn affects how other parts of the system that depend on the service behave. Cascading failu