How a 2% Latency Spike Collapses a 20-Service System and How to Prevent It

Last week, we modeled cascading database connection pool exhaustion in a distributed microservices architecture. No servers were killed. No regions failed. No database crashed. But the system still collapsed. The Architecture We simulated a realistic production-style topology: • API Gateway • Load Balancer • 12 stateless services • Shared database primary + 3 read replicas • Cache layer • Message broker • External payment API Each service was configured with: • 50 max DB connections • 3 retries (exponential backoff) • 2-second timeout • Shared connection pools per instance This is a completely normal backend architecture. Nothing exotic. The kind of system running at thousands of companies right now. Simulation 1 — Healthy Baseline Under steady-state conditions, the system behaves exactly as expected: • Collapse Probability: 3% — virtually negligible • Retry Amplification: 1.2x — minimal overhead • Cascade Depth: 2 layers — shallow, contained • Availability: >99% • Pool Utilization: 32