Building Reliable Software: Planning for Things to Break

We often joke that software is usually implemented in two steps: the first 80% of the time is spent on making it work, and then the second 80% of the time is spend on making it work well. People mistake demos, proofs-of-concept, and walking skeletons for products because the optimistic path is often realized in full, so under ideal lab conditions, a PoC behaves just like the full product. At Saleor , where I act as a CTO, we spend a significant part of our engineering effort embracing the different failure states and making sure the unhappy paths are covered as well as the happy ones. Embracing the Failure Because it does not matter how good your software is or how expensive your hardware is, something will eventually break. The only systems that never break are ones that are never used. Amazon's AWS spends more money on preventive measures than you will ever be able to, and yet, a major outage took out the entire us-east-1 region just last October. In 2016, the world's largest particl