Zero downtime is not merely the result of launching the system. Behind it lies many mundane choices made long before the launching day, choices that nobody present has credit for because the whole point is that nothing observable takes place. The sudden traffic increase that tests the system does not announce its arrival, hence the previous decisions must have been taken prior to its occurrence. 

What fails from heavy traffic is rarely what one might expect. People would assume the database would be the first to break under load, which, sometimes, it might be the case. Often, it is something smaller that fails, such as a downstream service with a fixed timeout rejecting requests. Consequently, any client that is instructed to retry again triggers an avalanche of additional requests, because one slow response is interpreted as several. This is called a retry storm and is the easiest way for a system functioning at the very edge of the capabilities to break down, since, in fact, the affected requests do not correspond to the increase of actual customer demand but are repetitions of the previously failed request. 

Redundancy is, of course, the solution, but teams often miscalculate its implementation, for the other system may fail, just like the first one. True redundancy means failures in different systems do not overlap; that is, there are different databases, different geographic locations, different routes, each capable of functioning independently of the other. In contrast, graceful degradation is like more reserved redundancy; for instead of stopping the processes altogether when hitting a problem in the upstream system, one still serves an imperfect version of the product with limited functionality. Thus, some recommendations disappear from the list, cache files provide outdated information, some minor features are non-operational, yet the main experience is still functional.

The catch with graceful degradation is that it’s a code path nobody exercises in normal operation, which means it’s also a code path that quietly rots. The only way to gain confidence in this code is to test it under more or less real-life conditions. A backup called graceful degradation must be tested before an emergency occurs, as it turns into a myth if it has never been triggered before.

Monitoring most of the time is the case when people do not take reliability into account when designing systems. Many teams create their post-incident dashboards after what happened as they can only make use of the previous experience and what they have learned from it instead of thinking about how to design the dashboard before the incident itself. The better version gets designed alongside the system itself: what needs to page a human immediately, what can wait until morning, and what threshold actually means something is wrong instead of just being noisy. An alert that fires constantly for nothing gets muted eventually, and a muted alert is worse than no alert, because it creates the illusion that someone’s watching.

For a successful consumer app, the term “high traffic” is usually not about the drastic growth that people usually visualize. Instead, it implies a launch burst, when one application unexpectedly gains rapid success within a particular platform or with the marketing launch coinciding with the regular release of the app. The studios that hold up aren’t the ones that predicted the exact spike. They’re the ones whose architecture never assumed there wouldn’t be one.