Everything breaks eventually. The difference between a mature organisation and a fragile one is not whether outages happen but how it behaves while they do. This chapter uses three real Google incidents, one caused by a test, one by a change, and one by a process, to extract the rules for emergency response. It is short on theory and long on lessons.

The common thread is that the right response was a mix of pre-prepared process and the willingness of individual engineers to stop and ask for help.

In one minute

• Don't panic. You are not alone, the sky is not falling, and most incidents are a matter of following the incident process. Pull in more people early if you are out of your depth.
• Test-induced emergency. A disaster test on a distributed MySQL-backed system was meant to fail over one replica and instead cascaded to take out the authentication system, causing a global outage. Lesson: tests should be scoped, you need a tested abort plan, and the test team should know how the system is used by others.
• Change-induced emergency. A configuration change to protect against abuse was pushed everywhere at once and caused almost every externally facing service to crash-loop. Lesson: roll out gradually, keep the ability to roll back, and make sure that your emergency tooling does not depend on the thing that just broke.
• Process-induced emergency. An automated turndown process, when fed a slightly wrong input by a human, started wiping servers across the fleet. Lesson: automation should refuse surprising scopes and have rate limits, and humans should have a way to stop it.
• Keep a history of outages. Postmortems are the only way everyone learns from one person's incident. Read other teams' postmortems too.
• Ask the big, improbable questions. "What if the building loses power?" is worth asking before it happens.
• Encourage proactive testing. Disaster tests find these failures while someone is watching; real life finds them at night.

What to do when systems break

Panic produces bad decisions. The first responder should remember that they have a process to follow and a team to call. Pulling people in when things look serious is always correct; the worst case is that they go back to bed. Google's incident process (chapter 14) exists so that the on-caller does not have to decide how to organise the response while also fixing the problem.

Test-induced emergency

Google runs proactive tests to find weaknesses. In this case, a test team planned to take down a replica of a distributed MySQL database to confirm failover. The failover worked, but because of a hidden dependency the test also disrupted a critical authentication service, and in minutes much of the company's internal and external services were affected. The team recognised the scope quickly and aborted, but reverting took longer than expected because the recovery plan had not been exercised.

Lessons: the responders had a plan to abort and used it, which saved the day. The responders also had the benefit of a prepared test plan so they knew what had been changed. What went wrong was the understanding of dependencies: nobody had mapped who relied on the system under test. Every disaster test needs a scope, a blast radius estimate, and a rehearsed abort.

Change-induced emergency

A change to a protective infrastructure configuration was pushed globally in one step. It had a bug that caused the receiving servers to crash on startup, and because the config was pushed to nearly everything, nearly everything crashed. Within seconds, external users saw errors. Engineers noticed quickly through monitoring and through their own inability to reach internal tools, which were also down. The fix was to roll back, but the rollback was slowed by the fact that the tooling used to push configuration depended on infrastructure that was itself down.

Lessons: the monitoring and paging worked, the responders were quick, and once the rollback went out recovery was fast. What went wrong was the deployment: a protective change was itself the risk, it was not canaried, and the emergency tooling had a circular dependency on the broken system. Roll out gradually. Make sure that at least one path to undo a change does not depend on the change having worked.

Process-induced emergency

An automated system for turning down machines was fed a request that, through a combination of human error and an over-permissive interface, matched many more machines than intended. The automation started wiping disks across the fleet. The on-caller noticed through a spike in failed health checks, worked out what was happening, and stopped the process. Recovery took considerable effort because the wiped machines had to be reinstalled and the data they hosted re-replicated.

Lessons: the automation had been built to be fast and efficient at exactly the thing it did, with no notion of "this is more than usual". Automation that destroys should be conservative, should check the size of what it is about to do against a sane bound, should proceed at a rate that lets humans notice and stop it, and should have an obvious big red button.

All problems have solutions

Systems break. That is life. Keep calm, follow the process, pull in help. Do not be afraid to take a drastic action like rolling back or diverting all traffic, because the cost of a brief disruption is lower than the cost of a long outage. Record what happened in a postmortem so that the organisation learns, and read postmortems from other teams because their failure modes are yours in waiting. Finally, ask the big questions in advance. The answer to "what would we do if the datacenter burned down" is cheap to write while the datacenter is fine.