Automation is the mechanism by which SRE keeps toil under 50 percent. This chapter takes a step back and asks what automation is for, how it matures over the life of a system, and when it goes wrong. The thesis is that the end state of automation is not a well-maintained script but a system that no longer needs the script: automatic rather than automated.

Two long case studies, Google's MySQL deployment for Ads and the Borg cluster turnup process, show the path from a human running commands to a system that runs itself.

In one minute

• Value of automation: consistency (humans make mistakes doing the same thing twice), a platform (automation can be extended, measured and shared), faster repairs (a fix applied automatically shortens MTTR), faster action than any human, and time saved for whoever would have done the task.
• Google tends to automate more than many organisations because of its scale and because its products are software, but it also automates where it is cheap and skips where it is not.
• Five levels of a use case: 1) no automation, a human does it; 2) externally maintained system-specific automation, an engineer's personal script; 3) externally maintained generic automation, a shared tool everyone uses; 4) internally maintained system-specific automation, shipped with the service; 5) systems that do not need automation because they handle their own failures.
• Ads MySQL on Borg: moving the master database onto Borg meant masters got rescheduled, so failover had to become automatic. Hand failover took 30 to 90 minutes; the Decider system brought it to under 30 seconds, and the hardware budget shrank by 95 percent.
• Cluster turnup: went from a human running a checklist, to Prodtest checks, to idempotent fix scripts, to services that configure themselves on first boot. Each step both removed toil and discovered latent inconsistency.
• Automation can cause harm. A disk-wipe automation at Google misread an empty response from a failing locking service as "no machines are protected" and started erasing disks. The lesson is to design automation that fails safe and that humans can stop.

The value of automation

Beyond saving the time of the person who used to do the work, automation gives consistency: the same inputs produce the same outputs, which humans cannot guarantee. It is a platform: once a task is automated it can be measured, extended, and composed with other automation. It can repair faults faster than a human can be paged. It can act faster than any human ever could, for example rotating a certificate across a thousand machines. The time savings are real but are often the least important benefit.

Automation processes vary along three axes: competence (how accurately they do the job), latency (how quickly), and relevance (how much of the real-world process they cover). Scripts that handle 80 percent of cases well and hand the rest to a human are common and useful.

The use cases and their evolution

The hierarchy of automation classes is the chapter's mental model. A team usually starts with no automation, moves to scripts that live with the engineer, then to a shared tool the team maintains, then to automation that lives inside the service so that it is tested and released with it, and ideally ends with a service that handles its own failures and no longer needs the external tool at all. Database failover, which the Ads case study walks through, follows exactly this path.

Google does not automate everything. The decision should weigh the cost of the automation against the cost and risk of the manual task, and recognise that automation itself needs maintenance. A good heuristic: automate what is done frequently and has a well-understood correct answer.

Case study: Ads MySQL

Google's Ads system ran on a few dozen MySQL replica sets, with manual master failover. When the databases moved onto Borg to improve utilisation, task rescheduling made master failures a regular occurrence. The team built a failover daemon called Decider that could promote a replica safely in less than 30 seconds for 95 percent of failures. This unlocked moving the remaining fleet to Borg, which freed 95 percent of the dedicated hardware budget, and it also drove the team to make the application tolerant of a master going away, which it had never had to be.

Case study: cluster turnup

Bringing a new cluster into service was once a checklist with hundreds of steps executed by a human. It took weeks and the result was inconsistent. The first fix was Prodtest, a set of Python unit tests that checked each precondition and flagged what was wrong. The next step was to write a fix for each test that was idempotent and automatically run when the test failed. Over time, the fix scripts were absorbed into the services themselves, so a new cluster of a service configured itself when it came up. The turnup time dropped from weeks to days, then hours. A side benefit was that the test suite exposed configuration drift that nobody knew about.

Automation that bites back

The warning story: an automation that wiped disks of decommissioned machines consulted a locking service to find which machines were protected. The locking service was being drained and returned an empty list. The automation interpreted "no protected machines" literally and began erasing the disks of live machines. The fault was a design that treated an empty answer as a safe answer, and that had no sanity check on the size of the action it was about to take. Automation must be conservative in what it destroys, should rate-limit itself, and should stop and ask when the world looks unexpected.

Recommendations

Build automation as a platform, not a pile of scripts. Test it like software. Prefer services that fix themselves. Assume the automation will one day be wrong and make sure that when it is, the blast radius is bounded and a human can stop it.