Lessons Learned from Other Industries
Interviews with engineers from nuclear, aviation, medical, manufacturing and lifeguarding backgrounds, comparing how each industry handles preparedness and disaster testing, postmortem culture, automation, structured and rational decision making, and what SRE does differently.
Google did not invent reliability. Nuclear plants, aircraft, hospitals and factories have been engineering for safety for decades. For this chapter the authors interviewed Google SREs who previously worked in those industries and asked what carried over and what did not. The comparison sharpens what is distinctive about SRE: the willingness to accept risk for velocity, and the use of blameless postmortems where other industries use blame and regulation.
It is the chapter most often skipped, and it is worth reading because it explains the cultural choices behind the technical ones.
In one minute
- Industries compared: nuclear power, aviation (air traffic control and avionics), medical devices and health care, manufacturing and lifeguarding. Each has a different cost of failure, from inconvenience to death, and a different regulatory environment.
- Preparedness and disaster testing: every industry rehearses. Nuclear and aviation use simulators and mandatory drills with role-specific checklists. SRE's equivalents are DiRT disaster tests, Wheel of Misfortune and production load tests. The difference is that SRE tests in production, because its failures are recoverable and its production is too complex to simulate.
- Postmortem culture: aviation has a strong blameless reporting culture (the NTSB, anonymous near-miss reporting). Nuclear and medicine have more blame, driven by liability and regulation. SRE deliberately chose the aviation model.
- Automating away repetitive work and operational overhead: nuclear and aviation automate heavily but keep humans in the loop for judgement, and are cautious about automation that surprises operators. SRE pushes automation further because the systems are software and the cost of a mistake is usually lower.
- Structured and rational decision making: all industries use checklists, defined roles and decision trees to counter the effects of stress. SRE's incident command system comes directly from emergency services.
- The contrasts: SRE accepts a non-zero failure rate as a design parameter (the error budget), moves fast, and lets engineers make changes without external approval. Regulated industries cannot. SRE also has an advantage in that its systems can be observed in far greater detail than a reactor.
Meet our industry veterans
The interviewees include engineers who had worked at nuclear facilities, in air traffic systems, on medical equipment, in high-volume manufacturing and as lifeguards. The chapter uses them as witnesses rather than as authorities, asking what their old industry would think of SRE practice.
Preparedness and disaster testing
In the nuclear industry, operators train continuously on simulators, run scenario drills with regulators watching, and use a defence-in-depth philosophy where every layer assumes the others have failed. Aviation similarly uses simulators for every pilot, and air traffic control has rigid procedures for every contingency. Medical device makers test failure modes under regulatory requirement. Lifeguards drill rescues until they are reflexive.
SRE does the same with different tools: DiRT exercises simulate datacenter loss and test the company's response, including the human parts like communication; Wheel of Misfortune drills individuals; load testing and chaos-style fault injection test the software. The main difference is that SRE often tests in production because the real system is the only accurate simulator, and because its failures are, by design, recoverable.
Relentless organisational focus on safety. In nuclear and aviation, safety is the top organisational value and is funded as such. SRE's analogue is the error budget: not safety at any cost, but a defined, measured tolerance.
Attention to detail. Checklists, double checks and sign-offs appear in every industry. SRE's code review, canary and launch checklist are the same idea.
Redundant capacity. Every industry overprovisions critical systems. SRE's N+2 provisioning and multi-region deployment are the same principle.
Simulations and live drills. Discussed above; the point is that practice must be regular and realistic.
Training and certification. Regulated industries require formal qualification. SRE has the on-call checklist and sign-offs (chapter 28), less formal but with the same intent.
Focus on detailed requirements and design. Regulated industries spend most of their effort before building. SRE spends more effort after, because software can be changed cheaply and observed in detail.
Postmortem culture
Aviation's culture is the model: incidents and near-misses are reported without fear of punishment, investigations by independent bodies look for systemic causes, and findings are published for the whole industry. Health care has historically been blame-oriented, and the interviewees described how this suppressed reporting. The nuclear industry sits between, with strong internal review but a regulatory context that discourages candour. SRE chose blameless postmortems because the interviewees' experience, and the research behind aviation safety, says that blame hides information and systems improve only when information flows.
Automating away repetitive work and operational overhead
Nuclear plants and aircraft are heavily automated, and both industries have learned that automation that operators do not understand is dangerous: the Three Mile Island and several aviation incidents involved operators confused by what the automation was doing. The lesson SRE takes is to make automation legible and stoppable (chapters 7 and 13). The difference is degree: because SRE systems are software and failures are usually recoverable, SRE automates more aggressively, including automation that makes production changes without a human in the loop.
Structured and rational decision making
Under stress, people fall back on habit. Every industry counters this with structure: the incident command system in emergency services, cockpit resource management in aviation, surgical checklists in medicine. SRE borrowed the incident command system directly (chapter 14) and uses playbooks and the hypothetico-deductive method (chapter 12) to keep troubleshooting rational. The interviewees noted that SRE's version is less formal and relies more on individual judgement, which works because of the quality of hiring and the blameless culture, and might not transfer to an organisation without those.
Conclusions
SRE shares most of its toolkit with older safety-critical industries: drills, checklists, defence in depth, defined roles, blameless learning. What is distinctive is the explicit acceptance of risk as a budget to be spent, the pace of change, the testing in production, and the extent of automation. Those choices are available to SRE because software failures are usually recoverable, observability is cheap, and changes are fast. The chapter's warning is that SRE should remember why it can afford its approach and notice when a system, such as one holding user data, deserves the stricter model.