Postmortem Culture: Learning from Failure
Why Google writes blameless postmortems, when one is required, how to collaborate on and review one, how to build a culture that rewards learning from failure, and how to tell whether the culture is working.
An incident costs money and sleep. A postmortem is how you make sure it also buys something: a written record of what happened, why, and what will be done so that it does not happen the same way again. This chapter is about the cultural conditions that make that record honest, which comes down to one rule: blameless.
Blamelessness is not softness. It comes from the observation that people make the decisions that seem right given what they know, so a postmortem that ends with "engineer X should have been more careful" has learned nothing about the system that led X to that decision.
In one minute
- A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root causes, and the follow-up actions to prevent recurrence.
- Triggers that require one: user-visible downtime or degradation beyond a threshold, data loss of any kind, on-call intervention (rollback, rerouting traffic), resolution time above a threshold, a monitoring failure (incident discovered manually). Stakeholders can also request one.
- Blameless means focusing on contributing causes without indicting any person or team for bad behaviour. You cannot fix people, but you can fix systems and processes that let a reasonable person make a bad decision.
- Postmortems are collaborative: written in a shared document, with real-time commenting, by everyone who was involved, and reviewed by senior engineers before publication to check that the data is complete, the impact is assessed, the root cause is deep enough, the action items are appropriate and prioritised, and the outcome has been shared.
- Action items get bugs with owners and priorities; the postmortem is not finished until they are filed.
- Culture is built through visible practices: postmortem of the month newsletters, a postmortem reading group, Wheel of Misfortune exercises replaying incidents, and leaders praising a well-written postmortem publicly.
- Common pushback: "they are too much work" (answered by templates and tooling), "nobody reads them" (answered by sharing widely), "people are embarrassed" (answered by blamelessness and by leaders writing their own).
- Measure it: survey whether postmortems improved culture, whether action items got done, and whether the same incident recurred.
Google's postmortem philosophy
Every significant incident gets a postmortem, with "significant" defined in advance by the triggers above so that nobody has to decide in the moment whether an outage was embarrassing enough. The purpose is learning, not punishment. The chapter contrasts a blaming write-up ("the engineer didn't read the runbook") with a blameless one ("the runbook was not linked from the alert and had not been updated after the migration"). The second yields an action item; the first yields fear, and fear yields engineers who hide problems.
Blameless does not mean no accountability. The team is accountable for fixing the system. It means that the question is "what about the situation made this action seem reasonable" rather than "who messed up".
Collaborate and share knowledge
Google's postmortem template is a shared document with sections for summary, impact, root causes and trigger, resolution, detection, action items, lessons learned (what went well, what went wrong, where we got lucky), and a timeline. Multiple people edit it at once, with comments for open questions. Tooling adds real-time collaboration, an open commenting system, and email notifications so the author can gather input quickly.
Before publishing, a senior engineer reviews it. The review checks that key incident data has been collected, that impact is quantified (users affected, revenue, duration), that the analysis went to a real root cause rather than stopping at the trigger, that the action items are concrete and prioritised, and that the document has been shared with everyone who should learn from it. Once reviewed, it goes to a wide internal mailing list and the team's postmortem archive.
Introducing a postmortem culture
Writing postmortems is a habit that has to be built. Google uses several reinforcing mechanisms. The postmortem of the month highlights an especially well-written document in a newsletter. A Google+ community, and later internal discussion groups, share postmortems and best practice across teams. A reading group picks a postmortem and discusses it like a case study. Wheel of Misfortune turns an old postmortem into a role-play exercise for on-call training. Senior leaders write and share their own postmortems to show that it is safe.
The chapter addresses the usual objections. Postmortems are time-consuming, so Google invests in templates, automation that pulls in timelines from chat and monitoring, and the review process, which keeps them focused. "Nobody reads them" is countered by distribution and by linking them from the alerts and runbooks they are relevant to. The discomfort of writing about your own mistake is exactly what blamelessness exists to defuse, and it is reinforced by praising the author rather than grilling them.
Conclusion and ongoing improvements
Google measures postmortem culture with surveys asking whether the process is seen as useful and whether people feel safe writing them, and by tracking whether action items are closed and whether similar incidents recur. The results are used to adjust the templates and the review process. The closing point is that a postmortem culture is itself a system that needs maintenance: it decays if leaders stop modelling it, and it improves when the organisation treats each incident as tuition it has already paid.