Chapter 13 dealt with the technical response to an outage. This chapter deals with the human one: how a group of smart engineers coordinates during a crisis without getting in each other's way. Google adapted its process from the Incident Command System used by US firefighters and emergency services. The chapter contrasts a badly run incident with a well-run one to show why structure matters more than skill.

The roles here, incident commander, operations lead, communications lead and planning lead, are what the on-caller is expected to set up as soon as an incident is declared.

In one minute

• An unmanaged incident has these features: sharp focus on the technical problem with nobody looking at the whole, poor communication (nobody knows what others are doing), and freelancing (engineers making changes without coordination). The result is a long outage and conflicting fixes.
• Recursive separation of responsibilities. Each participant has a role and stays in it. Roles can be delegated and sub-divided as the incident grows.
• Roles: Incident Command owns the overall state, assigns roles, removes roadblocks, and does not touch the system. Operational work is the only team modifying the system. Communication sends periodic updates to stakeholders and keeps the incident document current. Planning handles longer-term tasks: filing bugs, ordering food, arranging handoffs, tracking divergences from normal to revert later.
• A recognised command post: a chat channel or room where everyone knows to go. Chat gives a timestamped record for the postmortem.
• A live incident state document: a shared doc that the incident commander or communications lead keeps current, with the current theory, actions taken, who is in which role, and what is next.
• Clear, live handoff: when the incident commander changes, say so explicitly and have the incoming person acknowledge.
• When to declare: if a second team is needed, if the outage is visible to customers, or if the issue is unsolved after an hour of concentrated effort. Err on the side of declaring; a managed non-incident costs almost nothing.
• Best practices: prioritise (stop the bleeding, restore service, preserve evidence), prepare (document the process), trust (give full autonomy within roles), introspect (notice your own stress and ask for help), consider alternatives, practice, and change it around (rotate roles).

Unmanaged incidents

The chapter's first story: an on-caller gets paged about a service failing in one datacenter, then another. They dive in, start trying fixes, other engineers notice and start trying their own fixes, management asks for updates in the same channel, and someone makes a change that interferes with someone else's. Hours later, the service is partly up, nobody knows exactly which change fixed it, and the postmortem is impossible to write. Nobody did anything wrong individually; the process failed.

Elements of incident management

Separation of responsibilities. The incident commander's most important property is that they do not fix the problem. Their job is to know the state, assign work, and unblock. The operations lead and their team are the only ones allowed to change production. Communications owns outward messages and the status document so that engineers are not interrupted by questions. Planning thinks about what will be needed in six hours. As the incident grows, each lead can delegate, and the structure stays the same at each level.

Command post. Everyone involved joins one channel. Chat beats voice because it leaves a record and lets people join late and catch up. A dedicated bot can record events and remind about handoffs.

Live incident document. A shared document, editable by everyone, is the single source of truth. It should include the current summary, timeline, roles, actions in flight, and the next check-in time. It becomes the skeleton of the postmortem.

Handoff. Incidents outlast shifts. The outgoing commander states explicitly that they are handing off, the incoming one confirms, and the channel is told. The same applies to every role.

A managed incident

The same incident, run with structure: the on-caller declares, becomes incident commander, pulls in a colleague as operations lead and another to handle communications. Work is divided (one person checks the recent push, one checks traffic). Changes are announced before being made. Management gets updates from the communications lead on a schedule. When the fix is found, it is rolled out carefully, the document records it, and the postmortem is mostly written already.

When to declare an incident

Declare early. A good rule is to declare if any of three things is true: you need to involve a second team, the outage is visible to customers, or you have been trying to solve it for an hour without success. Teams that use the process for small events are good at it when the big one comes; teams that save it for emergencies fumble.

Best practices

Prioritise: stop the bleeding, restore service, preserve the evidence for root-causing, in that order. Prepare: write down the process, agree templates, and run drills. Trust: within a role, people have full authority; the commander does not second-guess the operations lead's technical calls. Introspect: if you notice you are overwhelmed, say so and get help; that is the process working, not failing. Consider alternatives: step back periodically and ask whether the current approach still makes sense or whether a different action, such as a full rollback, would be better. Practice: the roles must be rehearsed. Change it around: rotate who plays which role so the team can staff any of them.