Dealing with Interrupts
Why operational load interrupts engineering work: pages, tickets and ongoing operational activities as the three kinds, how cognitive flow state works and what interrupts cost, polarising time so each person is either fully on-interrupts or fully on-projects, and concrete tactics for each kind.
Chapter 5 said SREs must spend half their time on engineering. This chapter is about what makes that hard in practice: not a shortage of hours but a shortage of uninterrupted hours. An engineer who is interrupted every twenty minutes does not get half a day of engineering; they get none. The chapter looks at where interrupts come from and proposes a simple policy: polarise time, so that a person is either handling interrupts or doing project work, never both.
The chapter is short and practical, and it is the part of the book most directly applicable to any engineering team.
In one minute
- Operational load comes in three kinds: pages (production alerts with an SLO-driven response time), tickets (requests from customers or other teams, hours to days), and ongoing operational responsibilities (pushes, flag flips, consultations, meetings).
- Flow state is the condition of being fully absorbed in a task. It takes time to enter and is destroyed by an interrupt. Engineering work requires it; interrupt handling does not. Mixing the two means doing both badly.
- Metrics: a team's interrupt load can be measured (pages per shift, tickets per week, hours of ongoing work), and compared with its headcount to decide how many people must be dedicated to it.
- Polarise time. Assign interrupt duty to specific people for specific periods (the on-caller handles pages, a ticket rotation handles tickets) and shield everyone else. The person on duty does nothing else and does not feel guilty about it.
- Pages: the on-caller handles them, should not also have project work expected that week, and should write the postmortems. Secondary on-call handles overflow, not the whole team.
- Tickets: give them to a rotation or to the on-caller if the volume allows. Triage daily. Watch for tickets that are really toil and automate them. Make the ticket queue visible so load is measured.
- Ongoing responsibilities: schedule them. Batch pushes into a push day with a push owner. Make consulting a scheduled office hour rather than a drive-by.
- If interrupt load exceeds what a rotation can handle, that is a signal: the service is too noisy or the team is too small, and it should be fixed, not absorbed.
Managing operational load
Each kind of load has a different time scale and a different right response. Pages need minutes and are inherently interrupt-driven; the on-call rotation exists to contain them. Tickets need hours to days and can be batched. Ongoing operational work is predictable and can be scheduled. The mistake teams make is to spread all three across everyone, so that everyone is a little bit on-call all the time.
Factors in determining how interrupts are handled
How much load is there, in each category? What is the SLO for responding? How many people can handle each kind (some tickets need a specialist)? And what is the cost of distraction to the engineers who are not on duty? The chapter's point is that the last factor is usually ignored, and it is the largest.
Imperfect machines
Humans are not good at context switching. Entering a focused state takes time, and once interrupted, returning takes more. A person with project work and a steady stream of interrupts ends up frustrated and slow at both. The chapter borrows the term flow state and argues that an engineering organisation should treat the uninterrupted time of its engineers as a resource to be protected, the same as it protects CPU or disk.
Cognitive flow state. Flow is reachable for both kinds of work: an on-caller handling a sequence of incidents can be in flow if that is all they are doing. The damage comes from switching between kinds.
Do one thing well. Polarise: when you are on interrupts, do interrupts and nothing else; when you are on projects, be unreachable for interrupts. This may mean a person spends a week doing only tickets and then three weeks doing only project work, which is far more productive than four weeks of a quarter of each.
Distractibility. Email, chat and meetings are interrupts too. A person on project time should be allowed to close them.
On-call
The on-caller is on interrupts. They should have no project expectations for the shift and should be told so explicitly. They own pages, and ideally tickets as well if the volume fits. They write the postmortems for the incidents of their shift. The secondary should handle only what the primary cannot, so that only one person's week is consumed. Where a team's page load exceeds what one person can handle, the right move is to fix the service or push back, not to make everyone half on-call.
Tickets
Tickets drift in and create a feeling that someone should do them, which turns into everyone glancing at the queue. Assign them. Options: the on-caller takes them, or there is a separate ticket rotation, or the team holds a daily triage where tickets are given owners. The ticket queue should be visible and its size tracked, because a growing queue is evidence of toil that should be automated or pushed back to its source. Many tickets are the same request repeated, which is a tool waiting to be written.
Ongoing responsibilities
Pushes, configuration changes, and consultations are predictable. Batch them: a weekly push day with a named push engineer, a documented process, and the rest of the team uninvolved. Offer consultation as scheduled office hours rather than ad hoc questions, and route common questions to documentation.
Be on interrupts, or not on interrupts
The chapter's closing recommendation is that managers be explicit about who is on interrupts, publish it, and defend the time of everyone else. The on-interrupts person can be generous with their time because it is all allocated to this; the project person can be unavailable without guilt because someone else is covering. Teams that do this report both more engineering output and better operational response, because the person handling interrupts is actually paying attention to them.