Being On-Call
What on-call is for, how Google balances it in time and quantity, the psychology of a page (intuition versus deliberate thought), avoiding inappropriate operational load, and the resources an on-caller must have.
On-call is the sharp end of SRE: a human who is responsible for the service right now, with a pager. This chapter is about making that role sustainable. It sets quantitative limits on how much on-call a person should do, explains why calm rather than heroism is the goal, and names the resources every on-caller needs so that they can act well under stress.
The chapter's data points, two incidents per shift, 25 percent of time on-call, a few minutes to acknowledge, are what people mean when they say Google's on-call is well balanced.
In one minute
- The on-call engineer takes care of their assigned operations by managing outages and performing or vetting production changes, within a response time set by the service's SLO (around five minutes for user-facing services, 30 minutes for less sensitive ones).
- A faster response time costs more because it needs more engineers in the rotation. Response time also bounds achievable availability: a five-minute response cannot deliver 99.99 percent alone, so automation and redundancy have to carry the rest.
- Balance in quantity: at least 50 percent of an SRE's time on engineering, no more than 25 percent on-call, the remainder on other operational work. A single-site rotation needs eight people (one week in eight), or six per site in a two-site multi-region rotation. Fewer than this means too much time on-call; more means skills decay.
- Balance in quality: at most two incidents per 12-hour shift, so each gets root-caused and written up properly. If the pager is noisier, something must change: reduce alerts, or add developers to the rotation.
- Compensation: time off in lieu or cash, with a cap, so that on-call stays expensive to the organisation and nobody takes on too much.
- Stress pushes people toward intuitive, fast reactions, which are usually wrong in novel situations. Google wants deliberate, rational action, which needs clear escalation paths, a well-defined incident procedure, and a blameless postmortem culture.
- Operational overload (too many pages) is fixed by SLO-based alerting, deduplication and grouping, and pushing back on noisy services. Operational underload (too few pages) erodes skills, so Google runs exercises like Wheel of Misfortune and keeps teams in rotations even when the service is quiet.
Introduction and life of an on-call engineer
Being on-call means being available to respond to production incidents within minutes. The on-caller acknowledges the page, triages, and either resolves the problem or escalates. Not every page is an outage; some are for approving a risky change or answering a question from another team. The on-caller is also the person who files the postmortem when something significant happens. Google usually runs two rotations per service, primary and secondary, so there is always a fallback.
Response time is defined by the SLO. A service with a 99.99 percent target has about 13 minutes of downtime per quarter, so the human cannot be the whole story; the on-caller is one layer on top of self-healing and redundancy.
Balanced on-call
The 25 percent limit on on-call time is derived from the goal of 50 percent engineering: on-call plus other operational work must fit in the other half. Google counts a week on-call in eight weeks as the practical maximum, which gives the eight-person single-site rotation. Multi-site rotations ("follow the sun") avoid night pages and need six per site. Teams that are too small to staff this should merge rotations with another team or get developers into the rotation.
The two-events-per-shift limit comes from asking what a person can handle well. Each incident needs time to respond, to fix, to restore, and to write up. If the pager fires more often, the writeups do not get done and the same incident recurs. If a shift regularly has zero events, the team is losing practice and probably over-staffed.
Feeling safe
Modern research describes two modes of thought: intuitive and automatic, which is fast but error-prone in unfamiliar situations, and rational and deliberate, which is slow but reliable. Pages produce stress hormones that push people into the first mode. Google's position is that an on-caller should feel able to take the time to reason, and the way to make that true is organisational. Clear escalation paths mean you are never alone with a problem. A documented incident management procedure (chapter 14) means you do not have to invent a process while the site is down. A blameless postmortem culture means you can take the right action without fear of being punished if it does not work.
Concretely: the on-caller should be able to page a secondary, page a senior SRE, or declare an incident at any time without asking permission, and the organisation should treat each of those as normal.
Avoiding inappropriate operational load
Overload. When a team receives too many pages, the pages themselves are the problem. Alerts should be tied to SLO violations rather than causes; related alerts should be grouped so one page covers one problem; and when a service is noisy, the team can hand pages back to the developers or, in extreme cases, give the pager back entirely until the service is fixed. Chapter 30 describes the embedded-SRE intervention for this. Developers in the rotation also align incentives: a developer who gets paged fixes the bug.
Underload. A team that is rarely paged becomes unable to handle the pages it does get. Google counters this with practice: Wheel of Misfortune exercises replay past incidents as role-play, DiRT disaster recovery tests inject real failures in a controlled way, and SREs are encouraged to rotate through on-call for several related services.
Conclusions
On-call done well is a few pages a week, each of which gets fixed properly, handled by a rested engineer who knows exactly whom to call and what to do. On-call done badly is a treadmill that burns out engineers and never improves the service. The difference is not heroics; it is limits that management enforces and resources the team invests in ahead of time.