Eliminating Toil
What toil is and is not, why Google caps it at 50 percent of an SRE's time, why it is harmful to careers, morale and velocity, and how the engineering half of the job is defined.
An SRE team's reason to exist is to do engineering that reduces future operational work. This chapter names the enemy: toil, the manual, repetitive, automatable work that running a service generates by default. It explains the 50 percent rule from chapter 1 in detail and makes the case that letting toil creep is a slow-motion failure of the whole model.
Toil is not a synonym for "work I do not like". The chapter is precise about the definition so that teams can measure and argue about it honestly.
In one minute
- Overhead is administrative work not tied to running a service: HR paperwork, meetings, reviews, training. Not toil.
- Toil is work tied to running a production service that is manual, repetitive, automatable, tactical, has no enduring value, and scales linearly with service growth. It need not have all those properties; the more it has, the more clearly it is toil.
- Google's target is that toil stays below 50 percent of each SRE's time, averaged over the team. Surveys at the time found about 33 percent, with the biggest item being interrupts (non-urgent messages and emails), then on-call, then releases and pushes.
- Toil is not always bad. In small, predictable doses it is calming and gives engineers a sense of accomplishment. The problem is when it crowds out engineering.
- Engineering work is the opposite of toil: creative, novel, requires judgement, produces permanent improvement, and lets the team run a bigger service with the same headcount.
- SRE work falls into four buckets: software engineering (writing code: automation, tools, frameworks), systems engineering (configuring, tuning, consulting on architecture), toil, and overhead.
- Too much toil causes career stagnation, low morale, confusion about what SRE is, slow progress, bad precedent for other teams, attrition and breaches of faith with engineers who were hired to code.
What is toil
The six properties are the checklist to apply. Manual: a human runs the script, so the time spent is still toil even if the script does the work. Repetitive: a task you do for the first time is not toil. Automatable: if a machine could do it as well as you, it is toil; if it needs human judgement it is probably not. Tactical: interrupt-driven and reactive rather than strategic. No enduring value: the service is in the same state after you finish. Scales with the service: if the work grows with users or traffic, it is toil. Note that routine but valuable work, like a release that is mostly automated, is still toil for the portion a human has to drive.
Why less is better
The 50 percent cap comes from the reason SRE exists: if the team was fully operational it would be a conventional ops team with better salaries. Fifty percent is a ceiling, not a target. Teams that are below it have more room to invest in automation that pushes it lower still.
When a team measures its toil and finds it over the line, the fix is management action: push work back to developers, hire, or deliberately stop doing something. The chapter warns against the quiet alternative of letting individuals absorb it, because that is how toil becomes invisible.
What qualifies as engineering
Engineering work needs judgement, is guided by strategy, and leaves the system better than before. The examples given are writing automation, building a monitoring framework, writing a service's load balancing configuration from a reusable template, or consulting on the architecture of a new system. The distinguishing feature is leverage: one unit of work reduces many future units.
Is toil always bad?
A little toil is fine, and some engineers genuinely enjoy it. The chapter's concern is the cumulative effect. SREs who spend their time on toil do not learn, their work is not valued at promotion time, newcomers see the wrong model of the job, the service's reliability stops improving, other teams expect SRE to absorb their ops work, and the best people leave. Each of these is slow, which is why it has to be measured rather than felt.
The chapter ends with the pitch: if everyone on the team eliminates a little toil each week, the service becomes cleaner, the on-call rotation becomes quieter, and SRE keeps its identity as an engineering discipline. Let's invent more, and toil less.