Embedding an SRE to Recover from Operational Overload
A playbook for rescuing a team drowning in operational work by embedding an experienced SRE for a few months: the first phase of learning the service, the second of sharing context, the third of driving change, and what the embedded SRE leaves behind.
Sometimes a team ends up with so much operational load that it cannot dig itself out. The on-call is constant, the postmortems are not written, the toil keeps growing, and nobody has time to automate because everyone is firefighting. Google's remedy is to embed a senior SRE from elsewhere in the team for a defined period. This chapter is the playbook for that engagement, written as advice to the embedded engineer.
The chapter is as much about culture change as about technical fixes: the embedded SRE's main lever is showing the team a different way of working.
In one minute
- Signs of operational overload: the team is paged often and reacts rather than fixes, postmortems are skipped or shallow, the same incidents recur, toil is above 50 percent, and the team describes itself as an ops team.
- Phase 1, learn the service and get context: shadow on-call, read postmortems and alerts, talk to everyone, and look for kindling (things that will cause fires) and emergency behaviour (reactions that are habitual rather than reasoned).
- Common kindling: knowledge gaps (only one person knows a subsystem), services developed by SRE that have quietly become products, "common" alerts that are not acted on, capacity shortfalls, and relationships with developers that have gone sour.
- Phase 2, share context: write a good postmortem for the team as a model, sort fires into triggered (someone did something) and arising (the system did it), and make the difference visible. Explain the SRE principles as reasons, not rules.
- Phase 3, drive change: start with the basics (an SLO if there is none), get help clearing kindling (hand work back to the developers, automate obvious toil), explain your reasoning at every step, ask leading questions rather than giving orders.
- The embedded SRE should leave the team with a few habits that stick: a postmortem culture, an SLO, a prioritised list of engineering work, and a sense that operational load is a problem to be solved rather than a condition to be endured.
Phase 1: learn the service and get context
The embedded SRE arrives as an outsider and must spend the first period listening. Shadow the on-caller. Read the last few months of alerts and the postmortems that exist. Interview every team member about what takes their time and what frightens them. Look at the monitoring configuration and at the runbooks.
Identify the largest stress points. Where does the team spend its time, and where does it feel the most pressure? Often these are different; the loud pager and the quiet backlog of tickets both matter.
Identify kindling. Knowledge gaps, where a subsystem is owned by one person who is always paged for it. Services built by the SRE team as tools that now have external users and need to be run like products. Alerts that fire often and are acknowledged without action, which trains the team to ignore pages. Capacity planning that has fallen behind so that the service runs hot. Relationships with the developer team that have become adversarial, so that SRE is seen as blocking and developers stop sharing plans.
Identify emergency behaviour. Teams under stress develop reflexes: restart it, page the expert, roll back without understanding. These are rational short-term responses that prevent long-term fixes. Note them without judgement.
Phase 2: sharing context
The embedded SRE now has a picture, and the job is to give it back to the team in a way they can act on.
Write a good postmortem for the team. Pick a recent incident and write the postmortem the way it should be done: blameless, deep on root cause, with concrete action items. Share it and discuss it. It is a demonstration of what the team should be producing and a first action-item list.
Sort fires according to type. Triggered fires are caused by a human action: a push, a config change, a manual intervention. Arising fires come from the system without a human trigger: capacity, dependencies, latent bugs. The two need different fixes (process and automation for triggered, engineering for arising), and making the split visible shows the team where the leverage is.
Phase 3: driving change
Start with the basics. If there is no SLO, write one with the team and the developers. The SLO gives the team a reason to say no and a reason to prioritise, and it gives the developers a shared target.
Get help clearing kindling. The embedded SRE does not have time to fix everything. Delegate: ask the developers to take back a noisy component, ask the team to automate one piece of toil, ask management for time. Fix one piece of kindling at a time and show the result.
Explain your reasoning. Every change should come with a why. The team needs to understand the principles so they can apply them after the embedded SRE leaves. A rule imposed without reasoning is dropped the moment the pressure returns.
Ask leading questions. Rather than telling the team what to do, ask: why do we page on this? What would happen if we didn't? Who is going to fix the root cause of yesterday's incident? Questions build the habit of reasoning; answers build dependence.
Conclusion
An embedded engagement lasts a few months and ends. The measure of success is not how many fires the embedded SRE put out but whether the team is still improving six months later. That means the team should leave the engagement with an SLO, a working postmortem habit, a prioritised engineering backlog, a healthier relationship with the developers, and the belief that operational overload is a problem they can solve.