A new SRE arrives knowing how to write software and some systems fundamentals, and has to become someone who can be paged for a service they did not build. This chapter is about how to get them there deliberately rather than by throwing them at the pager. It describes a training programme that mixes structured study with early, bounded exposure to production, and it is explicit that the goal is a reasoning engineer, not someone who memorises runbooks.

The chapter is a manager's guide, but the practices in it, especially Wheel of Misfortune and postmortem reading, are the ones every SRE is expected to take part in.

In one minute

• Two bad patterns: trial by fire, where the newcomer learns by being on-call for things they do not understand, and menial work, where they shadow and do tickets but never learn the architecture. Both produce slow, anxious on-callers.
• Good training is cumulative and orderly: a structured programme on how the systems work (reading, lectures, labs), paired with targeted project work that gets the newcomer into the code and the production environment with a mentor.
• Train for reverse engineering (reading an unfamiliar system from its config, logs and RPC flow) and statistical and comparative thinking (is this metric abnormal, what changed), not for memorising procedures. Improvisational skills are what matter in a novel incident.
• Five practices that produce good on-callers: read and discuss past postmortems, run disaster role-playing (Wheel of Misfortune), break real things and fix them in a controlled environment, let the newcomer shadow on-call and then be shadowed, and run a bootcamp or on-call learning checklist with sign-offs.
• Stellar reverse engineers and improvisational thinkers are the aspirational outcome: engineers who can walk into a system they have never seen and understand it.
• After on-call: the engineer keeps learning through postmortems, through on-call for related services, through writing training material for the next person, and through project work that uses what they have learned.

Initial learning experiences: the case for structure over chaos

A newcomer handed a pager with no preparation learns that on-call is frightening and that the way to survive is to follow runbooks mechanically. A newcomer stuck on tickets for months learns that SRE is drudgery and leaves. Google's prescription is a structured path with a clear end point: an on-call learning checklist that lists the systems, tools and procedures a person must be able to use, with each item signed off by a mentor. Structured does not mean slow; the newcomer is in production early, on bounded tasks, with a mentor.

Learning paths that are cumulative and orderly. Start with the basic infrastructure that everything uses (the cluster manager, the storage, the RPC layer), then the team's own services, then the incident process. Each layer should be learnable from documentation and from exercises, and each exercise should build on the previous.

Targeted project work, not menial work. Give the newcomer a project that forces them to read the system's code and configuration and to make a change that goes to production. Good projects: a monitoring improvement, a small automation, a fix to a known issue. The project teaches the environment and gives the person a sense of ownership.

Creating stellar reverse engineers and improvisational thinkers

The chapter identifies what separates a strong on-caller from a weak one. Reverse engineers can take an unfamiliar system and work out how it is put together, by reading its configs, tracing its RPCs, and finding its logs. Train this by having the newcomer explain a system they were not taught, starting from the production configuration. Statistical and comparative thinkers know what normal looks like and how to tell whether the current state differs: compare this cluster to that one, compare this week to last week, look at the distribution rather than the mean. Improvisational artists have a repertoire of general actions (roll back, drain traffic, add capacity, degrade) and know when each applies, so they are not stuck when the runbook does not match.

Five practices for aspiring on-callers

A hunger for failure: reading and sharing postmortems. Postmortems are the richest training material a team has. Have the newcomer read the archive and discuss selected ones with a mentor: what was the trigger, how was it detected, what would you have done.

Disaster role-playing. Wheel of Misfortune: a senior engineer plays the system and the monitoring, the newcomer plays the on-caller, and an old incident is replayed step by step with the newcomer deciding what to do. The rest of the team watches and discusses. It is cheap, safe, and uncomfortable in the right way.

Break real things, fix real things. In a test environment, or in production with safeguards, inject a fault and have the newcomer find and fix it. This builds confidence that the tools work and that the person can use them under mild pressure.

Documentation as apprenticeship. The on-call checklist doubles as documentation. Having newcomers improve the documentation for the next person is both a learning exercise and a service to the team.

Shadow on-call early and often. First the newcomer shadows a primary, seeing pages and discussing responses without responsibility. Then the roles reverse: the newcomer is primary with the experienced engineer shadowing. Only after that do they go on-call alone.

On-call and beyond: rites of passage, and practising continuing education

Going on-call for the first time is a milestone and teams often mark it. But training does not stop there. Services change, incidents teach new lessons, and the on-caller of today is the mentor of tomorrow. The chapter recommends regular Wheel of Misfortune sessions for the whole team, rotation through related services' on-call, and pointing experienced engineers at the hardest training problem: writing down what they know so it can be taught.

Closing thoughts

Fast, safe training is a deliberate investment. A team that ramps new SREs in a few months with structure will have calmer on-callers, fewer repeat incidents, and a bench of people who can take on new services. A team that relies on trial by fire pays for it in every incident for years.