A launch is the moment a product meets real traffic, and it is where most of the risks in this book meet at once. Google created a specialist SRE role, the Launch Coordination Engineer (LCE), to shepherd launches through a checklist that captures everything the organisation has learned. This chapter describes the role, the checklist, and the techniques the checklist recommends, with the Shakespeare service as the example.

For interview purposes, the checklist sections are a good framework for "how would you launch this service" questions.

In one minute

• A launch is any new code or significant change that is visible to users. At Google's scale there are many per day, and the LCE team exists to make them reliable without becoming a bottleneck.
• LCEs are SREs who consult on launches: they know the infrastructure, the common failure modes, and the other teams involved. Their job is lightweight, robust, thorough, scalable and adaptable review, and they act as a liaison between the product team and SRE.
• The launch checklist has questions and action items in each section: architecture and dependencies, integration (DNS, load balancing), capacity planning, failure modes, client behaviour (abusive clients, retries), processes and automation, development process, external dependencies (vendors, partners), and rollout planning.
• Techniques for reliable launches: gradual and staged rollouts (by percentage of users, by region, by canary), feature flag frameworks to turn features on and off without a release, dealing with abusive client behaviour (the service must be able to shed or throttle a misbehaving client), and overload behaviour and load tests before launch.
• Launch process evolution: the checklist must be curated, or it grows until nobody reads it. Google kept it short by removing items as the infrastructure made them unnecessary, and by making approval criteria explicit.
• Problems that arise: scalability of the LCE team, differing requirements across product areas, and the risk of the process becoming a rubber stamp.

Launch coordination engineering

LCE is a consultancy model. A product team planning a launch fills in a questionnaire; the LCE reviews it, holds a meeting, and produces action items. The LCE does not approve or veto; they ensure the right questions are asked and that other teams who will be affected (network, storage, abuse) know in advance. Because LCEs see many launches, they also spread knowledge of what went wrong last time.

Setting up a launch process

The criteria for a good process: lightweight so small launches are not slowed; robust so that it catches obvious errors; thorough so that important details are considered; scalable so that it handles many launches; adaptable so that it fits common and unusual launches alike. Google balances these with simplicity (a few clear steps), a high-touch approach for the risky launches, and fast common paths for the routine ones.

The launch checklist

The checklist is a set of questions, each followed by an action. Examples: Architecture: what is the request flow? Does it use the standard infrastructure? Integration: is there a new DNS name, does it need load balancing configuration, are there new external endpoints? Capacity: what is the expected traffic at launch and in six months, has it been load-tested, what is the plan if demand is ten times the estimate? Failure modes: what happens if each dependency fails, is there a single point of failure, is there a tested degraded mode? Client behaviour: can clients create a flood by retrying, and are there backoff and jitter rules? Processes and automation: who is on-call, is there a runbook, are there automated rollbacks? Development process: has the code been reviewed, is it under continuous integration, is config in version control? External dependencies: vendor SLAs, partner readiness, legal review? Rollout planning: the stages, the metrics to watch at each stage, and the criteria for proceeding or rolling back.

Selected techniques for reliable launches

Gradual and staged rollouts. Launch to a small percentage of users or to a single region, watch the metrics, and increase. Combine with canaries for the binary and A/B measurements for the feature.

Feature flag frameworks. Separate deploying code from enabling it. Flags let the team enable a feature for internal users, then one percent, then everyone, and turn it off instantly if something goes wrong, without a rollback. They also let the team cheaply run experiments.

Dealing with abusive client behaviour. Clients that retry aggressively, that poll too often, or that were written against a mock can take down a service at launch. The service should be able to identify and throttle them, and client libraries should have sane defaults.

Overload behaviour and load tests. Before launch, load test to find the capacity and the behaviour past it. Confirm that the service degrades gracefully and that the monitoring will show the overload.

Development of LCE

The process had to evolve. Early checklists became long, and teams began to see them as bureaucracy. Google pruned questions that infrastructure improvements had made moot, moved routine checks into automated tooling, and kept the human review for judgement calls. The LCE team also had to scale: with many product areas, specialists developed per area, and a training programme produced new LCEs. The tension the chapter admits is between keeping the process relevant and avoiding it becoming a rubber stamp, which is resolved by continuous curation and by LCEs owning the outcome rather than the paperwork.

Conclusion

Launches are where the principles of the book become practice. The LCE model turns them from one-off events into a repeatable process, and the checklist is a living summary of the lessons from every previous launch.