Release engineering is a distinct discipline at Google: specialists who understand source control, compilers, build configuration, test automation, packaging and deployment. This chapter describes how they work with SRE and product developers to make releasing safe, repeatable and fast. The goal is that any engineer can release, but the system guarantees that the release was built from the right code, tested, and deployed in a controlled way.

Most of the specifics are about Google's internal Rapid system, but the four philosophies and the branching strategy are general enough to describe any good release pipeline.

In one minute

• Release engineers define the whole path from a commit in the source repository to a running binary: build rules, testing, packaging, deployment.
• Self-service: teams define and run their own releases. Release engineering provides the tooling, not a gatekeeper.
• High velocity: build frequently so each release has fewer changes to debug. Some teams build hourly and pick a release; some deploy every green build (push on green).
• Hermetic builds: the build is insensitive to what is installed on the build machine. Same revision anywhere produces the same binary. Compilers, libraries and tools are pinned and checked in.
• Enforcement of policies and procedures: gated operations with ACLs on approving code changes, specifying release actions, creating a release, approving the integration proposal, deploying, and changing build configuration.
• Branching: a release is cut from mainline at a specific revision. Fixes are made in mainline then cherry-picked into the release branch. Release branches are never merged back.
• Testing: continuous testing on mainline, plus the full test suite at the chosen revision before a release is cut.
• Packaging: Midas Package Manager (MPM) packages are named, versioned and signed, and labelled (dev, canary, production).
• Rapid is Google's release automation: blueprints define the workflow, which runs integration, build, test, package, and deploy stages, with Sisyphus or similar handling rollout.
• Configuration management has four options, all of which keep config in source control with strict review.

The role of a release engineer

Release engineers work with SREs on best practice for build and release, with developers on build rules and test strategy, and own the tools that make it happen. Google has them because getting releases wrong is one of the main causes of outages, and because the problem is big enough to deserve specialists. A product's build and release workflow is defined collaboratively, but it is the release engineering team that keeps it consistent across hundreds of products.

Philosophy

Self-service. At Google's scale a central team pushing every release would be a bottleneck, so each product team owns its release process using shared tooling. Releases are truly automatic, and a human intervenes only when something goes wrong.

High velocity. Frequent releases mean fewer changes between versions, so when something breaks there is less to bisect. Teams pick a cadence: hourly builds with a release chosen from them, or push on green where every build that passes all tests is deployed.

Hermetic builds. Builds depend only on known versions of compilers, libraries and build tools, all of them in the repository. Two people building the same revision on different machines get identical outputs. This also lets an old release be rebuilt exactly to make a security fix.

Enforced policies. Gated operations require approval through ACLs, and every action is logged, producing an audit trail of who did what in the release.

Continuous build and deployment

Building. Blaze (open-sourced as Bazel) builds everything; targets declare their dependencies, and the build identifies itself with a build ID, timestamp and revision.

Branching. Mainline receives all development. For a release, a branch is cut at a revision. Bugs found afterwards are fixed on mainline and cherry-picked into the branch. Release branches are never merged back, which prevents accidental inclusion of unrelated changes.

Testing. A continuous test system runs unit tests on every mainline commit, and the full suite is run again at the revision chosen for release so the release engineer knows exactly what passed.

Packaging. MPM builds a package from a Blaze rule, names it, versions it, and signs it. Labels like canary or production are attached and can be moved between versions to signal which should be installed.

Rapid. A blueprint written in a domain-specific language configures the release: when to branch, what tests to run, how to package, where to deploy. Rapid runs these stages as a pipeline, with parallel builds and workers, and rolls out through a deployment system. Rollout can be gradual (canary, then a fraction, then everything) with automatic stop on monitoring signals.

Configuration management

Configuration changes are a major outage source, so Google treats them like code. Four strategies: keep config in mainline and release it with the binary; keep config in a separate package versioned together with the binary (same MPM label, deployed together); keep config in a separate package with its own release cycle; or read config from an external store such as Chubby, Bigtable or a file in Colossus, so it can change without a release. Each has trade-offs between flexibility and the risk of a mismatch between config and code. Whatever the choice, every change must be reviewed and traceable.

Conclusions

The practices scale down as well as up. Start release engineering early in a product's life, because retrofitting a safe process onto a shipping system is much harder. Even a small team benefits from a hermetic build, from a release branch that is never merged back, and from treating configuration as code.