Testing is cheaper than paging. This chapter puts testing in SRE terms: a bug caught by a test has zero MTTR, so each test you add raises the mean time between failures of the running service. It then covers the traditional test pyramid, the production-only kinds of testing (configuration tests, stress tests, canaries), how to build a testing culture into a legacy system, and how to think about the confidence a test gives you in probabilistic terms.

The chapter also has a long worked example where an SRE team tests a tool that edits production configuration, chosen to show how you test the things that are most dangerous.

In one minute

• Testing bounds the cost of failure: a bug found in test has an MTTR of zero, and the more of those you find, the longer the MTBF of production.
• Traditional tests run offline. Unit tests check a function or class. Integration tests check assembled components, often with mocked dependencies. System tests check an undeployed whole: smoke (sanity) tests, performance tests, regression tests.
• Production tests run against the live service. Configuration tests compare the running config to the intended one. Stress tests find the limits (how full can the database get, how many QPS before latency breaks). Canary tests run the new version on a small fraction of tasks and watch it before proceeding.
• Build a test environment with hermetic, versioned builds, and a continuous build that runs the tests on every commit so that breakage is attributed to a single change.
• Testing at scale: test the tools (automation scripts and config generators are software and need tests too), test disaster recovery (does the backup restore?), and expect the speed of testing to bound release velocity.
• Think probabilistically: a release that has passed a given test suite has a certain probability of being fine. Testing more scenarios narrows the uncertainty; the goal is to make the error rate of releases small compared with the error budget.
• Production probes (black-box monitoring) are tests too, and the one kind that keeps running after launch.
• The worked example: a tool that changes config in a stored database is made safe by separating read from write, testing the read side exhaustively, and building a dry-run mode for the write side.

Types of software testing

The classic pyramid still applies. Unit tests are numerous and fast, and are the only ones you can run on every save. Integration tests catch mismatches between components. System tests validate end-to-end behaviour: a smoke test checks the trivial path to catch obvious breakage before spending time on more expensive tests, a performance test checks that latency and throughput have not regressed, and a regression test is a gallery of past bugs that must not return.

Production testing starts where the pyramid ends. Configuration tests verify that what is deployed is what source control says; a drift between the two is a common root cause. Stress tests find breaking points before traffic does. Canaries deploy a new version to a small fraction of the fleet and compare its metrics with the rest; they are the production equivalent of a controlled experiment and are the standard gate on a Google rollout. The chapter also notes that rollouts should be ordered: canary, then a datacenter, then the fleet, with the time at each step chosen so that a problem would be visible before proceeding.

Creating a test and build environment

For a new project, write tests as you go. For an inherited one, the chapter advises prioritising: identify the most critical components and the most frequently broken ones, and test those first. Documentation about what the system does is itself a kind of test, because it tells you what the tests should check. A hermetic build and a continuous integration system are prerequisites, because a flaky environment makes every test result ambiguous.

Testing at scale

At scale, the artefacts to test multiply. Configuration files generated by tools need tests of the generator. Automation that acts on production needs tests that it refuses to do dangerous things. Disaster recovery has to be exercised, not assumed. The chapter introduces the idea that for a given release, the test coverage defines an envelope within which you are confident the release behaves, and anything outside the envelope is an unknown.

The speed of testing matters too. Tests that take hours mean a release process that takes days, which lowers velocity and increases the size of each release, which increases risk. Investing in fast tests is a reliability investment.

Testing deployed configuration and the probabilistic view

The chapter steps back to define what a test buys you. A system with a certain set of tests has a certain probability of a bad release. If the SLO allows a certain number of bad releases a quarter, you can estimate whether the current test regime is adequate and where to add tests to buy the biggest reduction in that probability. Releases and tests should be seen as a loop: a production incident shows a scenario the tests did not cover, and a new test is added so that class of incident has zero MTTR from then on.

Expect testing to fail

Test flakiness, environments that diverge from production, and tests that nobody updates will all happen. Treat test infrastructure like production infrastructure: monitor it, fix it, and have someone own it.

Case study: testing a production configuration tool

SREs built a tool to modify a configuration database that drives production. Because a mistake would affect everything downstream, they structured the tool so that the read and parse logic was isolated and covered by exhaustive unit tests, the transformation logic was testable with synthetic inputs, and the write path had a dry-run mode that printed the diff without applying it. Integration tests ran the tool against a copy of real data. They added a sanity check that refused to apply a change touching more than a configured fraction of entries. This mirrors the lesson from chapters 7 and 13: the most dangerous software deserves the most tests and the most conservative defaults.

Conclusion

Testing is how SRE turns the vague wish for reliability into something that can be measured and improved before users are involved. Every incident should leave behind a test, and every test suite should be fast enough that it does not slow the release cadence that the error budget allows.