Most people assume that more reliability is always better. This chapter argues that past a certain point it is not: each extra nine costs exponentially more, the user cannot perceive the difference, and the engineering effort would have bought features instead. SRE's job is to manage risk, not to eliminate it.

The chapter then shows how to pick a target, how to measure against it, and how to turn the gap between the target and perfection into a budget that two teams with different incentives can share.

In one minute

• Unreliable systems lose user trust, but reliability has a cost in redundant hardware, opportunity cost of engineering time, and slower launches. Beyond the point where users notice, more nines are pure cost.
• Google aims for a service to be reliable enough, then stops. The target is set by the business, not by engineering pride.
• Time-based availability: uptime divided by total time. Useful when a service is either up or down. Breaks when the service is globally distributed and always partially up.
• Aggregate availability: successful requests divided by total requests. Works for distributed services and for things that are not request-driven, by picking the right unit of work.
• Consumer services pick a target by asking: what level of availability do users need, what do they do when the service is down, how do they use it, and what does the alternative cost.
• Infrastructure services have conflicting clients (some want low latency, some want throughput) and may need to expose multiple service levels at different prices, as Bigtable does with its low-latency and batch clusters.
• An error budget is 100 percent minus the SLO. It is spent on launches, experiments and planned maintenance. When it runs out, launches stop until the service recovers.
• The budget turns the dev versus SRE argument into a shared decision: both teams want to spend it all, and neither wants to overspend.

Managing risk

Reliability is not free. Redundant machines, replication, and engineers who build risk-reducing features rather than user-visible ones all cost money. Google therefore treats reliability as a continuum and tries to place each service at the right point. Moving from 99.9 percent to 99.99 percent is roughly ten times harder and ten times more expensive, while the user's own phone, Wi-Fi and ISP are already worse than either.

Measuring service risk

A target you cannot measure is a wish. The traditional metric is time-based: 99.99 percent availability allows about 52 minutes of downtime a year. For a globally distributed service that is always serving some fraction of traffic, "down" is not well defined, so Google uses request success rate instead. A 99.99 percent target becomes "no more than one failed request in ten thousand", over a quarter. This also works for batch, pipeline and storage systems once you define a unit of work (a record, a byte, a job).

Risk tolerance of services

Consumer services. The product manager and SRE together consider the required availability, whether different failures have different impacts (a short full outage versus a long partial one, as with Ads frontends versus Ads reporting), what the service costs, and what other metrics like latency matter. The chapter's example is that if raising availability by one nine would increase revenue by 0.01 percent, that bounds how much you should spend to get it.

Infrastructure services. Bigtable is used by low-latency serving systems and by throughput-hungry batch jobs. One cannot be made happy without making the other unhappy, if they share a cluster with one SLO. Google's answer is to partition: offer low-latency clusters with deliberately kept spare capacity, and high-throughput clusters with much higher utilisation and less consistent latency, at different prices. Let the client choose.

Latency and cost. Latency goals also differ. An interactive request has a tight budget; an asynchronous or batch request can wait. The cost of serving each is different and should be billed accordingly.

Error budgets

Product developers are rewarded for velocity, SREs for reliability. Without an agreed number, every disagreement about software fault tolerance, testing rigour, push frequency or canary duration becomes a negotiation. The error budget replaces negotiation with arithmetic:

1. Product management sets an SLO for how reliable the service needs to be each quarter.
2. A neutral monitoring system measures actual uptime.
3. The difference is the remaining budget.
4. As long as budget remains, new releases can be pushed.

If the budget is exhausted, releases are frozen until the service is within SLO again, except for high-priority fixes. Both teams now own the same goal: ship as much as possible without blowing the budget. Developers start to care about canaries and tests because they protect their own release velocity. SREs stop being the department of no. A network outage that consumes the budget is treated like any other spend, which pushes the team to reduce that risk too.

The chapter closes by noting the budget is a tool for balancing, not a target to hit exactly. Being far under budget is itself a signal that the team is being too conservative and could ship faster.