Chapter 3 said reliability targets should be deliberate. This chapter explains how to express them. Google uses three terms, in a strict order: an indicator is a measurement, an objective is a target for that measurement, and an agreement is a contract about what happens if the target is missed.

The chapter is the single most interview-relevant part of the book. Expect to be asked to define an SLI for a real service, pick a sensible SLO, and explain why averages are the wrong aggregation.

In one minute

• SLI (indicator): a carefully defined quantitative measure of some aspect of service, such as request latency, error rate, throughput, availability or durability.
• SLO (objective): a target value or range for an SLI, such as "99 percent of requests under 100 ms" or "99.9 percent availability over a quarter".
• SLA (agreement): an explicit or implicit contract with users that includes consequences of missing the SLO, usually financial. SRE does not write SLAs; it helps avoid triggering them. The rule of thumb: if there is no penalty, it is an SLO, not an SLA.
• Pick a handful of indicators, not dozens. Four to five is plenty. Choosing the right ones for the system type: serving systems care about availability, latency and throughput; storage about latency, availability and durability; big data about throughput and end-to-end latency. Every system cares about correctness.
• Aggregate as distributions and report percentiles (50th, 99th, 99.9th). Averages hide tail latency and bursts.
• Standardise definitions: aggregation interval, aggregation region, measurement frequency, which requests are included, how data is acquired, and data access latency.
• Choosing targets: do not pick based on current performance, keep it simple, avoid absolutes like "infinite scale", have as few SLOs as possible, and perfection can wait.
• SLOs set user expectations. Keep a safety margin by publishing a looser external SLO than the internal one. Do not overachieve, or users will depend on the better performance. Chubby deliberately takes itself down when it is too reliable.

Service level terminology

An SLI is a measurement. The common ones are request latency, error rate (as a fraction of requests), system throughput, and availability (fraction of well-formed requests that succeed, or for storage, durability). Ideally the SLI measures what the user experiences, but often that is hard, so a proxy is used, such as server-side latency rather than client-perceived latency.

An SLO is "SLI at or below (or within) some value". Some SLIs have natural targets (request latency), others do not (QPS is driven by users, not chosen). The SLO should be published so users know what to expect; an unstated SLO makes users assume far more than you intend.

An SLA adds consequences. Google Search has no SLA because there is no contract; Google for Work does. SRE's role with SLAs is to make sure the SLOs they contain can actually be met.

Indicators in practice

What you and your users care about. Do not monitor everything you can. Start from the user's perspective and choose a few SLIs that reflect it. A user-facing system wants availability, latency and throughput. Storage adds durability. Big data cares how much data can be processed and how long it takes. All systems care about correctness, though it is usually a property of the data and not an SRE responsibility.

Collecting indicators. Server-side metrics via Borgmon or Prometheus plus log analysis cover most needs. Some things, like a broken JavaScript bundle, are only visible from the client, so client-side probing or instrumentation is required.

Aggregation. A per-minute average request rate can hide a burst that saturated the service for seconds. A per-request latency average can look fine while 5 percent of requests are twenty times slower. Treat metrics as distributions and look at the tail. User studies suggest that a slow tail matters more than slightly higher median latency, so a 99.9th percentile target is often a better SLO than a median target.

Standardise. Agree once on aggregation intervals ("averaged over 1 minute"), regions ("all tasks in a cluster"), measurement frequency ("every 10 seconds"), scope ("all HTTP requests except health checks"), acquisition method ("measured at the load balancer") and data access latency ("time to first byte").

Objectives in practice

Define from what users care about, not from what is easy to measure. Starting from easy metrics yields SLOs that are met while users suffer.

Defining objectives. Use a clear form: "99 percent (over 1 minute) of Get RPCs complete in under 100 ms (measured across all backends)". It is fine to have several thresholds, such as 90 percent under 1 ms, 99 percent under 10 ms, 99.9 percent under 100 ms, to express a latency curve. Throughput and latency often trade off, so the SLO must specify the demand: "99.9 percent of writes under 100 ms at up to 1,000 writes per second".

Choosing targets. The chapter gives five rules. Do not pick a target based on current performance, because it locks you into a system that happens to behave that way. Keep it simple, since complicated aggregations hide behaviour. Avoid absolutes. Have as few SLOs as possible; if you cannot win an argument with an SLO, drop it. Perfection can wait; start loose and tighten as you learn.

Control measures. The loop is: monitor the SLIs, compare to the SLO, decide if action is needed, take it. For example, latency rises, dashboards show a Bigtable slowdown, you add capacity or shard.

SLOs set expectations. Users build on what they observe. Two tactics help. Keep a safety margin by advertising a looser SLO than the one you work to internally, so you can absorb a slow quarter. And do not overachieve. Chubby was so reliable that teams stopped handling its failures; Google now injects planned Chubby outages each quarter to keep dependents honest. Throttling also prevents users from relying on better-than-SLO throughput.

Agreements in practice

Writing an SLA is a business and legal job. SRE contributes by asking what happens if the SLO is missed and ensuring that the measurement infrastructure to prove compliance exists. Be conservative in what you promise to users, because it is far easier to loosen a promise than to tighten one after people have come to depend on it.