Monitoring is how a service tells its owners that something is wrong before the users do. This chapter is a philosophy of monitoring rather than a tool guide: what to collect, what to page on, and how to keep the whole thing from collapsing under its own weight. The four golden signals and the rules for a good page are the two things people quote from it.

Google learned these lessons by replacing hand-written alerting rules and giant checklists with a smaller set of carefully chosen symptoms. The chapter also includes two worked examples, Bigtable SRE and Gmail, where over-alerting had to be deliberately dialled back.

In one minute

• Monitoring is collecting, processing, aggregating and displaying real-time quantitative data about a system.
• White-box monitoring uses metrics exposed by the system's internals (logs, handlers, profiling). Black-box monitoring tests externally visible behaviour as a user would see it.
• Outputs are alerts (pages that need action now, tickets, email) and dashboards. A root cause is a defect which, if fixed, gives confidence the event will not recur in the same way.
• Reasons to monitor: long-term trend analysis, comparison over time or experiment groups, alerting, dashboards, and ad hoc retrospective debugging.
• Rules for pages: every page should be actionable, require intelligence, and be urgent. Pages that are a mechanical reaction should be automated. Pages for things that can wait should be tickets.
• Distinguish symptoms (what is broken, as the user sees it) from causes (why). Page on symptoms. Use causes as supporting information on dashboards.
• The four golden signals: latency, traffic, errors, saturation. Monitor all four and page when any is bad.
• Worry about the tail: use histograms and bucketed latency counts rather than averages.
• Pick resolution per metric: CPU every second is wasteful, disk fullness every minute is enough. Sample frequently, aggregate for storage.
• Keep it as simple as possible, no simpler. Rules that catch real incidents should be simple, predictable and reliable. Prune what is rarely used.

Why monitor

Beyond the obvious ("tell me when something breaks"), monitoring supports analysing long-term trends (how fast is the database growing, how fast is daily active user count rising), comparing over time (is this version faster than last week's), building dashboards for the golden signals, and ad hoc debugging during incidents. Alerting is only one of five jobs, and an alerting system designed without the others in mind tends to be poor at all of them.

A monitoring system should never require a human to interpret any part of the alerting domain. Interpretation is the software's job; the human is told only when action is needed.

Setting reasonable expectations

Monitoring complex systems is a significant engineering investment. Google's SRE teams with ten to twelve members usually have one or two people whose main job is monitoring. Google moved away from systems that try to infer causality (magic dependency hierarchies) and toward simpler, faster ones. Dashboards and alerting rules are kept simple so that when a page fires at 3 a.m., the on-call can understand it immediately.

Symptoms versus causes

"What's broken" is a symptom ("I'm serving HTTP 500s"). "Why" is a cause ("the database is refusing connections"). The same cause may show up as several symptoms and vice versa. Pages should be driven by symptoms because they measure user impact; causes belong on dashboards to speed up diagnosis. Black-box monitoring is symptom-oriented and catches real, active problems; white-box monitoring gives you imminent problems and masked failures (such as retries hiding errors) and is required for debugging. Use black-box for paging on things that affect users, and white-box for everything else.

The four golden signals

Latency: time to serve a request. Separate the latency of successful requests from that of failures; a fast 500 is not good news. Traffic: demand, usually requests per second, or sessions, or transactions, chosen per system. Errors: rate of failed requests, explicit (500s), implicit (200s with wrong content) or by policy (anything over one second is an error). Some errors are only visible end to end. Saturation: how full the service is, measured against its most constrained resource. Many systems degrade before they hit 100 percent, so set a utilisation target. Saturation can also be predictive ("your database will fill in four hours").

Worrying about the tail

If a service reports mean latency it hides the slow tail. The cheap fix is a histogram: count requests into buckets (0 to 10 ms, 10 to 30 ms, 30 to 100 ms, and so on) and alert on the bucket distribution. The chapter also warns about a related trap: a metric collected at one-minute granularity can hide a one-second CPU spike that caused tail latency.

Choosing resolution and keeping it simple

Per-second CPU load is cheap to sample and expensive to store at full resolution. Sample often, then aggregate for retention. Keep the number of rules small and make each rule simple, predictable and reliable; prune data collection, aggregation and alerting that is rarely used. Google's experience is that a monitoring system that is too clever becomes fragile, and that its failure is then its own outage.

Tying it together

The chapter's questions for a new rule: does it detect an otherwise undetected condition that is urgent, actionable and user-visible? Can I ever ignore it, and if so, how do I avoid that scenario? Does it definitely indicate harm, or could it be benign? Can I act on it, and is the action urgent? Could it be automated? Is it short-term relief or a long-term fix? And is the on-call being paged by several alerts for the same problem?

Pages should be rare, actionable and urgent. If they are not, the on-call learns to ignore them, and then the monitoring system has failed at its only critical job.

Monitoring for the long term

The Bigtable SRE example: an over-reliant alerting setup meant on-callers were paged on a mix of symptoms and causes, and had learned to live with constant pages. The fix was to temporarily loosen the SLO, remove emails and tickets that were not actionable, and point engineering effort at the underlying performance problem instead. The Gmail example: a small fraction of tasks failed in a way that needed manual intervention, so the team wrote a tool to do the intervention and considered the trade-off between short-term pages and long-term automation. In both cases, the choice to take a short-term hit in page quality to fund a permanent fix is called out as the correct one.