Chapter 6 said what to monitor; this chapter shows how Google actually does it. Borgmon was built alongside Borg to replace per-check scripts with a system that collects numeric time series from every task, computes derived series with rules, and fires alerts when rules say so. Prometheus is its direct descendant, so the model will be familiar: metrics as labelled time series, a query language over them, and alerting as just another rule.

The chapter is dense with Borgmon syntax. The important part is the model: variables exported by the binary, scraped by a collector, stored as time series, aggregated across tasks with rules, and alerted on with duration conditions.

In one minute

• Traditional monitoring runs a check script per condition and alerts on failure. It does not scale and gives no history. Borgmon treats monitoring as the collection and processing of time series.
• Every Google binary exposes its exported variables (varz) on an HTTP handler as name value pairs. Borgmon scrapes them on a schedule and the common library instruments HTTP servers, RPC servers and so on for free.
• Storage is an in-memory time-series arena, a fixed-size table of (timestamp, value) points per time series, checkpointed to disk and shipped to the Time-Series Database (TSDB) for long-term history.
• A time series is identified by a set of labels: var= (the variable name), job=, service=, zone= and the task instance. A query returns a vector of time series matching the labels.
• Rules are the programming language. They compute new time series from existing ones, such as a rate over 10 minutes and a sum across tasks. Rules are evaluated in a loop at the scrape interval.
• Counters (monotonically increasing) are preferred over gauges because a missed scrape loses nothing from a counter but loses a sample from a gauge.
• Alerting rules are rules whose output is a boolean plus a for duration, so a brief blip does not page. Alerts go to the Alertmanager, which deduplicates, groups, routes and inhibits.
• Scale by sharding: datacenter-level Borgmons scrape tasks and aggregate; a global Borgmon scrapes the datacenter Borgmons, so a few machines cover a planet.
• Prober does black-box monitoring by sending synthetic requests and exporting the results as variables so Borgmon can alert on them.
• Rules must be maintained like code: use templates, test them, and keep per-service specifics in separate files.

The rise of Borgmon

Before Borgmon, each team had its own scripts. Borgmon made monitoring a shared service: common instrumentation in libraries, common collection, and a common rule language. A key design decision was that Borgmon never knows what a variable means; it just collects numbers. Meaning is supplied by rules. This decoupling let one monitoring system serve very different services.

Instrumentation and collection

A binary exports variables through an HTTP handler, formatted as plain text. The application code increments a counter or sets a gauge; the library does the rest. Borgmon reads a list of targets from BNS, scrapes each one every few seconds, and records both the values and whether the scrape succeeded. Scrape failures are themselves useful signal: a task that stops answering its varz endpoint is probably unhealthy.

Storage and the time-series arena

The arena is a block of memory holding the most recent points for each series; a datacenter-sized Borgmon holds about twelve hours of one-minute data in roughly 17 GB. Older data goes to TSDB on disk. Because the arena is in memory, rule evaluation over millions of series is fast enough to run every few seconds.

Labels and vectors

Series are named by labels, which Borgmon attaches from the target (job, instance, zone) and from the variable itself (map-valued variables export one series per key). A rule's input is a vector of series that match a label set, so http_requests across a job gives one series per task and sum by job collapses them into one. This is exactly the Prometheus data model.

Rule evaluation

Rules are expressions over vectors, evaluated repeatedly. Typical pipeline: compute per-task rate over 10 minutes (rate()), sum across tasks to get the job's QPS, compute the error ratio as errors divided by total, and compare to a threshold. Intermediate series are stored just like scraped ones, so dashboards can plot them. Rules are composable but can become slow or circular, so the chapter recommends keeping them simple and testing them.

Alerting

An alert rule has a condition and a for duration. The alert enters pending when the condition becomes true and fires only once it has stayed true for the duration, which filters flapping. Fired alerts carry labels and templated annotations and are sent to Alertmanager, which groups related alerts, suppresses an alert when a more severe one is active (inhibition), and routes by labels to pager, chat or ticket. Alertmanager also handles silences for known conditions and maintenance.

Sharding the topology

One Borgmon cannot scrape every task on the planet. Google runs a Borgmon per datacenter (or per cluster) that scrapes local tasks and exports aggregated series, and a global Borgmon that scrapes those. Alerts that need a global view, such as total error ratio across all regions, live at the global level; task-level alerts stay local. Borgmons can also scrape each other for redundancy.

Black-box monitoring

Borgmon only knows what the binary tells it. Prober sends real requests to the service from outside, checks the response, and exports success and latency as variables. Borgmon scrapes Prober, so the same alerting pipeline covers end-to-end user-visible failures.

Maintaining the configuration

Rules are code. Google separates generic rules (how to compute error ratio for any HTTP server) from service-specific configuration (which jobs, which thresholds), uses macros to avoid repetition, and tests rules against synthetic data. The result is a monitoring setup that grows by adding a few lines of config for each new service instead of copying hundreds.

Ten years on

Borgmon's model of labelled time series plus a rule language proved durable. The chapter notes its influence on Riemann, Heka, Bosun and Prometheus, and Prometheus is now the standard way to get the Borgmon experience outside Google.