A single incident gets a postmortem. Patterns across hundreds of incidents need a different tool. This short chapter describes Outalator, Google's system for collecting every alert sent to every on-call, grouping them into incidents, tagging them, and producing reports. It is the data layer that makes the on-call balance in chapter 11 measurable rather than anecdotal.

Outside Google, the equivalent is whatever your paging tool exports plus some tagging discipline. The point of the chapter is what you can learn once the data exists.

In one minute

• Outalator receives a copy of every alert notification (pages, emails, tickets), stores them, and lets a human group related alerts into one incident.
• Benefits of grouping: a single incident with many alerts is one event in reports rather than fifty, the on-call sees one thing to resolve, and the alert queue is a timeline for the postmortem.
• Tagging incidents with free-form tags (cause:network, action:rollback, bogus for a false alert) lets a team query which kinds of problems are costing them, and which alerts never need action.
• Annotations and escalation: engineers add notes to an incident as they work, and can escalate it to another team's queue so that handoffs leave a record.
• Reporting: per-team and per-service counts of pages and incidents over time. This is how a team proves that it is over the two-incidents-per-shift limit, or that a noisy alert should be deleted.
• Unexpected uses: finding shared root causes across teams, spotting alerts that fire together and should be merged, and a searchable history to answer "have we seen this before" during an incident.

Escalator and Outalator

Google's first tool, Escalator, made sure a page was acknowledged: if the primary did not respond within a configured time, it paged the secondary. Outalator extended this to store the alerts, show them in a queue per team, and let the on-caller mark them as resolved, grouped, or false. Multiple queues exist, one per team or service, and an incident can be moved between them.

Aggregation

Real incidents produce many alerts: one per datacenter, one per symptom, one per dependent service. Without aggregation each is a separate thing to acknowledge and each inflates the incident count. In Outalator, the on-caller selects the related alerts and merges them into one incident, which then has one timeline and one resolution. This is cheap to do during the incident and it makes the historical record far more useful.

Tagging

Tags are free text, lightly structured by convention, such as cause: and action: prefixes. A tag like bogus marks an alert that fired for no good reason, and over a quarter the count of bogus tags per alert rule is a direct list of rules to fix or delete. Other tags record the affected service, whether the alert was actionable, and whether a postmortem was written.

Analysis

The reports answer the questions a team lead actually asks. How many pages per shift? Which rules produce the most pages? What fraction of pages were actionable? How did the on-call load change after last month's cleanup? The same data lets an organisation compare teams and notice that one is drowning. Because Outalator stores the alert content, engineers can also search for a symptom during a live incident and find how it was resolved last time.

Unexpected benefits

Once all teams' incidents live in one system, cross-team patterns appear: a network issue that paged five teams shows up as five incidents at the same minute, which argues for a shared alert and a shared postmortem. Alerts that always fire together are candidates for merging into one rule. Managers get an objective view of operational load that is independent of how loudly a team complains. And incident data becomes an input to decisions about where to spend engineering time, which is what the 50 percent rule needs in order to be enforceable.