Availability is about whether users can reach the service. Integrity is about whether the data is still there when they do. The chapter argues that from the user's point of view a service with intact data that is temporarily down is fine, and a service that is up but has lost their data is a catastrophe, so integrity deserves at least as much engineering. The provocative framing: nobody wants backups; what everyone wants is restores.

The two case studies, a Gmail outage in 2011 that was recovered from tape, and a Google Music deletion in 2012 that affected a small fraction of users, show the defence-in-depth approach working under real conditions.

In one minute

• Users define integrity as "the data I wrote is the data I read, forever". A momentary unavailability is forgivable; permanent data loss is not. Google's internal target for some data is effectively eleven nines of durability.
• Delivering a recovery system, not a backup system: a backup that has never been restored is unverified. Design, measure and test restores, with a recovery time objective (RTO) and recovery point objective (RPO).
• Twenty-four combinations of failure: a root cause (user action, operator error, application bug, infrastructure defect, hardware fault, site disaster) times a scope (wide or narrow) times a rate (big bang or slow and gradual). Each combination needs a different defence, and the slow, narrow, application-bug combination is the hardest because it is detected late.
• Challenges: replication is not a backup (it faithfully copies the deletion), the sheer size of data makes full copies slow and expensive, consistency across many services is hard, and retention of backups has legal and cost limits.
• Defence in depth, three layers: 1) soft deletion (and lazy deletion) so user and application errors are recoverable from a trash state for weeks; 2) backups and their restores, tiered by speed and cost and tested regularly; 3) early detection, with out-of-band data validation that compares data across sources and alerts on inconsistency.
• Recovery must be rehearsed. Google runs the pipelines that validate restores continuously, so the question is never "do backups work" but "when did they last work".
• The 24 combinations are too many to defend against individually. Instead apply the SRE principles: defence in depth, trust but verify, hope is not a strategy, and treat data integrity as a service with its own SLOs.
• Case studies: Gmail 2011, a software bug deleted data for a small fraction of accounts, restored from tape over several days, with the lesson that the recovery procedure had to scale. Google Music 2012, a privacy-protecting deletion pipeline bug removed tracks for a fraction of users, detected by validation, restored from offline backups in two waves, with the lesson that detection and parallel restore pipelines are what bound the damage.

Data integrity's strict requirements

Integrity is a property that users assume. Google's position is that the data must survive bugs, operator error, hardware failure and disasters, and must be recoverable in a time the user considers acceptable. The chapter distinguishes availability of data (can I read it right now) from integrity (is it correct and complete), and notes that durability numbers like eleven nines only make sense with a recovery system behind them.

Choosing the data integrity strategy is a product decision with a cost. Different data has different value, so the recovery objectives differ: a user's email needs a quick and complete restore; aggregate analytics can tolerate loss.

Google SRE objectives in maintaining data integrity and availability

The first objective is to have a working recovery system, measured by restore success rate and restore time, not by backup completion. The second is to anticipate the failure modes, which the 24-combination grid organises. The third is to accept that the hardest cases are the slow leaks: an application bug that corrupts one record in a million for months before anyone notices, because by the time it is found the backups that predate it may have expired.

Challenges listed: the cloud environment uses many layers of storage, each with its own deletion semantics and replication; replication and redundancy protect against hardware loss but not against bugs or operators; APIs let users and other services delete data at scale; and data volume means a full restore takes longer than the SLO allows unless it is designed for parallelism.

How Google SRE faces the challenges of data integrity

The 24 combinations. Root causes are user action, operator error, application bug, infrastructure defect, hardware fault, and site disaster. Scope is wide or narrow. Rate is big bang or slow and gradual. A wide, big-bang, site-disaster loss is dramatic but easy to detect and recover from a replica. A narrow, gradual, application-bug loss is what breaks naive strategies.

First layer: soft deletion. When a user deletes something, mark it deleted and hide it; purge it only after a delay (Gmail's trash, for example). For applications deleting on behalf of users, use lazy deletion with a further delay. This turns the most common causes, user mistake and application bug, into recoverable events without any restore.

Second layer: backups and their related recovery methods. Backups are the tool for everything soft deletion misses. Design them around the restore: how much data can you lose (RPO), how long can restore take (RTO), and how far back do you need to go. Google uses tiers: fast online replicas for recent data, slower and cheaper storage for older snapshots, and offline tape for the last line of defence. Critically, restores must be exercised regularly, and recovery time must be measured at scale, because restoring one account is different from restoring millions.

Third layer: early detection. Out-of-band validation pipelines read the data and check it against invariants or against other copies. Gmail runs validators that compare metadata with message storage. The goal is to shrink the gap between corruption and detection, because the shorter the gap, the more likely a clean backup still exists. Validators themselves need to be low-noise, or they will be ignored, and they need to be updated when the application changes.

Knowing that data recovery will work. Continuously run restore tests. Alert when they fail. Treat the recovery system as production with its own SLO.

Case studies

Gmail, February 2011. A bug in a storage update deleted data for about 0.02 percent of users. Soft deletion did not help because the bug operated below that layer. Replication did not help because it replicated the deletion. Tape backups did help: Google restored the affected accounts over several days, which was far longer than anyone wanted, and the postmortem focused on making the restore pipeline parallel and faster. Lesson: the last layer of defence was there, and it was the only one that worked, and its speed had never been tested at that scale.

Google Music, March 2012. A privacy-driven pipeline that purged deleted tracks had a bug and removed tracks that users had not deleted, for a small fraction of users. Detection came from users and from validators. Restoration had two phases: an initial recovery from near-line copies for most tracks, then a larger effort to restore the remainder from offline tape, which involved bringing back tapes from offsite storage. The lesson was that detection latency and restore parallelism bound the damage, and that the pipeline that deletes data permanently deserves the same scrutiny as the one that stores it.

General principles of SRE as applied to data integrity

Beginner's mind: do not assume you understand how your system can fail; the 24 combinations are a checklist to argue against. Trust but verify: validate the data that other systems hand you. Hope is not a strategy: test restores. Defence in depth: soft deletion, backups and validation together, because each alone has a hole.

Conclusion

The service is only as good as its data. Build recovery as a first-class system, assume every layer will fail, and know how long restore takes before you need it.