Before describing how Google runs services, the book has to describe what it runs them on. This chapter is a glossary of Google's production environment at the time the book was written. Most of the names are Google-internal, but each one has an open-source or cloud equivalent that interviewers will expect you to map to.

Read it as a map of the layers: hardware, cluster management, storage, networking, and the shared software infrastructure that lets a small team run a globally distributed service.

In one minute

• Machine means hardware or a VM. Server means a piece of software that implements a service. A server runs on machines, and machines are interchangeable.
• Physical hierarchy: tens of machines in a rack, racks in a row, rows form a cluster, a datacenter holds several clusters, nearby datacenters form a campus.
• Within a datacenter, Jupiter fabric gives a Clos network with 1.3 Pb/s of bisection bandwidth, so any machine can talk to any other at full speed. B4 is the backbone between datacenters.
• Borg is the cluster manager (ancestor of Kubernetes). Users submit jobs made of tasks. Borg picks machines, starts tasks, restarts them when they die, and relaunches them elsewhere if a machine fails. Tasks are found through the Borg Naming Service (BNS).
• Storage: local disk and flash sit at the bottom, D is a file server per machine, Colossus (successor to GFS) is the cluster-wide replicated filesystem, and Bigtable, Spanner and Blobstore are the databases built on top.
• Networking: a global OpenFlow-based software-defined network, Bandwidth Enforcer for bandwidth allocation, and GSLB for load balancing at DNS, user-service and RPC levels.
• Shared systems: Chubby for locking and leader election (Paxos, five replicas), Borgmon for monitoring, Stubby for RPC with protocol buffers.
• The Shakespeare example service shows how all of it combines: a MapReduce indexing job, a Bigtable index, frontends and backends in several datacenters, GSLB directing users, and Borg keeping the right number of replicas alive.

Hardware

Google builds its own hardware and arranges it so that failure is expected and boring. Because the network within a datacenter is non-blocking, locality rarely matters, and Borg can place a task anywhere. Datacenters have their own power and cooling, and a campus groups them geographically.

Managing machines

Borg is the system that turns a pile of machines into a pool. A job describes how many copies of a binary to run and what resources each needs. Borg finds room, starts the tasks, and reschedules them when they die or when a machine needs maintenance. Because tasks move, their addresses change, so every task is registered in BNS under a name like /bns/<cluster>/<user>/<job>/<task>, and clients look up the current address through that name. Borg also enforces resource limits and kills tasks that exceed them, which is a recurring source of incidents elsewhere in the book.

Storage

The layers stack up. D serves the disks of one machine. Colossus replicates data across a cluster and provides the filesystem that everything else uses. On top of Colossus, Bigtable is a sparse, distributed, persistent multidimensional sorted map (petabytes, eventual consistency across datacenters). Spanner adds SQL-like semantics and real consistency worldwide. Blobstore handles large unstructured objects. Each layer is a separate service with its own SRE team, which is why the book talks so often about dependencies between services.

Networking

The datacenter network and the backbone are controlled centrally with OpenFlow. Bandwidth Enforcer allocates bandwidth between services so one cannot starve another. GSLB answers three questions: which datacenter should this user's DNS lookup point to, which service in that datacenter should take the request, and which backend task should handle an RPC. Chapters 19 and 20 go into these in detail.

Other system software

Chubby provides a filesystem-like interface for locks and small amounts of strongly consistent data. It runs Paxos across five replicas and is the thing most Google services use for leader election and for discovering the current master. Borgmon scrapes metrics from every server, aggregates them, and feeds dashboards and alerts (chapter 10). All inter-process communication uses Stubby, an RPC framework over protocol buffers that serialise more compactly than XML and are the schema language for most Google data.

The software environment

Google services are multithreaded, run many tasks on many machines, export metrics for Borgmon, and talk to each other via RPC. Even a call within the same process is often an RPC so that it can be moved out later. The code lives in one repository, builds hermetically, and is deployed through the release process of chapter 8.

Shakespeare: a sample service

Suppose you want to answer "which plays contain the word 'Hamlet'". A MapReduce job reads the complete works, maps each word to its locations, reduces into an index, and writes it to Bigtable. The serving path is a frontend that parses the query, backends that read the index, GSLB that routes users to a healthy datacenter, and Borg that keeps enough replicas of each running. If a datacenter goes down, GSLB sends users elsewhere and the SRE team scales up the remaining regions. The same anatomy, data pipeline plus serving stack plus traffic steering plus cluster management, applies to almost every service in the book.