Inside a datacenter, a client task holding a request has to pick a backend task to send it to. This chapter is about how Google's RPC layer does that: which backends a client should even connect to, how to tell when a backend is unhealthy, and how to spread load so that no task is much busier than another. The payoff is in the numbers: naive round robin led to a 2x spread in CPU between the busiest and least busy task, and weighted round robin brought it close to even.

The chapter is one of the most reusable in the book because the same problems appear in any service mesh or client-side load balancer.

In one minute

• Goal: distribute requests so that backend tasks are evenly loaded, to minimise tail latency and wasted capacity, while using few connections.
• Identify bad tasks with flow control: a client tracks active requests per backend and stops sending to one that is above a limit. Simple but blunt, and only triggers once a backend is badly overloaded.
• Better: lame duck state. A backend reports healthy, refusing connections (unhealthy), or lame duck, meaning it is still serving existing requests but asking clients not to send new ones. Lame duck enables graceful shutdown: the task stops taking new work, drains, and exits with no errors.
• Subsetting: each client connects to only a subset of backends, to limit connections and memory. Random subsetting produces a very uneven spread (some backends in many subsets, some in few). Deterministic subsetting shuffles the backend list once per round and hands consecutive slices to clients, which gives a near-perfect spread.
• Simple round robin fails because requests differ in cost, clients differ in rate, and machine performance varies, giving a 2x spread in utilisation in Google's measurements.
• Least-loaded round robin picks the backend with the fewest active requests from this client, but each client sees only its own requests, and a failing backend that errors instantly looks lightly loaded and attracts more traffic (the sinkholing problem).
• Weighted round robin uses backend-provided utilisation (CPU, QPS, error rate) returned in responses to adjust per-backend weights, and brought the spread down to near uniform.

The ideal case

In an ideal datacenter, every backend task would be equally loaded at all times, so a service's capacity equals the sum of its tasks and no task is a latency hotspot. Reality diverges because requests are not uniform, clients are not uniform, and machines are not uniform. The design question is how much information clients and backends should exchange to correct for that.

Identifying bad tasks: flow control and lame ducks

A client cannot tell a slow backend from a dead one without a signal. The first mechanism is per-backend active request counts: if a backend has more than, say, 100 outstanding requests from this client, it is marked unhealthy and skipped. This protects against a backend that has stopped responding but reacts late and can mark a merely busy backend as bad.

Lame duck is the richer signal. Backends have three states as seen by a client. Healthy: accepting requests normally. Refusing connections: not listening or not reachable. Lame duck: listening and serving, but telling clients to stop sending new requests. A task enters lame duck when it is about to shut down. Clients stop sending, in-flight requests complete, and the task exits cleanly. Because this is a positive signal rather than a timeout, it is fast and precise, and it makes rolling restarts invisible to users. Clients periodically health-check backends so they can leave the unhealthy and lame duck states promptly.

Limiting the connections pool with subsetting

A service with 10,000 clients and 1,000 backends cannot have every client connected to every backend: that is ten million connections, each with memory and health-check cost. Each client connects to a subset, typically 20 to 100 backends.

Random subsetting is tempting and bad: with random choice the number of clients per backend varies widely, so some backends receive far more load than others. The chapter's measurements show a range from a backend appearing in about half the expected number of subsets to another appearing in twice as many.

Deterministic subsetting fixes this. Clients are numbered. The backend list is shuffled with a seed derived from the round number (client index divided by the number of subsets per round), and each client takes a contiguous slice of the shuffled list based on its position within the round. Every round uses the whole backend list exactly once, so each backend appears in the same number of subsets, give or take one for the partial last round. The chapter includes the Python implementation; it is short enough to memorise for an interview.

Load balancing policies

Simple round robin. Each client cycles through its subset. Problems: small subsetting means the number of clients per backend differs slightly; clients have different request rates, so a backend with many busy clients is overloaded; requests have different costs, in Google's measurements varying by three orders of magnitude; and machines differ in speed and in what else is running on them. Measured result: the most loaded task uses about twice the CPU of the least loaded.

Least-loaded round robin. Each client tracks active requests per backend and picks the one with the fewest. This adapts to request cost but has two flaws. Each client only sees its own requests, so two clients may both pile onto the same backend. And a backend that fails fast, returning errors immediately, has few outstanding requests and so attracts more traffic, which the chapter calls sinkholing. Google counts errors as active to blunt this, but the fundamental limitation of client-local information remains.

Weighted round robin. Backends include their current utilisation (CPU rate, QPS, error rate) in RPC responses. Clients compute a capability score per backend and distribute requests proportionally, adjusting as scores change. Because the signal comes from the backend, it reflects load from all clients, not just this one, and a sick backend reports high utilisation or errors and is weighted down. In the measurements, the CPU spread across tasks collapsed to within a few percent of uniform.

Putting it together

Google's RPC stack combines all of this: deterministic subsetting to limit connections, lame duck signalling for graceful restarts, and weighted round robin for distribution. The lesson for anyone building a client-side balancer is that the backend has information the client lacks, and the cheapest way to share it is to piggyback on the responses already flowing.