Load balancing tries to spread traffic evenly, but it cannot create capacity. Sooner or later a service gets more than it can handle. This chapter is about making sure that when that happens, the service degrades predictably rather than collapsing, and that the right requests are the ones that get dropped. It is the prerequisite for the cascading failures chapter that follows.

The ideas here, criticality, adaptive throttling, and the retry budget, are implemented inside Google's RPC framework so every service gets them for free.

In one minute

• Avoiding overload is a goal of load balancing, but overload still happens, and every service must be built to handle it gracefully.
• Measure load in cost, not queries per second. QPS varies with request mix; CPU consumption is a stable proxy for the cost of a request on most Google services. Provision in CPU and set per-customer limits in CPU.
• Per-customer limits: each backend has a global quota per customer, split among tasks, and rejects a customer's requests beyond it, protecting other customers.
• Client-side adaptive throttling: a client tracks requests sent and requests accepted over a window and probabilistically rejects locally when the ratio shows the backend is rejecting most of them, so the backend is not swamped by requests it will refuse. The reject probability is max(0, (requests - K * accepts) / (requests + 1)) with K around 2.
• Criticality: every request carries one of four levels: CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE. Backends shed lower criticality first, quotas can be per-criticality, and criticality propagates along the RPC chain.
• Utilisation signals: CPU is the default, but services can define their own (executor load average, memory pressure). Shedding starts when utilisation crosses a threshold, in criticality order.
• Handling overload errors: retry is right if a small fraction of tasks are overloaded; wrong if the whole datacenter is, in which case push back to the caller or fail over to another datacenter.
• Retry rules: per-request retry budget (about 3 tries), per-client retry budget (retries at most 10 percent of requests), and a retry count on the request so backends can reject one that has been retried too many times. Retry at the layer right above the failure, not at every layer.
• Load from connections matters too: health checks and idle connections cost CPU and memory; batch clients should use a proxy.

The pitfalls of "queries per second"

A query is not a unit of work. A request that reads one row and one that aggregates a million are both one QPS. Google models capacity in CPU time because for most services CPU correlates well with the real cost, and other resources (memory, network) can be bounded by also provisioning in CPU. Provision capacity and quotas in that unit and the model stays accurate when the request mix shifts.

Per-customer limits

A backend owned by one team is used by many customers. Each customer gets a quota that is agreed in advance and enforced per backend task (the global quota divided among tasks). When a customer exceeds its quota during overload, its excess requests are rejected with an overload error, while other customers within quota are unaffected. This stops one customer's bug from degrading everyone.

Client-side throttling

Rejecting a request still costs the backend CPU to parse it and reply. If a backend is already overloaded, the rejections themselves can keep it that way. Adaptive throttling moves the rejection to the client. Each client keeps two counters over the last two minutes: requests attempted and requests accepted by the backend. Normally they are equal. When the backend starts rejecting, accepts fall behind, and the client rejects new requests locally with probability (requests - K * accepts) / (requests + 1). With K = 2 the client allows twice as many requests as the backend is accepting, which lets it probe for recovery while still shedding most of the excess. Lower K is more aggressive. This has proven effective across Google because it needs no coordination.

Criticality

Not all requests are equal. CRITICAL_PLUS is for requests whose failure is a serious user-visible outage. CRITICAL is the default for production traffic. SHEDDABLE_PLUS is for things where partial unavailability is acceptable, like batch jobs that retry later. SHEDDABLE is for traffic that can disappear for extended periods. A service loads shed starting from the bottom. Quotas are set per criticality, so a customer can be allowed more CRITICAL than SHEDDABLE traffic. The RPC system propagates criticality automatically, so a backend four levels deep knows whether this call is on a user-facing path, and a service can lower the criticality of calls it makes on its own behalf.

Utilisation signals

Backends decide to shed based on utilisation. CPU is the standard signal, but Google also uses an executor load average, which counts active threads and smooths it over time, so a service that spends time blocked on I/O can still detect that it is saturated. Services with other bottlenecks define their own signals. The policy is to shed SHEDDABLE traffic early, then SHEDDABLE_PLUS, and only to reject CRITICAL when the task would otherwise fail.

Handling overload errors

An overload error can mean two things. If only some tasks are overloaded, retrying to another task is right, and the load balancer will take the task out of rotation soon. If the whole datacenter is overloaded, retries make it worse, and the right response is to let the error reach the caller or to redirect to another datacenter. The chapter is firm that retries must be governed: a per-request budget of about three attempts, a per-client budget of 10 percent of requests so that a dying backend does not triple a client's load, and a retry counter in the request metadata so a backend can see that a request has already been retried twice and reject it outright. Retries should happen at the layer directly above the failure; if every layer retries three times, a request six layers deep is retried hundreds of times.

Load from connections

Connections have their own cost: memory per connection, periodic health checks, and the CPU of TCP handshakes. Thousands of small batch jobs each opening a connection to a shared backend can overload it with no requests at all. Google's fixes include batch proxies that multiplex many clients onto fewer connections, and keeping health check frequency proportional to the client's traffic.

Conclusions

Overload handling is a design property, not a patch. Services should measure load in a cost unit, know the criticality of what they are serving, shed the least important work first, and never retry without a budget. Done well, a service in overload serves its most important requests at full quality and nothing else, and recovers as soon as demand falls.