"How would you scale this" and "what happens if this fails" are the two questions behind every system design round, and the DevOps version is more concrete than the developer version: which layer absorbs which failure, what setting makes it do so, and what you measured. Job posts for platform roles list rate limiting, quotas, abuse controls, Redis, Aurora, Cloudflare edge and WebSocket scaling, and the 100xDevs cohort spends five weeks on queues, WebSockets, rate limiting, captchas and DDoS. The worked example on 200,000 WebSocket connections is the kind of estimate you will be asked to do on a whiteboard.

Vertical and horizontal scaling, and where state goes

Vertical scaling is a bigger machine: more vCPUs, more memory. It is the first move because it needs no code change, and the last one available, because the largest instance is a hard ceiling and a single machine is a single failure. Horizontal scaling is more machines behind a load balancer. It has no ceiling but one condition: a request must be able to land on any replica, so the replica holds no state between requests. Sessions go to Redis, files to S3, work-in-progress to a queue, records to the database. An Auto Scaling group or a Kubernetes HPA can add replicas in a minute, but only if the new replica needs nothing from the old one.

Load balancing and health checks are the glue. The load balancer sends traffic only to targets that pass a health check and removes a failing replica after interval times threshold (an assumed tuning: 10 s and 3). The check must test what the service does, not just that a port is open, or a replica with a dead database connection keeps receiving traffic.

A resilient request path from edge CDN and WAF through rate limiter, load balancer, stateless replicas, cache, queue and workers to a database with replicas, with the failure each layer absorbs and the setting that makes it work

Caching

A cache answers repeated reads from memory instead of the database. Cache-aside is the pattern you will be asked about: the app checks the cache, on a miss reads the database, writes the value into the cache with a TTL (time to live), and returns it. Writes go to the database and either delete the cache key (invalidation) or overwrite it. Redis and its open-source fork Valkey are the usual in-memory stores; ElastiCache runs both on AWS. A CDN is a cache at the edge for HTTP responses: static files always, and any GET with a Cache-Control: public, max-age=60 header that is the same for every user.

Two numbers describe a cache: hit ratio and p99 latency. A 90 percent hit ratio means the database sees 10 percent of reads; raising it to 95 percent halves the database load again. The failure mode is the cache stampede: a hot key expires, 2,000 requests in the same 50 ms all miss, and all 2,000 go to the database at once. The fix is single-flight: the first miss takes a short lock (SET lock:key 1 NX EX 5), loads and fills the cache, and the others wait for the fill or serve the stale value. Add jitter to TTLs (300 + random(0, 60) seconds) so keys set together do not expire together.

Queues and async work

A queue sits between the API and the workers so the API answers in 20 ms with 202 Accepted while the real work (send the email, call the payment provider) happens later. It absorbs bursts (the queue grows, workers catch up), isolates failures (workers can be down for ten minutes and nothing is lost), and lets the two sides scale separately.

The semantics decide the design. Almost every queue gives at-least-once delivery: a worker that crashes after doing the work but before acknowledging will see the message again. So consumers must be idempotent: processing the same message twice gives the same result, usually by storing the message id with the result and skipping duplicates. Messages that fail repeatedly go to a dead-letter queue (DLQ) after N attempts so they stop blocking the others and a human can look. Backpressure is the queue telling the producer to slow down: a bounded queue, a 429 from the API when depth is above a threshold, or simply the lag metric that the autoscaler reads.

# Redis streams: the 100xDevs worker pattern in redis-cli
XGROUP CREATE jobs workers $ MKSTREAM                        # create the stream and group
XADD jobs MAXLEN ~ 100000 * type email user 42               # producer: trim to about 100k entries
XREADGROUP GROUP workers w1 COUNT 10 BLOCK 5000 STREAMS jobs >   # worker: 10 new messages, wait up to 5 s
XACK jobs workers 1699999999999-0                            # done; without this the entry stays pending
XPENDING jobs workers - + 10                                 # what is stuck, and for how long
XAUTOCLAIM jobs workers w2 60000 0-0 COUNT 10                # w2 takes over entries idle for over 60 s

Scaling WebSocket servers

An HTTP request lasts milliseconds; a WebSocket connection lasts hours and lives in one process's memory. That changes three things. First, sticky routing: the load balancer keeps a connection on the node that holds it, and the 100xDevs week 44 pattern routes by room or user id so one node holds everyone in a room. Second, fan-out across instances: when a message must reach users spread over 17 nodes, the sender publishes once to Redis pub/sub or a stream, every node subscribes, and each node delivers to the connections it holds. Third, limits per node: each connection costs a file descriptor and memory (an assumed 20 to 50 KB in Go or Node with buffers), so a node has a connection ceiling set by ulimit -n, memory and the CPU needed for ping frames; keep it at 60 percent so one node's loss fits on the others.

A deploy must drain gracefully: the pod stops accepting new connections, sends a close frame to existing ones at a limited rate, and clients reconnect with jittered backoff to the other pods. Without the rate limit and the jitter, a deploy of one pod becomes a reconnect storm that overloads the rest. Set terminationGracePeriodSeconds to cover the drain (60 to 120 seconds), use a preStop hook to flip readiness first, and use a PodDisruptionBudget so nodes drain one pod at a time.

Rate limiting, abuse controls and DDoS

Rate limiting caps how many requests a client may make, to protect the service and to make abuse expensive. Two algorithms cover the interview:

• Token bucket: a bucket of capacity B refills at R tokens per second; each request takes one token; an empty bucket means 429. It allows a burst of B and a steady rate of R. NGINX limit_req with burst=20 nodelay is one; so is the AWS API Gateway throttle. Most edge limiters are token buckets.
• Sliding window: count requests in the last W seconds (a log of timestamps, or two fixed windows weighted by overlap); above N, refuse. It is strict about the rate with no burst allowance, and after a burst it refuses everything until the window passes, which the code example below shows.

Apply limits at three places: at the edge per IP (Cloudflare or AWS WAF rate-based rules, the first line, cheap, catches scrapers), at the gateway per API token or user (the fair-share line, 1,000 requests per minute per token as an assumed quota), and per operation for expensive endpoints (5 password resets per hour per account). Quotas are limits over a day or a month, enforced from a counter in Redis (INCR with EXPIRE). Abuse controls beyond rate: a captcha or a managed bot score on signup and login, device and IP reputation, and a kill switch per token. The 100xDevs weeks 45 and 47 cover exactly this chain: rate limit, captcha, then DDoS.

A DDoS (distributed denial of service) sends traffic from thousands of sources, so per-IP limits do not help. Volumetric floods (tens of Gbps of UDP) can only be absorbed at the edge: an anycast CDN spreads the traffic over many points of presence and drops it there; AWS Shield Standard does this for every customer, Shield Advanced adds application-layer detection and a response team; Cloudflare does both. Application-layer floods (millions of valid-looking HTTP requests) need the WAF: managed bot rules, challenge pages, and rate rules keyed on a fingerprint rather than an IP. The rule for your own architecture: nothing with a public IP except the edge, so there is nothing else to hit.

# NGINX token bucket per client IP: 10 requests per second, bursts of 20 served without delay, then 429
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
limit_req_status 429;
server {
    location /api/ {
        limit_req zone=api burst=20 nodelay;
        proxy_pass https://app;
    }
}

Timeouts, retries, circuit breakers, bulkheads

Every remote call needs a timeout, both to connect and to read, set from the dependency's measured p99 plus a margin; without one, a slow dependency holds every thread and the slowness spreads. Retries recover from transient failures but are selfish: each retry is extra load on a service that is already failing. The retry storm is three layers each retrying three times, which turns one request into 27 at the bottom, and a brief blip into an outage. The rules: retry only idempotent calls, at one layer, a small number of times, with exponential backoff and jitter (wait random(0, base * 2^attempt), capped), and with a retry budget (retries may be at most 10 percent of requests). A circuit breaker counts failures per dependency: after N failures it opens and fails fast without calling, after a cooldown it goes half-open and lets one trial call through, and closes again on success. It turns a dead dependency from a timeout on every request into an instant error, and gives the dependency room to recover. Bulkheads isolate resources per dependency: a separate connection pool or goroutine limit for each, so a slow payment provider cannot consume the pool the search endpoint needs.

The Tail at Scale idea: at fan-out, latency is decided by the slowest call. If a page calls 100 backends and each has a 1 percent chance of a slow (p99) response, the page is slow 63 percent of the time (1 minus 0.99 to the power 100). That is why p99 matters more than the average for anything that fans out, and why hedged requests help: send the same request to a second replica after the p95 delay and take the first answer, which cuts the tail for a few percent of extra load.

Database scaling

The database holds the state, so it is the hardest layer to scale. In order of cost: connection pooling first. Each Postgres connection is a process with memory, and max_connections is often 100 to 500. A pool in the app (10 connections per pod) or a pooler in front (PgBouncer, RDS Proxy) keeps the count fixed. The classic incident: a deploy with maxSurge doubles the pod count for a minute, every new pod opens its 10 connections, 40 pods times 10 is 400, the limit is 300, and the new pods crash on too many connections, so the rollout stalls. Size pools so pods times pool size plus surge stays under the limit, or put RDS Proxy in front. Next, read replicas: Aurora gives a writer endpoint and a reader endpoint that balances across up to 15 replicas; the app sends SELECTs that tolerate replica lag to the reader. Then caching in front of reads, then sharding, which splits rows across databases by a key such as tenant id, moves joins across shards into the application, and is a multi-quarter project; do it when a single writer cannot keep up after the cheaper moves.

Multi-AZ, multi-region and disaster recovery

RPO (recovery point objective) is how much data you may lose, as time since the last good copy; RTO (recovery time objective) is how long until service is back. The business sets them; the design follows. Backups are the floor: automated snapshots daily with a 7 to 35 day retention, point-in-time recovery enabled, copied to another region, and restored on a schedule into a scratch account, because an untested backup is a guess. DNS failover with Route 53: a health check (request interval 30 s by default, 10 s for fast; failure threshold 3 by default) on the primary endpoint, and a failover record set that answers with the secondary when the check fails; keep the TTL at 60 s so clients move within a minute of the DNS change. The DR runbook is the list of commands, in order, with the person who runs each and the expected time, rehearsed every quarter: freeze writes, promote the secondary database, repoint DNS, scale the secondary region, verify with synthetic checks, announce.

Capacity planning and chaos testing

Plan from numbers: peak requests per second (the busiest hour of the busiest day plus growth over the planning horizon), the measured capacity of one replica at the latency target (load test one pod: say 400 requests per second at p99 under 200 ms), then replicas equals peak divided by per-replica capacity, divided by a utilisation target of 0.6 to 0.7, plus one zone's worth so losing an AZ does not drop below peak. Do the same for connections, queue depth and database IOPS. Then test the plan: chaos testing kills a pod, a node, an AZ or a dependency on purpose, with a small blast radius, and checks that the system did what the design said. Start cheap: delete a pod and confirm no errors; add 300 ms of latency to the database with Toxiproxy and confirm the breaker opens; cordon every node in one AZ and confirm the PDBs keep the service up.

A WebSocket service with 200,000 concurrent connections.

Step 1, connections per node. One Go pod handles an assumed 20,000 connections comfortably: at 40 KB per connection that is 800 MB of memory, plus ping frames every 30 s, which is 20,000 / 30 = 667 small writes per second, trivial CPU. Set the file descriptor limit above 25,000.

Step 2, nodes needed with headroom. The bare minimum is 200,000 / 20,000 = 10 pods. Run at 60 percent utilisation so a lost AZ (one third of pods) or a deploy surge fits: 200,000 / (20,000 x 0.6) = 16.7, so 17 pods, about 11,765 connections each. Spread over 3 AZs with a topology spread constraint; losing one AZ leaves 12 pods holding 16,667 each, under the 20,000 ceiling.

Step 3, fan-out for a message to 10 percent of users. 20,000 recipients, 1 KB payload. The publisher writes once to Redis pub/sub; Redis sends one copy to each of the 17 subscribed pods: 17 KB out of Redis. Each pod delivers to its own share, 20,000 / 17 = 1,176 connections, 1.2 MB of client egress per pod. At 50 such messages per second, Redis egress is 50 x 17 KB = 850 KB/s, under 1 MB/s, and client egress across the fleet is 50 x 20,000 x 1 KB = 1,024 MB/s, which is 8.2 Gbit/s, about 0.48 Gbit/s per pod: the client side, not Redis, is the bandwidth cost, and node network capacity is what you size.

Step 4, drain time on a deploy. A pod holds 11,765 connections. Closing them all at once makes 11,765 clients reconnect in the same second. Instead the pod closes 500 per second: 11,765 / 500 = 23.5 seconds to drain. Clients reconnect with a random 0 to 30 s delay, so the reconnect rate on the other pods is about 11,765 / 30 = 392 per second, which a fleet of 16 pods absorbs. With maxSurge: 25% (25 percent of 17 is 4.25, rounded up to 5) the rollout replaces about 5 pods per wave: ceil(17 / 5) = 4 waves, each 24 s of drain plus about 30 s for the new pod to be Ready, about 216 seconds, 3.6 minutes for the full rollout. terminationGracePeriodSeconds must be at least 24 s plus margin; set 60.