Design a horizontally scalable in-memory key-value cache that application servers use to offload reads from their databases, with sub-millisecond gets/sets, TTLs and eviction.

• 1M ops/s, 80% reads. A node does ~200k ops/s → ≥ 5 nodes for throughput.
• Data: 50M keys × 1 KB avg = 50 GB. With 64 GB nodes and ~70% usable → 2 nodes for capacity; throughput dominates, so run 8–10 nodes.
• Replication factor 2 doubles memory → ~16–20 nodes.
• Network: 1M × 1 KB = 1 GB/s aggregate — fine over 10/25 GbE spread across nodes.

GET   /cache/{key}                 → 200 value | 404
PUT   /cache/{key}?ttl=300         body: value
DELETE /cache/{key}
POST  /cache/{key}/incr?by=1
POST  /cache/{key}/cas             { expectedVersion, value }

In practice a binary protocol (RESP / memcached text) over persistent TCP connections, with a smart client library that knows the ring.

Per node: hash table key → {value, expiresAt, lastAccess, version} plus an eviction structure (doubly linked list for LRU, or sampled-LRU / TinyLFU for lower overhead). Cluster metadata: the hash ring (node id → virtual node positions), stored in a coordination service (ZooKeeper/etcd) or gossiped.

1. Client-side routing with consistent hashing: each key hashes to a position on a ring; the first node clockwise owns it. Each physical node gets 100–200 virtual nodes to balance load. Adding a node moves only ~1/N of keys.
2. Node internals: single-threaded event loop per core (Redis) or multi-threaded with striped locks (memcached); slab allocator to fight fragmentation; LRU via clock/sampling rather than an exact list.
3. Expiry: lazy on access + a background sampler that deletes a random slice of expired keys each tick.
4. Replication: each key also written to the next node on the ring (RF=2). Reads go to the primary; on failure, the replica.
5. Consistency with the DB: cache-aside (app reads DB on miss and populates), with write-invalidate on updates. Avoid write-through for hot-write data.
6. Membership: heartbeats via gossip or a coordinator; a node marked dead is removed from the ring and its replica promoted.

Cache stampede (thundering herd on a hot key expiry): use request coalescing (one loader, others wait), probabilistic early expiry (refresh slightly before TTL with probability rising as expiry nears), or serve stale while revalidating.

Hot keys: detect via per-key access counters; replicate hot keys to multiple nodes and let clients pick randomly, or add an L1 in-process cache with a 1–5 s TTL.

Large values: chunk values > 1 MB across keys or reject them; large values block the event loop.

Consistency: never do read-modify-write from the app; use server-side INCR/CAS.

• Node crash: keys move to the replica; misses spike briefly; DB must handle the temporary extra load (cap with a circuit breaker).
• Network partition / split brain: two nodes both think they own a range; with cache-aside this only causes stale reads, not corruption — acceptable.
• Memory exhaustion: eviction policy kicks in; monitor eviction rate as the key health metric.
• Cold start after full outage: pre-warm from a snapshot (RDB) or replay recent access logs; ramp traffic gradually.