Design a durable, high-throughput publish-subscribe log where producers append messages to topics and independent consumer groups read them at their own pace, with ordering within a partition and configurable retention.

• 1M msgs/s × 1 KB = 1 GB/s ingest. A broker can sustain ~100–200 MB/s sequential disk writes with page-cache-assisted reads → ~10 brokers for ingest, ×3 replication = ~30 brokers (or fewer, bigger ones).
• Retention 7 days: 1 GB/s × 604,800 s ≈ 600 TB × RF 3 = 1.8 PB.
• Partitions: target ≤ 10 MB/s per partition for consumer parallelism → 100+ partitions per hot topic.

produce(topic, key?, value, headers) → (partition, offset)      // batched, async with acks=all|1|0
consume: poll() → records; commit(offsets) (auto or manual)
admin: createTopic(name, partitions, replicationFactor, retention)

Wire protocol is binary over TCP; clients are partition-aware and talk directly to the leader broker.

Topic → N partitions; each partition is an append-only log on disk split into segment files (00000000000123.log + sparse .index). Message = offset, timestamp, key, value, headers. Consumer group state: (group, topic, partition) → committed offset, stored in an internal compacted topic. Cluster metadata (leaders, ISR) in the controller (ZooKeeper or KRaft).

1. Brokers host partitions. Each partition has one leader and R−1 followers; producers/consumers talk to the leader. Followers fetch from the leader and are in the ISR (in-sync replica set) while caught up.
2. Produce path: client batches by partition (chosen by hash(key) or round-robin) → leader appends to its log → waits for ISR acks per acks setting → responds. Sequential I/O + zero-copy sendfile make this cheap.
3. Consume path: consumers pull (not push) from leaders using offsets; long-poll for low latency. A group coordinator assigns partitions to consumers and rebalances on membership change.
4. Offsets are committed by consumers to the coordinator; replay is just seeking to an earlier offset.
5. Retention: delete old segments by time/size, or log compaction (keep latest value per key) for changelog topics.
6. Controller elects leaders when a broker dies (unclean election disabled to avoid data loss).

Why pull: consumers control rate (backpressure for free), batching is natural, and replay is trivial.

Ordering is per partition only; to order all events for an entity, key by that entity's id.

Exactly-once: idempotent producers (sequence numbers per producer/partition) + transactions spanning produce and offset commit; consumers read read_committed.

Rebalance pain: stop-the-world rebalances on every membership change; cooperative/incremental rebalancing and static membership reduce it.

Backpressure: consumer lag is the key metric; producers see backpressure via bounded in-flight buffers.

• Leader broker dies: controller promotes an ISR follower; producers refresh metadata and retry; no acknowledged data lost with acks=all and min.insync.replicas ≥ 2.
• Consumer crashes before commit: partition reassigned; messages since the last commit are re-delivered → consumers must be idempotent.
• Disk full: broker stops accepting; alerts on disk and retention misconfiguration.
• Slow follower falls out of ISR → writes still succeed on the remaining ISR; alert on under-replicated partitions.