Design a real-time messaging service supporting 1:1 and group chats, message history, read receipts, online presence and push notifications for offline users.

• 1B msgs/day ≈ 12k/s avg, 60k/s peak.
• 100 B avg message + 200 B metadata → 300 GB/day; 5-year retention ≈ 550 TB → Cassandra/HBase-class store.
• 50M concurrent WebSocket connections; a well-tuned server holds ~500k–1M → 50–100 chat servers plus headroom.
• Presence heartbeats every 30 s → 1.7M heartbeat ops/s — must be cheap (in-memory, Redis).

WebSocket messages (client ↔ chat server):

→ { type:"send", clientMsgId, conversationId, text|mediaId }
← { type:"ack", clientMsgId, serverMsgId, ts }
← { type:"message", serverMsgId, conversationId, senderId, text, ts }
→ { type:"read", conversationId, uptoMsgId }
→ { type:"typing", conversationId }

REST for non-realtime:

GET  /v1/conversations/{id}/messages?before={msgId}&limit=50
POST /v1/conversations            { memberIds[] }
POST /v1/media/upload → presigned URL

messages   (conversation_id, message_id TIMEUUID, sender_id, body, media_url, created_at)
             PK (conversation_id, message_id)   -- Cassandra, clustered by message_id DESC
conversations (id, type, created_at)
members    (conversation_id, user_id, last_read_msg_id, joined_at)
user_conversations (user_id, conversation_id, last_msg_ts)  -- inbox listing
presence   Redis: user_id → {status, last_seen}   TTL 60s
sessions   Redis: user_id → [chat_server_id ...]  (which server holds the socket)

message_id is a time-ordered id (Snowflake/TIMEUUID) generated by the chat server, which gives per-conversation ordering by generation time.

1. Connection layer: clients keep a WebSocket to a chat server (sticky via L4 LB). A session registry (Redis) maps user → server(s).
2. Send path: chat server assigns message_id, writes to the message store (Cassandra, RF=3) and publishes to a message queue. Ack to sender only after the store write succeeds.
3. Delivery: a router looks up recipients in the session registry and forwards to their chat servers. Offline recipients get a push via the notification service; the message waits in the store and syncs on reconnect.
4. Group fan-out: for small groups (≤ 500) fan-out on write to each member's inbox and socket; for huge channels use fan-out on read (members pull the channel timeline).
5. Sync/pagination: clients track the last seen message_id per conversation and pull deltas on reconnect.
6. Presence: heartbeats update Redis with TTL; presence changes published to friends via pub/sub, rate-limited.
7. Media: upload directly to S3 via presigned URL; the message carries the object key.

Ordering: per-conversation order comes from the time-ordered id assigned by whichever chat server handled the send; clock skew between servers can reorder near-simultaneous messages by a few ms — acceptable. For strict order, route all sends for a conversation through one partition of the queue.

Multi-device: the session registry holds a list of sockets; deliver to all; read receipts are per user, not per device.

Exactly-once for the user: clientMsgId is idempotent on the server so retries after a dropped ack don't duplicate.

Read receipts are just messages of another type; throttle typing indicators to 1 per 3 s.

• Chat server dies: clients reconnect (exponential backoff with jitter) to another server; the registry entry expires via TTL; undelivered messages are in the store, so nothing is lost.
• Queue outage: sends still persist to the store; delivery is delayed until the queue recovers; the client sees a tick (stored) but not double tick.
• Store write failure: sender gets no ack, client retries with the same clientMsgId.
• Split delivery (recipient's server unreachable): router retries, then falls back to push; recipient syncs on reconnect.