Design a service that lets internal teams send notifications to users over push (iOS/Android), SMS and email, with templates, user preferences, rate limits and delivery tracking.

• 100M/day ≈ 1,200/s average; bursts of 50k/s during campaigns.
• Message size ~1 KB → 100 GB/day of event data; keep 30 days hot → 3 TB.
• Provider throughput: FCM/APNs handle 10k+/s per connection pool; SMS providers often cap at hundreds/s per account → SMS needs its own queue with its own rate limiting.
• Device tokens: 200M devices × 200 B = 40 GB, sharded by user id.

POST /v1/notifications
{
  "idempotencyKey": "order-123-shipped",
  "recipients": { "userIds": [..] } | { "segmentId": "..." },
  "channels": ["push","sms"],
  "priority": "high",
  "template": "order_shipped", "params": { "orderId": 123 },
  "scheduleAt": null
}
→ 202 { "notificationId": "n_abc" }

GET /v1/notifications/{id}/status → per-recipient, per-channel status
PUT /v1/users/{id}/preferences   { push: true, sms: false, quietHours: {...} }

notifications(id, idempotency_key UNIQUE, template, params, priority, created_at)
deliveries(id, notification_id, user_id, channel, status, attempts, provider_msg_id, updated_at)
user_preferences(user_id, channel, enabled, quiet_start, quiet_end)
device_tokens(user_id, platform, token, updated_at)
templates(name, locale, channel, body)

deliveries is the high-volume table — partition by day, index on (user_id, updated_at).

1. Notification API validates, deduplicates on idempotencyKey, persists, and publishes to Kafka topics per priority × channel (e.g. push.high, sms.low).
2. Segment expansion: for segment sends, a resolver streams user ids from the user service into per-user messages (fan-out happens here, not in the API).
3. Channel workers consume, apply preferences and rate limits, render templates, and call the provider. Each channel has its own worker pool and concurrency so a slow SMS provider never starves push.
4. Provider adapters with circuit breakers, per-provider retries with backoff, and failover to a secondary provider (Twilio → SNS).
5. Delivery status: provider webhooks/callbacks update deliveries; a reconciliation job handles missing callbacks.
6. Scheduling: scheduleAt messages go to a delayed queue / scheduler service that republishes at the right time.

Exactly-once feel: at-least-once from Kafka + idempotency at two levels — the API key (caller retries) and a (notification_id, user_id, channel) unique constraint on deliveries (worker retries).

Priority isolation: separate topics and worker pools; scale high-priority workers independently and cap low-priority throughput during bursts.

Preference checks late: evaluate opt-outs and quiet hours in the worker right before sending — a user may opt out between enqueue and send.

Batching: FCM supports multicast to 500 tokens per call; batch by template to cut provider calls 100×.

• Provider outage: circuit breaker opens, traffic fails over to the secondary provider; if none, messages stay in Kafka (retention ≥ 24 h) and drain later.
• Worker crash mid-send: message not acked → redelivered → dedupe key prevents a second send if the first actually reached the provider.
• Stale device tokens: provider returns 'unregistered' → delete the token immediately to avoid wasting quota.
• Kafka lag during a campaign: dashboards on consumer lag per topic; auto-scale workers on lag.