Design a system that collects logs from thousands of services, makes them searchable within seconds, supports filtering by fields and time, and retains data cost-effectively for months.

• 10 TB/day ≈ 115 MB/s ≈ 200k events/s at 500 B each; peaks 1M/s.
• Index expansion ~1.2× raw for full-text; hot tier 3 days = 36 TB on SSD; warm 30 days = 300 TB on HDD (compressed ~5× → 60 TB); cold in object storage.
• Shards: keep index shards 20–40 GB; 10 TB/day → ~300 shards/day across the hot nodes.

Agents → POST /v1/ingest (batched, gzip)   { events:[{ts, service, level, message, fields{...}}] }
GET  /v1/search?q=level:ERROR AND service:checkout AND "timeout"&from=now-15m&to=now&limit=100
GET  /v1/aggregate?q=...&groupBy=status&interval=1m
WS   /v1/tail?q=...
POST /v1/alerts  { query, threshold, window }

Index per (tenant, day) → shards; document = {@timestamp, service, host, level, message, fields...}
Field mappings: keyword fields (exact/aggregatable), text fields (tokenised), timestamp
Tiering metadata: index → tier (hot/warm/cold), size, docs, retention_until
Kafka topic per tenant/priority as the ingest buffer

1. Agents (Fluent Bit/Vector) tail files/containers, parse to JSON, batch and ship to the ingest gateway with backpressure and local disk buffering.
2. Gateway authenticates, applies quotas, and writes to Kafka (buffer; decouples bursts from the index).
3. Processors consume, enrich (geo, k8s metadata), redact secrets/PII, sample noisy debug logs, and bulk-index into the search cluster (Elasticsearch/OpenSearch or a columnar log store like Loki/ClickHouse).
4. Index lifecycle: hot (SSD, writes + recent queries) → warm (HDD, read-only, force-merged) → cold (snapshots in object storage, searchable on demand) → delete.
5. Query service routes by time range to the right tier, applies tenant filters, caches aggregations; live tail subscribes to the Kafka stream with a filter.
6. Alerting evaluates saved queries periodically or on the stream.

Structured logging at the source is the biggest win: fields become keyword columns; message stays text.

Sampling & redaction happen in processors so the index never stores secrets and debug noise is cut by 90%.

Time-based indices make retention a delete of whole indices and let queries skip irrelevant shards entirely.

Columnar vs inverted index: inverted index (ES) for arbitrary text search; columnar/label-based (Loki, ClickHouse) is much cheaper when most queries filter by labels and time and grep the rest.

• Search cluster down: Kafka retains hours/days of logs; processors resume; live tail still works from Kafka.
• Agent host crash: at most the unflushed buffer is lost (seconds).
• Hot node loss: shard replicas (RF=2) on hot; warm/cold restored from snapshots.
• Retention misconfig (disk full): index write block → alerts on disk watermarks; emergency deletion of oldest indices.