Design a system that recommends items (products/videos) to users on the home page and item pages, using their history and item similarity, with real-time signals and A/B-testable models.

• 1B events/day ≈ 12k/s into the feature pipeline.
• Precomputed candidates: 100M users × 500 ids × 8 B = 400 GB → KV store; refreshed daily plus real-time updates for active users.
• Item embeddings: 50M × 128 floats × 4 B = 25 GB → ANN index (HNSW) sharded across a few nodes.
• Online ranking: 500 candidates × ~100 features per request; 20k QPS → 10M feature lookups/s → feature store must be an in-memory KV with batching.

GET /v1/recs/home?userId=&limit=20&context={device,page}     → { items:[{itemId, score, reason}], modelVersion }
GET /v1/recs/similar?itemId=&limit=20
POST /v1/events   { userId, itemId, type: view|click|purchase, ts }   (batched from clients)

events            Kafka → data lake (Parquet) ; user_history (user_id → recent item ids, Redis)
item_features     (item_id, category, price, popularity_7d, embedding)       -- feature store
user_features     (user_id, affinity vector, segment, recent_items)          -- feature store (online + offline)
candidates        (user_id → [item_id...]) precomputed (KV)
similar_items     (item_id → [item_id...]) precomputed from co-occurrence / embeddings
ann_index         HNSW over item embeddings
experiments       (exp_id, variants, allocation)

1. Event ingestion: client/server events → Kafka → (a) stream processor updating online features (recent items, session vectors) in Redis; (b) data lake for offline training.
2. Offline pipeline (daily/hourly): train collaborative-filtering / two-tower models → produce user and item embeddings, item-to-item similarity tables, and per-user candidate lists → publish to the feature store, ANN index and candidate KV.
3. Online serving on each request:
   • Candidate generation (fast, high recall): precomputed user candidates + ANN neighbours of the user's session embedding + similar-items of recently viewed + trending — merged to ~500.
   • Filtering: remove seen/purchased, out-of-stock, policy violations.
   • Ranking: fetch features for user × candidates from the feature store; score with the ranking model (served via a model server); apply diversity/business re-ranking.
   • Explanations attached from the candidate source.
4. Experimentation: request assigned to a variant by user id hash; models/config resolved per variant; metrics logged with variant ids.
5. Caching: home recs cached per user for a few minutes, invalidated by strong signals (purchase).

Two-stage architecture (retrieve then rank) is the standard: retrieval is cheap and broad; ranking is expensive and narrow.

Feature consistency: the same feature definitions must be used offline (training) and online (serving) — a feature store solves the training/serving skew.

Real-time: session embedding = average of recently viewed item embeddings, updated in-stream; ANN over it captures 'what you're doing right now'.

Cold start: new users → popularity by segment/geo; new items → content-based embeddings from metadata until interaction data exists.

Diversity & rules re-rank after scoring: category caps, price spread, promoted slots.

• Feature store down: serve precomputed candidate order (no re-ranking) or popularity.
• Model server down: fall back to previous model version or heuristic score.
• Stale candidates (pipeline failed): keep serving yesterday's; alert on staleness.
• Feedback loop (only recommending popular items): exploration slots (ε-greedy/bandits) in the re-ranker.