Design a distributed crawler that starts from seed URLs, downloads pages, extracts links, and keeps crawling the web — storing page content for a search index — while being polite to websites and avoiding traps.

• 1B pages/month ≈ 400 pages/s average; with ~500 KB avg page → 200 MB/s ingest, 500 TB/month stored (compress ~5× → 100 TB).
• URL frontier: ~10B known URLs × 100 B = 1 TB — must be on disk, not memory.
• Seen-URL check: 10B URLs in a Bloom filter at 1% FP ≈ 12 GB — fits in memory across a few nodes.
• DNS: 400 req/s is trivial, but DNS latency (~50 ms) is the hidden bottleneck without caching.

Mostly internal:

POST /crawl/seeds        { urls[] , priority }
GET  /crawl/status/{domain}
Internal: frontier.enqueue(url, priority, notBefore) / frontier.next(workerId)
Storage: putPage(url, fetchedAt, headers, bodyHash, bodyRef)

url_frontier     (url_hash PK, url, domain, priority, next_fetch_at, last_fetched_at, depth)
domain_state     (domain, robots_txt, crawl_delay, last_access_at, in_progress)
pages            (url_hash, fetched_at, http_status, content_hash, storage_key)  -- content in object storage
content_hashes   (simhash → url_hash)   -- near-duplicate detection

1. URL frontier: two-level queue system. Front queues by priority (page rank, freshness); back queues one per host to enforce politeness — each host queue is checked out by exactly one worker at a time with a crawl_delay between fetches. Implemented on Kafka/Redis + disk-backed store.
2. Fetchers: pull a URL, resolve DNS via a local cache, honour robots.txt (cached per domain), download with timeouts and size limits.
3. Parser/extractor: parse HTML, extract links and text, normalise URLs (lowercase host, strip fragments, resolve relative).
4. Dedup: URL-seen test (Bloom filter + exact check in the frontier store); content dedup via SimHash so mirrors/near-copies aren't stored twice.
5. Storage: raw content to object storage; metadata to the pages table; a downstream indexer consumes new pages.
6. Scheduler: re-enqueue pages with next_fetch_at based on observed change frequency.
7. Trap protection: max depth, max URLs per domain, URL length limits, pattern detection for calendar-style infinite links.

Politeness is the hard part: the mapping host → single back queue → single worker guarantees one in-flight request per host; a min-heap of (next_allowed_time, host) decides which back queue to serve next.

Prioritisation: PageRank-like score + update frequency; news sites re-crawled hourly, static docs monthly.

Distributed frontier: partition hosts across frontier nodes by hash; a worker talks to one partition; rebalancing moves whole hosts.

Content freshness vs cost: adaptive re-crawl — if a page didn't change in N visits, back off exponentially.

• Worker crash mid-fetch: URL lease with visibility timeout; re-queued after expiry; fetches are idempotent.
• Frontier node loss: frontier persisted to disk/replicated; rebuildable from the pages table + seeds if needed.
• Being blocked/banned: respect Retry-After and 429/503; back off per domain; identify with a clear user-agent and contact URL.
• Crawler trap: depth and per-domain URL limits stop runaway growth; alert on domains exceeding thresholds.