Design a service where teams register jobs (one-off at time T, or recurring by cron), which run on a worker fleet with retries, timeouts, dependencies and observability — with no job run twice or silently dropped.

• 10M runs/day ≈ 115/s average; bursts at :00 minutes can be 10k/s (everyone schedules on the hour) → the dispatcher must handle spikes via a queue.
• Job table 100k rows; run history 10M rows/day × 1 KB = 10 GB/day → partition by day, retain 30 days hot.
• Due-job scan: index on next_run_at; a poll every second reads the next few thousand rows.

POST /v1/jobs   { name, schedule: "0 2 * * *" | { runAt }, command|payload, timeoutSec, retries, maxConcurrency, dependsOn[] }
GET  /v1/jobs/{id}  /  PUT  /  DELETE
POST /v1/jobs/{id}/trigger
GET  /v1/jobs/{id}/runs?status=&limit=
GET  /v1/runs/{runId}/logs

jobs      (job_id, owner, schedule, payload, timeout_sec, max_retries, max_concurrency, next_run_at, enabled)
runs      (run_id, job_id, scheduled_for, started_at, finished_at, status, attempt, worker_id, output_ref)
           UNIQUE (job_id, scheduled_for)     -- prevents duplicate runs of the same tick
deps      (job_id, depends_on_job_id)
locks     (job_id, lease_until, holder)     -- or use the runs table + DB row locks

1. API/Job service persists jobs and computes next_run_at from the cron expression.
2. Scheduler (dispatcher): a small set of replicas; leader (via ZooKeeper/etcd lease or DB advisory lock) polls jobs WHERE next_run_at <= now() AND enabled every second, and for each: inserts a runs row (job_id, scheduled_for) (unique constraint = exactly one run per tick), advances next_run_at, and publishes the run to a queue. Alternatively partition jobs by hash across several active schedulers, each owning a range — removes the single leader bottleneck.
3. Queue (Kafka/SQS) buffers spikes; priority topics.
4. Workers pull runs, claim with a lease (started_at, worker_id, heartbeat), execute with the timeout, stream logs to object storage, mark succeeded/failed. On failure, re-enqueue with backoff up to max_retries.
5. Dependencies: a run is enqueued only when upstream runs for the same logical date succeeded; a DAG evaluator checks on each completion event.
6. Watchdog: reaps runs whose lease expired (worker died) and re-queues them.
7. UI/observability: run history, DAG view, alerts on failure/SLA miss.

Exactly one run per tick: the unique (job_id, scheduled_for) insert is the guard; two schedulers racing still yield one row.

Misfires: if the scheduler was down and ticks were missed, policy per job: run once (coalesce) or run all missed ticks.

Time zones/DST: store schedules with an explicit tz; compute next_run in that tz.

Idempotency: at-least-once is inherent (worker may die after doing the work but before marking success); job owners must make jobs idempotent; provide the run_id in the payload to help.

Concurrency limits: check running count for the job before dispatch; skip or queue.

• Scheduler leader dies: lease expires; another replica takes over within seconds; missed ticks handled by misfire policy.
• Worker dies mid-run: heartbeat stops; watchdog re-queues after the lease; job runs again (idempotency required).
• Queue outage: scheduler keeps creating run rows with queued status; a replay job publishes them when the queue returns.
• Clock skew between schedulers: use DB time (now()) for due checks, not local clocks.
• Poison job (always fails): retries exhausted → dead status + alert; never blocks other jobs.