Cron is one of the oldest Unix utilities and one of the hardest to make reliable in a distributed setting. This chapter walks through how Google built a cron service that runs jobs across a whole datacenter with the reliability of a single machine's cron, and the design decisions that fall out. The key question is deceptively simple: if the scheduler crashes mid-launch, do you run the job again, or not at all?

The chapter is a good exercise in applying chapter 23: a tiny amount of critical state (which launches have happened) managed with consensus.

In one minute

• Single-machine cron reads crontab, forks jobs at the scheduled time, and keeps no state about what has run. If the machine is down, the job is skipped. This is acceptable for one machine and not for a service.
• Some jobs are idempotent (a GC run) and can be safely repeated; others are not (sending a billing email). Some are fine to skip; others are not. The scheduler must let the owner choose, and the rule is that skipping is safer than double-running for most jobs, because a skipped job can be rerun by hand and a double-run cannot be undone.
• Distributed cron must track launches, which is state that must survive the scheduler's failure. Options: store it externally in a distributed filesystem, or keep it in the cron service itself using Paxos.
• Google's design: a cron service with several replicas; a leader elected by Paxos performs the scheduling and launching; followers track the leader's state through the Paxos log and take over on failure.
• The leader records two synchronisation points per launch in the Paxos log: "launch starting" before issuing the call to the datacenter scheduler, and "launch finished" after it returns. A new leader looks for launches that started but did not finish and resolves them by querying the scheduler, assuming the owner's choice if that is inconclusive.
• Resolution of partial failures: the datacenter scheduler (Borg) is treated as a source of truth about whether a job exists, so the cron service can check rather than guess. Job names include the scheduled time to make lookups unambiguous.
• Storing the state: keep it small. Only the current configuration and recent launch records are in Paxos; logs are snapshotted and stored in local disk and a distributed filesystem.
• Running large cron: the leader must not become a bottleneck; keep per-job work small and push the heavy lifting to the datacenter scheduler.
• Thundering herd: everyone schedules at midnight or on the hour. Google added a ? character to the crontab syntax that means "any value, pick one deterministically", which spreads load.

Cron and reliability perspective

The crontab format is well known; the reliability trade-off is not. For a single machine, cron's failure mode is to skip jobs while down, and because it keeps no state it never double-runs. Moving to a service changes what "down" means and introduces the possibility of the scheduler dying after launching a job but before recording that it did so. The job owner knows which failure mode is worse for their job, and the scheduler should let them say so.

Cron at large scale

Jobs run as Borg jobs, so cron's job is to tell Borg to start a job at the right time. Cron itself needs to be more reliable than the jobs it schedules, which means it must run as a replicated service, and the replicas must agree on what has been launched.

Building cron at Google

Tracking the state of cron jobs. Two choices: keep the state outside cron in a replicated store, or keep it in cron and make cron a Paxos group. Google chose the latter to limit dependencies, since cron is a low-level service that many other services depend on.

The use of Paxos. Cron runs as a small group of replicas. Paxos elects a leader; only the leader launches jobs. The leader writes its intentions to the Paxos log so followers have an up-to-date view of which launches are in flight, and when the leader fails a follower takes over with that knowledge. Leader election uses a lease, and the leader must stop acting when it loses the lease.

The roles of the leader and the follower. The leader schedules, talks to Borg, and writes to the log. Followers replay the log and are ready to become leader. Both keep the configuration in memory so a new leader can act immediately.

Resolving partial failures. Launching a job may require several RPCs to Borg, and the leader may die between any two of them. Hence the two sync points: the launch intent is logged before the first RPC, and completion after the last. On takeover, a new leader with an incomplete launch asks Borg whether the job exists. If Borg's answer is definitive, cron finishes or records accordingly; if not, cron applies the owner's preference of skip or rerun. Naming jobs by their scheduled time makes it possible to ask Borg a precise question.

Storing the state. Paxos logs grow forever if not compacted. Cron snapshots its state periodically, keeps the snapshot on the local disk of each replica for fast restart, and also writes it to a distributed filesystem for recovery when all replicas fail. The state kept is deliberately tiny so that snapshots and replays are quick.

Running large cron. A single leader could be a bottleneck if each launch were expensive. Launches are cheap RPCs, and Borg does the heavy work. Still, the thundering herd of thousands of jobs scheduled at the same instant stresses both cron and Borg. The ? extension spreads launches across the hour or the day with a deterministic choice so that the owner can predict it. The chapter also warns against jobs that themselves spawn many jobs at once and against retry loops in misconfigured crontabs.

Summary

A distributed cron is a distributed consensus problem with a small amount of state. Decide per job whether skipping or double-running is worse, record intent before acting and completion after, make the downstream system queryable so you can recover from a partial launch, keep the consensus state small, and spread the schedule to avoid everyone starting at once.