Designing a Job Scheduler: Remember the Work, Expect Retries

The Monday some parents got two reports, and some got none

Aditi runs the platform for VidyaSetu, a school-management service used by 600 schools. Every Monday at 08:00, each school's weekly progress reports are generated and emailed to parents. One Monday goes badly. A scheduler restart at 07:59 means some schools' reports never go out, and a network hiccup means some parents get two copies. (Aditi, VidyaSetu and every number here are made up for teaching.)

Her design starts from a school timetable. A timetable says when lessons should begin, but someone still has to open the classroom and teach. A job scheduler has the same split: it decides when work is due, and workers do it.

Three ideas must be kept apart:

• The job definition is "every Monday at 08:00, Asia/Kolkata."
• A scheduled run is "this Monday's report run": one occurrence.
• An attempt is "the second try at this Monday's run, after the first worker crashed."

Keeping these three separate is what makes scheduling easy to reason about, and every incident below came from mixing them up.

In 30 seconds, as an architect would frame it: "I model definitions, runs and attempts as separate records, with a unique key on (job, schedule version, scheduled instant) so rediscovery never duplicates a run. Scanners find work due at or before now, and create the run and an outbox entry in one transaction. A publisher feeds a queue, and workers claim runs with a lease and a fencing number. Because leases can't stop a paused worker from waking up, the target checks the fencing number and the effect is idempotent. Time zones, misfire policy and cancellation are explicit product rules."

1. The three records

• A lease gives a worker temporary permission to act on a run.
• A fencing number increases every time a new owner takes over, so the resources it touches can reject work from an old owner.

"Two scheduler copies both created Monday's run." VidyaSetu runs two scheduler processes for safety, and both noticed Monday 08:00 at the same moment. The fix is a unique constraint on (job_id, schedule_version, scheduled_instant). Both may discover the occurrence, but only one insert succeeds, and the other gets a conflict and moves on. A manual "run now" needs its own request identity, rather than pretending to be a scheduled occurrence.