Distributed Job Scheduler

A distributed job scheduler runs tasks at specific times or on recurring schedules (cron), across a fleet of workers, without missing or duplicating runs. Think of it as cron for a whole company: reports, billing runs, reminders, and cleanup jobs, many millions per day.

Clarify requirements

Functional

• Submit one-off jobs ("run at T") and recurring jobs (cron expressions).
• Jobs carry a payload and a target (an HTTP endpoint, a queue, or a container/task).
• Retries with backoff on failure; max attempts; dead-lettering.
• Query job status and history; cancel or pause jobs.
• Optional: priorities, dependencies between jobs (DAGs) as an extension.

Non-functional

• 100M job executions/day, with peaks at round times (every hour at :00, midnight).
• Start a job within ~1–5 seconds of its scheduled time.
• At-least-once execution with deduplication so effects happen once.
• Highly available: no single scheduler failure stops jobs.
• Durable: accepted jobs are never lost.