Distributed Task Scheduling
Work that shouldn't run inline with a request needs somewhere else to run — and something to decide where and when.
Advanced
TopNotchNote
Notes that matter
work that shouldn't run inline with a request
Some work genuinely doesn't belong in the request-response path. Regenerating a restaurant's weekly sales report, resizing every photo a restaurant just uploaded, or running nightly reconciliation between the payments ledger and the orders table — none of these should make a customer wait, and none of them need to run this millisecond. A distributed task scheduler is the system responsible for taking a large pool of such tasks and matching them to available compute resources efficiently.
where this shows up
| Context | Example |
|---|---|
| Single machine | The OS scheduler deciding which process gets the CPU next. |
| Cloud infrastructure | A Kubernetes-style scheduler placing containers onto available nodes based on resource requests. |
| Large distributed systems | A batch job scheduler running millions of background tasks — report generation, image processing, data pipeline steps — across a fleet of worker machines. |
This page focuses on the third case, since it's the one that shows up constantly in system design discussions of large platforms.
what the scheduler needs to do
| Requirement | What it means |
|---|---|
| Accept and track tasks | Users (or other services) submit tasks; the scheduler tracks each one's current state (queued, running, succeeded, failed). |
| Allocate resources fairly | Match each task to a worker with available capacity, without starving low-priority tasks indefinitely. |
| Handle failure | A worker that crashes mid-task should have that task rescheduled elsewhere, not silently lost. |
| Bounded waiting time | A task shouldn't sit in the queue indefinitely just because higher-priority tasks keep arriving. |
core architecture
| Component | Role |
|---|---|
| Task queue | Holds submitted tasks awaiting assignment — often itself a distributed messaging queue, sometimes with multiple priority tiers. |
| Scheduler | Matches queued tasks to available workers, based on each worker's current load and each task's resource requirements and priority. |
| Worker pool | The machines that actually execute tasks and report status back. |
| Task metadata store | Tracks each task's current state, retry count, and results — needed so a crashed scheduler can resume from where it left off rather than losing track of in-flight work. |
scheduling policies
| Policy | Behavior |
|---|---|
| FIFO | Tasks run in submission order — simple, but a large low-priority batch job can delay urgent small tasks behind it. |
| Priority-based | Higher-priority tasks jump ahead of lower-priority ones in the queue. |
| Fair-share | Capacity is divided across tenants/teams so one heavy user of the scheduler can't starve everyone else, even without explicit priorities. |
Priority-based scheduling alone risks starving low-priority tasks entirely if high-priority work never stops arriving — a common mitigation is priority aging, where a task's effective priority rises the longer it waits, guaranteeing it eventually runs.
retries and idempotency, again
A worker can crash mid-task, and the scheduler generally can't know for certain whether the task's side effects already completed. The safe default is to reschedule the task elsewhere and retry it — which means, exactly as with message queue consumers, every task handler needs to be idempotent, safe to run more than once without corrupting results (e.g. re-resizing an already-resized photo should just overwrite the same output, not create a duplicate).