Repository navigation
cron: catch up runs that came due while the machine was paused - #536
Merged
Merged
Conversation
The scheduler waits on the event loop's monotonic timer but decides what is due by the wall clock. Across host sleep, suspend or a VM snapshot restore the monotonic clock stands still while the wall clock is stepped forward, so a run that came due during the pause either waited out a timer armed before it or was dropped as missed once it started more than the misfire grace late. The cron service now samples both clocks every 5 seconds, and again right before the scheduler looks for due jobs, so whichever runs first after a resume sees the step. On a forward step of 5 seconds or more, every job it scheduled whose next run came due during the pause moves to its next fire time after the pause (interval jobs stay on their anchored grid), and the jobs with catchup plus the source runners run once through the normal wrappers, so gates, lock and the run log apply. Jobs with catchup: false, cleanup and the wakeup sweep only move on. A catch-up waits for one still running, including the startup catch-up.
The clock watcher awaited the whole catch-up, so it stopped sampling while the caught-up runs went on, and a second pause in that time was noticed only when the scheduler next looked. The watcher and the scheduler's hook now share one check that never waits: the triggers move at once and the catch-up runs as a tracked task. Moving the triggers also takes one job at a time: a job that cannot be moved is logged and left to the scheduler, and the jobs after it still move instead of being dropped as late runs.
The scheduler starts a job at most once at a time, but it only counts the runs it started itself. A catch-up after a pause ran through the run wrapper directly, so a run still going across the pause got a concurrent second run, and a catch-up still going when the job's next fire came got a scheduled twin; without lock that is two agent runs working the same queue. Both run wrappers now record every run in progress, by job id and by how it was started (schedule, catch-up or manual). A catch-up is skipped while any run of the job is going, as the scheduler skips a fire that finds the previous run going, and a scheduled fire is skipped while a catch-up of the job is going. A manual run is never skipped. Each skip is logged at INFO. The set of runs in progress is kept apart from the wrappers so other ways of starting a run can use the same check.
A catch-up kept the job objects it found when the pause was noticed, and could wait a long time for an earlier catch-up to finish. By then a reload might have removed or changed a job, or the scheduler might have run it on its moved trigger, and the catch-up still ran the old job, again; a job due in two queued pauses ran twice. A pause now queues job ids, once each, with the number of runs each job had started. When the catch-up gets its turn it looks every id up on the scheduler: a job that is gone is skipped, a changed one runs as it is now, and one that has started a run since it was queued is not caught up on top of it. Each skip is logged.
Moving every trigger that came due during a pause also moved one due a few seconds before the resume. The scheduler used to start such a run late, inside its 30 second misfire grace; now a job with catchup: false skipped it without a trace, although the docs said it still ran. A machine woken just in time for a job, and resumed a few seconds after it came due, skipped the very job it was woken for. A trigger now moves only when its run came due more than the grace before the resume; anything later is left to the scheduler, which runs it as it always did. A job with catchup: false that is moved gets a missed row in cron_logs for the run it skipped, as a late run does. The daily cleanup catches up once like the jobs that want it: on a machine that is mostly paused it may otherwise never run.
The catch-up runs started within seconds of a resume. On a laptop the network is often not back yet at that point (Wi-Fi rejoining, DNS), so a source runner or an agent turn started then failed, was logged as an error and waited for its next slot. The runs now start 15 seconds after the pause is noticed. Moving the triggers past the pause still happens at once, so the scheduler never sees the old fire times, and a run that comes due on its moved trigger within those seconds runs normally and is not caught up again. The log line for a pause says when its catch-up will start.
stop() cancelled the clock watcher and the catch-ups and waited for them with no limit, while the scheduler kept running: it could start runs, and its clock check could start new catch-ups that nobody would cancel. A task that did not wind down kept the daemon from exiting. stop() now marks the service as stopping and pauses the scheduler first. From then on the clock check, the watcher and a catch-up still waiting for its turn do nothing. The cancelled tasks get as long as the gateway gives its own background tasks to finish; whatever is left is logged and the scheduler shuts down anyway. A second stop() returns at once.
A jump of exactly five seconds counts as a pause and one just under it does not, read off clocks the test controls alone, so a changed comparison no longer goes unnoticed. The service tests all ran in UTC. New York now covers both 2026 DST changes: the local clock jumping an hour either way is not a pause, and neither is a wall clock set back an hour as daylight saving time ends. A pause across the spring change moves an hourly, a nightly and a four-hourly job each to its right next run.
The overview of the pause handling still said every trigger that came due is moved; one that the scheduler would still run late stays where it is.
alex-clickhouse
added this pull request to stack #540
October 21, 2026 14:46
alex-clickhouse
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cron: catch up runs that came due while the machine was paused
APScheduler arms its next fire on the monotonic clock, which does not advance while a machine sleeps, is suspended, or is restored from a snapshot. After such a pause a job armed "in 3 hours" fired 3 hours after the resume and was then dropped as missed. Nerve now detects the pause and recovers.
catchup: truejobs and source runners then run once, concurrently, after a short settle delay for the network;catchup: falsejobs are recorded as missed;cleanupcatches up once. Jobs that came due within the scheduler's own 30 s grace are left to it.stop()stops the watcher and waits with a bound.docs/cron.md("After a pause").Tests: 31 new in
tests/test_cron_resume.py(fake clocks against a real scheduler and SQLite: a 10-min crontab job over a 3 h step runs once, interval jobs keep their grid,catchup: falsenot run,lockrespected, a 2 s step ignored, DST-backward steps ignored, overlap and stale-catch-up cases). Full suite 6,667 passed, 39 skipped; ruff unchanged.🤖 Generated with Claude Code