fix(cron): only advance next_run_at for jobs that are actually dispatched - #60946
Open
wildnewton wants to merge 2 commits into
Open
fix(cron): only advance next_run_at for jobs that are actually dispatched#60946wildnewton wants to merge 2 commits into
wildnewton wants to merge 2 commits into
Conversation
added 2 commits
July 8, 2026 22:10
Add failing test demonstrating that when a due job is skipped by the in-flight dedup guard (already running from a previous tick), its next_run_at is still advanced via advance_next_run, causing it to skip an entire scheduling window without ever executing. The test asserts advance_next_run is NOT called for skipped jobs.
…ched Previously, advance_next_run was called for ALL due jobs before the in-flight dedup guard in tick(). When a job was silently skipped (already running from a previous tick), its next_run_at was still advanced, causing it to skip an entire scheduling window without ever executing. Now advance_next_run is called only inside _submit_with_guard, after the job is successfully claimed for dispatch. Skipped jobs keep their next_run_at unchanged so they are retried on the next tick. Fixes: job 488dec122600 skipped at 2026-07-08 21:00, next_run_at advanced to 2026-07-09 without execution.
teknium1
reviewed
Jul 10, 2026
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for isolating the pre-guard advancement bug; current main still has it at cron/scheduler.py:3554-3560, before the already-running return at cron/scheduler.py:3627-3630.
Problems
cron/scheduler.py:2937advances beforepool.submitat:2948. Current main now handles a shutdown race wherepool.submitfails and releases the in-memory claim (cron/scheduler.py:3641-3653) so the job stays due. The proposed ordering would still persist an advance when that submission never succeeds, contrary to the PR's "actually dispatched" contract.
Suggested changes
- Integrate this move with the current guarded submission path so rejected submissions leave
next_run_atunchanged without introducing a worker-start race. - Add a regression test for a
pool.submitshutdown failure assertingadvance_next_runwas not called, in addition to the already-running case.
Automated hermes-sweeper review.
| # Advance next_run_at NOW — this job is being dispatched, so the | ||
| # scheduler won't pick it up again until after it completes. | ||
| # (mark_job_run overwrites next_run_at on completion.) | ||
| advance_next_run(job_id) |
Contributor
There was a problem hiding this comment.
advance_next_run still runs before pool.submit. Current main has a shutdown-race path where submission fails and the in-memory claim is released; in that case this persistent advance means a job that was never enqueued is no longer due. Integrate this with the current submit-failure handling and add a regression test.
19 tasks
This was referenced Aug 1, 2026
spfcraze
added a commit
to spfcraze/hermes-agent
that referenced
this pull request
Aug 1, 2026
The scheduler's pre-dispatch loop called advance_next_run per due job — one full load_jobs() + one full save_jobs() of the jobs file each — so N due jobs cost N reads + N writes of the whole file (gateway-restart catch-up or co-scheduled bursts). advance_next_runs() does one load + at most one save for the whole due set with identical per-job semantics; advance_next_run() is now a thin wrapper over it. Measured (50 due recurring jobs, real jobs file): 107.9 ms -> 2.5 ms (45x; 50 loads + 50 saves -> 1 + 1). Tests: batch advances recurring and skips one-shots, single load + save I/O pin (fails pre-fix — no such function), no save when nothing advances, and per-job wrapper semantics unchanged. Related: NousResearch#60946 and NousResearch#75833 both restructure this loop's call site for correctness — neither addresses the I/O cost, and this batch primitive composes with either dispatch design; happy to rebase onto whichever lands first.
kshitijk4poor
pushed a commit
that referenced
this pull request
Aug 2, 2026
The scheduler's pre-dispatch loop called advance_next_run per due job — one full load_jobs() + one full save_jobs() of the jobs file each — so N due jobs cost N reads + N writes of the whole file (gateway-restart catch-up or co-scheduled bursts). advance_next_runs() does one load + at most one save for the whole due set with identical per-job semantics; advance_next_run() is now a thin wrapper over it. Measured (50 due recurring jobs, real jobs file): 107.9 ms -> 2.5 ms (45x; 50 loads + 50 saves -> 1 + 1). Tests: batch advances recurring and skips one-shots, single load + save I/O pin (fails pre-fix — no such function), no save when nothing advances, and per-job wrapper semantics unchanged. Related: #60946 and #75833 both restructure this loop's call site for correctness — neither addresses the I/O cost, and this batch primitive composes with either dispatch design; happy to rebase onto whichever lands first.
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
The scheduler's pre-dispatch loop called advance_next_run per due job — one full load_jobs() + one full save_jobs() of the jobs file each — so N due jobs cost N reads + N writes of the whole file (gateway-restart catch-up or co-scheduled bursts). advance_next_runs() does one load + at most one save for the whole due set with identical per-job semantics; advance_next_run() is now a thin wrapper over it. Measured (50 due recurring jobs, real jobs file): 107.9 ms -> 2.5 ms (45x; 50 loads + 50 saves -> 1 + 1). Tests: batch advances recurring and skips one-shots, single load + save I/O pin (fails pre-fix — no such function), no save when nothing advances, and per-job wrapper semantics unchanged. Related: NousResearch#60946 and NousResearch#75833 both restructure this loop's call site for correctness — neither addresses the I/O cost, and this batch primitive composes with either dispatch design; happy to rebase onto whichever lands first.
Open
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a cron job is silently skipped (not dequeued for execution because it's already running from a previous tick), the scheduler still advances
next_run_atviaadvance_next_run(). This means the job never retries and appears to have run when it hasn't.Evidence: job
488dec122600was skipped at 2026-07-08 21:00 andnext_run_atadvanced to 2026-07-09 without execution.Root Cause
In
tick(),advance_next_run()was called for ALL due jobs BEFORE the in-flight dedup guard (_submit_with_guard). When a job was skipped (already in_running_job_ids), itsnext_run_athad already been advanced.Fix
Move
advance_next_run()from the pre-dispatch loop into_submit_with_guard, calling it ONLY when the job is successfully claimed for dispatch (not already running). Skipped jobs keep theirnext_run_atunchanged so they are retried on the next tick.Changes
cron/scheduler.py: Moveadvance_next_runfrom pre-dispatch loop into_submit_with_guard, only after successful claimtests/cron/test_scheduler.py: Add two tests:advance_next_runcalledadvance_next_runcalled (at-most-once semantics preserved)TDD Commits
test(cron): RED — skipped job still has advance_next_run calledfix(cron): only advance next_run_at for jobs that are actually dispatched