fix(cron): malformed job records no longer freeze the scheduler (class fix) - #61723
Merged
Conversation
A cron record authored by a direct jobs.json edit that bypassed
add_job() can lack an "id" key (older writers used "job_id"). Every
site in _get_due_jobs_locked indexes job["id"] eagerly — both the
logging helpers (job.get("name", job["id"]) evaluates the default
argument unconditionally) and the 'for rj in raw_jobs: if rj["id"] ==
job["id"]' persistence loops. A single malformed record therefore
raised KeyError mid-tick, aborting the entire scan before save_jobs()
ran. Result: healthy jobs' fast-forwarded next_run_at was computed in
memory then discarded on the exception unwind, freezing the whole
profile's scheduler in a per-minute loop (observed dormant for weeks).
Fix: normalize id-less records at the top of _get_due_jobs_locked before
anything keys off job["id"] — recover the id from a drifted "job_id"
key when present, else synthesize one via uuid4, and persist. This
repairs the whole bug class at the source rather than guarding each of
the ~12 downstream index sites.
Adds a regression test that fails with KeyError on the current code and
passes with the fix, asserting a healthy sibling job is still returned
when an id-less record shares the store.
A job record in jobs.json can have a non-dict 'schedule' value (null, string,
etc.) from direct edit or old writers.
In _get_due_jobs_locked:
schedule = job.get('schedule', {})
kind = schedule.get('kind')
This (and direct schedule['kind'] in compute_next_run etc.) raises and
aborts the entire due-jobs scan before save_jobs() or advancing next_run_at
for healthy jobs. Exactly the same failure mode as the id-less job P1.
Fix: normalize non-dict schedules to {} early (before any use), matching the
defense added for id-less records. Also added defensive guards in compute
functions.
Added regression test that a bad schedule does not crash and healthy sibling
is still returned.
Refs similar pattern in #61382.
One bad next_run_at value in jobs.json aborts the due-jobs scan with ValueError from fromisoformat, before any save_jobs, so siblings lose progress (fast-forwards etc). Early normalization in _get_due_jobs_locked + defensive parses in compute_next_run / _recoverable_oneshot_run_at. Added test_bad_next_run_at_does_not_crash_or_block_sibling_jobs.
… class Structural completion of the malformed-job freeze fixes (#61382 id-less, #61525 non-dict schedule, #61581 bad next_run_at): wrap the per-job body of _get_due_jobs_locked in try/except so any FUTURE malformed-field variant degrades to skipping that one job for the tick instead of aborting the scan before save_jobs() and freezing the whole profile's scheduler. Also: restore test_repeated_concurrent_runs_accumulate_completed_count to TestMarkJobRunConcurrency (accidentally re-parented by the #61581 diff), add a containment regression test, and AUTHOR_MAP for hydracoco7. E2E: one jobs.json carrying all five malformed shapes (drifted job_id, missing id, null schedule, garbage next_run_at, non-string last_run_at) plus a healthy sibling — single tick contains all five, sibling fires, repairs persist, second tick stable. 670 cron tests green.
alt-glitch
added
type/bug
Collaborator
Related: #61382 (id-less job repair, @hydracoco7), #61525 (non-dict schedule, @necoweb3), #61581 (malformed next_run_at/last_run_at, @necoweb3) — this PR consolidates all three (authorship preserved) and adds the structural per-job try/except containment guard none of them had. Also related to the closed superset #51267 (per-job quarantine) and the narrowest open #50377 (guards the exact fromisoformat parse). Same silent-whole-cron-scheduler-freeze family; this is the canonical class-fix. Maintainer to pick between this consolidation and the individual source PRs. |
This was referenced Jul 10, 2026
Open
4 tasks
This was referenced Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A single malformed record in
cron/jobs.jsoncan no longer freeze the whole scheduler. The due scan trusted persisted job shape; one bad record (missingid, non-dictschedule, unparseablenext_run_at/last_run_at) raised mid-loop beforesave_jobs(), discarding every sibling's recovered state — the scheduler re-crashed identically every tick, forever.Consolidates three contributor fixes (cherry-picked, authorship preserved) and adds the structural guard none of them had:
job_id, else synthesize)schedulenormalizationnext_run_at/last_run_atstripping +compute_next_runhardening (cron portion only; its unrelated process_registry change was left for separate review)Changes
cron/jobs.py: three shape normalizations at the top of_get_due_jobs_locked+ per-job containment guardtests/cron/test_jobs.py: three contributor regression tests + containment test; restoredtest_repeated_concurrent_runs_accumulate_completed_countto its original class (accidentally re-parented by the fix(cron): malformed next_run_at no longer freezes the scheduler #61581 diff)scripts/release.py: AUTHOR_MAP for hydracoco7Validation
Infographic