fix(cron): guard stale-entry removal with liveness check - #62013
Closed
liuhao1024 wants to merge 1 commit into
Closed
fix(cron): guard stale-entry removal with liveness check#62013liuhao1024 wants to merge 1 commit into
liuhao1024 wants to merge 1 commit into
Conversation
19 tasks
Contributor
|
Thanks for submitting the first focused fix for #62002. Reviewed against current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes a race condition in cron one-shot job stale-entry recovery where
get_due_jobs()deletes a job while its run is still alive.The one-shot dispatch-limit guard (#38758) removes stale entries when
completed >= timesand therun_claimTTL is expired. However, it only checked the claim age as a liveness signal. A one-shot whose run legitimately outlives the TTL (default 1800s) — for example, stalled on network I/O or a laptop that slept mid-run — satisfies the same condition while the run is still executing.This caused the job record to be deleted mid-flight:
cronjob(action='list')shows the job as gone → looks like it never ranmark_job_run()finds nothing →last_run_at/last_status/last_delivery_errorare lostThe fix adds a same-process liveness check before removal: consult
cron.scheduler.get_running_job_ids()to see if the job is still executing in this process. If it is, skip removal regardless of claim age. This covers the common single-gateway case where the ticker and run share the same process.Related Issue
Fixes #62002
Type of Change
Changes Made
cron/jobs.py: Inget_due_jobs(), before removing a stale one-shot entry, check if the job is still executing in this process viacron.scheduler.get_running_job_ids(). If it is, skip removal and log that the run is still alive.How to Test
repeat=1and a prompt that stalls >1800s (e.g., blackhole the provider endpoint after the first tool call).run_claimage > TTL.get_due_jobs()logs "one-shot dispatch limit reached but still executing in this process — skipping removal" and does NOT delete the job.mark_job_run()should find the job and updatelast_run_at/last_status/last_delivery_errorcorrectly.cronjob(action='list')shows the job with correct completion status.Regression test: Existing cron tests pass (416 tests in
tests/cron/test_jobs.py).Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/cron/ -qand all tests pass (416 passed, 1 pre-existing flaky test unrelated to this change)test_jobs.pytests; same-process check guards deletion)Documentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/A