fix(cron): keep scheduler alive when job-store persistence fails - #5376
Open
rickererer wants to merge 1 commit into
Open
fix(cron): keep scheduler alive when job-store persistence fails#5376rickererer wants to merge 1 commit into
rickererer wants to merge 1 commit into
Conversation
A single OSError from _save_store() (disk full, permission change, locked file) escaped _on_timer's try/finally and killed the asyncio timer task, because _arm_timer() sits outside the block. All scheduled jobs silently stopped until restart or a manual re-arm via add_job/update_job/remove_job. Move _arm_timer() into the finally block and guard the whole tick body (including _load_store, which can persist during agent-binding migrations) so a transient persistence failure is logged and retried on the next tick instead of killing the scheduler. Add test_save_store_failure_does_not_kill_scheduler to cover the failure path that existing tests (which mock _arm_timer) never exercised.
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a silent failure mode where a single persistence error (e.g. disk full, permission change, locked file) inside
CronService._on_timerpermanently kills the cron scheduler:_save_store()raises, the exception escapes thetry/finally, and_arm_timer()— which sits outside the block — is never called again, so no further ticks are scheduled until the process restarts or a user manually re-arms the timer viaadd_job/update_job/remove_job/enable_job.Root cause
nanobot/cron/service.py,_on_timer():_save_store()->_atomic_write()performsopen()/fsync()/os.replace(); anyOSErrorpropagates.finallyblock only decrements_active_executions; the re-arm (_arm_timer()) lives after the block.tick()creates the asyncio task with no exception handling, so the exception kills_timer_task("Task exception was never retrieved")._load_jobspreserves corrupt stores as.corrupt-<ts>backups); the write path had none.Changes
nanobot/cron/service.py:_arm_timer()into thefinallyblock so the next tick is always scheduled, even on unexpected failures._save_store()in its owntry/exceptthat logs and keeps the in-memory store; the next tick retries the save.store is Nonebranch (it no longer needs its own_arm_timer()sincefinallycovers it).tests/cron/test_cron_service.py:test_save_store_failure_does_not_kill_scheduler: forces_save_storeto raise, then asserts the service is still re-armed and can run a due job on the next healthy tick.Validation
python -m pytest tests/cron/test_cron_service.py -qpython -m ruff check nanobot/cron/service.py tests/cron/test_cron_service.pyBehavior notes