Skip to content

fix(cron): release tick lock before running jobs - #32786

Closed
Ordinath wants to merge 1 commit into
NousResearch:mainfrom
Ordinath:upstream-fix/cron-release-tick-lock-before-execution
Closed

fix(cron): release tick lock before running jobs#32786
Ordinath wants to merge 1 commit into
NousResearch:mainfrom
Ordinath:upstream-fix/cron-release-tick-lock-before-execution

Conversation

@Ordinath

Copy link
Copy Markdown

Summary

  • release the global cron .tick.lock after selecting due jobs and advancing next_run_at
  • keep job execution outside the scheduler metadata critical section
  • guard the final cleanup path so it only unlocks/closes the fd when still held
  • add a regression test proving a second tick can acquire the lock while a first tick is still running a slow job

Why

A long-running or wedged cron job can currently hold ~/.hermes/cron/.tick.lock for the entire job runtime. Later ticks return 0 with the lock held, so unrelated due jobs and smoke tests stay stuck behind the slow job.

This reproduces the scheduler wedge described in #3752 and matches the lock-scope bug discussed in #27492 / #29350.

Test plan

  • uv run pytest tests/cron/test_scheduler.py::TestParallelTick -q -o addopts=''

Result: 4 passed in 1.33s

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management labels May 26, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #27492 (same fix: release cron tick file lock before job execution). Also overlaps with #21901 (same lock narrowing + heartbeat ticker) and closed #29350 (same lock narrowing + stream timeouts). See also original issue #3752.

@Ordinath

Copy link
Copy Markdown
Author

Closing this PR for now. Opened prematurely during local diagnosis; will revisit only if/when there is explicit agreement on the fix.

@Ordinath Ordinath closed this May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants