Skip to content

Align Windows _jobs_lock timeout with POSIX polling pattern - #62763

Open
rahulrao85 wants to merge 2 commits into
NousResearch:mainfrom
rahulrao85:fix/windows-jobs-lock-timeout
Open

Align Windows _jobs_lock timeout with POSIX polling pattern#62763
rahulrao85 wants to merge 2 commits into
NousResearch:mainfrom
rahulrao85:fix/windows-jobs-lock-timeout

Conversation

@rahulrao85

Copy link
Copy Markdown

Summary

The POSIX (fcntl) path in _jobs_lock()\ polls LOCK_NB with a 30-second bounded timeout before gracefully degrading to in-process-only locking (fix for issue #60703 — P1: cron silently dies on lock contention). The Windows (msvcrt) path used a blocking LK_LOCK call with ~10s internal retry and a generic WARNING log on failure.

This change replaces LK_LOCK with LK_NBLCK in a polling loop matching the POSIX pattern: same 30-second _JOBS_LOCK_TIMEOUT_SECONDS, same ERROR-level timeout log, same \lock_fd.close()\ degradation.

Changes

\cron/jobs.py:294-295\ → replaced single blocking LK_LOCK call with a polling loop using LK_NBLCK:

  • 30-second bounded wait with 100ms sleep between attempts
  • Explicit \logger.error()\ on timeout with filename and duration
  • Graceful degradation to in-process-only locking on timeout (same as POSIX)
  • \lock_fd.seek(0)\ inside loop ensures correct lock position on retry

Related

@rahulrao85
rahulrao85 force-pushed the fix/windows-jobs-lock-timeout branch from d307dfd to da1ff1d Compare July 11, 2026 17:35

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for carrying the #60703 bounded-lock guarantee over to Windows. Current main still has the blocking LK_LOCK call at cron/jobs.py:226; the POSIX polling implementation was added by 7ecc822e1 and the Windows branch was not included.

Problems

  • There is no regression coverage for the new Windows path. tests/cron/test_ticker_stall_60703.py:45 skips the whole bounded-lock module when fcntl is unavailable, and this PR changes only cron/jobs.py.

Suggested changes

  • Add a platform-independent mocked-msvcrt test for both successful LK_NBLCK acquisition and deadline degradation (ERROR log plus critical-section entry).

Automated hermes-sweeper review.

Comment thread cron/jobs.py
@@ -292,7 +292,29 @@ def _jobs_lock():
break
time.sleep(0.1)
elif msvcrt is not None:
getattr(msvcrt, "locking")(lock_fd.fileno(), getattr(msvcrt, "LK_LOCK"), 1)
_deadline = time.monotonic() + _JOBS_LOCK_TIMEOUT_SECONDS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add regression coverage for this LK_NBLCK path. The existing bounded-lock tests are module-skipped without fcntl (tests/cron/test_ticker_stall_60703.py:45), so they do not cover Windows. A fake msvcrt can verify retry-to-deadline, ERROR logging, degraded critical-section entry, and an uncontended acquisition without requiring Windows CI.

@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows P2 Medium — degraded but workaround exists labels Jul 11, 2026
@rahulrao85

Copy link
Copy Markdown
Author

Tests added as requested — \ ests/cron/test_windows_jobs_lock.py\ with a fake msvcrt module covering three scenarios:

  • Lock contention times out and degrades (ERROR log + degraded critical-section entry)
  • Uncontested LK_NBLCK acquisition is fast and silent
  • Transient contention recovers without triggering the timeout path

All 3 tests pass locally.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 11, 2026
The POSIX (fcntl) path polls LOCK_NB for 30 seconds with explicit
timeout logging before degrading to in-process-only locking (fix
for issue NousResearch#60703). The Windows (msvcrt) path used a blocking LK_LOCK
with ~10s internal retry and a generic warning on failure.

Replace LK_LOCK with LK_NBLCK in a polling loop matching the POSIX
pattern (same 30-second _JOBS_LOCK_TIMEOUT_SECONDS, same degradation
behavior, same ERROR-level log message). This ensures consistent
timeout handling across platforms and gives the Windows path the
same bounded contention resilience that POSIX has.

Signed-off-by: Rahul Rao <rahulrao85@gmail.com>
The bounded-lock tests in test_ticker_stall_60703.py are module-skipped
when fcntl is unavailable, so they never cover the Windows msvcrt path.

Add a new test file with a fake msvcrt module that exercises LK_NBLCK
polling on any platform:

- test_lock_times_out_and_degrades: always-fail mock verifies 30-second
  bounded timeout, ERROR logging, and degraded critical-section entry.
- test_uncontended_lock_is_fast_and_silent: immediate LK_NBLCK success.
- test_lock_recovers_after_transient_contention: 1 failure then success
  must not trigger the timeout path.

Signed-off-by: Rahul Rao <rahulrao85@gmail.com>
@rahulrao85
rahulrao85 force-pushed the fix/windows-jobs-lock-timeout branch from d166690 to b828a04 Compare July 25, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants