Skip to content

fix(cron): preserve heartbeated external fire claims - #105724

Open
bzarzoza wants to merge 1 commit into
NousResearch:mainfrom
bzarzoza:fix/cron-live-fire-claim-sweep
Open

bzarzoza wants to merge 1 commit into
NousResearch:mainfrom
bzarzoza:fix/cron-live-fire-claim-sweep

Conversation

@bzarzoza

@bzarzoza bzarzoza commented Sep 8, 2026 •

Copy link
Copy Markdown

Problem

A restart-safe external cron worker can legitimately run longer than the local in-flight guard allowance. The local process has no live Future for that worker, so sweep_stale_inflight() can force-release the job after 30 minutes even while the worker is heartbeating its durable fire_claim. The subsequent completion clears the claim and the still-running worker discards its result as stale.

This was reproduced on two external cron jobs; each ran for about 30m46s and ended with Fire claim ownership lost; stale result was discarded.

Fix

  • Read the current durable fire claim under its existing per-job fence.
  • Do not age-release a missing/finished local Future while that durable claim remains live.
  • Preserve the existing stale-wedge recovery once the durable claim expires.

Tests

scripts/run_tests.sh tests/cron/test_recurring_wedge_selfheal.py -k 'live_durable_fire_claim_outlives_local_future or stale_claim_self_heals_and_redispatches or guard_stats_reported'

3 passed.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management labels Sep 8, 2026
@Enough1122

Copy link
Copy Markdown

AI code review — automated review for reference; please use your judgment.

PR #105724 — preserve heartbeated external fire claims in cron sweep

Correct race fix: before the age-based sweep reaps a stale in-flight entry, it checks the durable fire claim (scheduler.py:36 via new live_fire_claim_owner in jobs.py:9), so a restart-safe external worker that outlives the local Future keeps its claim. Test simulates the exact wedge (cleared futures + 6h-old running_since + live claim → sweep returns []).

  • Non-blocking — the from cron.jobs import live_fire_claim_owner import sits inside the sweep loop body; hoist to module top (or at least function top) to avoid re-executing the import per stale job.
  • Non-blocking — every stale candidate now costs a _under_fire_fence + _with_job round-trip (file lock + read). Sweeps over many stale jobs pay O(n) fenced reads; acceptable, but consider batching the claim check if sweep latency ever matters.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants