fix(cron): never stale-remove a one-shot whose run is still alive (#62002)

[PRATHAMESH75]

What does this PR do?

Stops get_due_jobs()'s one-shot stale-entry recovery (#38758) from deleting a job record while its run is still alive.

The recovery treats completed >= times plus an expired run_claim (#59229) as proof the claiming tick died. But the TTL is only an age heuristic: a run stalled on network I/O — or a laptop that slept mid-run — legitimately outlives it while very much alive. In the incident in the issue, a one-shot's provider stream dead-stalled for ~40 minutes; the job record was deleted mid-flight, cronjob(action='list') showed the job gone, and when the run finally completed (and delivered successfully), mark_job_run() found nothing to update — last_run_at / last_status / last_delivery_error were lost. Had delivery failed, there would have been zero durable trace.

The fix adds the two liveness signals proposed in the issue:

1. Same process (the common single-gateway case, and the incident's): before removing a dispatch-limit-reached one-shot, get_due_jobs consults the scheduler's get_running_job_ids() (lazily imported — the scheduler already imports cron.jobs, so a module-level import would be circular). If the job is still running in this process it is slow, not stale: keep the entry and skip it this tick. Once the run is actually gone, recovery removes the entry exactly as before.
2. Cross process: run_job's monitor loop now refreshes run_claim.at every 60s while the run is alive, via a new heartbeat_run_claim() in cron/jobs.py (called from the existing 5s poll; the HERMES_CRON_TIMEOUT=0 unlimited branch, which previously blocked without polling, now polls for one-shots too). An expired claim therefore really does mean "the owner died", and the TTL keeps its short dead-owner-detector semantics.

mark_job_run() still clears the claim on completion; for a run that never stamped a claim (e.g. a manual run) the heartbeat is a no-op. No new env vars or config keys; no behavior change for recurring jobs or for genuinely dead ticks.

Related Issue

Fixes #62002

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• cron/jobs.py: new _job_running_in_this_process() helper (lazy scheduler import) and a liveness guard in front of the "one-shot dispatch limit reached — removing stale due entry" removal in get_due_jobs(); new heartbeat_run_claim(job_id) that refreshes run_claim.at under the jobs lock and no-ops when the job or claim is gone.
• cron/scheduler.py: run_job() heartbeats the one-shot's run claim every 60s from the inactivity-monitor poll loop, and the unlimited-timeout (HERMES_CRON_TIMEOUT=0) branch now polls for one-shots instead of blocking so the heartbeat still runs; imports time and heartbeat_run_claim.
• tests/cron/test_jobs.py: three new tests — the incident's store shape (completed 1/1, claim older than TTL) survives while the running set holds the job and is removed once it doesn't; a heartbeat-refreshed claim keeps a long run claimed past the original TTL horizon so mark_job_run lands on a live record; heartbeat is a safe no-op without a claim.

How to Test

1. scripts/run_tests.sh tests/cron/test_jobs.py tests/cron/test_scheduler.py → all pass (128 + 155 tests, 0 failed).
2. Regression shape: tests/cron/test_jobs.py::TestGetDueJobs::test_stale_maxed_oneshot_kept_while_running_in_this_process reproduces the incident (repeat 1/1, run_claim older than the TTL) and asserts the record is kept while the run is alive in-process — and cleaned up as before once it isn't.
3. Full cron suite: scripts/run_tests.sh tests/cron/ → 673 passed, 0 failed (macOS).

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass (28 pre-existing failures on plain upstream/main in my environment are unchanged by this PR; every cron test passes)
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS (Darwin 25.5.0)

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A (docstrings/comments in the changed functions)
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A (no config changes)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A (pure-Python, no platform-specific paths or process APIs)
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A (no tool surface change)

Credits

The fix follows the remediation proposed by @jeff-mettel in #62002 — the same-process get_running_job_ids() check and the run-claim heartbeat are their design; this PR implements it (with tests). Thanks for the detailed incident analysis and log forensics.

Infographic

[teknium1]

Maintainer review note: #62013 by @liuhao1024 was the earliest focused submission of the same-process liveness guard. #62014 is the selected implementation because it also heartbeats the claim for cross-process schedulers; both contributors are credited for resolving #62002.