Skip to content

fix(cron): keep a run's real outcome when an unknown-fenced delivery meets a claim miss - #136066

Draft
liuhao1024 wants to merge 2 commits into
NousResearch:mainfrom
liuhao1024:liuhao/cron-bugfix-132078
Draft

liuhao1024 wants to merge 2 commits into
NousResearch:mainfrom
liuhao1024:liuhao/cron-bugfix-132078

Conversation

@liuhao1024

Copy link
Copy Markdown

What does this PR do?

A cron execution whose delivery the durable queue fenced unknown (the restart-safe worker's 300s wait budget ran out while the gateway was still mid-send) could still be closed as failed: Interrupted by shutdown before terminal completion. when the runner tail died afterwards and the post-delivery fire-claim sample missed. In the reporter's fleet the delivery receipt demonstrably landed ~9 minutes before the tail died and the gateway process was up the whole time — no shutdown ever occurred.

The gap was in the #105861 fall-through: it exempts a claim miss only when the delivery returned no error. An unknown outcome returns an error string while the gateway's at-most-once send may still land (by design the queue never retries it), so a delivered run fell into _record_fire_ownership_lost and got overwritten with the synthetic shutdown string, polluting last_status, error streaks and incident ledgers.

This PR:

  • stamps job["_delivery_outcome_unknown"] in the queue branch of _deliver_result when the queue row fenced the send unknown (same in-band pattern as the existing _notification_all_targets_suppressed marker, re-derived per attempt and cleared at the top of every _deliver_result call), and
  • extends the existing post-delivery fall-through to that marker: the run keeps its real outcome (_finish_completed_run). An ok run records last_status="delivery_failed" with last_error=None and the streak untouched while last_delivery_error carries the honest "outcome is unknown" text; a failed run keeps its real error. The queue row remains where the uncertainty is recorded. An explicit transport cancel stays fail-closed, exactly as before.

The synthetic _OWNERSHIP_LOST_INTERRUPTED wording itself is left untouched — #116463 already reworks that message and would conflict.

Related Issue

Fixes #132078

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • cron/scheduler_delivery.py: queue branch stamps _delivery_outcome_unknown when get_status reports unknown; _deliver_result pops a stale marker up front.
  • cron/scheduler.py: the post-delivery ownership-lost check (_run_one_job_body) extends the Cron agent jobs: successful delivery overwritten with "Interrupted by shutdown before terminal completion." when fire-claim heartbeat lapses mid-run #105861 fall-through to an outcome-unknown delivery; the warning distinguishes "a completed" from "an outcome-unknown" delivery.
  • cron/scheduler_delivery_redact.py: _redact_cron_payload split out unchanged (re-exported from scheduler_delivery) purely to offset the file-size ratchet, since scheduler_delivery.py is over its cap and may only go down.
  • tests/cron/test_fire_claim_lost_after_delivery.py: two regression tests — the full _drive harness with an unknown-fenced delivery and a post-delivery claim-sample miss (ok and failure-notice variants), and the queue-branch stamping/clearing behaviour.

How to Test

  1. python -m pytest tests/cron/test_fire_claim_lost_after_delivery.py -q — 9 passed. On main without the fix, the three new tests fail: the run records last_status="error" / last_error="Interrupted by shutdown before terminal completion." (and the queue branch stamps nothing).
  2. python -m pytest tests/cron/ -q — 1476 passed, 11 skipped; the only failure, test_atomic_paused_creation.py::test_invalid_creation_is_rejected_without_writes, reproduces identically on a clean checkout of upstream/main on this machine (hermes_bootstrap._settle_interrupted_update probes the checkout's .git at import time and trips the home-I/O guard) and is unrelated to these files.
  3. python scripts/check --base upstream/main — 11 checks, ok.

Observed result: after the fix, the unknown-fenced run's on-disk record is last_status="delivery_failed" / last_error=None / failure_streak=0 with the "outcome is unknown" text preserved in last_delivery_error — the run's real outcome survives the claim miss instead of the synthetic shutdown failure.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • python scripts/check passes (the blocking lint and code-health checks CI runs)
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 15.2 (arm64), Python 3.14

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

N/A — no skill added.

Screenshots / Logs

WARNING  cron.scheduler:scheduler.py Job '0e1cc36cb905': fire claim ownership lost after an outcome-unknown delivery; recording the delivered run's terminal status

…meets a claim miss

NousResearch#132078: a restart-safe worker's 300s wait budget fences a mid-send
delivery "unknown" (cron/delivery_queue._terminalize_wait_timeout) even
though the gateway's at-most-once send may still land — the reporter's
fleet saw the receipt land 9 minutes before the runner tail died. The
post-delivery fire-claim sample then missed, and the NousResearch#105861 fall-through
only exempted a delivery with NO error, so the delivered run was closed
"failed: Interrupted by shutdown before terminal completion." for a
shutdown that never happened, polluting error streaks and incident
ledgers.

The queue branch of _deliver_result now stamps
job["_delivery_outcome_unknown"] when the queue row fenced the send
unknown (same pattern as _notification_all_targets_suppressed, re-derived
per attempt), and the post-delivery ownership check extends the existing
fall-through to that case: the run keeps its real outcome (ok run records
delivery_failed with the streak untouched; failed run keeps its real
error), while the queue row remains where the uncertainty is recorded.
An explicit transport cancel stays fail-closed; the synthetic message
wording is left to NousResearch#116463.
Offset the file-size ratchet for the delivery-outcome marker added in
this PR: scheduler_delivery.py is over its cap and may only go down, so
the self-contained payload redaction helper moves to a topical sibling
and is re-exported from the original module (callers and monkeypatching
of the old name are unchanged).
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Oct 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

2 participants