Skip to content

fix(cron): alert on restart-safe worker dispatch failures (salvage #123432) - #124013

Merged
kshitijk4poor merged 6 commits into
NousResearch:mainfrom
kshitijk4poor:fix/p1-0926-c3-cron-dispatch-failure-notice
Sep 26, 2026
Merged

kshitijk4poor merged 6 commits into
NousResearch:mainfrom
kshitijk4poor:fix/p1-0926-c3-cron-dispatch-failure-notice

Conversation

@kshitijk4poor

Copy link
Copy Markdown

A failed handoff to the restart-safe cron worker now opens a cron_incidents row and sends the job's failure notice, instead of failing silently. Fixes #123401. This salvages #123432 by @liuhao1024.

Why

When the external-worker handoff failed, run_one_job() (cron/scheduler.py) recorded the failure on the job and in executions.db, then returned before any operator-facing path ran. There was no incident and no notice. In #123401, a 9.5-hour outage produced 106 failed executions and zero notifications, including for jobs whose only purpose is alerting.

Changes

  • Incident and notice (@liuhao1024, fix(cron): surface external-worker dispatch failures through the incident path #123432): the dispatch-failure branch now goes through the existing _deliver_crash_failure, the same path a crashed run uses. That covers incident dedup, cooldown and the delivery outcome in the ledger.
  • Bookkeeping always closes: the call sits inside the existing try/finally, so finish_execution still runs if delivery raises. There's no new try/except swallowing. This is defensive and not pinned by a test (the stack keeps to two tests); the probe covers it.
  • No false alert after the handoff: once the worker has spawned and acknowledged the job, a later wait failure only updates bookkeeping. It opens no "dispatch failed" incident and sends no ping, because the worker may have run the job. Its last_error reads failed after handoff. A pre-handoff failure keeps the existing dispatch failed text.

Validation

probe (isolated HOME) main this PR
handoff fails before ack incidents 0, notices 0 incidents 1, notices 1, exec failed/delivered
handoff succeeds 0 new 0 new
failure notice send fails (502) nothing recorded incident detected, delivery_error recorded
failure after the worker acked n/a 0 incidents, 0 notices, bookkeeping recorded
  • Tests: tests/cron/test_restart_safe_worker.py + tests/cron/test_run_one_job.py pass on the rebased head.
  • Two invariant tests. Each fails with its production change reverted:
    • dispatch failure opens an incident and delivers the notice
    • a post-handoff failure is bookkeeping-only, with the new label

Known limit

The alert is not gated on the fire-claim owner. A non-owner that lost the claim can still alert on a pre-handoff failure. The triage rated this low severity, and it is left for a follow-up.

Credit

liuhao1024 and others added 6 commits September 26, 2026 20:00
…dent path

A failed restart-safe handoff in run_one_job() recorded the failure on the
job and in the executions ledger, then returned before any incident or
delivery path ran: no cron_incidents row, no failure-lane notice. Route the
dispatch-failure branch through _deliver_crash_failure() so it opens the
same job+signature incident and delivers the same failure notice as any
other job failure, with the existing alerted-cooldown withholding repeats.
A notice-path exception no longer loses the bookkeeping: mark_job_run and
finish_execution still run with a "failed" delivery outcome (NousResearch#123401).

(cherry picked from commit e5b5969)
_deliver_crash_failure is already best-effort internally (incident upsert,
_deliver_result and _mark_incident_alerted each swallow their own errors),
and the sibling crash path in run_one_job calls it without a wrapper. The
extra try/except was a defensive layer that could only mask a genuine bug
in the notice helper; drop it so both call sites match. The existing
try/finally still guarantees finish_execution runs after mark_job_run.

webtecnica submitted an equivalent independent fix in NousResearch#123433.

Fixes NousResearch#123401

Co-authored-by: webtecnica <webtecnica@gmail.com>
…ivery raises

The dispatch-failure branch called _deliver_crash_failure before the
try/finally that closes the execution row. If delivery ever raised,
finish_execution was skipped and the row stayed claimed until dead-owner
recovery. Run delivery inside the try so the pre-existing finally still
terminalizes the attempt; delivery_error/outcome default to None when it
raises.
_launch_external_cron_worker returns through _wait_for_external_cron_worker
once the worker is spawned/acknowledged, and that waiter can raise (e.g. the
adopted worker died and recovery could not terminalize the row). Such an
error landed in the same run_one_job except branch as a genuine dispatch
failure, so a job that actually ran got a false "Restart-safe cron worker
dispatch failed" incident and failure ping on top of whatever the worker
itself delivered.

Wrap waiter failures in a dedicated _ExternalWorkerPostHandoffError and skip
_deliver_crash_failure for it, keeping the base bookkeeping-only behaviour
(mark_job_run + finish_execution, the latter a no-op on a worker-owned row).
Pre-handoff dispatch failures still open an incident and notify.
run_one_job's dispatch-failure handler already routes an
_ExternalWorkerPostHandoffError through bookkeeping only (no incident, no
ping), but the recorded error still read "Restart-safe cron worker dispatch
failed", which is misleading in last_error / executions.db for a worker that
was spawned and may have run. Compute post_handoff first and label it
"Restart-safe cron worker failed after handoff: ..."; the pre-handoff prefix
is unchanged.
The fold that stops post-handoff waiter failures from raising a false
"dispatch failed" incident/ping had no regression coverage. Drive
run_one_job through a real _wait_for_external_cron_worker whose body raises
and assert: no cron incident, no delivery, one failed mark_job_run carrying
the post-handoff label, and the execution row terminalized as failed.
Red with the post-handoff routing reverted (an incident is opened).
@kshitijk4poor
kshitijk4poor enabled auto-merge (rebase) September 26, 2026 14:36
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management P1 High — major feature broken, no workaround sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Sep 26, 2026
@kshitijk4poor
kshitijk4poor merged commit ffe5cf0 into NousResearch:main Sep 26, 2026
80 of 82 checks passed
kshitijk4poor pushed a commit that referenced this pull request Oct 3, 2026
run_one_job's pre-handoff dispatch-failure branch (#124013) calls
_deliver_crash_failure before _run_one_job_body, which is where the
firing profile's secret scope is installed for delivery. Under a
multiplexed gateway the notice then resolves the home channel through
get_secret with no scope, raises UnscopedSecretError, and never leaves.

Wrap the notice in _install_fire_secret_scope/_reset_fire_secret_scope,
the pair _run_one_job_body and record_unknown_worker_outcome already use.

(cherry picked from commit 5e0735b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P1 High — major feature broken, no workaround sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Cron dispatch failures create no incident and send no failure notification

3 participants