fix(cron): surface acknowledged worker failures and release stale claims

[unsupportedpastels]

Summary

Fixes #129254

Surface post-acknowledgement worker failures: record the worker's exit cause, preserve its stderr diagnostic, retire the matching fire claim, and invoke the job's existing failure-notification lane.

The execution ledger deliberately remains unknown: an acknowledged worker may already have performed side effects. The job records an error. This does not automatically retry or claim to identify/prevent the reporter's initiating startup crash.

Root cause

After a worker adopts its execution and dies, the parent recovers its row to terminal unknown, then returns without recording the job outcome. The fire claim and empty job error survive. Separately, the payload cleanup unlinks stderr before Python prints an escaping exception, so an outer traceback wrapper cannot preserve that diagnostic.

Fix

• Carry forward the targeted dead-owner terminalizer and observed exit status from fix(cron): record a lost manual run instead of reporting a restart that never happened #128616; never terminalize a live worker.
• Read the existing bounded/redacted stderr tail after a lost worker exits. Keep exceptional captures until that read; successful workers still clean up their own captures.
• Project a recovered unknown run onto the job through the existing incident/failure-delivery path. Hold its fire-owner fence through notification and bookkeeping so an already-recorded run, replay, or newer fire is not overwritten.
• Preserve diagnostics when a concurrent recovery sweep wins the ledger transition first.
• Keep new outcome logic in cron/scheduler_worker_failure.py; normal worker completion and unrelated waiter-error semantics remain unchanged.

Prior work and credit

• #128616, @Finn763: cherry-picked d3f3571f8c6f4d2f188e90151d8f8393f049a573 with -x, retaining the original author. Its exit-evidence/claim fix is the base; this adds failure delivery and closes its payload-exception stderr gap.
• #120391, @DresvyanskiyDenis: recovery projection and failure-delivery approach, hand-ported with verified Co-authored-by: Denis Dresvyanskiy <dresvyanskiydenis@gmail.com>. This version holds the fire fence over the side effect and adds worker diagnostics.
• #129256, @dskwe, with @ehz0ah's reviews: identified the post-ack stderr unlink ordering and, at e322e76, records a worker-side traceback and releases the owner-fenced claim when run_one_job raises. This PR reaches the same claim/cause outcome from the parent, so it also covers a hard worker exit and sends the job's failure notice. No code copied. One deliberate difference: the ledger stays unknown rather than failed, because run_one_job may already have had side effects before it raised.
• #126355, @dahux-net: managed-store relaunch fixes that are a plausible initiating cause of this crash class. Separate mechanism, not superseded here.
• #120347, #108806, and closed #108920: broader recovery projections reviewed, not incorporated. Their periodic/restart scope remains separate; fix(cron): project recovered worker failures #120347 also failed the already-recorded-success cross-test.
• #124847 and #124855 address bootstrap re-exec in fix(cron): adopted external worker can re-exec after deleting one-shot payload, leaving run unknown #124827; #127931 addresses dead fire-owner identity. These are not proven causes of this report and are not superseded by touching the same file.
• Pre-ack proposal #109294 and existing merged work #113219, #124013 (carrying #123432), #116689, #126344, and #106733 informed the comparison. Existing redaction, atomic acknowledgement, dependency activation, pre-ack notices, and claim-reclaim behavior are retained.

Test plan

Canonical runner, with HERMES_PYTHON=/home/mark/.hermes/hermes-agent/venv/bin/python:

• scripts/run_tests.sh tests/cron/ --file-retries 0 -j 8 on base 7239625ae1b: baseline 1,411 passed / 3 skipped; this branch 1,422 passed / 3 skipped; no failures.
• Two new invariant test functions (eleven cases): real child death and payload exception through the shared claimed-job entry, direct and builtin execution creation, concurrent recovery on both wait-return and timeout branches, repeat/new-owner fencing, and refusal to terminalize a live worker.
• RED checks: restoring main production files makes all four core failure-visibility cases fail; restoring fix(cron): record a lost manual run instead of reporting a restart that never happened #128616 production also makes all four fail. Restored the committed fix and reran the suite.
• Separate cross-PR harness executes the real module entry, including fix(cron): preserve external worker startup diagnostics #129256's wrapper: 4/4 cases fail on main and each open candidate (including fix(cron): preserve external worker startup diagnostics #129256 at e322e76 and fix(cron): relaunch with dependencies, never mid-handoff, and keep the dead worker's stderr #126355); 4/4 pass on this branch. This is controlled fault injection, not reproduction of the reporter's unknown crash cause.
• fix(cron): record a restart-safe worker's unknown outcome on the job #120391's two behavioral tests pass on the final salvage, including preserving a run the worker recorded before dying.
• python3 scripts/check-windows-footguns.py --all: passed.
• python -m ruff check cron/scheduler.py cron/executions.py cron/scheduler_worker_failure.py tests/cron/test_restart_safe_worker.py tests/cron/test_restart_safe_worker_death_evidence.py: passed.
• git diff --check origin/main..HEAD: passed; contributor author-email mapping check passed.

Scope and risk

No real Telegram delivery, native Windows/macOS runtime run, or hosted CI is claimed yet. Notification tests exercise the existing delivery boundary without sending messages.

This covers the surviving parent observing its worker's death, as in the report. If the gateway itself has already died, it cannot project or notify the failure here; broader periodic recovery remains separate. A failed worker may leave its stderr capture if no parent survives. No interpreter-selection, lazy-install policy, schema, or automatic retry changes. Rollback is reverting the four implementation commits.

[arkheioncorp]

Revisão: PR #129748

Verdict: APPROVE

Análise

Resolve #128509: um worker externo que morre após adoção deixava o execution como "unknown" com causa genérica "Scheduler restarted...", e o run_one_job retornava True (pulando mark_job_run), bloqueando o fire_claim para o próximo fire manual.

Mudanças principais:

1. terminalize_dead_owner() em cron/executions.py: terminaliza um execution para "unknown" com causa real observada pelo waiter, com guards contra live workers e handoff adoption grace.
2. cron/scheduler_worker_failure.py (new): record_unknown_worker_outcome() e external_worker_exited_reason() para visibilidade de worker failure.
3. Modificações em _wait_for_external_cron_worker_body(): agora usa terminalize_dead_owner() antes de fallback para recover_interrupted_executions().
4. run_one_job(): trata post-handoff failures via record_unknown_worker_outcome().
5. Testes novos em test_restart_safe_worker_death_evidence.py cobrindo múltiplos cenários de morte de worker.

Security scan

Nenhum hardcoded secret, injection, eval/exec, ou deserialization encontrado. Código de scheduler interno.

Qualidade

• A lógica está correta: o waiter observa a morte do worker e registra a causa real.
• Os guards em terminalize_dead_owner() (live worker check, handoff grace) são essenciais e bem implementados.
• Testes muito completos cobrindo hard exit, payload exception, concurrent recovery, timeout variants.

Sugestões (non-blocking)

Nenhuma.

Aprovado.

[unsupportedpastels]

Self-review (COMMENT; own PR). Independent read-only review approved exact head 166ea75 on base 7239625; exact-head CI: run 36793140949 green, including All required checks pass, Python tests, ruff/ty lints, Windows/macOS OS-specific tests and contributor attribution; also cron suite 1,422 passed / 3 skipped on a local trial merge with current main 6633626. Evidence: repro across main and 11 candidate PRs, RED checks against main and #128616 production, cron suite 1,422 passed / 3 skipped (baseline 1,411 / 3). Authorship: Finn763's commit preserved via cherry-pick -x; Denis Dresvyanskiy credited by Co-authored-by trailer.