fix(cron): a restart-safe worker that dies before its ack reports its own stderr instead of a bare exit 1 (#112729)

[teknium1]

When a restart-safe cron worker dies before it acknowledges the handoff, the dispatch error now carries the worker's own stderr tail (e.g. the ModuleNotFoundError traceback) instead of just exit 1.

• cron/scheduler.py::_launch_external_cron_worker: worker stderr goes to <home>/cron/external-workers/<execution_id>.stderr (0o600) instead of DEVNULL; on a pre-ack exit the redacted tail is appended to the RuntimeError, which already flows into last_error, the executions ledger and the gateway log (Restart-safe cron worker dispatch failed: …). The file rides in handoff_files, so every exit path removes it.
• cron/scheduler_diagnostics.py::external_worker_stderr_tail: 1500-char tail through redact_sensitive_text(force=True, redact_url_credentials=True).
• One invariant test in tests/cron/test_restart_safe_worker.py (real subprocess that fails at import; asserts the cause is in the message and no exec-1.* handoff file survives).

Validation

Live probe: real _launch_external_cron_worker under a temp HERMES_HOME, dispatch argv rewritten to an interpreter that fails at import and exits 1 before writing the ack:

before (origin/main 9796235): cron external worker exited before ownership acknowledgement (exit 1)
after: cron external worker exited before ownership acknowledgement (exit 1); worker stderr: Traceback (most recent call last): … ModuleNotFoundError: No module named 'cron_missing_module'
Both: leftover handoff files: [].

scripts/run_tests.sh tests/cron/test_restart_safe_worker.py → 24 passed; the new test is red with the source swapped back to origin/main (23 passed, 1 failed).

Why this and not the proposed interpreter change

The issue's mechanism does not hold: CPython does not resolve symlinks in sys.executable (measured: a venv whose bin/python3 is a symlink reports sys.executable=<venv>/bin/python3, sys.prefix=<venv>, and a child spawned with sys.executable keeps the venv — matching @KeyArgo's analysis and @holny's withdrawal on the thread). The reporter's python -m cron.scheduler --help control is not the worker's launch path either. What IS real is that the worker's actual exit-1 cause was unrecoverable because its stderr went to DEVNULL and its own log handler installs only after cron.scheduler imports — so this PR makes the next such report self-diagnosing. #112731 (@KoNit-K, $VIRTUAL_ENV/bin/python preference) is not salvaged: it is a no-op wherever sys.executable is already the venv interpreter and VIRTUAL_ENV need not name the gateway's venv under systemd.

Part of #112729 — the reporter's install still needs the captured stderr from a re-run to identify the real failure.

Infographic

Review follow-up

• MINOR A (orphaned <execution_id>.stderr capture when the gateway dies mid-run in the restart-safe topology) → fixed @44030f683cbda: the acknowledged worker unlinks ack_path.with_suffix(".stderr") in its post-run finally; pre-ack paths are unchanged so a worker dying before its ack still gets its stderr reported. test_external_worker_adopts_execution_and_runs_payload_once now seeds the sibling and asserts it is gone.
• MINOR B (two comments still said stderr is DEVNULL: systemd-run failure branch, __main__ setup_logging motivation) → fixed @44030f683cbda; git grep DEVNULL -- cron/scheduler.py now matches only the real stdin/stdout DEVNULL args and the reworded comment.

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 44030f6 — fix(cron): acknowledged worker removes its own stderr captur}

⚠️ Warnings

CI timings · View report · View job

Wall time 13m45s vs 6m8s (+124.2%). 6 job(s) slower, 3 faster, 3 unchanged.

• OS-specific tests / macOS-only tests: +12.0s
• Python lints / Windows footguns (blocking): +11.0s
• Python tests / e2e: +6.0s
• Detect affected areas: -6.0s
• Python tests / Run tests: -4.0s