Cron external workers no longer relaunch after their ack and leave the run unknown (#124827, salvage #133720)

[kshitijk4poor]

A restart-safe cron external worker no longer relaunches itself after its ownership ack, which left the adopted run unknown and dropped the job. The worker now finishes its hermes_bootstrap boot, including a pending source-update relaunch or interrupted-pull re-exec, before it reads its payload, so a relaunch replays the whole worker.

Fixes #124827 (salvage #133720)

What happened

On main, the worker read and deleted its payload, adopted the execution and published the .ready ack. Only then did run_one_job import run_agent, which was the worker's first import hermes_bootstrap. If a source update was pending, prepare_launch relaunched the worker into an -I interpreter that ignores the pinned PYTHONPATH. worker_bootstrap() had already popped _HERMES_CRON_WORKER_BOOT, so the new process skipped the PM dependency boot, died on ruamel, and its payload was already gone. An interrupted hermes update pull re-exec had the same post-ack timing. This was observed in production on a self-managed WSL install, and it has been the behaviour since the worker marker landed (#122936).

Change

• cron/worker_bootstrap.py
  • worker_bootstrap() reads the marker (get) instead of consuming it.
  • New finish_worker_boot() imports hermes_bootstrap, whose import settles an interrupted pull and finishes a pending source update. It then pops the marker.
• cron/scheduler.py: finish_worker_boot() is the first statement of the --external-worker-file branch, before argparse, the payload read, adoption and the ack. It adds 1 line.
• tests/cron/test_restart_safe_worker.py
  • The boot-order test now runs with the marker set.
  • A new parametrized test (source update, interrupted pull) asserts the relaunch happens with the payload unread, no ack and the marker set.
• cron/AGENTS.md: the hardening invariant describes the new order. Its wording comes from fix(cron): a source-update relaunch replays the external worker before its ack #133720 and still names run_agent's recovery; a one-line refresh naming hermes_bootstrap waits on approval for protected-file edits.

Credit

@mzyas (first-time contributor) wrote the fix and both tests in #133720. Their 2 commits are cherry-picked with authorship kept (4d3b390df0, 873dd42631), and contributors/emails/mzykr@yahoo.co.jp maps them.

Our follow-ups:

• 8a33ea0f86 drops the explicit _early_recovery call. Main's hermes_bootstrap._settle_interrupted_update (teknium1, f1def901fb/fd73042f2d/99e954838f, 10-03/04) already runs that repair at import.
• e1d335d276 fixes docstring wording and reverts comment rewrap churn.
• 06c02c54ff adds a test that a boot without a relaunch consumes the marker (review).

Verification

Head 06c02c54ff on main 152aa695bb.

Live probe (real imports, clone test interpreter, cron.scheduler as __main__)	main	this branch
order of hermes_bootstrap vs payload open	payload first, bootstrap later (after the ack)	bootstrap (marker set), then payload
prepare_launch returns a relaunch	relaunch after payload consumed + ack	RELAUNCH payload_exists=True ack_exists=False marker=1
no relaunch pending	runs	runs (unchanged)

• Tests: tests/cron/test_restart_safe_worker.py, tests/pm/test_runtime.py, tests/cron/test_scheduler.py: 134 passed, 4 skipped (Linux-only), 0 failed. The PR's tests on main: 3 failed, 1 passed.
• Mutations: reverting the fix, dropping the bootstrap import, consuming the marker at package init, or dropping the __main__ call each turns the tests red.
• ruff, the Windows footgun scan, git diff --check and code-health are clean (cron/scheduler.py +1 line).
• Review on the final head:
  • hermes-pr-review 2a/2b/2c plus the reuse, quality and efficiency lenses, a thermo-nuclear pass and idiom_delta all approve.
  • A second-model verifier (gpt-6.1-sol) returned no findings on the final head.

Maintainer notes

• Every worker now pays the hermes_bootstrap import before its ack (measured 0.05–0.09 s cold). no_agent jobs did not pay it before. This is deliberate: the relaunch has to happen pre-ack whatever the job type.
• A marked worker runs the dependency boot twice (package init, then hermes_bootstrap). It is idempotent and bounded. The package-init call must stay because cron.jobs needs dependencies before __main__ runs.
• Known and unchanged (identical on main): run_agent.py's own import-time recovery can still re-exec after the ack if an updater held custody at worker start and died before run_agent was imported. That code is on the update-recovery surface, so it is left for a separate change.
• Other open PRs for fix(cron): adopted external worker can re-exec after deleting one-shot payload, leaving run unknown #124827: fix(cron): suppress bootstrap relaunch inside an adopted external worker #124847, fix(cron): stop an adopted external worker from re-exec'ing and losing its payload (#124827) #124855, fix(cron): import run_agent before consuming worker handoff payload #124886, fix(cron): relaunch with dependencies, never mid-handoff, and keep the dead worker's stderr #126355. Each will be closed with credit once this lands; the PR description of fix(cron): a source-update relaunch replays the external worker before its ack #133720 explains how each differs.

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 06c02c5 — test(cron): pin that a booted worker drops the marker before}

debug info

CI timings

CI timings · View report · View job

Wall time 14m32s vs 44m28s (-67.3%). 9 job(s) slower, 5 faster, 2 unchanged.

• Python tests / Run tests (1/2): +184.0s
• Python tests / Run tests (2/2): +85.0s
• OS-specific tests / macOS-only tests (2/2): +33.0s
• OS-specific tests / Windows-only tests: -25.0s
• OS-specific tests / macOS-only tests (1/2): +18.0s

[greptile-apps]

Confidence Score: 5/5

[Critical risk] Cron worker boot and relaunch sequence for external workers.

The PR appears safe to merge; the missing marker-cleanup coverage is now present.

Summary

External cron workers now finish hermes_bootstrap before reading their payload or publishing their ownership ack. A relaunch preserves the payload and worker marker.

• External cron workers finish bootstrap before they read their payload.
• The contributor email now maps to mzyas.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["Marked worker starts"] --> B["Package entry boots dependencies"]
    B --> C["__main__ calls finish_worker_boot"]
    C --> D["Import hermes_bootstrap"]
    D --> E{"Relaunch needed?"}
    E -- Yes --> F["Restart with payload and marker intact"]
    F --> A
    E -- No --> G["Remove worker marker"]
    G --> H["Read payload, adopt run, publish ack"]
    H --> I["Run job"]

Loading

_{Reviews (2) · Last reviewed commit: "test(cron): pin that a booted worker dro..." · Reviewed by Greptile}