Repository navigation
Conversation
…l worker On a self-managed install the restart-safe cron worker is spawned with the store Python, which owns no third-party dependencies. The sanitizer drops the runtime site-packages and the tree pin only re-adds the checkout, so the worker dies at its first import (No module named 'ruamel') before publishing its ownership ack. Restore the dependency environment this process actually activated, next to the tree, derived from sys.path (never a home-scoped read) and gated on a real venv so nothing is invented for installs that already own their dependencies. Fixes NousResearch#122222
alt-glitch
added
type/bug
|
Independent verification on the PR head (0a86331): restart-safe-worker suite 30/30 green on Linux. Restoring activated dependency site-packages for the external worker keeps the restarted worker on the right environment. No findings. |
Local verification against the fix in this PRApplied this diff on top of Before the fix (repo root only on After the fix (this PR, End-to-end against a live gateway (before → after):
Both were failing with the One note for reviewers comparing against #122238: this implementation derives the dependency path from the process's own |
|
Independent verification of the same failure mode converged on this same shape — restoring the activated Mechanism confirmed at a real boundary: Three edges around the fix, none blocking the reported topology:
Also a nit: the diff drops the trailing newline at end of |
andrexibiza
left a comment
andrexibiza
left a comment
There was a problem hiding this comment.
Restoring the dependency path fixes the reported store-Python import failure without rebuilding the child environment from raw os.environ. Keep that, the wheel/pipx early return, and the existing execution-adoption protocol. There is a separate lifetime gap in the handoff: the child receives the dependency directory, but not the protection that keeps that directory available.
Reviewed head: 0a8633160e8b9767823cb2dc1895bf2689910b69.
P2 — protect the dependency generation before the worker starts importing it
The new insertion at scheduler_worker_env.py:105–110 forwards a pathname. It does not acquire or transfer a lifetime lease for its generation.
The actual launch path builds sys.executable -m cron.scheduler, sanitizes the environment, applies this pin, then starts a separate process. That module begins application/dependency imports before normal dependency activation.
The existing PM activation contract does more than add a path: activate_dependencies() resolves the committed environment, leases it under the install-lock/reselection protocol, and processes it with site.addsitedir() before application imports. In runtime_state.py, that lease is a process-owned kernel lock. The collector can delete an unselected, lease-managed generation once it is old enough and no process holds its lease. PYTHONPATH does not keep that lock alive.
The failing sequence is:
- A gateway is still running on G1 after PM has selected G2. G1 is older than the collector's normal 86,400-second threshold.
- The gateway gives a new external worker G1's dependency path.
- The gateway exits while the child is still importing, before the child has performed any later bootstrap activation. Its lease is released.
- Collection removes G1 while the child is alive. The child's next import from G1 fails.
I reproduced that sequence with the production pin, lease, collector and lock implementations in an isolated fixture. A genuine child-held lease prevents deletion; merely keeping the child alive does not.
This is conditional, not a claim that ordinary jobs still fail with the reported ruamel error. The direct-entry/bootstrap-order gap predates this PR; restoring import reachability does not close its lifetime requirement. Also, agent workers are not necessarily unleased forever: run_agent.py imports hermes_bootstrap, which may activate dependencies later. That cannot protect imports already attempted before reaching it.
Please establish an installation-bound dependency generation and its lifetime protection at the worker entry, before the first application/dependency import. Reuse PM's activation/lease machinery and keep protection continuous across parent exit and the child's required use. A directory name or pyvenv.cfg is not a retention permit. If the intended behavior is to boot the latest committed generation instead of retaining the parent's generation, select and lease it before importing—not halfway through a dependency graph loaded from G1. Preserve the sanitized environment and the existing handoff, adoption and recovery fences.
The regression needs the real external-worker entry, an old still-running gateway generation, a newly selected generation, controlled parent exit and actual collection. Assert that the required generation remains usable throughout the handoff. Include a child-held lease, a still-selected generation, a younger generation and a legacy unmanaged generation as controls. Exercise pre-ack failure/recovery without turning uncertain execution into a duplicate job.
Verification
I re-ran the retained characterization on Linux / Python 3.13.5 before submitting this review. It uses three complete modules pinned to the reviewed source blobs:
cron/scheduler_worker_env.py:026c3209bb1aab0f653b1513994c10f005667cf9.hermes_cli/runtime_state.py:7a74b6e9d5fd65cf768f723f6e382ffaa55fe3c1.pm/filesystem.py:f7dc30b48239706f37e94ad9aa8be984d9eeedf0.
The pin, collector, lease functions, kernel locks, child processes, imports and deletion are real. Install-path/selection metadata and the child job body are fixtures. This is not full cron.scheduler/systemd E2E, a complete repository checkout, store-Python 3.14 acceptance, or native Windows/macOS verification.
The five lifetime characterizations confirmed that the parent lease initially retains G1; releasing it allows deletion and a late ModuleNotFoundError in the unleased child; a child lease prevents both; and selected, young and legacy unmanaged generations remain retained. The sixth case confirms the already-discussed .pth limitation: path-only forwarding does not perform site.addsitedir() activation. That is not a second newly discovered security vulnerability.
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest test_generation_boundary.py -q -s \
-k 'not required_invariant' --basetemp=<fresh disposable directory>
6 passed, 1 deselected in 4.25s; exit 0
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest test_generation_boundary.py -q -s \
-k required_invariant --basetemp=<different disposable directory>
1 failed, 6 deselected in 0.61s; exit 1
The failing assertion is the required property: a live worker must not lose the generation supplied by its pin. The six passing characterizations reproduce and bound the failure; they do not mean it is fixed. Harness SHA-256: 31fd98ccd533469a18de7687d63ab53c8707ebbd801aa52e980cf468acec8bf6.
The added tests cover path derivation and a stubbed launch, not collection during handoff. The 30/30 Linux report and author's live-job verification remain useful evidence of the original fix. I did not independently execute those suites or live jobs, and claim no full adjacent-suite, packaged-runtime or independent 5×2×3 acceptance.
Coordination and acceptance
JoanDuarte's #122222 is the original failure analysis and path-restoration proposal; retain that credit alongside 123456hyu's implementation and verification here.
The comparison with webtecnica's #122238 needs an exact-source qualification. At reviewed head 663403201ba7c9fce7e47a0a72637b0e81238adf, its diff also scans sys.path and checks a pyvenv.cfg ancestor; it does not use the selected_venv() spawn-time strategy described in the comparison comment. Both reviewed diffs change cron/scheduler_worker_env.py and tests/cron/test_restart_safe_worker.py for the same issue. Consolidate that implementation choice and preserve the relevant contributions rather than landing parallel helpers. The lifetime requirement applies whichever import-restoration implementation is retained.
The recorded PR base is 138e33d51f521cde606f7e6a00cdc9a203275ed3. The original review compared against main 59004a62356f3a4697ab0fe8ad5086d2b405e2a6, with merge base 8e3b0d1566d12ca9b593ba6627c0c70ef2359e52; those are comparison pins, not a claim that main has stopped moving. Publication revalidation still finds the same head, open/non-draft/mergeable, with one commit and two changed files. Main movement alone is not a finding or an automatic rebase requirement.
Exact-head workflows were rechecked and remain action_required, not green and not failed executed tests: CI 36095196098, Docker 36095195874, and Nix 36095195853. The sole commit therefore has no green acceptance receipt. I cannot authorize those workflows.
The relevant architecture is generation ownership and exact realization under #93943, #90145 and #92618; PM already owns the activation/retention mechanism. This is not a request to finish unrelated fleet work before fixing cron. The bounded review covers dependency realization, sanitized environment composition, process lifetime, collection and the handoff/recovery path. It does not establish a remote privilege escalation, containment escape or whole-repository security clearance. The actual worker-entry lifetime regression remains the missing proof.
| pinned = [root] | ||
| dependency = _activated_dependency_site_packages() | ||
| if dependency is not None: | ||
| pinned.append(str(dependency)) |
There was a problem hiding this comment.
P2 — forwarding the dependency directory does not retain its generation. This child enters cron.scheduler directly and begins dependency imports before normal bootstrap activation. If the parent exits in that interval, its process-owned lease disappears; an old, unselected lease-managed generation can then be collected while the worker remains alive. The isolated production pin/lease/collector probe reproduced deletion followed by ModuleNotFoundError; a child-held lease prevented it. Establish and protect the installation-bound generation before application imports, using the existing PM activation/lease contract. Later run_agent bootstrap does not protect earlier imports. Add actual external-worker parent-exit/collection coverage with child-lease, selected, young and legacy controls. This is a conditional lifetime gap in the restored path, not a claim that all cron runs fail.
There was a problem hiding this comment.
Addressed in #126344, which carries this PR's pin and lease commits with authorship preserved. The child-held lease is now established by PM's own activate_dependencies in the cron package prelude, before any application import. test_marked_worker_keeps_its_generation_through_a_rotation checks it against the real collector: the worker's generation survives a newer commit while the worker lives, and is collected after it exits.
|
Independent verification from a real self-managed install, in case it helps this land. Install shape (Ubuntu 24.04.5): PM store Python ( A/B, gateway-like env — parent carries the activated generation on
After applying it, that install's next fires complete through the external worker path ( One caveat worth checking before merge. Running the suite from that same install — the test process carries the generation on That is If you want the spawn path to stay clear of home reads entirely, the committed selection already names the environment ( Failing set, for reference: |
… entry (NousResearch#122290 P2) The pin in scheduler_worker_env.py restores the site-packages *directory* into the child's PYTHONPATH, but a directory name is not a retention permit. The PM collector removes an unselected, lease-managed generation once it is old enough and no process holds its lease, so between the gateway's exit and the worker's next import the very generation the worker needs can be collected away. New cron/worker_bootstrap.py closes that gap at the worker entry, before the first application/dependency import: - lease_dependencies(): acquire the committed generation's kernel lease (the same hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses), so the collector's 'no process holds the lease' safety condition keeps the generation alive for the worker's whole lifetime. Unconditional and idempotent. - activate_worker_dependencies(): delegate to pm.environments.activate_dependencies (the same resolution + site.addsitedir + sys.path reordering the gateway's boot uses) so the worker's own imports resolve without waiting for the gateway's handoff. Gated on --external-worker-file in argv, so the in-process gateway path keeps its exact launch contract and the runner-owns-its-own-deps case stays a no-op. Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both degrade to a logged no-op when no dependency generation is committed. Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway, _no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard failures in this suite are pre-existing on a clean checkout and unrelated. Fixes the P2 lifetime finding in @andrexibiza's review of NousResearch#122290.
P2 lifetime finding — addressed in commit
|
CI: re-run needed on head
|
alt-glitch
added
the
needs-decision
|
@123456hyu Patience. Pinging people for this does not help your cause. |
|
Noted — dropping the CI ping. The P2 lease fix is in |
alt-glitch
removed
the
needs-decision
… entry (#122290 P2) The pin in scheduler_worker_env.py restores the site-packages *directory* into the child's PYTHONPATH, but a directory name is not a retention permit. The PM collector removes an unselected, lease-managed generation once it is old enough and no process holds its lease, so between the gateway's exit and the worker's next import the very generation the worker needs can be collected away. New cron/worker_bootstrap.py closes that gap at the worker entry, before the first application/dependency import: - lease_dependencies(): acquire the committed generation's kernel lease (the same hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses), so the collector's 'no process holds the lease' safety condition keeps the generation alive for the worker's whole lifetime. Unconditional and idempotent. - activate_worker_dependencies(): delegate to pm.environments.activate_dependencies (the same resolution + site.addsitedir + sys.path reordering the gateway's boot uses) so the worker's own imports resolve without waiting for the gateway's handoff. Gated on --external-worker-file in argv, so the in-process gateway path keeps its exact launch contract and the runner-owns-its-own-deps case stays a no-op. Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both degrade to a logged no-op when no dependency generation is committed. Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway, _no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard failures in this suite are pre-existing on a clean checkout and unrelated. Fixes the P2 lifetime finding in @andrexibiza's review of #122290.
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it.
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
|
Independent verification on macOS (arm64) — same failure mode, and the first non-Linux data point in this thread Every report on #122222 / #122529 so far is Linux (Ubuntu, Arch, Debian/LXC, aarch64). Adding a Darwin/arm64 row: it reproduces identically, and restoring the activated generation's Install shape
Spawn-boundary A/B Exact child command ( Line-for-line the same chain as the existing reports, on a platform they don't cover. End-to-end, not just the import: on this host, before the dependency path was restored, the job store showed scheduled jobs landing with the worker-stderr traceback and never reaching their ownership ack; after restoring the activated generation's Supporting the P2 (generation lifetime) with a field observation We run a local workaround on this same seam, and it hit this class of problem directly:
The Not re-verified here: Windows, and the in-tree Happy to re-run both legs against a new head on request. |
alt-glitch
added
the
needs-decision
This PR makes the restart-safe cron external worker survive on a self-managed (shell-installer / PM) install in two parts.
Minor:
Unverified / please confirm:
|
Handoff note: the lease findings on
|
… entry (NousResearch#122290 P2) The pin in scheduler_worker_env.py restores the site-packages *directory* into the child's PYTHONPATH, but a directory name is not a retention permit. The PM collector removes an unselected, lease-managed generation once it is old enough and no process holds its lease, so between the gateway's exit and the worker's next import the very generation the worker needs can be collected away. New cron/worker_bootstrap.py closes that gap at the worker entry, before the first application/dependency import: - lease_dependencies(): acquire the committed generation's kernel lease (the same hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses), so the collector's 'no process holds the lease' safety condition keeps the generation alive for the worker's whole lifetime. Unconditional and idempotent. - activate_worker_dependencies(): delegate to pm.environments.activate_dependencies (the same resolution + site.addsitedir + sys.path reordering the gateway's boot uses) so the worker's own imports resolve without waiting for the gateway's handoff. Gated on --external-worker-file in argv, so the in-process gateway path keeps its exact launch contract and the runner-owns-its-own-deps case stays a no-op. Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both degrade to a logged no-op when no dependency generation is committed. Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway, _no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard failures in this suite are pre-existing on a clean checkout and unrelated. Fixes the P2 lifetime finding in @andrexibiza's review of NousResearch#122290. (cherry picked from commit 429f0dd)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
worker_bootstrap() logged a failed activate_dependencies() and let the worker carry on through the inherited PYTHONPATH pin. That path is not leased: once the gateway releases its generation the collector deletes it under the live worker and the next dependency import dies mid-job (NousResearch#122936 review, reproduced with a committed generation that has no site-packages). The failure now propagates, so the worker exits before its ownership ack and _launch_external_cron_worker reports the dispatch failure with the worker's stderr. PM's own no-op cases (venv, wheel, Nix, nothing committed under a venv) still return normally. Tests drive the real PM boot in a marked child: it imports a sentinel that exists only in the committed generation, and when an update commits a newer one mid-job the real collector keeps the worker's generation until the worker exits. A damaged selection exits before cron.jobs is imported. 123456hyu made the same prelude move and fail-closed change on NousResearch#122290 (dff16e1); this carries them without the argv gate and source-order test. Co-authored-by: 123456hyu <123456hyu@users.noreply.github.com>
… entry (#122290 P2) The pin in scheduler_worker_env.py restores the site-packages *directory* into the child's PYTHONPATH, but a directory name is not a retention permit. The PM collector removes an unselected, lease-managed generation once it is old enough and no process holds its lease, so between the gateway's exit and the worker's next import the very generation the worker needs can be collected away. New cron/worker_bootstrap.py closes that gap at the worker entry, before the first application/dependency import: - lease_dependencies(): acquire the committed generation's kernel lease (the same hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses), so the collector's 'no process holds the lease' safety condition keeps the generation alive for the worker's whole lifetime. Unconditional and idempotent. - activate_worker_dependencies(): delegate to pm.environments.activate_dependencies (the same resolution + site.addsitedir + sys.path reordering the gateway's boot uses) so the worker's own imports resolve without waiting for the gateway's handoff. Gated on --external-worker-file in argv, so the in-process gateway path keeps its exact launch contract and the runner-owns-its-own-deps case stays a no-op. Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both degrade to a logged no-op when no dependency generation is committed. Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway, _no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard failures in this suite are pre-existing on a clean checkout and unrelated. Fixes the P2 lifetime finding in @andrexibiza's review of #122290.
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
… entry (#122290 P2) The pin in scheduler_worker_env.py restores the site-packages *directory* into the child's PYTHONPATH, but a directory name is not a retention permit. The PM collector removes an unselected, lease-managed generation once it is old enough and no process holds its lease, so between the gateway's exit and the worker's next import the very generation the worker needs can be collected away. New cron/worker_bootstrap.py closes that gap at the worker entry, before the first application/dependency import: - lease_dependencies(): acquire the committed generation's kernel lease (the same hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses), so the collector's 'no process holds the lease' safety condition keeps the generation alive for the worker's whole lifetime. Unconditional and idempotent. - activate_worker_dependencies(): delegate to pm.environments.activate_dependencies (the same resolution + site.addsitedir + sys.path reordering the gateway's boot uses) so the worker's own imports resolve without waiting for the gateway's handoff. Gated on --external-worker-file in argv, so the in-process gateway path keeps its exact launch contract and the runner-owns-its-own-deps case stays a no-op. Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both degrade to a logged no-op when no dependency generation is committed. Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway, _no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard failures in this suite are pre-existing on a clean checkout and unrelated. Fixes the P2 lifetime finding in @andrexibiza's review of #122290.
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
… entry (NousResearch#122290 P2) The pin in scheduler_worker_env.py restores the site-packages *directory* into the child's PYTHONPATH, but a directory name is not a retention permit. The PM collector removes an unselected, lease-managed generation once it is old enough and no process holds its lease, so between the gateway's exit and the worker's next import the very generation the worker needs can be collected away. New cron/worker_bootstrap.py closes that gap at the worker entry, before the first application/dependency import: - lease_dependencies(): acquire the committed generation's kernel lease (the same hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses), so the collector's 'no process holds the lease' safety condition keeps the generation alive for the worker's whole lifetime. Unconditional and idempotent. - activate_worker_dependencies(): delegate to pm.environments.activate_dependencies (the same resolution + site.addsitedir + sys.path reordering the gateway's boot uses) so the worker's own imports resolve without waiting for the gateway's handoff. Gated on --external-worker-file in argv, so the in-process gateway path keeps its exact launch contract and the runner-owns-its-own-deps case stays a no-op. Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both degrade to a logged no-op when no dependency generation is committed. Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway, _no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard failures in this suite are pre-existing on a clean checkout and unrelated. Fixes the P2 lifetime finding in @andrexibiza's review of NousResearch#122290. (cherry picked from commit 429f0dd)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
worker_bootstrap() logged a failed activate_dependencies() and let the worker carry on through the inherited PYTHONPATH pin. That path is not leased: once the gateway releases its generation the collector deletes it under the live worker and the next dependency import dies mid-job (NousResearch#122936 review, reproduced with a committed generation that has no site-packages). The failure now propagates, so the worker exits before its ownership ack and _launch_external_cron_worker reports the dispatch failure with the worker's stderr. PM's own no-op cases (venv, wheel, Nix, nothing committed under a venv) still return normally. Tests drive the real PM boot in a marked child: it imports a sentinel that exists only in the committed generation, and when an update commits a newer one mid-job the real collector keeps the worker's generation until the worker exits. A damaged selection exits before cron.jobs is imported. 123456hyu made the same prelude move and fail-closed change on NousResearch#122290 (dff16e1); this carries them without the argv gate and source-order test. Co-authored-by: 123456hyu <123456hyu@users.noreply.github.com>
… entry (NousResearch#122290 P2) The pin in scheduler_worker_env.py restores the site-packages *directory* into the child's PYTHONPATH, but a directory name is not a retention permit. The PM collector removes an unselected, lease-managed generation once it is old enough and no process holds its lease, so between the gateway's exit and the worker's next import the very generation the worker needs can be collected away. New cron/worker_bootstrap.py closes that gap at the worker entry, before the first application/dependency import: - lease_dependencies(): acquire the committed generation's kernel lease (the same hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses), so the collector's 'no process holds the lease' safety condition keeps the generation alive for the worker's whole lifetime. Unconditional and idempotent. - activate_worker_dependencies(): delegate to pm.environments.activate_dependencies (the same resolution + site.addsitedir + sys.path reordering the gateway's boot uses) so the worker's own imports resolve without waiting for the gateway's handoff. Gated on --external-worker-file in argv, so the in-process gateway path keeps its exact launch contract and the runner-owns-its-own-deps case stays a no-op. Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both degrade to a logged no-op when no dependency generation is committed. Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway, _no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard failures in this suite are pre-existing on a clean checkout and unrelated. Fixes the P2 lifetime finding in @andrexibiza's review of NousResearch#122290. (cherry picked from commit 429f0dd)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
worker_bootstrap() logged a failed activate_dependencies() and let the worker carry on through the inherited PYTHONPATH pin. That path is not leased: once the gateway releases its generation the collector deletes it under the live worker and the next dependency import dies mid-job (NousResearch#122936 review, reproduced with a committed generation that has no site-packages). The failure now propagates, so the worker exits before its ownership ack and _launch_external_cron_worker reports the dispatch failure with the worker's stderr. PM's own no-op cases (venv, wheel, Nix, nothing committed under a venv) still return normally. Tests drive the real PM boot in a marked child: it imports a sentinel that exists only in the committed generation, and when an update commits a newer one mid-job the real collector keeps the worker's generation until the worker exits. A damaged selection exits before cron.jobs is imported. 123456hyu made the same prelude move and fail-closed change on NousResearch#122290 (dff16e1); this carries them without the argv gate and source-order test. Co-authored-by: 123456hyu <123456hyu@users.noreply.github.com>
… entry (#122290 P2) The pin in scheduler_worker_env.py restores the site-packages *directory* into the child's PYTHONPATH, but a directory name is not a retention permit. The PM collector removes an unselected, lease-managed generation once it is old enough and no process holds its lease, so between the gateway's exit and the worker's next import the very generation the worker needs can be collected away. New cron/worker_bootstrap.py closes that gap at the worker entry, before the first application/dependency import: - lease_dependencies(): acquire the committed generation's kernel lease (the same hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses), so the collector's 'no process holds the lease' safety condition keeps the generation alive for the worker's whole lifetime. Unconditional and idempotent. - activate_worker_dependencies(): delegate to pm.environments.activate_dependencies (the same resolution + site.addsitedir + sys.path reordering the gateway's boot uses) so the worker's own imports resolve without waiting for the gateway's handoff. Gated on --external-worker-file in argv, so the in-process gateway path keeps its exact launch contract and the runner-owns-its-own-deps case stays a no-op. Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both degrade to a logged no-op when no dependency generation is committed. Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway, _no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard failures in this suite are pre-existing on a clean checkout and unrelated. Fixes the P2 lifetime finding in @andrexibiza's review of #122290. (cherry picked from commit 429f0dd)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
worker_bootstrap() logged a failed activate_dependencies() and let the worker carry on through the inherited PYTHONPATH pin. That path is not leased: once the gateway releases its generation the collector deletes it under the live worker and the next dependency import dies mid-job (#122936 review, reproduced with a committed generation that has no site-packages). The failure now propagates, so the worker exits before its ownership ack and _launch_external_cron_worker reports the dispatch failure with the worker's stderr. PM's own no-op cases (venv, wheel, Nix, nothing committed under a venv) still return normally. Tests drive the real PM boot in a marked child: it imports a sentinel that exists only in the committed generation, and when an update commits a newer one mid-job the real collector keeps the worker's generation until the worker exits. A damaged selection exits before cron.jobs is imported. 123456hyu made the same prelude move and fail-closed change on #122290 (dff16e1); this carries them without the argv gate and source-order test. Co-authored-by: 123456hyu <123456hyu@users.noreply.github.com>
* upstream/main: fix(desktop): keep inherited NODE_OPTIONS intact when adding the builder heap fix(desktop): pack no longer runs through cross-env (NousResearch#110121, NousResearch#103010) fix(cron): stop the external worker when PM refuses its dependency boot refactor(cron): rename the worker boot marker and correct the pin's docs fix(cron): boot the external worker's PM dependencies before cron.jobs docs(cron): record the external worker's self-boot contract fix(cron): boot the external worker through PM's committed selection fix(cron): hold the committed dependency generation's lease at worker entry (NousResearch#122290 P2) fix(cron): restore activated dependency site-packages for the external worker chore: map contributor emails
… entry (NousResearch#122290 P2) The pin in scheduler_worker_env.py restores the site-packages *directory* into the child's PYTHONPATH, but a directory name is not a retention permit. The PM collector removes an unselected, lease-managed generation once it is old enough and no process holds its lease, so between the gateway's exit and the worker's next import the very generation the worker needs can be collected away. New cron/worker_bootstrap.py closes that gap at the worker entry, before the first application/dependency import: - lease_dependencies(): acquire the committed generation's kernel lease (the same hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses), so the collector's 'no process holds the lease' safety condition keeps the generation alive for the worker's whole lifetime. Unconditional and idempotent. - activate_worker_dependencies(): delegate to pm.environments.activate_dependencies (the same resolution + site.addsitedir + sys.path reordering the gateway's boot uses) so the worker's own imports resolve without waiting for the gateway's handoff. Gated on --external-worker-file in argv, so the in-process gateway path keeps its exact launch contract and the runner-owns-its-own-deps case stays a no-op. Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both degrade to a logged no-op when no dependency generation is committed. Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway, _no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard failures in this suite are pre-existing on a clean checkout and unrelated. Fixes the P2 lifetime finding in @andrexibiza's review of NousResearch#122290. (cherry picked from commit 429f0dd)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
…e its ack The external worker first imported hermes_bootstrap through run_agent inside run_one_job, after it had read (and deleted) its payload and published its ownership ack. When the dependency inputs were not current, prepare_launch synced and relaunched the worker into an -I interpreter that ignores the pinned PYTHONPATH, and worker_bootstrap had already popped the boot marker: the new process died on `import ruamel` and the adopted run was left `unknown` (#122290 follow-up). The worker's __main__ now runs hermes_bootstrap (finish_worker_boot) before it touches the payload, and the marker is dropped only after that, so a relaunch replays the whole worker and boots its dependencies again. The call cannot live in worker_bootstrap(): during `-m cron.scheduler`'s package import sys.argv[0] is still "-m" and __main__ has no spec, so the relaunch command could not name the module. Covers the POSIX execv and Windows subprocess.call relaunch branches alike. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b537b59)
10 participants
What does this PR do?
Fixes #122222 — on a self-managed (shell-installer / PM) install, every scheduled
cron job failed before it could run. The restart-safe external worker is spawned
with the store Python, and after the subprocess sanitizer drops the runtime
site-packages, the
pin_hermes_tree_on_pythonpathcounter-measure re-adds onlythe checkout, leaving the worker with no path to its dependencies. It died at its
first import (
No module named 'ruamel') before publishing its ownership ack, soevery job landed
last_status: failed.This restores the dependency environment this process actually activated, next to
the tree:
cron/scheduler_worker_env.pyderives the activatedsite-packagesfrom thisprocess's own
sys.path(the same signalpm.environments.running_from_selected_environmentreads), excluding theinterpreter's own
pureliband gating on a real venv (pyvenv.cfgin anancestor). When none is found the behaviour is unchanged (tree-only), so
installs that already own their dependencies are unaffected.
Related Issue
Fixes #122222
Type of Change
Changes Made
cron/scheduler_worker_env.py:pin_hermes_tree_on_pythonpathnow alsorestores the activated dependency
site-packages(order: tree, dependency,kept entries); new
_activated_dependency_site_packages+_in_real_venvhelpers. Derivation reads only
sys.path— no home-scoped filesystem reads,so it stays clear of
home_io_guard.tests/cron/test_restart_safe_worker.py: covered both directions of thederivation (activated env restored; nothing invented when the runner owns its
deps or the entry is not a real venv) and relaxed the exact-list assertion in
test_launch_external_worker_pin_extends_the_sanitized_env_not_os_environto arelationship assertion (tree first, sanitizer-kept preserved).
How to Test
pytest tests/cron/test_restart_safe_worker.pyPYTHONPATH="$HERMES_HOME/hermes-agent" "$STORE" -m cron.scheduler --helpno longer raises
ModuleNotFoundError: No module named 'ruamel'.Notes
pytest tests/suite: this machine is a Windows devcheckout and the target failure mode is the self-managed runtime install shape.
The changed module's tests were run in the dev venv.
tests/cron/test_restart_safe_worker.pypasses for all pin/derivation tests(6 targeted + the existing pin tests). Two unrelated tests fail on Windows and
fail identically on a clean checkout before this change:
os.getuidis absent on Windows (a scoped-dispatch test that is Linux-only inpractice but lacks the platform marker), and
home_io_guardrejects the realC:\Users\...\hermeshome during_resolve_hermes_bin_dir. Neither touchesthe changed code path.
Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests pass — not fully (see Notes)Documentation & Housekeeping
docs/, docstrings) — or N/A (docstrings updated inline)cli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/Asite-packages/dist-packages+ apyvenv.cfgancestor probe that is layout-agnostic)