Repository navigation
fix(cron): external worker inherits parent Hermes site-packages - #122738
GabboAlarcon wants to merge 1 commit into
Conversation
Restart-safe external workers spawn as plain 'sys.executable -m cron.scheduler', skipping the runtime bootstrap that resolves third-party dependencies (standalone interpreter + hermes_bootstrap). The tree pin alone leaves imports like ruamel (via hermes_yaml) unresolvable and the worker dies before its ownership ack with ModuleNotFoundError. Extend pin_hermes_tree_on_pythonpath to also prepend the live parent's Hermes-owned site-packages dirs to the worker PYTHONPATH. No-op when none match (tree pin only, as before).
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
The underlying fix looks well-motivated — a 1. for entry in sys.path:
if "site-packages" not in entry or ".hermes" not in entry:
continue
The same probe on I want to be precise about severity: I did not find a path where a remote or untrusted party parts = Path(entry).parts
if "site-packages" not in parts or ".hermes" not in parts:
continueor, tighter, resolve the entry and require it to sit under the same Hermes home the worker is being 2. 3. No test is added for the new behaviour. The diff is one file, Minor:
Verification actually performed (on
Unverified / please confirm:
|
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
|
Closing as superseded by #126344, which fixes the same failure by carrying #122290/#122936. Inheriting the parent's site-packages alone leaves the generation unleased, so the collector could delete it under a running job. #126344 instead boots PM's |
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)
Restart-safe external cron workers spawn as plain
sys.executable -m cron.scheduler, skipping the runtime bootstrap that resolves third-party dependencies. On installs where the gateway runs on a standalone interpreter with runtime-resolved deps (pm-managed layout), the tree pin alone leaves imports likeruamel(viahermes_yaml<-agent.secret_scope<-cron.jobs) unresolvable. The worker dies before its ownership ack:ModuleNotFoundError: No module named 'ruamel'
observed as repeated
cron external worker exited before ownership acknowledgementfailures on a no-agent interval job (24 in a row in the field).This extends
pin_hermes_tree_on_pythonpathto also prepend the live parent's Hermes-owned site-packages dirs (existing dirs under a.hermestree) to the worker PYTHONPATH. Verified with a real run: failing streak reset to 0, job status ok. No-op when no such dirs exist (tree pin only, as before).