fix(cron): restore activated dependency site-packages in external worker PYTHONPATH by webtecnica · Pull Request #122238 · NousResearch/hermes-agent

webtecnica · 2026-09-25T03:55:41Z

Summary

Restores the activated dependency site-packages directory onto the external cron worker's PYTHONPATH when spawned under a self-managed install.

Root cause

Under a self-managed install, the gateway runs on the store Python and activates project dependencies by placing the selected generation's site-packages on sys.path. When spawning an external cron worker via sys.executable -m cron.scheduler, the subprocess sanitizer strips Hermes-owned pythonpaths, and pin_hermes_tree_on_pythonpath previously only restored repo_root. Since the store Python's own site-packages contains no third-party libraries, the worker crashed with ModuleNotFoundError (such as No module named 'ruamel') before publishing its ownership acknowledgement.

Fix

In cron/scheduler_worker_env.py, implemented _activated_site_packages() which scans sys.path for an activated site-packages directory residing inside a real virtual environment (pyvenv.cfg ancestor check) while excluding the interpreter's own purelib.
In pin_hermes_tree_on_pythonpath(), append the activated site-packages next to repo_root.
In tests/cron/test_restart_safe_worker.py, added test_pin_hermes_tree_restores_activated_dependency_site_packages and converted test_launch_external_worker_pin_extends_the_sanitized_env_not_os_environ to relationship assertions.

Test plan

Python syntax compilation: python -m py_compile cron/scheduler_worker_env.py tests/cron/test_restart_safe_worker.py (0 errors).
Unit tests: Verified red->green transition on test_pin_hermes_tree_restores_activated_dependency_site_packages.
Sabotage: Sabotaged _activated_site_packages to return None, verifying expected failure (1 failed), then restored to clean pass (3/3 passed).

Closes #122222

…ker PYTHONPATH

kiramakes · 2026-09-25T04:00:40Z

🤖 Hermes Agent automated review: PR reviewed. Diff analyzed (113 lines changed). Check CI status and manual review recommended.

arkheioncorp

Review: fix(cron): restore activated dependency site-packages in external worker PYTHONPATH

Verdict: APPROVE

Findings

Clean fix. The new _activated_site_packages() helper correctly walks sys.path, skips the interpreter's own purelib, and requires a real venv ancestor (pyvenv.cfg) — no accidental non-venv site-packages leaking in.
Good edge-case handling. OSError/RuntimeError/ValueError around Path.resolve() are caught; empty sys.path entries are skipped.
Test coverage. test_pin_hermes_tree_restores_activated_dependency_site_packages sets up a fake venv with pyvenv.cfg, a stray non-venv site-packages, and asserts the right ordering — exactly the scenario that motivated the fix.
No security concerns. No secrets, no shell injection, no unsafe deserialization.

One minor note: the existing test test_launch_external_worker_pin_extends_the_sanitized_env_not_os_environ was relaxed from an exact-equality assert to containment checks — this is the right call given the new variable insertion, but worth a glance to ensure the ordering intent (repo_root first, then activated, then existing) is preserved. The new test enforces that ordering explicitly, so it is covered.

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>

fastfinge · 2026-09-26T08:56:30Z

Production verification (self-hosted, PM-managed install)

Hit this on a real host: 14 scheduled jobs, every external worker died before its ownership ack since ~Sept 24 (326+ ModuleNotFoundError: No module named 'ruamel' failures/day — cron/__init__.py → cron.jobs → agent.secret_scope → utils.py → hermes_yaml → ruamel.yaml).

Applied this PR's approach locally (independently derived, nearly identical to _activated_site_packages) and verified end-to-end:

Before: PYTHONPATH=<repo> python -m cron.scheduler --help → immediate ruamel traceback, exit 1 (matches the gateway's Restart-safe cron worker dispatch failed: cron external worker exited before ownership acknowledgement log line)
After pinning the committed venv's site-packages: same spawn → clean --help, exit 0
In production: first worker dispatch after gateway rotation completed normally (Iceshrimp 502 watchdog, no_agent script job, previously failing every 5-min tick since Sept 24; jobs output resumed under ~/.hermes/cron/output/)

One robustness note for review: this implementation scans sys.path for a non-purelib site-packages under a pyvenv.cfg ancestor, which correctly handles the PM layout (~/.hermes/installs/<key>/environments/<hash>/venv/...). An alternative (what my local copy does) is resolving via pm.environments.committed_venv(repo_root) + site_packages(), which reads the same source of truth activate_dependencies uses at boot — marginally less heuristic if a stray foreign venv ever lands on sys.path, but either approach fixes the outage. Tests here cover the pin + sanitizer interaction well.

Either way — this PR unblocks every PM-era self-hosted install with restart-safe cron workers. Would love to see it merged; happy to re-test on my host after merge.

Tomsik01 · 2026-09-28T08:19:21Z

Can someone please approve this? 🙏🏻😭

goetterbote2342 · 2026-09-28T10:06:26Z

Independent confirmation from a live self-managed (PM / managed-runtime) install — Linux arm64 (Raspberry Pi), v0.21.5 @ 5912ed8, systemd-managed gateway.

Symptom before the fix, on every dispatch from the managed gateway (first failure 2026-09-28 09:45, every tick after):

ERROR cron.scheduler: Job '…': Restart-safe cron worker dispatch failed:
cron external worker exited before ownership acknowledgement (exit 1)
  ModuleNotFoundError: No module named 'ruamel'
  (cron.jobs → cron.env_settings → agent.secret_scope → utils → hermes_yaml → ruamel.yaml)

Repro of the spawn contract (store Python + checkout pinned, no dependency activation):

cd ~/.hermes/hermes-agent
<store-python> -m cron.scheduler --help                        # ModuleNotFoundError: No module named 'ruamel'
PYTHONPATH="$PWD:<generation-site-packages>" <store-python> -m cron.scheduler --help   # rc=0

hermes cron run <id> succeeded the whole time (in-process path), so the job itself was fine — the dispatch env was the only broken link.

I had independently implemented a local variant that resolves the path via pm.environments.committed_venv(project_root) → site_packages() instead of scanning sys.path; on this host both variants produce identical PYTHONPATH output. Your sys.path scan is the better shape — layout-agnostic, it also covers the pre-PM in-tree venv activation and does not depend on the PM API being importable at pin time.

Test evidence: tests/cron/test_restart_safe_worker.py + tests/pm/test_member_selection.py pass locally (34 tests), including a regression test asserting the pin carries the activated site-packages alongside the checkout.

One edge worth a test if you agree: a PYTHONPATH entry does not process that directory's .pth files (unlike site.addsitedir), so uv-editable members registered via .pth inside the generation are not visible to the worker. The checkout pin covers hermes itself and the full cron.scheduler import chain works live here, but a job whose code imports an editable-installed path dependency would still miss it.

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)

kshitijk4poor · 2026-09-28T14:16:32Z

Closing as superseded by #126344, which fixes the same failure: the restart-safe cron external worker dying with No module named 'ruamel' before its ownership ack. It carries #122290/#122936 and boots PM's own activate_dependencies in the cron package prelude, before cron.jobs. That leases the committed generation for the worker's lifetime, so the collector cannot delete it mid-job, and a broken activation fails the dispatch before the ack instead of running on unheld dependencies. Putting the dependency path on PYTHONPATH, or choosing a different interpreter, makes the import work but does not hold that lease.

Thank you @webtecnica, you're credited in the #126344 body. Yours was the earliest fix submitted for #122222, and the PR body credits it that way. If #126344 stalls, I'll reopen this.

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, #122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685 (@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)

@123456hyu

The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the subprocess sanitizer has just stripped, so the worker died at its first dependency import (`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership acknowledgement -- every scheduled job landed `last_status: failed`. Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency site-packages and leased the generation at the worker entry. Reworked onto PM's own machinery: * `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` + `site_packages()` for the generation, instead of deriving one from this process's `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active home, so the child-spawn path no longer reads the real home -- that probe tripped `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the uv-launcher, first-match-only and ordering edge cases of the old derivation go with it. * `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease + `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it holds the generation's lease for the process's life. It is gated on the `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch contract without argv identity inference. Verified live on PM's store Python against a real generation (A/B, same child): tree-only PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports, `running_from_selected_environment` True, and the worker holds a real kernel lease while it runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed. Tests: the two invariants that carry the contract -- the pin restores the committed generation (and invents nothing without one), and the boot runs only in the marked worker, once. The six construction tests that drove the old derivation are dropped with it. Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685 (@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed this same worker-dependency failure in parallel while this branch was rebuilt. None of their diffs is contained here -- the co-author trailers below record direction and duplicate-cluster credit, not lifted code; their roles are listed in the PR body. Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com> Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com> Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com> Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com> Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com> (cherry picked from commit 7c55715)

fix(cron): restore activated dependency site-packages in external wor…

6634032

…ker PYTHONPATH

arkheioncorp reviewed Sep 25, 2026

View reviewed changes

alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/cron Cron scheduler and job management sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Sep 25, 2026

This was referenced Sep 25, 2026

fix(kanban): carry the module argv's import root to the worker env #122420

Closed

cron external worker missing venv site-packages on PYTHONPATH (ModuleNotFoundError: ruamel) #122529

Open

ajiua mentioned this pull request Sep 25, 2026

[Bug]: cron external worker cannot import dependencies on self-managed installs — every scheduled job fails before its ownership ack #122222

Closed

123456hyu mentioned this pull request Sep 25, 2026

fix(cron): restore activated dependency site-packages for the external worker #122290

Closed

12 of 13 tasks

nhod mentioned this pull request Sep 25, 2026

fix(cron): pin the runtime dependency path for POSIX .py cron scripts #122742

Closed

6 of 7 tasks

ethernet8023 mentioned this pull request Sep 25, 2026

Cron jobs on self-managed installs no longer die before dispatch (#122222, salvage #122290) #122936

Open

12 of 13 tasks

strokesws mentioned this pull request Sep 25, 2026

fix(launch): publish launchers on a runtime that can spawn a child (kanban/cron workers) #123041

Open

jonpol01 mentioned this pull request Sep 26, 2026

fix(cron): carry the committed venv's site-packages in the worker PYTHONPATH pin #123738

Closed

kshitijk4poor mentioned this pull request Sep 26, 2026

fix(p1/p2): worker deps pin, sqlite uri encoding, remote perms, rollback carry guard #123374

Closed

liuhao1024 mentioned this pull request Sep 26, 2026

Cron external worker uses raw interpreter and loses runtime dependencies #124279

Open

mathcuei mentioned this pull request Sep 28, 2026

fix(cron): pin the external worker's runtime site-packages, not just repo root #126001

Closed

3 of 4 tasks

pan-long mentioned this pull request Sep 28, 2026

[Bug]: cron external worker exits before ownership acknowledgement after update — env sanitizer drops runtime site-packages (ModuleNotFoundError: ruamel) #125820

Open

alt-glitch added the area/install-update Installer, updater, packaging, wheels, doctor label Sep 28, 2026

alt-glitch added the needs-decision Awaiting maintainer decision before any implementation label Sep 28, 2026

kshitijk4poor mentioned this pull request Sep 28, 2026

Restart-safe cron jobs no longer die on 'No module named ruamel' under multiplex, and fail closed on a broken PM boot (#122222, salvage #122936) #126344

Merged

kshitijk4poor closed this Sep 28, 2026

Navigation Menu

Repository navigation

fix(cron): restore activated dependency site-packages in external worker PYTHONPATH - #122238

fix(cron): restore activated dependency site-packages in external worker PYTHONPATH#122238
webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/122222-cron-worker-dependencies

webtecnica commented Sep 25, 2026

Uh oh!

kiramakes commented Sep 25, 2026

Uh oh!

arkheioncorp left a comment

Uh oh!

fastfinge commented Sep 26, 2026

Uh oh!

Tomsik01 commented Sep 28, 2026

Uh oh!

goetterbote2342 commented Sep 28, 2026

Uh oh!

kshitijk4poor commented Sep 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

webtecnica commented Sep 25, 2026

Summary

Root cause

Fix

Test plan

Uh oh!

kiramakes commented Sep 25, 2026

Uh oh!

arkheioncorp left a comment

Choose a reason for hiding this comment

Review: fix(cron): restore activated dependency site-packages in external worker PYTHONPATH

Findings

Uh oh!

fastfinge commented Sep 26, 2026

Production verification (self-hosted, PM-managed install)

Uh oh!

Tomsik01 commented Sep 28, 2026

Uh oh!

goetterbote2342 commented Sep 28, 2026

Uh oh!

kshitijk4poor commented Sep 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants