Skip to content

fix(cron): external worker inherits parent Hermes site-packages - #122738

Closed
GabboAlarcon wants to merge 1 commit into
NousResearch:mainfrom
GabboAlarcon:fix/cron-worker-inherit-runtime-deps
Closed

GabboAlarcon wants to merge 1 commit into
NousResearch:mainfrom
GabboAlarcon:fix/cron-worker-inherit-runtime-deps

Conversation

@GabboAlarcon

Copy link
Copy Markdown

Restart-safe external cron workers spawn as plain sys.executable -m cron.scheduler, skipping the runtime bootstrap that resolves third-party dependencies. On installs where the gateway runs on a standalone interpreter with runtime-resolved deps (pm-managed layout), the tree pin alone leaves imports like ruamel (via hermes_yaml <- agent.secret_scope <- cron.jobs) unresolvable. The worker dies before its ownership ack:

ModuleNotFoundError: No module named 'ruamel'

observed as repeated cron external worker exited before ownership acknowledgement failures on a no-agent interval job (24 in a row in the field).

This extends pin_hermes_tree_on_pythonpath to also prepend the live parent's Hermes-owned site-packages dirs (existing dirs under a .hermes tree) to the worker PYTHONPATH. Verified with a real run: failing streak reset to 0, job status ok. No-op when no such dirs exist (tree pin only, as before).

Restart-safe external workers spawn as plain 'sys.executable -m
cron.scheduler', skipping the runtime bootstrap that resolves third-party
dependencies (standalone interpreter + hermes_bootstrap). The tree pin alone
leaves imports like ruamel (via hermes_yaml) unresolvable and the worker dies
before its ownership ack with ModuleNotFoundError.

Extend pin_hermes_tree_on_pythonpath to also prepend the live parent's
Hermes-owned site-packages dirs to the worker PYTHONPATH. No-op when none
match (tree pin only, as before).
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/cron Cron scheduler and job management sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Sep 25, 2026
ethernet8023 added a commit that referenced this pull request Sep 25, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, #122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685
(@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
@Enough1122

Copy link
Copy Markdown

AI code review — automated follow-up for reference; not a maintainer.

The underlying fix looks well-motivated — a -m cron.scheduler worker that cannot import third-party
deps is a real failure, and reading the entries off the live parent rather than guessing paths is
sensible. I confirmed the mechanism works. I do have one concern about the filter that selects which
paths get inherited, plus two smaller notes.

1. cron/scheduler_worker_env.py:44 — the ownership test is a substring match on the whole path,
so directories that are not Hermes-owned get inherited into the worker's PYTHONPATH.

for entry in sys.path:
    if "site-packages" not in entry or ".hermes" not in entry:
        continue

in on a str tests substrings anywhere in the path, not path components. The docstring at :37-38
says "Only existing dirs under a .hermes tree are returned", but the predicate admits anything whose
full path contains both substrings. Measured on this head, with a controlled sys.path:

ADMITTED  real hermes site-packages
rejected  user site-packages (NO .hermes)
ADMITTED  PROJECT dir named '.hermes'                 <- not hermes-owned
ADMITTED  unrelated dir w/ both substrings            <- not hermes-owned
ADMITTED  venv inside a .hermes tree
ADMITTED  user-writable site-packages under .hermes

…/workspace/.hermes/lib/site-packages is a plausible shape for a user project literally named
.hermes, and a backup directory such as my.hermes-backup/site-packages-old matches too. Because
these land ahead of the pre-existing PYTHONPATH (dict.fromkeys([root, *parent_site_pkgs, *existing]) at :67), a planted module in such a directory would shadow the real dependency for the
worker. Concretely, with ruamel.py written into a project-local .hermes/lib/site-packages:

worker PYTHONPATH (ordered; lower index = higher import precedence):
    [0] .../repo_root
    [1] .../workspace/.hermes/lib/site-packages      <- planted
    [2] .../other-project/.hermes-data/site-packages <- planted
    [3] /pre-existing
planted dir in worker PYTHONPATH : True at index 1

The same probe on upstream/main produces only [repo_root, /pre-existing] and has no
_parent_hermes_site_packages at all, so this exposure is introduced by this PR.

I want to be precise about severity: I did not find a path where a remote or untrusted party
controls a sys.path entry, so I am not claiming a remote exploit. The concern is that the filter is
looser than its docstring and looser than the module's own stated invariant — this module exists
specifically so that "user children must not see our tree" (:10-12), and the same reasoning applies
in reverse: entries that are not ours should not silently become ours. A component-wise test closes
it:

parts = Path(entry).parts
if "site-packages" not in parts or ".hermes" not in parts:
    continue

or, tighter, resolve the entry and require it to sit under the same Hermes home the worker is being
launched for (worker_env["HERMES_HOME"], which cron/scheduler.py:3525 already sets). Either
keeps the fix and drops the substring behaviour.

2. cron/scheduler_worker_env.py:64-65 — the early return means the fix does not apply in the
wheel / pipx layout, and that is the same layout where the docstring's ruamel example bites.
When
_installed_purelib() == repo_root the function returns the env untouched, so no site-packages are
inherited at all. That branch is deliberate and pre-existing (and the existing test at
tests/cron/test_restart_safe_worker.py:646-648 pins it), so I am not asking you to change it — but
it does mean the guarantee is "inherited outside the wheel layout", not unconditional. Worth a
sentence in the docstring so the next reader knows which configurations the fix covers.

3. No test is added for the new behaviour. The diff is one file, 28 insertions(+), 1 deletion(-), with no test change. The pre-existing assertion at
tests/cron/test_restart_safe_worker.py:642 is an exact-equality check on the entry list
(assert entries == [str(repo_root), str(tmp_path / "kept-by-sanitizer")]), which means it is
positionally sensitive to exactly this function's output. It passes today only because this test
environment's sys.path happens to contain no entry matching both substrings — on a developer
machine or CI image that does, the existing test would start failing on this PR. That is worth
knowing either way: a test that pins the list exactly will turn red on some hosts. A test that
monkeypatches sys.path with a controlled Hermes-shaped entry and asserts it is inherited would both
cover the fix and stabilise that assertion.

Minor:

  • cron/scheduler_worker_env.py:40 — import sys is inside the function while os and sysconfig
    are module-level imports (:18-19). Harmless, but inconsistent with the file's own style.
  • cron/scheduler_worker_env.py:46-47 — the if entry in found: continue dedupe is redundant with
    dict.fromkeys([...]) at :67, which already dedupes. Not a bug; just two mechanisms doing one
    job.
  • Ordering note, not a request: the inherited entries are placed before *existing, so a
    Hermes-owned site-packages wins over whatever the sanitizer left. Given the module's stated
    invariant that the sanitizer's "dropped runtime site-packages" decisions "stand" (:12-13), that
    ordering is the intended reversal — flagging only so the trade-off is explicit rather than
    incidental.

Verification actually performed (on 63b277d0948f9a532480cfd37a1f332590e4c179; no source file
modified, one worktree per side, isolated HERMES_HOME):

  • a controlled-sys.path probe listing exactly which directory shapes the new predicate admits;
  • a second probe planting ruamel.py in a project-local .hermes/lib/site-packages and reading the
    resulting ordered PYTHONPATH, with the purelib early-return stubbed out so the real code path
    ran;
  • the identical probe on upstream/main, which yields only [repo_root, /pre-existing];
  • the purelib == repo_root branch, confirming the env is returned untouched and no inheritance runs;
  • python -m pytest tests/cron/test_restart_safe_worker.py -k "pythonpath or worker_env or pin"
    on head: 2 passed (so this is not a test regression I am reporting, only a coverage gap);
  • git diff --stat against the PR base confirming the diff is confined to the one file with no test
    change.

Unverified / please confirm:

  • I could not determine whether any real deployment puts a non-Hermes directory matching both
    substrings on sys.path. That depends on how the gateway's runtime sys.path is assembled, which I
    traced only as far as this function and its call site at cron/scheduler.py:3541-3543.
  • I did not run the full cron test suite, only the two tests matching the -k filter above.
  • I have not verified that the third-party import the docstring names (ruamel via hermes_yaml) is
    genuinely absent from the worker's path without this change; I took the failure mode from the PR
    description and the module docstring rather than reproducing the original ModuleNotFoundError.
  • Whether inheriting the parent's full site-packages (rather than a computed subset) is acceptable
    under your threat model is a maintainer judgement; I have shown the filter is looser than documented,
    not that the approach is wrong.

kshitijk4poor pushed a commit to kshitijk4poor/hermes-agent that referenced this pull request Sep 28, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685
(@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
(cherry picked from commit 7c55715)
kshitijk4poor pushed a commit to kshitijk4poor/hermes-agent that referenced this pull request Sep 28, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685
(@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
(cherry picked from commit 7c55715)
@kshitijk4poor

Copy link
Copy Markdown

Closing as superseded by #126344, which fixes the same failure by carrying #122290/#122936. Inheriting the parent's site-packages alone leaves the generation unleased, so the collector could delete it under a running job. #126344 instead boots PM's activate_dependencies in the cron package prelude, leases the generation for the worker's lifetime, and fails closed on activation errors. You're credited in the PR body. Thank you. If #126344 stalls, I'll reopen this.

kshitijk4poor pushed a commit to kshitijk4poor/hermes-agent that referenced this pull request Sep 28, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685
(@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
(cherry picked from commit 7c55715)
OutThisLife pushed a commit that referenced this pull request Sep 28, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, #122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685
(@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
OutThisLife pushed a commit that referenced this pull request Sep 28, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, #122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685
(@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
kshitijk4poor pushed a commit to kshitijk4poor/hermes-agent that referenced this pull request Sep 28, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685
(@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
(cherry picked from commit 7c55715)
kshitijk4poor pushed a commit to kshitijk4poor/hermes-agent that referenced this pull request Sep 28, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685
(@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
(cherry picked from commit 7c55715)
kshitijk4poor pushed a commit that referenced this pull request Sep 28, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, #122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685
(@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
(cherry picked from commit 7c55715)
aryn-lacy pushed a commit to aryn-lacy/hermes-agent that referenced this pull request Sep 30, 2026
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, NousResearch#122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.

Salvaged from NousResearch#122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:

* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
  `site_packages()` for the generation, instead of deriving one from this process's
  `sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
  home, so the child-spawn path no longer reads the real home -- that probe tripped
  `tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
  uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
  `site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
  `pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
  holds the generation's lease for the process's life. It is gated on the
  `HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
  sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
  contract without argv identity inference.

Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.

Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.

Duplicate-cluster credit: NousResearch#122238 (@webtecnica, earliest), NousResearch#122382 (@Sakujakira), NousResearch#122685
(@etrisnanto), NousResearch#122719 (@aglowinthefield) and NousResearch#122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
(cherry picked from commit 7c55715)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P1 High — major feature broken, no workaround sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants