fix(kanban): carry the module argv's import root to the worker env by liuhao1024 · Pull Request #122420 · NousResearch/hermes-agent

liuhao1024 · 2026-09-25T07:16:26Z

What does this PR do?

On a store-python (PM) install, the kanban dispatcher resolves the worker's spawn argv through the module form (sys.executable -m hermes_cli.main) because find_spec("hermes_cli") succeeds in the gateway process — but that importability comes from the shim's in-process sys.path injection, which is invisible to the spawned child. The child runs the bare interpreter from the task workspace with a PYTHONPATH the subprocess-env factory has scrubbed of Hermes-owned entries, fails with ModuleNotFoundError: No module named 'hermes_cli' before doing any work, trips the dispatcher's circuit breaker, and the board auto-blocks (#122299; four independent reproductions across Linux, Windows and macOS installs).

The fix keeps the module-over-PATH ordering that da18c20226 introduced for safety (#111569 — a PATH hermes could be attacker-planted) and instead makes the selected module form actually resolvable for the child: when the argv leads with -m hermes_cli.main, the dispatcher prepends the running install's package root onto the worker's PYTHONPATH. This propagates exactly the context that made find_spec succeed in the parent. It is version-safe because the child re-execs this same interpreter — the PYTHONPATH entries the scrub removes are dangerous only for a different interpreter's compiled modules. Once hermes_cli.main is importable, its own bootstrap owns dependency activation as usual, so no site-packages are propagated (unlike the cron-side fix in #122238, whose -m cron.scheduler child imports third-party packages directly). A resolved shim path is left untouched: a shim owns its own imports.

Related Issue

Fixes #122299

Type of Change

🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

hermes_cli/kanban_db_dispatch.py: added _propagate_module_import_root() (module-form guard + case-normalized dedup + prepend of the package root derived from the running module's own location); called it in _default_spawn right after _worker_argv builds the command, before the systemd restart-safe wrapper.
tests/hermes_cli/test_kanban_db.py: two regression tests — helper semantics (module form prepends the root, existing entries survive, no duplication, shim paths untouched) and a real-subprocess check that a bare interpreter launched from a neutral cwd with only the propagated PYTHONPATH can resolve hermes_cli.

How to Test

pytest tests/hermes_cli/test_kanban_db.py -q — 53 passed, 1 skipped for the file (test_infrastructure_spawn_refusal_never_charges_the_card fails identically on a clean upstream/main checkout on this macOS host — systemd user-bus probe surface — and is untouched by this diff; it passes in CI).
pytest tests/hermes_cli/test_kanban_db.py -k "puts_package_root or importable_from_scrubbed" — the new tests pass; observed result: 2 passed.
Store-python shape from the report (manual, on a PM install): cd /tmp && <store-python> -m hermes_cli.main --version fails with ModuleNotFoundError; the same interpreter with PYTHONPATH=<repo> succeeds — which is exactly what the dispatcher now sets for the child. Observed result: the subprocess probe test replicates the green half of this mechanically.

Checklist

Code

I've read the Contributing Guide
My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
I searched for existing PRs to make sure this isn't a duplicate (fix(cron): restore activated dependency site-packages in external worker PYTHONPATH #122238 covers the cron-side sibling [Bug]: cron external worker cannot import dependencies on self-managed installs — every scheduled job fails before its ownership ack #122222 in cron/scheduler_worker_env.py — different subsystem, no file overlap)
My PR contains only changes related to this fix/feature (no unrelated commits)
I've run pytest tests/ -q and all tests pass (full file run: 53 passed / 1 skipped; the one failure noted above is pre-existing on clean upstream/main locally and outside this change's path)
I've added tests for my changes (required for bug fixes, strongly encouraged for features)
I've tested on my platform: macOS 15 (arm64); the code path is platform-neutral and the propagated-root guard uses os.pathsep/os.path.normcase for Windows

Documentation & Housekeeping

I've updated relevant documentation (README, docs/, docstrings) — or N/A (docstring on the new helper documents the invariant)
I've updated cli-config.yaml.example if I added/changed config keys — or N/A
I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
I've considered cross-platform impact (Windows, macOS) per the compatibility guide (path separator and case-insensitive dedup handled)
I've updated tool descriptions/schemas if I changed tool behavior — or N/A

The dispatcher's _resolve_hermes_argv proves hermes_cli importable in the gateway process, where a store-python shim has the repo root on sys.path in-process; the spawned bare 'sys.executable -m hermes_cli.main' child runs from the task workspace with a scrubbed PYTHONPATH and cannot import the package, dies before any work, and the board auto-blocks (NousResearch#122299). Keep the module-over-PATH ordering (da18c20, NousResearch#111569) and instead propagate the running install's package root onto the worker's PYTHONPATH; a resolved shim path keeps owning its own imports. Fixes NousResearch#122299

whyyagswhy · 2026-09-25T07:21:05Z

Independent verification on the PR head (46e36e2): kanban-db suite 54/54 green on Linux, 1 skipped. Carrying the module argv import root to the worker env keeps worker imports resolving. No findings.

liuhao1024 · 2026-09-25T07:25:04Z

Thanks @whyyagswhy for the independent verification on Linux — much appreciated. That complements the macOS side nicely: locally I also ran the real-subprocess probe (fresh process, neutral cwd, clean env with only PYTHONPATH pointing at the repo root) and the child resolved hermes_cli fine, and the helper-level tests cover prepend/keep-existing/dedup/no-shim-mutation semantics. Cross-platform confirmation is exactly what this change needed since the failure only shows up when the worker is spawned outside the repo layout.

connerstirne-gh · 2026-09-25T10:27:35Z

E2E-verified this fix on a real PM/source install (macOS 27.0, Apple Silicon, store Python 3.14.7 with sys.prefix == sys.base_prefix, gateway running via the launcher wrapper with PYTHONPATH=<checkout>:<committed-site-packages>), using the exact case matrix from #122487. Method: imported this branch's patched _propagate_module_import_root with the module root bound to the live checkout (what the real dispatcher computes), spawned the real worker argv [store python, -m, hermes_cli.main, --version] from a sanitized env (what _strip_hermes_owned_pythonpath leaves) in a worker-like cwd:

Case	Result
1. Sanitized env, no root (before fix)	rc=1 — `ModuleNotFoundError: No module named 'hermes_cli'` (the #122299/#122487 symptom)
2. Sanitized env + `_propagate_module_import_root`	rc=0 — boots, reports v0.21.5 @ `69948c0`
3. Control: full activated PYTHONPATH (root + committed site-packages)	rc=0
4. Resolved shim argv	untouched (no PYTHONPATH rewriting)
5. Root already present	no duplication, root stays first

One scope note for anyone landing on this issue: the propagated root is the dispatcher's own checkout root. On our install the child then activates the committed generation via hermes_cli.main's bootstrap and runs fine — no need to also propagate the committed site-packages path. The Bot-DM delivery runner (tools/bot_mode_dm.py) remains uncovered by this PR, as noted above.

This fix resolves the Kanban half of #122487 on our configuration. Thanks for the clean, well-tested patch.

liuhao1024 · 2026-09-25T10:37:03Z

Thanks @connerstirne-gh — this is a great E2E confirmation, especially since it runs the real worker argv from the env _strip_hermes_owned_pythonpath actually leaves, not a synthetic one. Case 1 reproducing the exact ModuleNotFoundError: hermes_cli symptom from #122299/#122487 and Case 2 booting to v0.21.5 @ 69948c00 is precisely the before/after the helper-level tests pin down; Cases 4/5 (shim argv untouched, no root duplication) confirm the propagation stays surgical.

Your scope note matches the intended boundary: the propagated root is the dispatcher's own checkout root, and the child activates the committed generation via its own bootstrap — which is why no site-packages propagation was added. The Bot-DM delivery runner (tools/bot_mode_dm.py) staying uncovered is the known limit noted upthread.

Two independent verifications now (Linux suite + this macOS E2E matrix) — appreciated.

megastruktur · 2026-09-25T13:00:57Z

E2E confirmation on a live Linux PM/source install (CachyOS, kernel 7.2, store Python 3.14.7, main @ 59004a62356, gateways under systemd user units with kanban.dispatch_in_gateway: true).

Before: every dispatcher spawn (<store python> -m hermes_cli.main -p <profile> ... chat -q "work kanban task ...") died with ModuleNotFoundError: No module named 'hermes_cli', two runs, then gave_up and the card auto-blocked.
Applied this PR's diff on top of main (applies cleanly), ran the two new regression tests (pass), restarted the dispatching gateway.
After: the same task was re-dispatched, and the worker booted, heartbeated and ran tool calls normally.

Thanks for the fix. It would be great to get this merged: on this install the kanban board was fully dead after the 2026-09-25 update.

liuhao1024 · 2026-09-25T13:09:42Z

Thanks @megastruktur — a third independent confirmation, and the first one capturing the full failure arc end to end: the dispatcher spawn dying twice with ModuleNotFoundError: No module named 'hermes_cli', the card going to gave_up and auto-blocking, and then the same task re-dispatching, booting, heartbeating and running tool calls normally once the diff was applied on top of main. The kanban.dispatch_in_gateway: true + systemd user units setup also exercises exactly the gateway-side spawn path this change targets.

With the two new regression tests passing on your install on top of the Linux suite run and the macOS E2E matrix upthread, the fix now has coverage on three platforms. Glad the board is alive again after the 2026-09-25 update.

dummey · 2026-09-26T20:19:55Z

I'll add another independent confirmation for my Windows 11 machine:

Component: hermes_cli/kanban_db_dispatch.py (worker spawn)
Platform: Windows 11, x86_64
Hermes install: C:\Users\Ricky\AppData\Local\hermes\hermes-agent
Interpreter: managed tools\python-3.14.7+20260901-win32-x64\python.exe

I have some left over tests tasks from V0.21.5 that were failing. Checked out your pr branch and they have run just fine. Retested with a new batch of tasks and they are working fine now too.

From my test task showing initially running into the problem (-22min), it self diagnosing (-22 min), and then resolving correctly on this PR branch (-6 min).

crashed
workout-buddy
1m
22 min. ago
pid 6844 exited with code 1 Worker's last output: "C:\\Users\\Ricky\\AppData\\Local\\hermes\\tools\\python-3.14.7+20260901-win32-x64\\python.exe: Error while finding module specification for 'hermes_cli.main' (ModuleNotFoundError: No module named 'hermes_cli')"

blocked
workout-buddy
19s
22 min. ago
Deterministic upstream Hermes spawn defect, NOT fixable from this repo or via environment vars. Fresh control card t_f0c3dab6 (created after all upgrades, scratch workspace, no repo) failed identically in 28s, eliminating stale-card/upgrade theories. Root cause: dispatcher runs inside gateway PID 22308 and spawns workers as `sys.executable -m hermes_cli.main` with cwd=workspace; sys.path is not inherited, so the child cannot import hermes_cli. PYTHONPATH is scrubbed (local.py:279), ignored under -I, and stripped as repo-root (local_pythonpath.py:142-144). Needs a fix in _resolve_hermes_argv/build_subprocess_env. Diagnostic card served its purpose - no code changes.

completed
workout-buddy
33s
6 min. ago
DIAGNOSTIC RESULT: SPAWN NOW WORKS. hermes_cli RESOLVES. The defect on t_ce65da6f and prior attempts of this card is NO LONGER REPRODUCIBLE.

liuhao1024 · 2026-09-26T20:23:46Z

Thanks @dummey — this is the first Windows-side confirmation, so it completes the platform matrix (Linux + macOS + now Windows 11), which matters here because the propagation is exactly the kind of thing that can drift per-OS.

The timeline you captured is a particularly clean before/after: the pre-fix card dying twice with ModuleNotFoundError: No module named 'hermes_cli' from the managed store interpreter — matching the original #122299/#122487 symptom — and then the diagnostic card on the PR branch reporting SPAWN NOW WORKS. hermes_cli RESOLVES, with your leftover V0.21.5 tasks and a fresh batch both running clean. Also nice that the worker's own self-diagnosis on the crashed card independently landed on the same root cause (child not inheriting the import root), which corroborates the fix target rather than just the symptom.

Appreciate you testing on the real managed-tools interpreter path (tools\python-3.14.7+...\python.exe) — that's the argv shape the helper-level tests pin down, so it's good to see it verified live on Windows.

#122299, #122487, #122500, salvage #122420) _resolve_hermes_argv proves hermes_cli importable in the GATEWAY, where the store-python shim / launcher bootstrap put the repo root on sys.path in-process. The worker env scrub (build_subprocess_env / served_profile_child_env) strips Hermes-owned PYTHONPATH entries, so the bare `sys.executable -m hermes_cli.main` child could not import the package the parent just proved importable and every worker died at birth (ModuleNotFoundError: hermes_cli) -> crashed x2 -> gave_up. Reuse cron's pin_hermes_tree_on_pythonpath (#112729) on the module argv only: a resolved shim path owns its imports. hermes_cli.main's bootstrap then activates the committed PM dependency generation as usual, so a store Python with an empty site-packages still boots. Same class in the Bot Chat delivery runner (cron/scheduler_delivery.py), which builds the module argv after served_profile_child_env.

liuhao1024 · 2026-10-01T19:16:23Z

Closing — this fix has been salvaged into main: 7e3b34c landed the module-argv import-root propagation from this PR, and 5392a83 later refined it to reuse cron's pin_hermes_tree_on_pythonpath (#112729) on the module argv only, covering #122299/#122487/#122500 and the dispatcher crash reported in #125649. Nothing further to carry here — thanks for the salvage.

liuhao1024 mentioned this pull request Sep 25, 2026

Kanban worker spawn dies with 'No module named hermes_cli' on PM/source installs: module-argv guard trusts find_spec, but the worker env strips the PYTHONPATH that provides it #122487

Closed

Wenfengcheng mentioned this pull request Sep 25, 2026

fix(bot-mode): activate committed dependencies in DM runner #122499

Closed

Alexandre78500 mentioned this pull request Sep 25, 2026

fix(kanban): bootstrap managed Hermes relaunches #122579

Open

strokesws mentioned this pull request Sep 25, 2026

fix(launch): publish launchers on a runtime that can spawn a child (kanban/cron workers) #123041

Open

dummey mentioned this pull request Sep 26, 2026

Windows: kanban workers crash with ModuleNotFoundError when gateway runs from venv (pip console_script launcher fails with restricted PATH) #30943

Open

alt-glitch added P3 Low — cosmetic, nice to have P2 Medium — degraded but workaround exists and removed P2 Medium — degraded but workaround exists P3 Low — cosmetic, nice to have labels Sep 26, 2026

diegolm2012-creator mentioned this pull request Sep 26, 2026

kanban dispatcher: worker spawn argv guard checks importability in parent, unsound for bare-interpreter child (ModuleNotFoundError: hermes_cli) #122299

Closed

alt-glitch mentioned this pull request Sep 27, 2026

fix(kanban): pin checkout on worker PYTHONPATH under the PM launcher #124793

Open

13 of 14 tasks

intellectronica mentioned this pull request Sep 27, 2026

fix(kanban): spawn workers through the installation-bound launcher bootstrap #125088

Open

dskwe mentioned this pull request Sep 27, 2026

fix(cron): launch the external worker on the committed dependency venv interpreter #125304

Closed

11 of 13 tasks

liuhao1024 mentioned this pull request Sep 27, 2026

fix(kanban): dispatcher workers crash at birth with ModuleNotFoundError under managed Python runtime #125649

Open

teknium1 mentioned this pull request Sep 28, 2026

Kanban workers boot on store-Python installs; unknown assignee, gh identity and dead-credential exits fixed (#122299 #122487 #122500 #122422 #122689, salvage #122420 #122567 #122714 #122716) #126153

Merged

liuhao1024 closed this Oct 1, 2026

Navigation Menu

Repository navigation

fix(kanban): carry the module argv's import root to the worker env - #122420

fix(kanban): carry the module argv's import root to the worker env#122420
liuhao1024 wants to merge 1 commit into
NousResearch:mainfrom
liuhao1024:liuhao/cron-bugfix-122299

liuhao1024 commented Sep 25, 2026

Uh oh!

whyyagswhy commented Sep 25, 2026

Uh oh!

liuhao1024 commented Sep 25, 2026

Uh oh!

connerstirne-gh commented Sep 25, 2026

Uh oh!

liuhao1024 commented Sep 25, 2026

Uh oh!

megastruktur commented Sep 25, 2026

Uh oh!

liuhao1024 commented Sep 25, 2026

Uh oh!

dummey commented Sep 26, 2026

Uh oh!

liuhao1024 commented Sep 26, 2026

Uh oh!

liuhao1024 commented Oct 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

liuhao1024 commented Sep 25, 2026

What does this PR do?

Related Issue

Type of Change

Changes Made

How to Test

Checklist

Code

Documentation & Housekeeping

Uh oh!

whyyagswhy commented Sep 25, 2026

Uh oh!

liuhao1024 commented Sep 25, 2026

Uh oh!

connerstirne-gh commented Sep 25, 2026

Uh oh!

liuhao1024 commented Sep 25, 2026

Uh oh!

megastruktur commented Sep 25, 2026

Uh oh!

liuhao1024 commented Sep 25, 2026

Uh oh!

dummey commented Sep 26, 2026

Uh oh!

liuhao1024 commented Sep 26, 2026

Uh oh!

liuhao1024 commented Oct 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants