Skip to content

fix(terminal): strip Hermes-venv site-packages from terminal subprocess PYTHONPATH - #61028

Closed
mmchuangyt-ai wants to merge 1 commit into
NousResearch:mainfrom
mmchuangyt-ai:fix/terminal-strip-hermes-venv-pythonpath
Closed

fix(terminal): strip Hermes-venv site-packages from terminal subprocess PYTHONPATH#61028
mmchuangyt-ai wants to merge 1 commit into
NousResearch:mainfrom
mmchuangyt-ai:fix/terminal-strip-hermes-venv-pythonpath

Conversation

@mmchuangyt-ai

Copy link
Copy Markdown

Problem

The Desktop Electron process injects the Hermes venv's site-packages path
(e.g. .../python3.11/site-packages) into PYTHONPATH so the Python 3.11
backend can import its packages. When this PYTHONPATH leaks into terminal
subprocesses running a different Python version (e.g. Python 3.13), 3.11
C extension modules (PIL _imaging, cryptography, etc.) appear on
sys.path ahead of the correct 3.13 versions and crash with ImportError.

Solution

Replace the existing ACTIVE_VENV_MARKER_VARS approach (which only
covered VIRTUAL_ENV/CONDA_PREFIX) with a new _strip_mismatched_site_packages
function that surgically filters PYTHONPATH:

  • Strip only paths under ~/.hermes/hermes-agent/venv/.../site-packages
  • Preserve the Hermes source root (ACTIVE_HERMES_ROOT)
  • Preserve all user-set PYTHONPATH entries

Applied to all three env builders in tools/environments/local.py:
_make_run_env, _sanitize_subprocess_env, and the PTY spawn builder.

Why not fix at the Desktop Electron level?

The root cause is in main.cjs:getVenvSitePackagesEntries() which adds
the venv's site-packages to PYTHONPATH — this is redundant because the
venv's own Python already knows its site-packages location. Fixing it
there requires rebuilding the Desktop app. The Python-side fix covers
all code paths regardless of how the environment was set up.

Verification

  • from PIL import Image no longer crashes (was ABI conflict with 3.11 _imaging)
  • httpx, cryptography load from correct 3.13 paths
  • User's own PYTHONPATH entries (/home/user/my-lib) preserved
  • VIRTUAL_ENV pointing at Hermes venv also stripped from subprocess env

…ss PYTHONPATH to prevent cross-version ABI conflicts

The Desktop Electron process injects the Hermes venv's site-packages path
(e.g. .../python3.11/site-packages) into PYTHONPATH so the Python 3.11
backend can import its packages. When this PYTHONPATH leaks into terminal
subprocesses running a different Python version (e.g. Python 3.13), 3.11
C extension modules appear on sys.path ahead of the correct 3.13 versions
and crash with ImportError (PIL _imaging, cryptography, etc.).

Replace the existing blunt pop of PYTHONPATH from _ACTIVE_VENV_MARKER_VARS
with a surgical Hermes-venv-aware filter:

- Parse each PYTHONPATH entry by path
- Strip only paths under ~/.hermes/hermes-agent/venv/.../site-packages
- Preserve the Hermes source root (needed for import hermes_cli)
- Preserve all user-set PYTHONPATH entries

The same filter is applied in all three env builders:
- _make_run_env (foreground terminal commands)
- _sanitize_subprocess_env (background/PTY spawns)
- PTY env builder

This preserves env_passthrough semantics and never silently discards the
user's own PYTHONPATH configuration.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists tool/terminal Terminal execution and process management backend/local Local shell execution labels Jul 8, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the Desktop-to-terminal environment leak. Current main still injects the Desktop venv site-packages into the backend environment at apps/desktop/electron/main.ts:1496, and the local terminal inherits os.environ through tools/environments/local.py:801, so the bug class remains relevant.

Problems

  • tools/environments/local.py:887 only detects "/site-packages". Desktop Windows builds venvRoot\\Lib\\site-packages with path.join at apps/desktop/electron/main.ts:1860, so the proposed filter leaves the Windows entry intact.
  • The VIRTUAL_ENV branch at proposed tools/environments/local.py:901 cannot run: every caller removes _ACTIVE_VENV_MARKER_VARS first (for example proposed lines 839-842).
  • The diff adds no regression tests for the three environment builders or Windows path handling.

Suggested changes

  • Compare normalized path components against the actual Hermes venv site-packages location, rather than matching a Unix-only substring.
  • Keep the existing unconditional marker stripping and remove the unreachable branch.
  • Add POSIX/Windows and all-builder coverage while verifying source-root and user PYTHONPATH entries survive.

Automated hermes-sweeper review.

if not entry:
continue
# Does this entry point at site-packages inside the Hermes venv?
if _hermes_venv and _hermes_venv in entry and "/site-packages" in entry:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Unix-only "/site-packages" check misses the Desktop Windows entry: current apps/desktop/electron/main.ts:1860 builds venvRoot\\Lib\\site-packages with path.join. Normalize/compare path components against the Hermes venv instead.

)

# --- VIRTUAL_ENV: remove if pointing at the Hermes venv ---
ve = env.get("VIRTUAL_ENV")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VIRTUAL_ENV has already been removed by the _ACTIVE_VENV_MARKER_VARS loop before this helper is called in every proposed caller, so this branch is unreachable.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 10, 2026
@Pawls

Pawls commented Jul 17, 2026

Copy link
Copy Markdown

_strip_mismatched_site_packages() looks like it's a no-op on Windows, which I think leaves the case in #57467/#65909 (both labeled platform/windows) still leaking.

The predicate tests "/site-packages" with a forward slash:

if _hermes_venv and _hermes_venv in entry and "/site-packages" in entry:

but the Windows PYTHONPATH entry the gateway injects uses backslashes, so that substring never matches. Running the predicate verbatim against the real values on my box (Hermes 0.18.2, Windows 11):

_hermes_venv               : C:\Users\<me>\AppData\Local\hermes\hermes-agent\venv
entry                      : C:\Users\<me>\AppData\Local\hermes\hermes-agent\venv\Lib\site-packages
_hermes_venv in entry      : True
"/site-packages" in entry  : False
=> stripped?               : False

The _hermes_venv in entry half matches fine (both sides come from os.path.join/the env, so both are backslashed) — it's only the hardcoded /site-packages that fails. Comparing on path components rather than a substring would cover both platforms, e.g.:

parts = Path(entry).parts
if _hermes_venv and _is_relative_to(Path(entry), Path(_hermes_venv)) and "site-packages" in parts:

Also worth noting os.path.normcase for the _hermes_venv in entry half, since Windows paths compare case-insensitively and HERMES_HOME casing isn't guaranteed to match what the launcher put on PYTHONPATH.

Second repro data point

Same leak, different C extension — a Python 3.14 child instead of 3.13:

numpy\_core\__init__.py", line 24, in <module>
    from . import multiarray
ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'

The 3.14 interpreter finds Hermes' 3.11 numpy on PYTHONPATH ahead of its own venv's, and _multiarray_umath.cp311-win_amd64.pyd won't load under 3.14. Setting the gateway's PYTHONPATH on an otherwise-healthy 3.14 venv reproduces it exactly; env -u PYTHONPATH clears it. Worth mentioning because the failure surfaces as "numpy is broken" rather than as a path-precedence problem — in my case an agent read it as a pre-existing environment fault and skipped a runtime verification step over it.

Re: #57470 as the companion fix

On this install PYTHONPATH has two independent sources, and gateway/run.py is only one of them. The gateway is started by a Scheduled Task → wscript → generated Hermes_Gateway.vbs (built by gateway_windows.py::_build_gateway_vbs_script / _resolve_detached_python), which sets it before Python starts:

env.Item("VIRTUAL_ENV") = "...\hermes-agent\venv"
env.Item("PYTHONPATH") = "...\hermes-agent;...\hermes-agent\venv\Lib\site-packages"
sh.Run "...\Python311\pythonw.exe -m hermes_cli.main gateway run", 0, False

That's deliberate — _resolve_detached_python's docstring explains uv venv launchers respawn a console python.exe and pop a visible terminal window, so it runs the base interpreter directly and rebuilds imports via PYTHONPATH. But it means removing the os.environ["PYTHONPATH"] mutation in gateway/run.py (#57470) won't by itself clean the env for scheduled-task-launched gateways. This PR's subprocess-boundary strip is what actually covers that, which is why the Windows matching bug seems worth fixing before merge.

@jeff-mettel

Copy link
Copy Markdown
Contributor

This PR addresses the defect later reported as #74817 (PYTHONPATH leaking from the gateway env into terminal-tool subprocesses, crashing cross-version compiled imports), and predates that issue by three weeks — but was not cross-referenced on it, so the issue timeline showed only the three later PRs (#74871, #74951, #78917). Posting here to create the link.

A full tabulation of the cluster (approaches, dates, the additional cron script-job surface, and the ordering fact that answers the Windows-cron propagation concern for factory-side strips) is at #78917 (comment).

Defect confirmed still live on main @ 3e6a081d: _ACTIVE_VENV_MARKER_VARS remains ("VIRTUAL_ENV", "CONDA_PREFIX") (tools/environments/local.py:349).


Filed by an AI agent (Claude Fable 5) operating autonomously on @jeff-mettel's behalf. Code references were verified against NousResearch/hermes-agent@3e6a081d before posting.

@teknium1

Copy link
Copy Markdown
Contributor

The selective PYTHONPATH filtering approach you pioneered here has been merged via #88182, with your original commit and authorship preserved in the history (cherry-picked through #78917, which fixed the Windows path-matching and venv-detection review items on top of your work). Thank you @mmchuangyt-ai — you had the right shape first: surgical removal of Hermes-owned entries while preserving user paths, which is exactly what landed.

Closing since the work is now on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/local Local shell execution P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants