Skip to content

fix(gateway): stop Windows gateway boot from adopting another Python's venv - #122805

Closed
g3org3yo wants to merge 1 commit into
NousResearch:mainfrom
g3org3yo:fix/windows-gateway-venv-python-mismatch
Closed

g3org3yo wants to merge 1 commit into
NousResearch:mainfrom
g3org3yo:fix/windows-gateway-venv-python-mismatch

Conversation

@g3org3yo

@g3org3yo g3org3yo commented Sep 25, 2026 •

Copy link
Copy Markdown

What fails

On Windows, a PM install (store interpreter + committed dependency generation) that still has the legacy pre-PM venv/ in the checkout kills the gateway's hosted-room worker minutes after boot:

ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'
  ...
  await asyncio.to_thread(self._start_hosted_room_worker_sync)
  File "...\gateway\run_startup.py", line 794, in _start_hosted_room_worker_sync
ERROR gateway.run: Supervised task hosted_room_worker died 5 times in rapid succession

pydantic imported fine (pure Python), but its compiled core came from a cp311 tree while the process ran the cp314 store interpreter. Hosted rooms stop working until the leftover tree is removed by hand. Seen on the update that moved the install to tools/python-3.14.7 while hermes-agent\venv\ (uv, 3.11) stayed behind.

Root cause

gateway/run.py::_ensure_windows_gateway_venv_imports() runs before MCP discovery and injects a venv's site-packages at sys.path[0:1] so detached runs see Hermes' packages. Its candidates are VIRTUAL_ENV then <project_root>/venv, and the legacy in-tree venv is accepted whatever interpreter built it — so its site-packages lands ahead of the generation hermes_bootstrap activation put on sys.path. Any package that exists in both trees then resolves from the old one, and its compiled extensions refuse to load.

The helper meant to catch exactly this cannot see it: pm.environments.venv_python_version() reads only pyvenv.cfg's version key, while uv venv — every Hermes-managed environment — writes version_info:

home = C:\Users\...\hermes\tools\python-3.14.7+202****0901-win32-x64
implementation = CPython
uv = 0.12.3
version_info = 3.14.7

So it returns None for all of them (on POSIX it also makes site_packages() date the tree from the caller's sys.version_info).

Reproduction (deterministic)

Store interpreter, PYTHONPATH=<checkout> (what a detached Windows gateway gets), checkout holding a leftover 3.11 venv/:

import sys, hermes_bootstrap                    # gateway boot: activates the PM generation
import gateway.run as gr
gr._ensure_windows_gateway_venv_imports()       # ...then the legacy venv is injected ahead of it
print([p for p in sys.path if "site-packages" in p])
import tui_gateway.server                       # what hosted_room_worker imports

Parent commit: site-packages lists hermes-agent\venv\Lib\site-packages first and the import raises ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'. With this PR: only the PM generation's tree is on sys.path and the import succeeds. Live check on the affected machine: hermes gateway restart → zero new pydantic_core / hosted_room_worker died entries over six minutes (before: 5 death-restarts, ~3 minutes after every boot).

Fix

  • _ensure_windows_gateway_venv_imports() prefers PM's committed environment (pm.environments.committed_venv) over VIRTUAL_ENV/<checkout>/venv, and skips any candidate whose recorded interpreter version differs from the running one — warning, naming the path, when that leaves nothing. A candidate without pyvenv.cfg keeps the historical behavior, and a legacy install with a matching venv is untouched.
  • pm.environments.venv_python_version() also reads version_info, so uv-built venvs are identifiable at all.

Tests

tests/gateway/test_windows_gateway_venv_injection.py — leftover skipped, matching venv still injected, marker-less venv unchanged, committed generation outranks the leftover.
tests/pm/test_venv_python_version.py — version_info, legacy version, missing/unparsable marker, and (POSIX) site_packages() using the venv's own version.

Both files fail on the parent commit (3 failed, 5 passed) and pass with the fix (8 passed, 1 skipped, Windows).

Full tests/pm on the machine that hit this is unchanged by the diff: 60 failed, 625 passed, 88 skipped, 102 errors before and after, with an identical failure/error ID set (pre-existing gaps in that checkout's legacy 3.11 dev venv — node/worker/network-bound tests), so nothing here regresses the suite.

…s venv

A PM install keeps one dependency generation per install (a store interpreter plus
its own environment) while the checkout may still hold the legacy pre-PM `venv/`,
built by uv for a different CPython. `_ensure_windows_gateway_venv_imports` walked
`VIRTUAL_ENV` then `<project_root>/venv` and inserted whichever site-packages it
found at sys.path[0:1] — ahead of the generation the process booted from — so on a
cp314 store interpreter `pydantic` resolved from the cp311 tree and its compiled
core could not load:

    ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'

The gateway then logged "Supervised task hosted_room_worker died 5 times in rapid
succession" and hosted rooms stopped working, on every update that moved the store
Python. Reproduced deterministically by importing `hermes_bootstrap`, calling the
helper, then importing `tui_gateway.server` with the store interpreter.

Two changes:

* `_ensure_windows_gateway_venv_imports` now prefers PM's committed environment
  (`pm.environments.committed_venv`) and skips any candidate whose recorded Python
  does not match the running interpreter, warning when that leaves nothing. A
  candidate without `pyvenv.cfg` keeps the historical behavior.
* `pm.environments.venv_python_version` also reads `version_info`, the key
  `uv venv` writes for every Hermes-managed environment. Reading only `version`
  returned None for all of them, which is what let a mismatched tree look
  acceptable in the first place (and dated POSIX site-packages from the caller).

Tests: tests/gateway/test_windows_gateway_venv_injection.py (leftover skipped,
matching venv still injected, marker-less venv unchanged, committed generation
outranks the leftover) and tests/pm/test_venv_python_version.py. Both fail on the
parent commit and pass with the fix.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/windows area/install-update Installer, updater, packaging, wheels, doctor sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows labels Sep 25, 2026
@Enough1122

Copy link
Copy Markdown

AI code review — automated review for reference; please use your judgment.

Right direction: pairing version with version_info is the correct half of the 122611 lesson, and the judgment is a parsed (major, minor), not a path/string test. But on main it does not merge green.

1. [blocker] On main, this PR's own test fails — main already fixed this bug. It landed 716efc89c0 (#122183) while the branch was open, rewriting the same function to if committed_venv(project_root) is not None: return (upstream/main:gateway/run.py:433). The edits touch adjacent lines and merge cleanly, so both survive: main's early return runs first, leaving everything this PR adds below unreachable. Merging ebf137294e into upstream/main (deterministic over two orderings): test_committed_generation_outranks_the_leftover FAILED, 1 failed / 4 passed. That test wants the committed generation on sys.path; main's contract is a PM-managed gateway gets no injection. Incompatible.

2. [blocker] except Exception around committed_venv (gateway/run.py:448-452) downgrades a load-bearing error to a silent legacy adopt. committed_venv deliberately raises for a broken selection (pm/environments.py:180,185,189,193); selected_venv:154 calls that "an error, not permission to load an older dependency set silently". Probed with a facts.json naming a generation lacking pyvenv.cfg: it raises, yet the legacy tree still lands on sys.path. Narrow the except to the import.

3. [blocker] The new tests run in no CI lane. test_windows_gateway_venv_injection.py:21 has only skipif(sys.platform != "win32", ...), but the Windows lanes list files via scripts/ci/list_os_marked_tests.py, which selects only files with mark.platforms(...) (:56): ... grep venv_injection → no match, of 168. Linux skips everything, so this Windows-only fix has no coverage. The merged sibling for the same function, test_windows_venv_import_guard.py:13, uses pytestmark = pytest.mark.platforms("windows") — the one-line fix. Reverting the key check reds 2 tests, but reversing the candidate order — the ordering the headline claims — leaves all 4 green. _fake_venv (:30) also only writes version_info, and unjudgeable trees fail open.

Relations. #122427 coexists — it never touches gateway/run.py, this never touches environments_adopt.py, so both its blockers stand. #123062 is the same function and conflicts there.

@g3org3yo g3org3yo closed this Sep 27, 2026
@g3org3yo
g3org3yo deleted the fix/windows-gateway-venv-python-mismatch branch September 27, 2026 13:19
@alt-glitch alt-glitch added os/windows Native Windows behaviour or breakage: PowerShell install, Windows paths, cmd.exe, WSL-vs-native diff and removed platform/windows labels Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/gateway Gateway runner, session dispatch, delivery os/windows Native Windows behaviour or breakage: PowerShell install, Windows paths, cmd.exe, WSL-vs-native diff P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants