From fda234a2101d02f3ffff8aee468972e22e17d03b Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 16:31:40 -0700 Subject: [PATCH 01/20] =?UTF-8?q?feat(windows):=20close=20native-Windows?= =?UTF-8?q?=20install=20gaps=20=E2=80=94=20crash-free=20startup,=20UTF-8?= =?UTF-8?q?=20stdio,=20tzdata=20dep,=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Native Windows (with Git for Windows installed) can now run the Hermes CLI and gateway end-to-end without crashing. install.ps1 already existed and the Git Bash terminal backend was already wired up — this PR fills the remaining gaps discovered by auditing every Windows-unsafe primitive (`signal.SIGKILL`, `os.kill(pid, 0)` probes, bare `fcntl`/`termios` imports) and by comparing hermes against how Claude Code, OpenCode, Codex, and Cline handle native Windows. ## What changed ### UTF-8 stdio (new module) - `hermes_cli/stdio.py` — single `configure_windows_stdio()` entry point. Flips the console code page to CP_UTF8 (65001), reconfigures `sys.stdout`/`stderr`/`stdin` to UTF-8, sets `PYTHONIOENCODING` + `PYTHONUTF8` for subprocesses. No-op on non-Windows. Opt out via `HERMES_DISABLE_WINDOWS_UTF8=1`. - Called early in `cli.py::main`, `hermes_cli/main.py::main`, and `gateway/run.py::main` so Unicode banners (box-drawing, geometric symbols, non-Latin chat text) don't `UnicodeEncodeError` on cp1252 consoles. ### Crash sites fixed - `hermes_cli/main.py:7970` (hermes update → stuck gateway sweep): raw `os.kill(pid, _signal.SIGKILL)` → `gateway.status.terminate_pid(pid, force=True)` which routes through `taskkill /T /F` on Windows. - `hermes_cli/profiles.py::_stop_gateway_process`: same fix — also converted SIGTERM path to `terminate_pid()` and widened OSError catch on the intermediate `os.kill(pid, 0)` probe. - `hermes_cli/kanban_db.py:2914, 3041`: raw `signal.SIGKILL` → `getattr(signal, "SIGKILL", signal.SIGTERM)` fallback (matches the pattern already used in `gateway/status.py`). ### OSError widening on `os.kill(pid, 0)` probes Windows raises `OSError` (WinError 87) for a gone PID instead of `ProcessLookupError`. Widened the catch at: - `gateway/run.py:15101` (`--replace` wait-for-exit loop — without this, the loop busy-spins the full 10s every Windows gateway start) - `hermes_cli/gateway.py:228, 460, 940` - `hermes_cli/profiles.py:777` - `tools/process_registry.py::_is_host_pid_alive` - `tools/browser_tool.py:1170, 1206` ### Dashboard PTY graceful degradation `hermes_cli/pty_bridge.py` depends on `fcntl`/`termios`/`ptyprocess`, none of which exist on native Windows. Previously a Windows dashboard would crash on `import hermes_cli.web_server` because of a top-level import. Now: - `hermes_cli/web_server.py` wraps the pty_bridge import in `try/except ImportError` and sets `_PTY_BRIDGE_AVAILABLE=False`. - The `/api/pty` WebSocket handler returns a friendly "use WSL2 for this tab" message instead of exploding. - Every other dashboard feature (sessions, jobs, metrics, config editor) runs natively on Windows. ### Dependency - `pyproject.toml`: add `tzdata>=2023.3; sys_platform == 'win32'` so Python's `zoneinfo` works on Windows (which has no IANA tzdata shipped with the OS). Credits @sprmn24 (PR #13182). ### Docs - README.md: removed "Native Windows is not supported"; added PowerShell one-liner and Git-for-Windows prerequisite note. - `website/docs/getting-started/installation.md`: new Windows section with capability matrix (everything native except the dashboard `/chat` PTY tab, which is WSL2-only). - `website/docs/user-guide/windows-wsl-quickstart.md`: reframed as "WSL2 as an alternative to native" rather than "the only way". - `website/docs/developer-guide/contributing.md`: updated cross-platform guidance with the `signal.SIGKILL` / `OSError` rules we enforce now. - `website/docs/user-guide/features/web-dashboard.md`: acknowledged native Windows works for everything except the embedded PTY pane. ## Why this shape Pulled from a survey of how other agent codebases handle native Windows (Claude Code, OpenCode, Codex, Cline): - All four treat Git Bash as the canonical shell on Windows, same as hermes already does in `tools/environments/local.py::_find_bash()`. - None of them force `SetConsoleOutputCP` — but they don't have to, Node/Rust write UTF-16 to the Win32 console API. Python does not get that for free, so we flip CP_UTF8 via ctypes. - None of them ship PowerShell-as-primary-shell (Claude Code exposes PS as a secondary tool; scope creep for this PR). - All of them use `taskkill /T /F` for force-kill on Windows, which is exactly what `gateway.status.terminate_pid(force=True)` does. ## Non-goals (deliberate scope limits) - No PowerShell-as-a-second-shell tool — worth designing separately. - No terminal routing rewrite (#12317, #15461, #19800 cluster) — that's the hardest design call and needs a separate doc. - No wholesale `open()` → `open(..., encoding="utf-8")` sweep (Tianworld cluster) — will do as follow-up if users hit actual breakage; most modern code already specifies it. ## Validation - 28 new tests in `tests/tools/test_windows_native_support.py` — all platform-mocked, pass on Linux CI. Cover: - `configure_windows_stdio` idempotency, opt-out, env-preservation - `terminate_pid` taskkill routing, failure → OSError, FileNotFoundError fallback - `getattr(signal, "SIGKILL", …)` fallback shape - `_is_host_pid_alive` OSError widening (Windows-gone-PID behavior) - Source-level checks that all entry points call `configure_windows_stdio` - pty_bridge import-guard present in `web_server.py` - README no longer says "not supported" - 12 pre-existing tests in `tests/tools/test_windows_compat.py` still pass. - `tests/hermes_cli/` ran fully (3909 passed, 9 failures — all confirmed pre-existing on main by stash-test). - `tests/gateway/` ran fully (5021 passed, 1 pre-existing failure). - `tests/tools/test_process_registry.py` + `test_browser_*` pass. - Manual smoke: `import hermes_cli.stdio; import gateway.run; import hermes_cli.web_server` — all clean, `_PTY_BRIDGE_AVAILABLE=True` on Linux (as expected). ## Files - New: `hermes_cli/stdio.py`, `tests/tools/test_windows_native_support.py` - Modified: `cli.py`, `gateway/run.py`, `hermes_cli/main.py`, `hermes_cli/profiles.py`, `hermes_cli/gateway.py`, `hermes_cli/kanban_db.py`, `hermes_cli/pty_bridge.py`, `hermes_cli/web_server.py`, `tools/browser_tool.py`, `tools/process_registry.py`, `pyproject.toml`, `README.md`, and 4 docs pages. Credits to everyone whose prior PR work informed these fixes — see the co-author trailers. All of the PRs listed in `~/.hermes/plans/windows-support-prs.md` fixing `os.kill` / `signal.SIGKILL` / UTF-8 stdio / tzdata / README patterns found the same issues; this PR consolidates them. Co-authored-by: Philip D'Souza <9472774+PhilipAD@users.noreply.github.com> Co-authored-by: Arecanon <42595053+ArecaNon@users.noreply.github.com> Co-authored-by: XiaoXiao0221 <263113677+XiaoXiao0221@users.noreply.github.com> Co-authored-by: Lars Hagen <1360677+lars-hagen@users.noreply.github.com> Co-authored-by: Luan Dias <65574834+luandiasrj@users.noreply.github.com> Co-authored-by: Ruzzgar Co-authored-by: sprmn24 Co-authored-by: adybag14-cyber <252811164+adybag14-cyber@users.noreply.github.com> Co-authored-by: Prasanna28Devadiga <54196612+Prasanna28Devadiga@users.noreply.github.com> --- README.md | 16 +- cli.py | 9 + gateway/run.py | 13 +- hermes_cli/gateway.py | 10 +- hermes_cli/kanban_db.py | 9 +- hermes_cli/main.py | 16 +- hermes_cli/profiles.py | 17 +- hermes_cli/pty_bridge.py | 13 +- hermes_cli/stdio.py | 134 ++++++ hermes_cli/web_server.py | 27 +- pyproject.toml | 6 + tests/tools/test_windows_native_support.py | 408 ++++++++++++++++++ tools/browser_tool.py | 9 + tools/process_registry.py | 6 +- website/docs/developer-guide/contributing.md | 12 +- website/docs/getting-started/installation.md | 29 +- .../docs/user-guide/features/web-dashboard.md | 2 +- .../docs/user-guide/windows-wsl-quickstart.md | 21 +- 18 files changed, 728 insertions(+), 29 deletions(-) create mode 100644 hermes_cli/stdio.py create mode 100644 tests/tools/test_windows_native_support.py diff --git a/README.md b/README.md index 004585826196e..be9bccdcbdc69 100644 --- a/README.md +++ b/README.md @@ -30,15 +30,27 @@ Use any model you want — [Nous Portal](https://portal.nousresearch.com), [Open ## Quick Install +### Linux, macOS, WSL2, Termux + ```bash curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash ``` -Works on Linux, macOS, WSL2, and Android via Termux. The installer handles the platform-specific setup for you. +### Windows (native, PowerShell) + +Run this in PowerShell: + +```powershell +irm https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.ps1 | iex +``` + +Native Windows requires **[Git for Windows](https://git-scm.com/download/win)** to be installed — Hermes uses its bundled Git Bash to run terminal commands, matching how Claude Code and other agents work on Windows. The installer handles everything else (uv, Python 3.11, Node.js, ripgrep, ffmpeg). + +The installer handles the platform-specific setup for you. > **Android / Termux:** The tested manual path is documented in the [Termux guide](https://hermes-agent.nousresearch.com/docs/getting-started/termux). On Termux, Hermes installs a curated `.[termux]` extra because the full `.[all]` extra currently pulls Android-incompatible voice dependencies. > -> **Windows:** Native Windows is not supported. Please install [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) and run the command above. +> **Windows:** Native Windows is supported — the PowerShell one-liner above installs everything. If you'd rather use WSL2, the Linux command works there too. Native Windows install lives under `%LOCALAPPDATA%\hermes`; WSL2 installs under `~/.hermes` as on Linux. The only Hermes feature that currently needs WSL2 specifically is the browser-based dashboard chat pane (it uses a POSIX PTY — classic CLI and gateway both run natively). After installation: diff --git a/cli.py b/cli.py index 08a9bb94ced66..0dea5f65b5cc8 100644 --- a/cli.py +++ b/cli.py @@ -12342,6 +12342,15 @@ def main( """ global _active_worktree + # Force UTF-8 stdio on Windows before any banner/print() runs — the + # Rich console prints Unicode box-drawing characters that would + # UnicodeEncodeError on cp1252. No-op on Linux/macOS. + try: + from hermes_cli.stdio import configure_windows_stdio + configure_windows_stdio() + except Exception: + pass + # Signal to terminal_tool that we're in interactive mode # This enables interactive sudo password prompts with timeout os.environ["HERMES_INTERACTIVE"] = "1" diff --git a/gateway/run.py b/gateway/run.py index 24ed6608955d4..44bd61744af34 100644 --- a/gateway/run.py +++ b/gateway/run.py @@ -15100,7 +15100,10 @@ async def start_gateway(config: Optional[GatewayConfig] = None, replace: bool = try: os.kill(existing_pid, 0) time.sleep(0.5) - except (ProcessLookupError, PermissionError): + except (ProcessLookupError, PermissionError, OSError): + # OSError covers Windows' WinError 87 "invalid parameter" + # for an already-gone PID — without this the probe loop + # busy-spins for the full 10s on every --replace start. break # Process is gone else: # Still alive after 10s — force kill @@ -15385,6 +15388,14 @@ def restart_signal_handler(): def main(): """CLI entry point for the gateway.""" + # Force UTF-8 stdio on Windows — gateway logs and startup banner would + # otherwise UnicodeEncodeError on cp1252 consoles. No-op on POSIX. + try: + from hermes_cli.stdio import configure_windows_stdio + configure_windows_stdio() + except Exception: + pass + import argparse parser = argparse.ArgumentParser(description="Hermes Gateway - Multi-platform messaging") diff --git a/hermes_cli/gateway.py b/hermes_cli/gateway.py index 5f95d0c204dd4..171ca6de38eb6 100644 --- a/hermes_cli/gateway.py +++ b/hermes_cli/gateway.py @@ -232,6 +232,10 @@ def _graceful_restart_via_sigusr1(pid: int, drain_timeout: float) -> bool: # Process still exists but we can't signal it. Treat as alive # so the caller falls back. pass + except OSError: + # Windows raises OSError (WinError 87 "invalid parameter") for + # a gone PID — treat the same as ProcessLookupError. + return True _time.sleep(0.5) # Drain didn't finish in time. return False @@ -458,6 +462,9 @@ def launch_detached_profile_gateway_restart(profile: str, old_pid: int) -> bool: break except PermissionError: pass + except OSError: + # Windows: gone PID raises OSError (WinError 87). + break time.sleep(0.2) subprocess.Popen( cmd, @@ -935,7 +942,8 @@ def stop_profile_gateway() -> bool: try: os.kill(pid, 0) _time.sleep(0.5) - except (ProcessLookupError, PermissionError): + except (ProcessLookupError, PermissionError, OSError): + # OSError covers Windows' WinError 87 for gone PIDs. break if get_running_pid() is None: diff --git a/hermes_cli/kanban_db.py b/hermes_cli/kanban_db.py index f905dd89af40d..8ce5436df8565 100644 --- a/hermes_cli/kanban_db.py +++ b/hermes_cli/kanban_db.py @@ -2911,7 +2911,10 @@ def _terminate_reclaimed_worker( if _pid_alive(pid): try: - kill(int(pid), signal.SIGKILL) + # signal.SIGKILL doesn't exist on Windows; fall back to SIGTERM + # (which maps to TerminateProcess via the stdlib shim). + _sigkill = getattr(signal, "SIGKILL", signal.SIGTERM) + kill(int(pid), _sigkill) info["sigkill"] = True except (ProcessLookupError, OSError): return info @@ -3035,7 +3038,9 @@ def enforce_max_runtime( time.sleep(0.5) if _pid_alive(pid): try: - kill(pid, signal.SIGKILL) + # signal.SIGKILL doesn't exist on Windows. + _sigkill = getattr(signal, "SIGKILL", signal.SIGTERM) + kill(pid, _sigkill) killed = True except (ProcessLookupError, OSError): pass diff --git a/hermes_cli/main.py b/hermes_cli/main.py index 70d15d4c0f2f1..695f2379ad2ed 100644 --- a/hermes_cli/main.py +++ b/hermes_cli/main.py @@ -7965,10 +7965,15 @@ def _service_restart_sec( print( f" ⚠ {len(_stuck)} gateway process(es) ignored SIGTERM — force-killing" ) + from gateway.status import terminate_pid as _terminate_pid for pid in _stuck: try: - os.kill(pid, _signal.SIGKILL) - except (ProcessLookupError, PermissionError): + # Routes through taskkill /T /F on Windows, + # SIGKILL on POSIX — _signal.SIGKILL doesn't + # exist on Windows so the old raw os.kill call + # used to crash the entire update path. + _terminate_pid(pid, force=True) + except (ProcessLookupError, PermissionError, OSError): pass # Give the OS a beat to reap the processes so the # watchers see them exit and respawn. @@ -8554,6 +8559,13 @@ def _build_provider_choices() -> list[str]: def main(): """Main entry point for hermes CLI.""" + # Force UTF-8 stdio on Windows before anything prints. No-op elsewhere. + try: + from hermes_cli.stdio import configure_windows_stdio + configure_windows_stdio() + except Exception: + pass + from hermes_cli._parser import build_top_level_parser parser, subparsers, chat_parser = build_top_level_parser() diff --git a/hermes_cli/profiles.py b/hermes_cli/profiles.py index 93928364c423a..17d99a613d1c4 100644 --- a/hermes_cli/profiles.py +++ b/hermes_cli/profiles.py @@ -758,7 +758,6 @@ def _cleanup_gateway_service(name: str, profile_dir: Path) -> None: def _stop_gateway_process(profile_dir: Path) -> None: """Stop a running gateway process via its PID file.""" - import signal as _signal import time as _time pid_file = profile_dir / "gateway.pid" @@ -769,19 +768,27 @@ def _stop_gateway_process(profile_dir: Path) -> None: raw = pid_file.read_text().strip() data = json.loads(raw) if raw.startswith("{") else {"pid": int(raw)} pid = int(data["pid"]) - os.kill(pid, _signal.SIGTERM) + # Route through terminate_pid so Windows uses the appropriate + # primitive (taskkill / TerminateProcess) — raw os.kill with + # _signal.SIGKILL raises AttributeError at import time on Windows, + # and raw os.kill with SIGTERM doesn't cascade to child processes + # the same way taskkill /T does. + from gateway.status import terminate_pid as _terminate_pid + _terminate_pid(pid) # graceful first # Wait up to 10s for graceful shutdown for _ in range(20): _time.sleep(0.5) try: os.kill(pid, 0) - except ProcessLookupError: + except (ProcessLookupError, OSError): + # OSError covers Windows' WinError 87 "invalid parameter" + # returned for an invalid/gone PID probe. print(f"✓ Gateway stopped (PID {pid})") return # Force kill try: - os.kill(pid, _signal.SIGKILL) - except ProcessLookupError: + _terminate_pid(pid, force=True) + except (ProcessLookupError, OSError): pass print(f"✓ Gateway force-stopped (PID {pid})") except (ProcessLookupError, PermissionError): diff --git a/hermes_cli/pty_bridge.py b/hermes_cli/pty_bridge.py index 66fdb4ac720b4..fcb858356bec2 100644 --- a/hermes_cli/pty_bridge.py +++ b/hermes_cli/pty_bridge.py @@ -7,11 +7,14 @@ Design constraints: -* **POSIX-only.** Hermes Agent supports Windows exclusively via WSL, which - exposes a native POSIX PTY via ``openpty(3)``. Native Windows Python - has no PTY; :class:`PtyUnavailableError` is raised with a user-readable - install/platform message so the dashboard can render a banner instead of - crashing. +* **POSIX-only.** This module depends on ``fcntl``, ``termios``, and + ``ptyprocess``, none of which exist on native Windows Python. Native + Windows ConPTY is a different API (Windows 10 build 17763+) and would + need a separate Windows implementation (``pywinpty``) — that's tracked + as a future enhancement. On native Windows, importing this module + raises :class:`ImportError` and the dashboard's ``/chat`` tab shows a + WSL-recommended banner instead of crashing. Every other feature in the + dashboard (sessions, jobs, metrics, config editor) works natively. * **Zero Node dependency on the server side.** We use :mod:`ptyprocess`, which is a pure-Python wrapper around the OS calls. The browser talks to the same ``hermes --tui`` binary it would launch from the CLI, so diff --git a/hermes_cli/stdio.py b/hermes_cli/stdio.py new file mode 100644 index 0000000000000..cfa27e2cab856 --- /dev/null +++ b/hermes_cli/stdio.py @@ -0,0 +1,134 @@ +"""Windows-safe stdio configuration. + +On Windows, Python's ``sys.stdout``/``sys.stderr`` default to the console's +active code page (often ``cp1252``, sometimes ``cp437``, occasionally ``cp932`` +on Japanese locales, etc.). Hermes's banners, tool output feed, and slash +command listings all contain Unicode: box-drawing characters (``─┌┐└┘├┤``), +mathematical and geometric symbols (``◆ ◇ ◎ ▣ ⚔ ⚖ →``), and user-supplied +text in any language. Printing those to a cp1252 console raises +``UnicodeEncodeError: 'charmap' codec can't encode character…`` and kills the +whole CLI before the REPL even opens. + +The fix is to force UTF-8 on the Python side and also flip the console's +code page to UTF-8 (65001). Both matter: Python-level only helps when +Python's stdout is a real TTY; code-page flipping lets subprocesses and +child Python ``print()`` calls agree on encoding. + +This module is a no-op on every non-Windows platform, and idempotent. +Entry points (``cli.py`` ``main``, ``hermes_cli/main.py`` CLI dispatch, +``gateway/run.py`` startup) call :func:`configure_windows_stdio` exactly +once early in startup. + +Patterns cribbed from Claude Code (``src/utils/platform.ts``), OpenCode +(``packages/opencode/src/pty/index.ts`` env injection), and OpenAI Codex +(``codex-rs/core/src/unified_exec/process_manager.rs``). None of those +actually flip the console code page — they rely on their runtime (Node or +Rust) writing UTF-16 to the Win32 console API and letting the terminal +sort it out. Python doesn't get that luxury. +""" + +from __future__ import annotations + +import os +import sys + +__all__ = ["configure_windows_stdio", "is_windows"] + + +_CONFIGURED = False + + +def is_windows() -> bool: + """Return True iff running on native Windows (not WSL).""" + return sys.platform == "win32" + + +def _flip_console_code_page_to_utf8() -> None: + """Set the attached console's input and output code pages to UTF-8. + + Uses ``SetConsoleCP`` / ``SetConsoleOutputCP`` via ``ctypes``. Failure + is silent — if there's no attached console (e.g. Hermes is running + behind a redirected stdout, under a service, or inside a PTY-less CI + runner) these calls simply return 0 and we move on. + + CP_UTF8 is 65001. + """ + try: + import ctypes + + kernel32 = ctypes.windll.kernel32 # type: ignore[attr-defined] + # Best-effort; if there's no console attached these just fail silently. + kernel32.SetConsoleCP(65001) + kernel32.SetConsoleOutputCP(65001) + except Exception: + # ctypes import, missing kernel32, or non-Windows — any failure here + # is non-fatal. We've still reconfigured Python's own streams below. + pass + + +def _reconfigure_stream(stream, *, encoding: str = "utf-8", errors: str = "replace") -> None: + """Reconfigure a text stream to UTF-8 in place. + + Uses ``TextIOWrapper.reconfigure`` (Python 3.7+). If the stream isn't + a ``TextIOWrapper`` (e.g. it's been redirected to an ``io.StringIO`` + during tests), we skip rather than blow up. + """ + try: + reconfigure = getattr(stream, "reconfigure", None) + if reconfigure is None: + return + reconfigure(encoding=encoding, errors=errors) + except Exception: + pass + + +def configure_windows_stdio() -> bool: + """Force UTF-8 stdio on Windows. No-op elsewhere. + + Idempotent — safe to call multiple times from different entry points. + + Returns ``True`` if anything was actually changed, ``False`` on + non-Windows or on a repeat call. + + Set ``HERMES_DISABLE_WINDOWS_UTF8=1`` in the environment to opt out + (for diagnosing encoding-related bugs by forcing the old cp1252 path). + """ + global _CONFIGURED + + if _CONFIGURED: + return False + if not is_windows(): + # Mark configured so repeated calls on POSIX are true no-ops. + _CONFIGURED = True + return False + + if os.environ.get("HERMES_DISABLE_WINDOWS_UTF8") in ("1", "true", "True", "yes"): + _CONFIGURED = True + return False + + # Encourage every child Python process spawned by the agent to also use + # UTF-8 for its stdio. PYTHONIOENCODING wins over the locale-based + # default in subprocesses. Don't override an explicit user setting. + os.environ.setdefault("PYTHONIOENCODING", "utf-8") + # PYTHONUTF8 = 1 enables UTF-8 Mode globally for any Python subprocess + # (PEP 540). Again, don't override an explicit setting. + os.environ.setdefault("PYTHONUTF8", "1") + + # Flip the console code page first so that any subprocess that + # inherits the console (e.g. a launched shell) also sees CP_UTF8. + _flip_console_code_page_to_utf8() + + # Reconfigure Python's own stdio wrappers so ``print()`` calls from + # this process round-trip emoji / box-drawing / non-Latin text. + # ``errors="replace"`` means a genuinely unencodable byte sequence + # gets a ``?`` rather than crashing the interpreter — we prefer + # degraded output over a stack trace. + _reconfigure_stream(sys.stdout) + _reconfigure_stream(sys.stderr) + # stdin is re-configured for completeness; Hermes's interactive + # input path uses prompt_toolkit which manages its own encoding, + # but batch/pipe input benefits from UTF-8 decoding on stdin too. + _reconfigure_stream(sys.stdin) + + _CONFIGURED = True + return True diff --git a/hermes_cli/web_server.py b/hermes_cli/web_server.py index 46786455ceab3..b3a4209b33732 100644 --- a/hermes_cli/web_server.py +++ b/hermes_cli/web_server.py @@ -2979,7 +2979,20 @@ async def get_models_analytics(days: int = 30): import re import asyncio -from hermes_cli.pty_bridge import PtyBridge, PtyUnavailableError +# PTY bridge is POSIX-only (depends on fcntl/termios/ptyprocess). On native +# Windows the import raises; catch and leave PtyBridge=None so the rest of +# the dashboard (sessions, jobs, metrics, config editor) still loads and the +# /api/pty endpoint cleanly refuses with a WSL-suggested message. +try: + from hermes_cli.pty_bridge import PtyBridge, PtyUnavailableError + _PTY_BRIDGE_AVAILABLE = True +except ImportError as _pty_import_err: # pragma: no cover - Windows-only path + PtyBridge = None # type: ignore[assignment] + _PTY_BRIDGE_AVAILABLE = False + + class PtyUnavailableError(RuntimeError): # type: ignore[no-redef] + """Stub on platforms where pty_bridge can't be imported.""" + pass _RESIZE_RE = re.compile(rb"\x1b\[RESIZE:(\d+);(\d+)\]") _PTY_READ_CHUNK_TIMEOUT = 0.2 @@ -3113,6 +3126,18 @@ async def pty_ws(ws: WebSocket) -> None: await ws.accept() + # On native Windows, the POSIX PTY bridge can't be imported. Tell the + # client and close cleanly rather than pretending the feature works. + if not _PTY_BRIDGE_AVAILABLE: + await ws.send_text( + "\r\n\x1b[31mChat unavailable: the embedded terminal requires a " + "POSIX PTY, which native Windows Python doesn't provide.\x1b[0m\r\n" + "\x1b[33mInstall Hermes inside WSL2 to use the dashboard's /chat " + "tab — the rest of the dashboard works here.\x1b[0m\r\n" + ) + await ws.close(code=1011) + return + # --- spawn PTY ------------------------------------------------------ resume = ws.query_params.get("resume") or None channel = _channel_or_close_code(ws) diff --git a/pyproject.toml b/pyproject.toml index bbc786b9801b3..bcdd5d9181909 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -36,6 +36,12 @@ dependencies = [ "edge-tts>=7.2.7,<8", # Skills Hub (GitHub App JWT auth — optional, only needed for bot identity) "PyJWT[crypto]>=2.12.0,<3", # CVE-2026-32597 + # Windows has no IANA tzdata shipped with the OS, so Python's ``zoneinfo`` + # (PEP 615) raises ``ZoneInfoNotFoundError`` for every non-UTC timezone + # out of the box. ``tzdata`` ships the Olson database as a data package + # Python resolves automatically. No-op on Linux/macOS (which have + # /usr/share/zoneinfo). Credits: PR #13182 (@sprmn24). + "tzdata>=2023.3; sys_platform == 'win32'", ] [project.optional-dependencies] diff --git a/tests/tools/test_windows_native_support.py b/tests/tools/test_windows_native_support.py new file mode 100644 index 0000000000000..1503bf11245c9 --- /dev/null +++ b/tests/tools/test_windows_native_support.py @@ -0,0 +1,408 @@ +"""Behavioral tests for Windows-specific compatibility fixes. + +Complements ``tests/tools/test_windows_compat.py`` (which does source-level +pattern linting) with cross-platform-mocked tests that exercise the actual +code paths Hermes takes on native Windows. + +Runs on Linux CI — every test mocks ``sys.platform``, ``subprocess.run``, +and ``os.kill`` as needed to simulate Windows behavior without requiring a +Windows runner. +""" + +from __future__ import annotations + +import importlib +import os +import signal +import subprocess +import sys +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + + +# --------------------------------------------------------------------------- +# configure_windows_stdio +# --------------------------------------------------------------------------- + + +class TestConfigureWindowsStdio: + """``hermes_cli.stdio.configure_windows_stdio`` wiring. + + The function must: + - be a no-op on non-Windows + - only configure once per process (idempotent) + - set PYTHONIOENCODING / PYTHONUTF8 without overriding explicit user settings + - reconfigure sys.stdout/stderr/stdin to UTF-8 on Windows + - flip the console code page to CP_UTF8 (65001) via ctypes + - respect HERMES_DISABLE_WINDOWS_UTF8 opt-out + """ + + @pytest.fixture(autouse=True) + def _reset_configured(self, monkeypatch): + """Reload the module before each test so the _CONFIGURED flag resets.""" + # Remove from sys.modules so import triggers a fresh load + sys.modules.pop("hermes_cli.stdio", None) + # Fresh import now; tests import from hermes_cli.stdio themselves, + # but this guarantees the module they get is a brand-new copy. + import hermes_cli.stdio as _s + _s._CONFIGURED = False + yield + sys.modules.pop("hermes_cli.stdio", None) + + def test_no_op_on_posix(self): + from hermes_cli import stdio + + assert stdio.is_windows() is False + result = stdio.configure_windows_stdio() + assert result is False + + def test_idempotent(self): + from hermes_cli import stdio + + stdio.configure_windows_stdio() + # Second call returns False because _CONFIGURED is set + assert stdio.configure_windows_stdio() is False + + def test_windows_path_sets_env_and_reconfigures_streams(self, monkeypatch): + from hermes_cli import stdio + + monkeypatch.setattr(stdio, "is_windows", lambda: True) + # Pretend the user has no prior setting + monkeypatch.delenv("PYTHONIOENCODING", raising=False) + monkeypatch.delenv("PYTHONUTF8", raising=False) + monkeypatch.delenv("HERMES_DISABLE_WINDOWS_UTF8", raising=False) + + reconfigure_calls = [] + + def fake_reconfigure(stream, *, encoding="utf-8", errors="replace"): + reconfigure_calls.append((stream, encoding, errors)) + + cp_calls = [] + + def fake_flip(): + cp_calls.append(True) + + monkeypatch.setattr(stdio, "_reconfigure_stream", fake_reconfigure) + monkeypatch.setattr(stdio, "_flip_console_code_page_to_utf8", fake_flip) + + result = stdio.configure_windows_stdio() + assert result is True + assert os.environ.get("PYTHONIOENCODING") == "utf-8" + assert os.environ.get("PYTHONUTF8") == "1" + assert len(cp_calls) == 1 # SetConsoleOutputCP path hit + assert len(reconfigure_calls) == 3 # stdout, stderr, stdin + + def test_respects_existing_env_var(self, monkeypatch): + """User's explicit PYTHONIOENCODING wins over our default.""" + from hermes_cli import stdio + + monkeypatch.setattr(stdio, "is_windows", lambda: True) + monkeypatch.setenv("PYTHONIOENCODING", "latin-1") + monkeypatch.setattr(stdio, "_reconfigure_stream", lambda *a, **kw: None) + monkeypatch.setattr(stdio, "_flip_console_code_page_to_utf8", lambda: None) + + stdio.configure_windows_stdio() + assert os.environ["PYTHONIOENCODING"] == "latin-1" + + @pytest.mark.parametrize("optout", ["1", "true", "True", "yes"]) + def test_disable_flag_short_circuits(self, monkeypatch, optout): + from hermes_cli import stdio + + monkeypatch.setattr(stdio, "is_windows", lambda: True) + monkeypatch.setenv("HERMES_DISABLE_WINDOWS_UTF8", optout) + + reconfigure_hit = [] + monkeypatch.setattr( + stdio, + "_reconfigure_stream", + lambda *a, **kw: reconfigure_hit.append(True), + ) + + result = stdio.configure_windows_stdio() + assert result is False + assert reconfigure_hit == [], "opt-out must skip stream reconfiguration" + + def test_reconfigure_stream_handles_missing_method(self, monkeypatch): + """StringIO-like objects without .reconfigure() must not blow up.""" + from hermes_cli import stdio + import io + + buf = io.StringIO() + # Must not raise + stdio._reconfigure_stream(buf) + + +# --------------------------------------------------------------------------- +# terminate_pid — the centralized kill primitive +# --------------------------------------------------------------------------- + + +class TestTerminatePidRoutingOnWindows: + """``gateway.status.terminate_pid`` must use taskkill /T /F on Windows. + + On Linux we can't reload gateway/status with sys.platform=win32 because + the module unconditionally imports ``msvcrt`` in that branch. Instead + we patch the module-level ``_IS_WINDOWS`` flag and ``subprocess.run`` + on the already-loaded module, which exercises the same branching code. + """ + + def test_force_uses_taskkill_on_windows(self, monkeypatch): + from gateway import status + + captured = {} + + def fake_run(args, **kwargs): + captured["args"] = args + result = MagicMock() + result.returncode = 0 + result.stderr = "" + result.stdout = "" + return result + + monkeypatch.setattr(status, "_IS_WINDOWS", True) + monkeypatch.setattr(status.subprocess, "run", fake_run) + status.terminate_pid(12345, force=True) + + assert captured["args"][0] == "taskkill" + assert "/PID" in captured["args"] + assert "12345" in captured["args"] + assert "/T" in captured["args"] + assert "/F" in captured["args"] + + def test_force_taskkill_failure_raises_oserror(self, monkeypatch): + from gateway import status + + def fake_run(args, **kwargs): + result = MagicMock() + result.returncode = 128 + result.stderr = "ERROR: The process cannot be terminated." + result.stdout = "" + return result + + monkeypatch.setattr(status, "_IS_WINDOWS", True) + monkeypatch.setattr(status.subprocess, "run", fake_run) + with pytest.raises(OSError, match="cannot be terminated"): + status.terminate_pid(12345, force=True) + + def test_graceful_on_windows_uses_os_kill_sigterm(self, monkeypatch): + """Non-force path calls os.kill with SIGTERM (Windows has no SIGKILL). + + ``terminate_pid(pid)`` with force=False bypasses the taskkill branch + and uses ``os.kill`` directly — so platform doesn't actually matter + for the signal choice. Verifies the getattr fallback works. + """ + from gateway import status + + captured = {} + + def fake_kill(pid, sig): + captured["pid"] = pid + captured["sig"] = sig + + monkeypatch.setattr(status.os, "kill", fake_kill) + status.terminate_pid(99, force=False) + + assert captured["pid"] == 99 + assert captured["sig"] == signal.SIGTERM + + def test_taskkill_not_found_falls_back_to_os_kill(self, monkeypatch): + """On Windows without taskkill (WinPE, containers), fall back gracefully.""" + from gateway import status + + captured = {} + + def fake_run(args, **kwargs): + raise FileNotFoundError(2, "taskkill not found") + + def fake_kill(pid, sig): + captured["pid"] = pid + captured["sig"] = sig + + monkeypatch.setattr(status, "_IS_WINDOWS", True) + monkeypatch.setattr(status.subprocess, "run", fake_run) + monkeypatch.setattr(status.os, "kill", fake_kill) + status.terminate_pid(42, force=True) + + assert captured["pid"] == 42 + assert captured["sig"] == signal.SIGTERM + + +# --------------------------------------------------------------------------- +# SIGKILL fallback pattern +# --------------------------------------------------------------------------- + + +class TestSigkillFallback: + """Modules that want SIGKILL must fall back to SIGTERM when absent.""" + + def test_getattr_fallback_works_when_sigkill_missing(self, monkeypatch): + """The `getattr(signal, "SIGKILL", signal.SIGTERM)` pattern.""" + # Build a stand-in signal module with no SIGKILL attribute + fake_signal = MagicMock() + del fake_signal.SIGKILL # ensure it's absent + fake_signal.SIGTERM = 15 + + result = getattr(fake_signal, "SIGKILL", fake_signal.SIGTERM) + assert result == 15 + + def test_getattr_fallback_prefers_sigkill_when_present(self): + """On POSIX the fallback is a no-op: real SIGKILL wins.""" + result = getattr(signal, "SIGKILL", signal.SIGTERM) + assert result == signal.SIGKILL + + @pytest.mark.parametrize( + "module_path, line_pattern", + [ + ("hermes_cli.kanban_db", 'getattr(signal, "SIGKILL", signal.SIGTERM)'), + ], + ) + def test_module_uses_getattr_fallback(self, module_path, line_pattern): + """Source-level check that our modules use the safe fallback.""" + rel = module_path.replace(".", "/") + ".py" + root = Path(__file__).resolve().parents[2] + source = (root / rel).read_text(encoding="utf-8") + assert line_pattern in source, ( + f"{rel} must use the getattr fallback pattern on its SIGKILL site" + ) + + +# --------------------------------------------------------------------------- +# OSError widening on os.kill(pid, 0) probes +# --------------------------------------------------------------------------- + + +class TestProcessRegistryOSErrorWidening: + """_is_host_pid_alive must treat Windows' OSError as 'not alive'.""" + + def test_oserror_treated_as_not_alive(self, monkeypatch): + from tools.process_registry import ProcessRegistry + + def fake_kill(pid, sig): + # Simulate Windows' WinError 87 for an unknown PID + raise OSError(22, "Invalid argument") + + monkeypatch.setattr("tools.process_registry.os.kill", fake_kill) + assert ProcessRegistry._is_host_pid_alive(12345) is False + + def test_permission_error_treated_as_not_alive(self, monkeypatch): + """Conservative: PermissionError also means 'not alive' (matches existing behavior).""" + from tools.process_registry import ProcessRegistry + + def fake_kill(pid, sig): + raise PermissionError(1, "Operation not permitted") + + monkeypatch.setattr("tools.process_registry.os.kill", fake_kill) + assert ProcessRegistry._is_host_pid_alive(12345) is False + + def test_zero_or_none_pid_returns_false_without_calling_kill(self, monkeypatch): + """No wasted syscall on falsy pids.""" + from tools.process_registry import ProcessRegistry + + kill_calls = [] + monkeypatch.setattr( + "tools.process_registry.os.kill", + lambda pid, sig: kill_calls.append(pid), + ) + assert ProcessRegistry._is_host_pid_alive(None) is False + assert ProcessRegistry._is_host_pid_alive(0) is False + assert kill_calls == [] + + def test_alive_pid_returns_true(self, monkeypatch): + from tools.process_registry import ProcessRegistry + + # os.kill returning None (default) means "probe succeeded → pid alive" + monkeypatch.setattr("tools.process_registry.os.kill", lambda pid, sig: None) + assert ProcessRegistry._is_host_pid_alive(os.getpid()) is True + + +# --------------------------------------------------------------------------- +# tzdata dependency +# --------------------------------------------------------------------------- + + +class TestTzdataDependencyDeclared: + """Windows installs must pull tzdata for zoneinfo to work.""" + + def test_pyproject_declares_tzdata_for_win32(self): + root = Path(__file__).resolve().parents[2] + source = (root / "pyproject.toml").read_text(encoding="utf-8") + # The dependency line should be conditional on sys_platform == 'win32' + # and should NOT be in the core dependencies for Linux/macOS. + assert ( + 'tzdata>=2023.3; sys_platform == \'win32\'' in source + or "tzdata>=2023.3; sys_platform == 'win32'" in source + or 'tzdata>=2023.3; sys_platform == "win32"' in source + ), "tzdata must be a Windows-only dep in pyproject.toml dependencies" + + +# --------------------------------------------------------------------------- +# README / docs consistency +# --------------------------------------------------------------------------- + + +class TestReadmeNoLongerSaysWindowsUnsupported: + """The README shouldn't claim native Windows isn't supported.""" + + def test_readme_does_not_say_not_supported(self): + root = Path(__file__).resolve().parents[2] + source = (root / "README.md").read_text(encoding="utf-8") + # Previous string (removed in this PR): "Native Windows is not supported" + assert "Native Windows is not supported" not in source, ( + "README.md still says native Windows is not supported — update the " + "install copy to reflect the PowerShell installer." + ) + + def test_readme_mentions_powershell_installer(self): + root = Path(__file__).resolve().parents[2] + source = (root / "README.md").read_text(encoding="utf-8") + assert "install.ps1" in source, ( + "README.md must point at scripts/install.ps1 for Windows users" + ) + + +# --------------------------------------------------------------------------- +# pty_bridge graceful import on Windows +# --------------------------------------------------------------------------- + + +class TestWebServerPtyBridgeGuard: + """The web server must not crash if pty_bridge can't import (Windows).""" + + def test_import_guard_present_in_source(self): + root = Path(__file__).resolve().parents[2] + source = (root / "hermes_cli" / "web_server.py").read_text(encoding="utf-8") + assert "_PTY_BRIDGE_AVAILABLE" in source + assert "except ImportError" in source, ( + "web_server.py must wrap the pty_bridge import in try/except ImportError" + ) + + def test_pty_handler_checks_availability_flag(self): + """The /api/pty handler must short-circuit when the bridge is unavailable.""" + root = Path(__file__).resolve().parents[2] + source = (root / "hermes_cli" / "web_server.py").read_text(encoding="utf-8") + assert "if not _PTY_BRIDGE_AVAILABLE" in source, ( + "/api/pty handler must return a friendly error when PTY is unavailable" + ) + + +# --------------------------------------------------------------------------- +# Entry points wire configure_windows_stdio +# --------------------------------------------------------------------------- + + +class TestEntryPointsConfigureStdio: + """cli.py, hermes_cli/main.py, gateway/run.py must call configure_windows_stdio.""" + + @pytest.mark.parametrize( + "relpath", + ["cli.py", "hermes_cli/main.py", "gateway/run.py"], + ) + def test_entry_point_calls_configure_stdio(self, relpath): + root = Path(__file__).resolve().parents[2] + source = (root / relpath).read_text(encoding="utf-8") + assert "configure_windows_stdio" in source, ( + f"{relpath} must call hermes_cli.stdio.configure_windows_stdio() " + "early in startup so Windows consoles render Unicode without crashing" + ) diff --git a/tools/browser_tool.py b/tools/browser_tool.py index c8cdedcf0b1fe..c7b16feb1171c 100644 --- a/tools/browser_tool.py +++ b/tools/browser_tool.py @@ -1175,6 +1175,10 @@ def _reap_orphaned_browser_sessions(): # Owner exists but we can't signal it (different uid). # Treat as alive — don't reap someone else's session. owner_alive = True + except OSError: + # Windows: gone PID raises OSError (WinError 87) instead + # of ProcessLookupError. Treat as dead to match POSIX. + owner_alive = False except (ValueError, OSError): owner_alive = None # corrupt file — fall through @@ -1211,6 +1215,11 @@ def _reap_orphaned_browser_sessions(): except PermissionError: # Alive but owned by someone else — leave it alone continue + except OSError: + # Windows raises OSError (WinError 87) for a gone PID — treat + # as dead and clean up, mirroring the ProcessLookupError branch. + shutil.rmtree(socket_dir, ignore_errors=True) + continue # Daemon is alive and its owner is dead (or legacy + untracked). Reap. try: diff --git a/tools/process_registry.py b/tools/process_registry.py index 0fc312185d164..a5fd5c8e519d3 100644 --- a/tools/process_registry.py +++ b/tools/process_registry.py @@ -407,7 +407,11 @@ def _is_host_pid_alive(pid: Optional[int]) -> bool: try: os.kill(pid, 0) return True - except (ProcessLookupError, PermissionError): + except (ProcessLookupError, PermissionError, OSError): + # OSError covers Windows' WinError 87 for a gone PID, and the + # ``WinError 5 Access denied`` case — treat both as "can't probe + # or process is gone", which matches the conservative + # "not alive" semantics callers already handle. return False def _refresh_detached_session(self, session: Optional[ProcessSession]) -> Optional[ProcessSession]: diff --git a/website/docs/developer-guide/contributing.md b/website/docs/developer-guide/contributing.md index 8cfa618ad6a09..f86262f21bd8f 100644 --- a/website/docs/developer-guide/contributing.md +++ b/website/docs/developer-guide/contributing.md @@ -95,7 +95,17 @@ pytest tests/ -v ## Cross-Platform Compatibility -Hermes officially supports Linux, macOS, and WSL2. Native Windows is **not supported**, but the codebase includes some defensive coding patterns to avoid hard crashes in edge cases. Key rules: +Hermes officially supports **Linux, macOS, WSL2, and native Windows** (via PowerShell install). Native Windows uses Git Bash (from [Git for Windows](https://git-scm.com/download/win)) for shell commands. A few features require POSIX kernel primitives and are gated: the dashboard's embedded PTY terminal pane (`/chat` tab) is WSL2-only. + +When contributing code, keep these rules in mind: + +- **Don't add unguarded `signal.SIGKILL` references.** It's not defined on Windows. Either route through `gateway.status.terminate_pid(pid, force=True)` (the centralized primitive that does `taskkill /T /F` on Windows and SIGKILL on POSIX), or fall back with `getattr(signal, "SIGKILL", signal.SIGTERM)`. +- **Catch `OSError` alongside `ProcessLookupError` on `os.kill(pid, 0)` probes.** Windows raises `OSError` (WinError 87, "parameter is incorrect") for an already-gone PID instead of `ProcessLookupError`. +- **Don't force the terminal to POSIX semantics.** `os.setsid`, `os.killpg`, `os.getpgid`, `os.fork` all raise on Windows — gate them with `if sys.platform != "win32":` or `if os.name != "nt":`. +- **Open files with an explicit `encoding="utf-8"`.** The Python default on Windows is the system locale (often cp1252), which mojibakes or crashes on non-Latin text. +- **Use `pathlib.Path` / `os.path.join` — never manually concat with `/`.** This matters less for strings the OS gives us back and more for strings we construct to hand to subprocesses. + +Key patterns: ### 1. `termios` and `fcntl` are Unix-only diff --git a/website/docs/getting-started/installation.md b/website/docs/getting-started/installation.md index 5ff5489f874c5..3ee5a3a3266e4 100644 --- a/website/docs/getting-started/installation.md +++ b/website/docs/getting-started/installation.md @@ -1,7 +1,7 @@ --- sidebar_position: 2 title: "Installation" -description: "Install Hermes Agent on Linux, macOS, WSL2, or Android via Termux" +description: "Install Hermes Agent on Linux, macOS, WSL2, native Windows, or Android via Termux" --- # Installation @@ -16,6 +16,20 @@ Get Hermes Agent up and running in under two minutes with the one-line installer curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash ``` +### Windows (native, PowerShell) + +Open PowerShell and run: + +```powershell +irm https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.ps1 | iex +``` + +The native Windows installer provisions `uv`, Python 3.11, Node.js 22, `ripgrep`, and `ffmpeg`, clones the repo under `%LOCALAPPDATA%\hermes\hermes-agent`, creates a virtualenv, and adds `hermes` to your **User PATH**. Restart your terminal (or open a new PowerShell window) after the install so PATH picks up. + +**Prerequisite:** Install [Git for Windows](https://git-scm.com/download/win) first. Hermes uses the bundled Git Bash to execute terminal commands — the same approach Claude Code and other coding agents take on Windows. If you install Git to a non-default location, set `HERMES_GIT_BASH_PATH` in your environment to point at `bash.exe`. + +If you prefer WSL2, the Linux installer above works inside it; both native and WSL installs can coexist without conflict (native data lives under `%LOCALAPPDATA%\hermes`, WSL data lives under `~/.hermes`). + ### Android / Termux Hermes now ships a Termux-aware installer path too: @@ -33,8 +47,17 @@ The installer detects Termux automatically and switches to a tested Android flow If you want the fully explicit path, follow the dedicated [Termux guide](./termux.md). -:::warning Windows -Native Windows is **not supported**. Please install [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) and run Hermes Agent from there. The install command above works inside WSL2. +:::note Windows Feature Parity + +Everything except the browser-based dashboard chat terminal runs natively on Windows: +- **CLI (`hermes chat`, `hermes setup`, `hermes gateway`, …)** — native, uses your default terminal +- **Gateway (Telegram, Discord, Slack, …)** — native, runs as a background PowerShell process +- **Cron scheduler** — native +- **Browser tool** — native (Chromium via Node.js) +- **MCP servers** — native (stdio and HTTP transports both supported) +- **Dashboard `/chat` terminal pane** — **WSL2 only** (uses a POSIX PTY; native Windows has no equivalent). The rest of the dashboard (sessions, jobs, metrics) works natively — only the embedded PTY terminal tab is gated. + +Set `HERMES_DISABLE_WINDOWS_UTF8=1` in your environment if you hit an encoding-related bug and want to fall back to the legacy cp1252 stdio path (useful for bisecting). ::: ### What the Installer Does diff --git a/website/docs/user-guide/features/web-dashboard.md b/website/docs/user-guide/features/web-dashboard.md index 5aa09b1c0572f..e796849858668 100644 --- a/website/docs/user-guide/features/web-dashboard.md +++ b/website/docs/user-guide/features/web-dashboard.md @@ -80,7 +80,7 @@ The **Chat** tab embeds the full Hermes TUI (the same interface you get from `he - Node.js (same requirement as `hermes --tui`; the TUI bundle is built on first launch) - `ptyprocess` — installed by the `pty` extra (`pip install 'hermes-agent[web,pty]'`, or `[all]` covers both) -- POSIX kernel (Linux, macOS, or WSL). Native Windows Python is not supported — use WSL. +- POSIX kernel (Linux, macOS, or WSL2). The `/chat` terminal pane specifically needs a POSIX PTY — native Windows Python has no equivalent, so on a native Windows install the rest of the dashboard (sessions, jobs, metrics, config editor) works but the `/chat` tab will show a banner telling you to use WSL2 for that feature. Close the browser tab and the PTY is reaped cleanly on the server. Re-opening spawns a fresh session. diff --git a/website/docs/user-guide/windows-wsl-quickstart.md b/website/docs/user-guide/windows-wsl-quickstart.md index e3c057d22d8da..59f6f4391aaae 100644 --- a/website/docs/user-guide/windows-wsl-quickstart.md +++ b/website/docs/user-guide/windows-wsl-quickstart.md @@ -7,7 +7,18 @@ sidebar_position: 2 # Windows (WSL2) Guide -Hermes Agent is developed and tested on **Linux** and **macOS**. Native Windows is not supported — on Windows you run Hermes inside **WSL2** (Windows Subsystem for Linux, version 2). That means there are effectively two computers in play: your Windows host, and a Linux VM managed by WSL. Most confusion comes from not being sure which one you're on at any moment. +Hermes Agent now supports **both** native Windows and WSL2. This page covers the WSL2 path; for the native PowerShell install see [Installation](../getting-started/installation.md#windows-native-powershell). + +**When to pick WSL2 over native:** +- You want to use the dashboard's embedded terminal (`/chat` tab) — that pane requires a POSIX PTY and is WSL2-only. +- You're doing POSIX-heavy development work and want your Hermes sessions to share the same filesystem / paths as your dev tools. +- You already have a WSL2 environment and don't want to maintain a second install. + +**When native is fine (or better):** +- Interactive chat, gateway (Telegram/Discord/etc.), cron scheduler, browser tool, MCP servers, and most Hermes features all run natively on Windows. +- You don't want to think about crossing the WSL↔Windows boundary every time you reference a file or open a URL. + +In WSL2 there are effectively two computers in play: your Windows host, and a Linux VM managed by WSL. Most confusion comes from not being sure which one you're on at any moment. This guide covers the parts of that split that specifically affect Hermes: installing WSL2, getting files back and forth between Windows and Linux, networking in both directions, and the pitfalls people actually hit. @@ -15,11 +26,13 @@ This guide covers the parts of that split that specifically affect Hermes: insta A Chinese-language walkthrough of the minimum install path is maintained on this same page — switch via the **language** menu (top right) and select **简体中文**. ::: -## Why WSL2 (and not "just Windows") +## Why WSL2 (vs. native Windows) + +The native Windows install runs in Windows directly: your Windows terminal (PowerShell, Windows Terminal, etc.), Windows filesystem paths (`C:\Users\…`), and Windows processes. Hermes uses Git Bash to run shell commands, which is how Claude Code and other agents handle Windows today — it sidesteps the POSIX-vs-Windows gap without a full rewrite. -Hermes assumes a POSIX environment: `fork`, `/tmp`, UNIX sockets, signal semantics, PTY-backed terminals, shells like `bash`/`zsh`, and tools like `rg`, `git`, `ffmpeg` that behave the way they do on Linux. Rewriting that for native Windows would be a full port — WSL2 gives you a real Linux kernel in a lightweight VM instead, and Hermes inside it is essentially identical to running on Ubuntu. +WSL2 runs a real Linux kernel in a lightweight VM, so Hermes inside it is essentially identical to running on Ubuntu. That's valuable when you want a real POSIX environment: `fork`, `/tmp`, UNIX sockets, signal semantics, PTY-backed terminals, shells like `bash`/`zsh`, and tools like `rg`, `git`, `ffmpeg` that behave the way they do on Linux. -Practical consequences of this choice: +Practical consequences of WSL2: - The Hermes CLI, gateway, sessions, memory, skills, and tool runtimes all live inside the Linux VM. - Windows programs (browsers, native apps, Chrome with your logged-in profile) live outside it. From 5486ad2f2ac57f3bde272ec94434d6e1c6727c8f Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 16:38:11 -0700 Subject: [PATCH 02/20] feat(windows-install): bundle portable MinGit instead of relying on winget MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User hit a real failure case: their system Git was in a half-installed state (can neither uninstall nor reinstall) and winget refused to work around it. We were one step away from shipping an installer that would have left users with exactly the problem he already had. What other agents do (reality check): - Claude Code: requires pre-installed Git; breaks if user doesn't have it. - OpenCode, Codex: don't need bash at all — PowerShell-first design. - Cline: uses whatever shell VSCode is configured with; installs nothing. None of them solve the "broken system Git" problem. We need to own our Git. Changes: - scripts/install.ps1::Install-Git: dropped winget path entirely. Now: (1) use existing git if present; (2) download portable MinGit from the official git-for-windows GitHub release to %LOCALAPPDATA%\hermes\git. No winget, no admin, no Windows installer registry, no system impact. - Added %LOCALAPPDATA%\hermes\git\{cmd,usr\bin} to User PATH so git + bash + POSIX coreutils (which, env, grep, …) resolve in fresh shells. - tools/environments/local.py::_find_bash: reorder so Hermes' portable MinGit install is checked BEFORE falling through to shutil.which("bash") or system install locations. This way a broken system Git can't hijack the bash lookup. - README + installation docs reworded to reflect the new story: "portable Git Bash, isolated from any system install, recoverable via rm -rf if it ever breaks." Recoverability: if Hermes' Git install ever breaks, ``Remove-Item %LOCALAPPDATA%\hermes\git`` and re-run the installer — no system impact, no uninstall drama, no winget to fight with. --- README.md | 4 +- scripts/install.ps1 | 164 ++++++++++++++++++- tools/environments/local.py | 16 +- website/docs/getting-started/installation.md | 10 +- 4 files changed, 181 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index be9bccdcbdc69..ead2e44917b08 100644 --- a/README.md +++ b/README.md @@ -44,9 +44,9 @@ Run this in PowerShell: irm https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.ps1 | iex ``` -Native Windows requires **[Git for Windows](https://git-scm.com/download/win)** to be installed — Hermes uses its bundled Git Bash to run terminal commands, matching how Claude Code and other agents work on Windows. The installer handles everything else (uv, Python 3.11, Node.js, ripgrep, ffmpeg). +The installer handles everything: uv, Python 3.11, Node.js, ripgrep, ffmpeg, **and a portable Git Bash** (MinGit, unpacked to `%LOCALAPPDATA%\hermes\git` — no admin required, completely isolated from any system Git install). Hermes uses this bundled Git Bash to run shell commands. -The installer handles the platform-specific setup for you. +If you already have Git installed, the installer detects it and uses that instead. Otherwise a ~45MB MinGit download is all you need — it won't touch or interfere with any system Git. > **Android / Termux:** The tested manual path is documented in the [Termux guide](https://hermes-agent.nousresearch.com/docs/getting-started/termux). On Termux, Hermes installs a curated `.[termux]` extra because the full `.[all]` extra currently pulls Android-incompatible voice dependencies. > diff --git a/scripts/install.ps1 b/scripts/install.ps1 index 144113d5a0f0f..a6154282c6df5 100644 --- a/scripts/install.ps1 +++ b/scripts/install.ps1 @@ -191,19 +191,167 @@ function Test-Python { return $false } -function Test-Git { +function Install-Git { + <# + .SYNOPSIS + Ensure Git (and Git Bash) are installed. Git for Windows bundles bash.exe + which Hermes uses to run shell commands. + + Priority order (deliberately simple — no winget, no registry, no system + package manager): + 1. Existing ``git`` on PATH — use it as-is (the common fast path). + 2. Download portable **MinGit** from the official git-for-windows GitHub + release and unpack it to ``%LOCALAPPDATA%\hermes\git`` — never touches + system Git, never requires admin, works even on locked-down machines + and machines with a broken system Git install. + + We deliberately skip winget because it fails badly when the system Git + install is in a half-installed state (partially registered, or uninstall- + blocked). Owning the Hermes copy of Git ourselves is predictable and + recoverable: if it ever breaks, ``Remove-Item %LOCALAPPDATA%\hermes\git`` + and re-running this installer fully recovers. + + After install we locate ``bash.exe`` and persist the path in + ``HERMES_GIT_BASH_PATH`` (User scope) so Hermes can find it in a fresh + shell without a second PATH refresh. + #> Write-Info "Checking Git..." - + if (Get-Command git -ErrorAction SilentlyContinue) { $version = git --version Write-Success "Git found ($version)" + Set-GitBashEnvVar return $true } - - Write-Err "Git not found" - Write-Info "Please install Git from:" - Write-Info " https://git-scm.com/download/win" - return $false + + # Download portable MinGit into $HermesHome\git. This always works as + # long as we can reach github.com — no admin, no winget, no reliance on + # the user's possibly-broken system Git install. + Write-Info "Git not found — downloading portable MinGit to $HermesHome\git\ ..." + Write-Info "(no admin rights required; isolated from any system Git install)" + + try { + $arch = if ([Environment]::Is64BitOperatingSystem) { "64-bit" } else { "32-bit" } + # Query the GitHub API for the latest git-for-windows release and pick + # the MinGit zip for our architecture. + $releaseApi = "https://api.github.com/repos/git-for-windows/git/releases/latest" + $release = Invoke-RestMethod -Uri $releaseApi -UseBasicParsing -Headers @{ "User-Agent" = "hermes-installer" } + $assetPattern = if ($arch -eq "64-bit") { "MinGit-*-64-bit.zip" } else { "MinGit-*-32-bit.zip" } + # Prefer the non-busybox MinGit — it ships the real bash.exe which is + # what Hermes' _find_bash() looks for. Busybox MinGit replaces bash + # with busybox.exe (ash), which Hermes is NOT tested against. + $asset = $release.assets | Where-Object { $_.name -like $assetPattern -and $_.name -notlike "*busybox*" } | Select-Object -First 1 + + if (-not $asset) { + throw "Could not find MinGit zip in latest git-for-windows release" + } + + $downloadUrl = $asset.browser_download_url + $tmpZip = "$env:TEMP\$($asset.name)" + $gitDir = "$HermesHome\git" + + Write-Info "Downloading $($asset.name) ($([math]::Round($asset.size / 1MB, 1)) MB)..." + Invoke-WebRequest -Uri $downloadUrl -OutFile $tmpZip -UseBasicParsing + + if (Test-Path $gitDir) { + Write-Info "Removing previous MinGit install at $gitDir ..." + Remove-Item -Recurse -Force $gitDir + } + New-Item -ItemType Directory -Path $gitDir -Force | Out-Null + Expand-Archive -Path $tmpZip -DestinationPath $gitDir -Force + Remove-Item -Force $tmpZip -ErrorAction SilentlyContinue + + # MinGit layout: cmd\git.exe + mingw64\bin\git.exe + usr\bin\bash.exe. + # (Note: MinGit puts bash under usr\bin, NOT bin like the full + # Git for Windows installer does. Our _find_bash() in + # tools/environments/local.py checks both locations.) + $gitExe = "$gitDir\cmd\git.exe" + if (-not (Test-Path $gitExe)) { + throw "MinGit extraction did not produce git.exe at $gitExe" + } + + # Add cmd\ to session PATH so the rest of this install run (which + # needs git clone) can see the new git.exe. + $env:Path = "$gitDir\cmd;$env:Path" + + # Also add to persisted User PATH so fresh shells see it. We add + # cmd\ (for git) and usr\bin\ (for bash + coreutils) — without usr\bin, + # bash-launched commands like `which`, `env`, `grep` etc. that Hermes + # tools call would be missing. + $newPathEntries = @("$gitDir\cmd", "$gitDir\usr\bin") + $userPath = [Environment]::GetEnvironmentVariable("Path", "User") + $userPathItems = if ($userPath) { $userPath -split ";" } else { @() } + $changed = $false + foreach ($entry in $newPathEntries) { + if ($userPathItems -notcontains $entry) { + $userPathItems += $entry + $changed = $true + } + } + if ($changed) { + [Environment]::SetEnvironmentVariable("Path", ($userPathItems -join ";"), "User") + } + + $version = & $gitExe --version + Write-Success "Git $version installed to $gitDir (portable, user-scoped)" + Set-GitBashEnvVar + return $true + } catch { + Write-Err "Could not install MinGit portable: $_" + Write-Info "" + Write-Info "Fallback: install Git manually from https://git-scm.com/download/win" + Write-Info "then re-run this installer. Hermes needs Git Bash on Windows to run" + Write-Info "shell commands (same as Claude Code and other coding agents)." + return $false + } +} + +function Set-GitBashEnvVar { + <# + .SYNOPSIS + Locate ``bash.exe`` from an already-installed Git and persist the path in + ``HERMES_GIT_BASH_PATH`` (User env scope) so Hermes can find it even before + PATH propagation completes in a newly-spawned shell. + #> + $candidates = @() + + # Our own portable MinGit install is ALWAYS checked first, so a broken + # system Git doesn't hijack us. If the user had a working system Git + # we'd have returned early from Install-Git's fast path and never called + # this with a system-Git-only installation anyway. + $candidates += "$HermesHome\git\usr\bin\bash.exe" # MinGit layout + $candidates += "$HermesHome\git\bin\bash.exe" # safety — non-MinGit portable layouts + + # git.exe on PATH can tell us where the install root is + $gitCmd = Get-Command git -ErrorAction SilentlyContinue + if ($gitCmd) { + $gitExe = $gitCmd.Source + # Git for Windows (full installer): \cmd\git.exe + \bin\bash.exe + # MinGit: \cmd\git.exe + \usr\bin\bash.exe + $gitRoot = Split-Path (Split-Path $gitExe -Parent) -Parent + $candidates += "$gitRoot\bin\bash.exe" + $candidates += "$gitRoot\usr\bin\bash.exe" + } + + # Standard system install locations as a final fallback. Note: + # ProgramFiles(x86) can't be referenced via ${env:...} string interpolation + # because of the parens — use [Environment]::GetEnvironmentVariable(). + $candidates += "${env:ProgramFiles}\Git\bin\bash.exe" + $pf86 = [Environment]::GetEnvironmentVariable("ProgramFiles(x86)") + if ($pf86) { $candidates += "$pf86\Git\bin\bash.exe" } + $candidates += "${env:LocalAppData}\Programs\Git\bin\bash.exe" + + foreach ($candidate in $candidates) { + if ($candidate -and (Test-Path $candidate)) { + [Environment]::SetEnvironmentVariable("HERMES_GIT_BASH_PATH", $candidate, "User") + $env:HERMES_GIT_BASH_PATH = $candidate + Write-Info "Set HERMES_GIT_BASH_PATH=$candidate" + return + } + } + + Write-Warn "Could not locate bash.exe — Hermes may not find Git Bash." + Write-Info "If needed, set HERMES_GIT_BASH_PATH manually to your bash.exe path." } function Test-Node { @@ -889,7 +1037,7 @@ function Main { if (-not (Install-Uv)) { throw "uv installation failed — cannot continue" } if (-not (Test-Python)) { throw "Python $PythonVersion not available — cannot continue" } - if (-not (Test-Git)) { throw "Git not found — install from https://git-scm.com/download/win" } + if (-not (Install-Git)) { throw "Git not available and auto-install failed — install from https://git-scm.com/download/win then re-run" } Test-Node # Auto-installs if missing Install-SystemPackages # ripgrep + ffmpeg in one step diff --git a/tools/environments/local.py b/tools/environments/local.py index f9094ee5b7909..0ee851dce17bd 100644 --- a/tools/environments/local.py +++ b/tools/environments/local.py @@ -189,6 +189,20 @@ def _find_bash() -> str: if custom and os.path.isfile(custom): return custom + # Prefer our own portable MinGit install first — this way a broken or + # partially-uninstalled system Git can't hijack the bash lookup. The + # install.ps1 installer always drops MinGit here when the user didn't + # already have a working system Git. + _local_appdata = os.environ.get("LOCALAPPDATA", "") + _hermes_portable_git = os.path.join(_local_appdata, "hermes", "git") if _local_appdata else "" + if _hermes_portable_git: + for candidate in ( + os.path.join(_hermes_portable_git, "usr", "bin", "bash.exe"), # MinGit layout + os.path.join(_hermes_portable_git, "bin", "bash.exe"), # non-MinGit portable + ): + if os.path.isfile(candidate): + return candidate + found = shutil.which("bash") if found: return found @@ -196,7 +210,7 @@ def _find_bash() -> str: for candidate in ( os.path.join(os.environ.get("ProgramFiles", r"C:\Program Files"), "Git", "bin", "bash.exe"), os.path.join(os.environ.get("ProgramFiles(x86)", r"C:\Program Files (x86)"), "Git", "bin", "bash.exe"), - os.path.join(os.environ.get("LOCALAPPDATA", ""), "Programs", "Git", "bin", "bash.exe"), + os.path.join(_local_appdata, "Programs", "Git", "bin", "bash.exe"), ): if candidate and os.path.isfile(candidate): return candidate diff --git a/website/docs/getting-started/installation.md b/website/docs/getting-started/installation.md index 3ee5a3a3266e4..6c56a4d8899b8 100644 --- a/website/docs/getting-started/installation.md +++ b/website/docs/getting-started/installation.md @@ -24,9 +24,15 @@ Open PowerShell and run: irm https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.ps1 | iex ``` -The native Windows installer provisions `uv`, Python 3.11, Node.js 22, `ripgrep`, and `ffmpeg`, clones the repo under `%LOCALAPPDATA%\hermes\hermes-agent`, creates a virtualenv, and adds `hermes` to your **User PATH**. Restart your terminal (or open a new PowerShell window) after the install so PATH picks up. +The installer handles **everything**: `uv`, Python 3.11, Node.js 22, `ripgrep`, `ffmpeg`, **and a portable Git Bash** (MinGit — a slim, self-contained Git for Windows distribution that Hermes uses for shell commands). It clones the repo under `%LOCALAPPDATA%\hermes\hermes-agent`, creates a virtualenv, and adds `hermes` to your **User PATH**. Restart your terminal (or open a new PowerShell window) after the install so PATH picks up. -**Prerequisite:** Install [Git for Windows](https://git-scm.com/download/win) first. Hermes uses the bundled Git Bash to execute terminal commands — the same approach Claude Code and other coding agents take on Windows. If you install Git to a non-default location, set `HERMES_GIT_BASH_PATH` in your environment to point at `bash.exe`. +**How Git is handled:** +1. If `git` is already on your PATH, the installer uses your existing install. +2. Otherwise it downloads portable **MinGit** (~45MB, from the official `git-for-windows` GitHub release) and unpacks it to `%LOCALAPPDATA%\hermes\git`. No admin rights required. Completely isolated — it won't interfere with any system Git install, broken or otherwise. + +**Why not use winget?** Earlier designs auto-installed Git via `winget install Git.Git`, but winget fails badly when a system Git install is in a partial or broken state (exactly when users need the installer to just work). The portable MinGit approach sidesteps winget, the Windows installer registry, and any existing system Git entirely. If the Hermes Git install itself ever breaks, `Remove-Item %LOCALAPPDATA%\hermes\git` and re-run the installer — no system impact, no uninstall drama. + +The installer also sets `HERMES_GIT_BASH_PATH` to the located `bash.exe` so Hermes resolves it deterministically in fresh shells. If you prefer WSL2, the Linux installer above works inside it; both native and WSL installs can coexist without conflict (native data lives under `%LOCALAPPDATA%\hermes`, WSL data lives under `~/.hermes`). From 1da89528e7f67a20d02fccfcd8b34a6b3fe8456f Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 16:46:37 -0700 Subject: [PATCH 03/20] fix(windows-editor): default EDITOR=notepad so /edit and Ctrl+X Ctrl+E work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-existing Windows bug surfaced while reviewing the portable-MinGit install: prompt_toolkit's Buffer.open_in_editor() falls back to POSIX absolute paths (/usr/bin/nano, /usr/bin/vi, /usr/bin/emacs) that don't exist on native Windows. When neither $EDITOR nor $VISUAL is set, Ctrl+X Ctrl+E ("open prompt in editor") and /edit both silently do nothing on Windows — the user hits the key, nothing happens, no error. This wasn't caused by MinGit (full Git for Windows doesn't fix it either, because the Windows Python subprocess call resolves `/usr/bin/nano` as `C:\usr\bin\nano`, which doesn't exist even with nano installed). Fixes: - hermes_cli/stdio.py::configure_windows_stdio now sets EDITOR=notepad on Windows if neither EDITOR nor VISUAL is set. notepad.exe is in every Windows install, works as a blocking editor (subprocess.call waits for the window to close), and writes back to the file. - hermes_cli/config.py (hermes config edit): reorder fallback list so Windows tries notepad first — previously nano led the list, which required Git Bash / WSL to be in PATH. - Users who want VSCode / Neovim / Notepad++ can still override via $env:EDITOR — that's checked before our default kicks in. Docstring spells out the common overrides. The Ink TUI (`hermes --tui`) already handled Windows correctly via ui-tui/src/lib/editor.ts falling back to notepad.exe on win32 — this commit brings the classic prompt_toolkit CLI into parity. 3 new tests in test_windows_native_support.py verify: - EDITOR=notepad gets set when unset on Windows - Explicit $EDITOR is respected - $VISUAL is respected (not overwritten by our default) --- hermes_cli/config.py | 16 ++++++-- hermes_cli/stdio.py | 46 ++++++++++++++++++++++ tests/tools/test_windows_native_support.py | 39 ++++++++++++++++++ 3 files changed, 97 insertions(+), 4 deletions(-) diff --git a/hermes_cli/config.py b/hermes_cli/config.py index 1e040c3685ba7..f38a795191a77 100644 --- a/hermes_cli/config.py +++ b/hermes_cli/config.py @@ -4696,11 +4696,19 @@ def edit_config(): # Find editor editor = os.getenv('EDITOR') or os.getenv('VISUAL') - + if not editor: - # Try common editors - for cmd in ['nano', 'vim', 'vi', 'code', 'notepad']: - import shutil + # Try common editors — order is platform-aware so Windows users + # land on a working editor (notepad) even without Git Bash or nano + # installed. On POSIX, prefer nano/vim over code/notepad because + # it's more likely to be present on headless / server systems. + import shutil + import sys as _sys + if _sys.platform == "win32": + candidates = ['notepad', 'code', 'vim', 'vi', 'nano'] + else: + candidates = ['nano', 'vim', 'vi', 'code', 'notepad'] + for cmd in candidates: if shutil.which(cmd): editor = cmd break diff --git a/hermes_cli/stdio.py b/hermes_cli/stdio.py index cfa27e2cab856..e010304d989de 100644 --- a/hermes_cli/stdio.py +++ b/hermes_cli/stdio.py @@ -92,6 +92,9 @@ def configure_windows_stdio() -> bool: Set ``HERMES_DISABLE_WINDOWS_UTF8=1`` in the environment to opt out (for diagnosing encoding-related bugs by forcing the old cp1252 path). + + Also sets a sensible default ``EDITOR`` on Windows if none is already + set — see :func:`_default_windows_editor`. """ global _CONFIGURED @@ -114,6 +117,16 @@ def configure_windows_stdio() -> bool: # (PEP 540). Again, don't override an explicit setting. os.environ.setdefault("PYTHONUTF8", "1") + # Set EDITOR to a working Windows default if neither EDITOR nor VISUAL + # is set. prompt_toolkit's ``open_in_editor`` falls back to POSIX-only + # paths (``/usr/bin/nano``, ``/usr/bin/vi``) that don't exist on + # Windows — Ctrl+X Ctrl+E and ``/edit`` silently do nothing there + # otherwise. This happens even with full Git for Windows installed, + # so it's not a MinGit-specific issue. + _default_editor = _default_windows_editor() + if _default_editor and not os.environ.get("EDITOR") and not os.environ.get("VISUAL"): + os.environ["EDITOR"] = _default_editor + # Flip the console code page first so that any subprocess that # inherits the console (e.g. a launched shell) also sees CP_UTF8. _flip_console_code_page_to_utf8() @@ -132,3 +145,36 @@ def configure_windows_stdio() -> bool: _CONFIGURED = True return True + + +def _default_windows_editor() -> str: + """Return a Windows-appropriate default for ``$EDITOR``. + + Priority order, first match wins: + + 1. ``notepad`` — ships with every Windows install, no deps, works as a + blocking editor (``subprocess.call(["notepad", file])`` blocks until + the user closes the window). This is the "always-works" default. + + The prompt_toolkit buffer's ``open_in_editor`` and Hermes's + ``hermes config edit`` both honour ``$EDITOR``. Users who prefer a + different editor can override: + + - VSCode: ``$env:EDITOR = "code --wait"`` (``--wait`` is critical; + without it the editor returns immediately and any input is lost) + - Notepad++: ``$env:EDITOR = "'C:\\Program Files\\Notepad++\\notepad++.exe' -multiInst -nosession"`` + - Neovim: ``$env:EDITOR = "nvim"`` (if installed) + + Set this before launching Hermes (User env var in Windows Settings, or + export in a PowerShell profile) and Hermes picks it up automatically. + """ + import shutil + + # notepad.exe is always in %SystemRoot%\System32 on Windows, so shutil.which + # will reliably find it. Return the bare name so prompt_toolkit's shlex + # split doesn't trip over a path containing spaces. + if shutil.which("notepad"): + return "notepad" + # On the extreme off-chance notepad is missing (WinPE, Nano Server), fall + # back to nothing and let prompt_toolkit's silent no-op do its thing. + return "" diff --git a/tests/tools/test_windows_native_support.py b/tests/tools/test_windows_native_support.py index 1503bf11245c9..fa1a0ed8e3ff4 100644 --- a/tests/tools/test_windows_native_support.py +++ b/tests/tools/test_windows_native_support.py @@ -73,6 +73,8 @@ def test_windows_path_sets_env_and_reconfigures_streams(self, monkeypatch): monkeypatch.delenv("PYTHONIOENCODING", raising=False) monkeypatch.delenv("PYTHONUTF8", raising=False) monkeypatch.delenv("HERMES_DISABLE_WINDOWS_UTF8", raising=False) + monkeypatch.delenv("EDITOR", raising=False) + monkeypatch.delenv("VISUAL", raising=False) reconfigure_calls = [] @@ -86,14 +88,51 @@ def fake_flip(): monkeypatch.setattr(stdio, "_reconfigure_stream", fake_reconfigure) monkeypatch.setattr(stdio, "_flip_console_code_page_to_utf8", fake_flip) + # Pretend notepad.exe is on PATH (it always is on real Windows hosts, + # but not on the Linux CI runner — mock it so the editor default + # survives). + monkeypatch.setattr(stdio, "_default_windows_editor", lambda: "notepad") result = stdio.configure_windows_stdio() assert result is True assert os.environ.get("PYTHONIOENCODING") == "utf-8" assert os.environ.get("PYTHONUTF8") == "1" + # EDITOR must be set so prompt_toolkit's open_in_editor finds + # a working program on Windows (it defaults to /usr/bin/nano). + assert os.environ.get("EDITOR") == "notepad" assert len(cp_calls) == 1 # SetConsoleOutputCP path hit assert len(reconfigure_calls) == 3 # stdout, stderr, stdin + def test_respects_existing_editor_var(self, monkeypatch): + """User's explicit EDITOR wins over our default.""" + from hermes_cli import stdio + + monkeypatch.setattr(stdio, "is_windows", lambda: True) + monkeypatch.setenv("EDITOR", "code --wait") + monkeypatch.setattr(stdio, "_reconfigure_stream", lambda *a, **kw: None) + monkeypatch.setattr(stdio, "_flip_console_code_page_to_utf8", lambda: None) + monkeypatch.setattr(stdio, "_default_windows_editor", lambda: "notepad") + + stdio.configure_windows_stdio() + assert os.environ["EDITOR"] == "code --wait" + + def test_respects_existing_visual_var(self, monkeypatch): + """VISUAL takes precedence over our EDITOR default too.""" + from hermes_cli import stdio + + monkeypatch.setattr(stdio, "is_windows", lambda: True) + monkeypatch.delenv("EDITOR", raising=False) + monkeypatch.setenv("VISUAL", "nvim") + monkeypatch.setattr(stdio, "_reconfigure_stream", lambda *a, **kw: None) + monkeypatch.setattr(stdio, "_flip_console_code_page_to_utf8", lambda: None) + monkeypatch.setattr(stdio, "_default_windows_editor", lambda: "notepad") + + stdio.configure_windows_stdio() + # EDITOR should NOT be set when VISUAL already is (prompt_toolkit + # checks VISUAL first anyway, but we also shouldn't override it). + assert os.environ.get("EDITOR", "") != "notepad" + assert os.environ["VISUAL"] == "nvim" + def test_respects_existing_env_var(self, monkeypatch): """User's explicit PYTHONIOENCODING wins over our default.""" from hermes_cli import stdio From eeb723fff24679ca21ce7a5576aa9a54373a65bc Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 17:29:31 -0700 Subject: [PATCH 04/20] =?UTF-8?q?feat(windows):=20close=20remaining=20POSI?= =?UTF-8?q?X-only=20landmines=20=E2=80=94=20TUI=20crash,=20kanban=20waitpi?= =?UTF-8?q?d,=20AF=5FUNIX=20sandbox,=20/bin/bash,=20npm=20.cmd=20shims,=20?= =?UTF-8?q?cwd=20tracking,=20detach=20flags?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second pass on native Windows support, driven by a systematic audit across five areas: POSIX-only primitives (signal.SIGKILL/SIGHUP/SIGPIPE, os.WNOHANG, os.setsid), path translation bugs (/c/Users → C:\Users), subprocess patterns (npm.cmd batch shims, start_new_session no-op on Windows), subsystem health (cron, gateway daemon, update flow), and module-level import guards. Every change is platform-gated — POSIX (Linux/macOS) behaviour is preserved bit-identical. Explicit "do no harm" test: test_posix_path_preserved_on_linux, test_posix_noop, test_windows_detach_popen_kwargs_is_posix_equivalent_on_posix. ## New module - hermes_cli/_subprocess_compat.py — shared helpers (resolve_node_command, windows_detach_flags, windows_hide_flags, windows_detach_popen_kwargs). All no-ops on non-Windows. ## CRITICAL fixes (would crash or silently break on Windows) - tui_gateway/entry.py: SIGPIPE/SIGHUP referenced at module top level would AttributeError on import on Windows, breaking `hermes --tui` entirely (it spawns this module as a subprocess). Guard each signal.signal() call with hasattr() and add SIGBREAK as Windows' SIGHUP equivalent. - hermes_cli/kanban_db.py: os.waitpid(-1, os.WNOHANG) in dispatcher tick was unguarded. os.WNOHANG doesn't exist on Windows. Gate the whole reap loop behind `os.name != "nt"` — Windows has no zombies anyway. - tools/code_execution_tool.py: AF_UNIX socket for execute_code RPC fails on most Windows builds. Fall back to loopback TCP (AF_INET on 127.0.0.1:0 ephemeral port) when _IS_WINDOWS. HERMES_RPC_SOCKET env var now accepts either a filesystem path (POSIX) or `tcp://127.0.0.1:` (Windows). Generated sandbox client parses both. - cron/scheduler.py: `argv = ["/bin/bash", str(path)]` hardcoded. Use shutil.which("bash") so Windows (Git Bash via MinGit) works, with a readable error when bash is genuinely absent. - 6 bare npm/npx spawn sites: tools_config.py x2, doctor.py, whatsapp.py (npm install + node version probe), browser_tool.py x2. On Windows npm is npm.cmd / npx is npx.cmd (batch shims); subprocess.Popen(["npm", ...]) fails with WinError 193. shutil.which(...) returns the absolute .cmd path which CreateProcessW accepts because the extension routes through cmd.exe /c. POSIX behaviour unchanged (shutil.which still returns the same path subprocess would resolve itself). ## HIGH fixes (silent misbehaviour on Windows) - tools/environments/local.py get_temp_dir: hardcoded /tmp returned on Windows meant `_cwd_file = "/tmp/hermes-cwd-*.txt"`, which bash wrote via MSYS2's virtual /tmp but native Python couldn't open. Result: cwd tracking silently broken — `cd` in terminal tool did nothing. Windows branch now returns `%HERMES_HOME%/cache/terminal` with forward slashes (works in both bash and Python, guaranteed no spaces). - tools/environments/local.py _make_run_env PATH injection: `/usr/bin not in split(":")` heuristic mangles Windows PATH (";" separator). Gate the injection behind `not _IS_WINDOWS`. - hermes_cli/gateway.py launch_detached_profile_gateway_restart: outer Popen + watcher-script Popen both used start_new_session=True, which Windows silently ignores. Watcher stayed attached to CLI's console, died when user closed terminal after `hermes update`, left gateway stale. Now branches through windows_detach_popen_kwargs() helper (CREATE_NEW_PROCESS_GROUP | DETACHED_PROCESS | CREATE_NO_WINDOW on Windows, start_new_session=True on POSIX — identical to main). ## MEDIUM fixes - gateway/run.py /restart and /update handlers: hardcoded bash/setsid chain crashes on Windows when user triggers /update in-gateway. Now has sys.platform=="win32" branch using sys.executable + a tiny Python watcher with proper detach flags. POSIX path is unchanged. - cli.py _git_repo_root: Git on Windows sometimes returns /c/Users/... style paths that break subprocess.Popen(cwd=...) and Path().resolve(). Added _normalize_git_bash_path() helper that translates /c/Users, /cygdrive/c, /mnt/c variants to native C:\Users form. POSIX no-op. _git_repo_root() now routes every result through it. - cli.py worktree .worktreeinclude: os.symlink on directories failed hard on Windows (requires admin or Developer Mode). Falls back to shutil.copytree with a warning log. ## Tests - 29 new tests in tests/tools/test_windows_native_support.py covering: subprocess_compat helpers, TUI entry signal guards, kanban waitpid guard, code_execution TCP fallback source-level invariants, cron bash resolution, npm/npx bare-spawn lint per-file, local env Windows temp dir, PATH injection gating, git bash path normalization, symlink fallback, gateway detached watcher flags. - One existing test assertion adjusted in test_browser_homebrew_paths: it compared captured Popen argv to the BARE `"npx"` literal; after the shutil.which() change argv[0] is the absolute path. New assertion checks the shape (two items, second is `agent-browser`) rather than the exact first-item string. Behaviour unchanged; test was too strict. All 56 tests pass on Linux (30 from previous commits + 26 new). 267 tests from the affected files/dirs (browser, code_exec, local_env, process_registry, kanban_db, windows_compat) all pass — zero regressions. tests/hermes_cli/ (3909 pass) and tests/gateway/ (5021 pass) unchanged; all pre-existing test failures confirmed unrelated via `git stash` re-run. ## What's still deferred (LOW priority) - Visible cmd-window flashes on short-lived console apps (~14 sites) — cosmetic, needs a follow-up pass once we have user reports. - agent/file_safety.py POSIX-only security deny patterns — separate hardening task. - tools/process_registry.py returning "/tmp" as fallback — theoretical; reachable only when all env-var candidates fail. --- cli.py | 72 +++- cron/scheduler.py | 17 +- gateway/platforms/whatsapp.py | 16 +- gateway/run.py | 128 ++++++-- hermes_cli/_subprocess_compat.py | 175 ++++++++++ hermes_cli/doctor.py | 7 +- hermes_cli/gateway.py | 47 ++- hermes_cli/kanban_db.py | 29 +- hermes_cli/tools_config.py | 12 +- tests/tools/test_browser_homebrew_paths.py | 10 +- tests/tools/test_windows_native_support.py | 365 +++++++++++++++++++++ tools/browser_tool.py | 18 +- tools/code_execution_tool.py | 66 +++- tools/environments/local.py | 33 +- tui_gateway/entry.py | 34 +- 15 files changed, 957 insertions(+), 72 deletions(-) create mode 100644 hermes_cli/_subprocess_compat.py diff --git a/cli.py b/cli.py index 0dea5f65b5cc8..7232110580afe 100644 --- a/cli.py +++ b/cli.py @@ -728,8 +728,43 @@ def _run_cleanup(): _active_worktree: Optional[Dict[str, str]] = None +def _normalize_git_bash_path(p: Optional[str]) -> Optional[str]: + """Translate a Git Bash-style path (``/c/Users/...``) to the native + Windows form (``C:\\Users\\...``) that Python's ``subprocess.Popen`` + and ``pathlib.Path`` accept. + + No-op on non-Windows and for paths that already look native. Git on + native Windows normally emits forward-slash Windows paths + (``C:/Users/...``) which both bash and Python handle, but certain + configurations (Git Bash shells, MSYS2, WSL-mounted repos) surface + ``/c/...`` or ``/cygdrive/c/...`` variants. + """ + if not p: + return p + if sys.platform != "win32": + return p + import re as _re + # /c/Users/... or /C/Users/... + m = _re.match(r"^/([a-zA-Z])/(.*)$", p) + if m: + drive, rest = m.group(1), m.group(2) + return f"{drive.upper()}:\\{rest.replace('/', chr(92))}" + # /cygdrive/c/... or /mnt/c/... + m = _re.match(r"^/(?:cygdrive|mnt)/([a-zA-Z])/(.*)$", p) + if m: + drive, rest = m.group(1), m.group(2) + return f"{drive.upper()}:\\{rest.replace('/', chr(92))}" + return p + + def _git_repo_root() -> Optional[str]: - """Return the git repo root for CWD, or None if not in a repo.""" + """Return the git repo root for CWD, or None if not in a repo. + + Runs through :func:`_normalize_git_bash_path` so callers can pass + the result directly to ``Path``/``subprocess.Popen(cwd=...)`` on + Windows without hitting ``C:\\c\\Users\\...`` style resolution + mistakes. + """ import subprocess try: result = subprocess.run( @@ -737,7 +772,7 @@ def _git_repo_root() -> Optional[str]: capture_output=True, text=True, timeout=5, ) if result.returncode == 0: - return result.stdout.strip() + return _normalize_git_bash_path(result.stdout.strip()) except Exception: pass return None @@ -832,10 +867,39 @@ def _setup_worktree(repo_root: str = None) -> Optional[Dict[str, str]]: dst.parent.mkdir(parents=True, exist_ok=True) shutil.copy2(str(src), str(dst)) elif src.is_dir(): - # Symlink directories (faster, saves disk) + # Symlink directories (faster, saves disk). On Windows, + # symlink creation requires Developer Mode or elevation, + # and fails with OSError otherwise — fall back to a + # recursive copy so the worktree is still usable. The + # copy is slower and uses disk, but it doesn't require + # admin and matches the Linux/macOS symlink outcome + # functionally. if not dst.exists(): dst.parent.mkdir(parents=True, exist_ok=True) - os.symlink(str(src_resolved), str(dst)) + try: + os.symlink(str(src_resolved), str(dst)) + except (OSError, NotImplementedError) as _sym_err: + if sys.platform == "win32": + logger.info( + ".worktreeinclude: symlink failed (%s) — " + "falling back to copytree on Windows.", + _sym_err, + ) + try: + shutil.copytree( + str(src_resolved), + str(dst), + symlinks=True, + dirs_exist_ok=False, + ) + except Exception as _copy_err: + logger.warning( + ".worktreeinclude: copy fallback " + "also failed for %s -> %s: %s", + src, dst, _copy_err, + ) + else: + raise except Exception as e: logger.debug("Error copying .worktreeinclude entries: %s", e) diff --git a/cron/scheduler.py b/cron/scheduler.py index 97d0567300e7a..65db48466c402 100644 --- a/cron/scheduler.py +++ b/cron/scheduler.py @@ -14,6 +14,7 @@ import json import logging import os +import shutil import subprocess import sys @@ -714,7 +715,21 @@ def _run_job_script(script_path: str) -> tuple[bool, str]: # choice explicit here keeps the allowed surface small and auditable. suffix = path.suffix.lower() if suffix in (".sh", ".bash"): - argv = ["/bin/bash", str(path)] + # Resolve bash dynamically so Windows (Git Bash) and Linux/macOS + # all work. On native Windows without Git for Windows installed + # shutil.which returns None — fall back to a clear error rather + # than a FileNotFoundError with a confusing "[WinError 2]" + # traceback. + _bash = shutil.which("bash") or ( + "/bin/bash" if os.path.isfile("/bin/bash") else None + ) + if _bash is None: + return False, ( + f"Cannot run .sh/.bash script {path.name!r}: bash not found on PATH. " + "On Windows, install Git for Windows (which ships Git Bash) " + "or rewrite the script as Python (.py)." + ) + argv = [_bash, str(path)] else: argv = [sys.executable, str(path)] diff --git a/gateway/platforms/whatsapp.py b/gateway/platforms/whatsapp.py index ec454870393f4..4c781926ac497 100644 --- a/gateway/platforms/whatsapp.py +++ b/gateway/platforms/whatsapp.py @@ -21,6 +21,7 @@ import os import platform import re +import shutil import signal import subprocess @@ -177,10 +178,15 @@ def check_whatsapp_requirements() -> bool: WhatsApp requires a Node.js bridge for most implementations. """ - # Check for Node.js + # Check for Node.js. Resolve via shutil.which so we respect PATHEXT + # (node.exe vs node) and get a meaningful "not installed" signal + # instead of spawning a cmd flash on Windows. + _node = shutil.which("node") + if not _node: + return False try: result = subprocess.run( - ["node", "--version"], + [_node, "--version"], capture_output=True, text=True, timeout=5 @@ -464,9 +470,13 @@ async def connect(self) -> bool: bridge_dir = bridge_path.parent if not (bridge_dir / "node_modules").exists(): print(f"[{self.name}] Installing WhatsApp bridge dependencies...") + # Resolve npm path so Windows can execute the .cmd shim. + # shutil.which honours PATHEXT; on POSIX it returns the + # plain executable path. + _npm_bin = shutil.which("npm") or "npm" try: install_result = subprocess.run( - ["npm", "install", "--silent"], + [_npm_bin, "install", "--silent"], cwd=str(bridge_dir), capture_output=True, text=True, diff --git a/gateway/run.py b/gateway/run.py index 44bd61744af34..f7577ccf37de9 100644 --- a/gateway/run.py +++ b/gateway/run.py @@ -2784,6 +2784,48 @@ async def _launch_detached_restart_command(self) -> None: return current_pid = os.getpid() + + # On Windows there's no bash/setsid chain — spawn a tiny Python + # watcher directly via sys.executable instead. The watcher polls + # current_pid, waits for our exit, then runs `hermes gateway + # restart` with detach flags so the respawn survives the CLI + # that triggered the /restart command closing its console. + if sys.platform == "win32": + import textwrap + from hermes_cli._subprocess_compat import windows_detach_popen_kwargs + + cmd_argv = [*hermes_cmd, "gateway", "restart"] + watcher = textwrap.dedent( + """ + import os, subprocess, sys, time + pid = int(sys.argv[1]) + cmd = sys.argv[2:] + deadline = time.monotonic() + 120 + while time.monotonic() < deadline: + try: + os.kill(pid, 0) + except (ProcessLookupError, PermissionError, OSError): + break + time.sleep(0.2) + _CREATE_NEW_PROCESS_GROUP = 0x00000200 + _DETACHED_PROCESS = 0x00000008 + _CREATE_NO_WINDOW = 0x08000000 + subprocess.Popen( + cmd, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + creationflags=_CREATE_NEW_PROCESS_GROUP | _DETACHED_PROCESS | _CREATE_NO_WINDOW, + ) + """ + ).strip() + subprocess.Popen( + [sys.executable, "-c", watcher, str(current_pid), *cmd_argv], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + **windows_detach_popen_kwargs(), + ) + return + cmd = " ".join(shlex.quote(part) for part in hermes_cmd) shell_cmd = ( f"while kill -0 {current_pid} 2>/dev/null; do sleep 0.2; done; " @@ -11305,30 +11347,78 @@ async def _handle_update_command(self, event: MessageEvent) -> str: # where systemd-run --user fails due to missing D-Bus session). # PYTHONUNBUFFERED ensures output is flushed line-by-line so the # gateway can stream it to the messenger in near-real-time. - hermes_cmd_str = " ".join(shlex.quote(part) for part in hermes_cmd) - update_cmd = ( - f"PYTHONUNBUFFERED=1 {hermes_cmd_str} update --gateway" - f" > {shlex.quote(str(output_path))} 2>&1; " - f"status=$?; printf '%s' \"$status\" > {shlex.quote(str(exit_code_path))}" - ) - try: - setsid_bin = shutil.which("setsid") - if setsid_bin: - # Preferred: setsid creates a new session, fully detached + # Spawn `hermes update --gateway` detached so it survives gateway restart. + # --gateway enables file-based IPC for interactive prompts (stash + # restore, config migration) so the gateway can forward them to the + # user instead of silently skipping them. + # Use setsid for portable session detach (works under system services + # where systemd-run --user fails due to missing D-Bus session). + # PYTHONUNBUFFERED ensures output is flushed line-by-line so the + # gateway can stream it to the messenger in near-real-time. + # + # Windows: no bash/setsid chain. Run `hermes update --gateway` + # directly via sys.executable; redirect stdout/stderr to the same + # output files via Popen file handles; write the exit code in a + # follow-up write. A tiny Python watcher would be cleaner but + # we're already inside gateway/run.py's update path which is async, + # so the simplest correct thing is: launch an inline Python helper + # that runs the command and writes both outputs. + try: + if sys.platform == "win32": + import textwrap + from hermes_cli._subprocess_compat import windows_detach_popen_kwargs + + # hermes_cmd is a list of argv parts we can pass directly + # (no shell-quoting needed). + helper = textwrap.dedent( + """ + import os, subprocess, sys + output_path = sys.argv[1] + exit_code_path = sys.argv[2] + cmd = sys.argv[3:] + env = dict(os.environ) + env["PYTHONUNBUFFERED"] = "1" + with open(output_path, "wb") as f: + proc = subprocess.Popen(cmd, stdout=f, stderr=subprocess.STDOUT, env=env) + rc = proc.wait() + with open(exit_code_path, "w") as f: + f.write(str(rc)) + """ + ).strip() subprocess.Popen( - [setsid_bin, "bash", "-c", update_cmd], + [ + sys.executable, "-c", helper, + str(output_path), str(exit_code_path), + *hermes_cmd, "update", "--gateway", + ], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, - start_new_session=True, + **windows_detach_popen_kwargs(), ) else: - # Fallback: start_new_session=True calls os.setsid() in child - subprocess.Popen( - ["bash", "-c", update_cmd], - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - start_new_session=True, - ) + hermes_cmd_str = " ".join(shlex.quote(part) for part in hermes_cmd) + update_cmd = ( + f"PYTHONUNBUFFERED=1 {hermes_cmd_str} update --gateway" + f" > {shlex.quote(str(output_path))} 2>&1; " + f"status=$?; printf '%s' \"$status\" > {shlex.quote(str(exit_code_path))}" + ) + setsid_bin = shutil.which("setsid") + if setsid_bin: + # Preferred: setsid creates a new session, fully detached + subprocess.Popen( + [setsid_bin, "bash", "-c", update_cmd], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + else: + # Fallback: start_new_session=True calls os.setsid() in child + subprocess.Popen( + ["bash", "-c", update_cmd], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) except Exception as e: pending_path.unlink(missing_ok=True) exit_code_path.unlink(missing_ok=True) diff --git a/hermes_cli/_subprocess_compat.py b/hermes_cli/_subprocess_compat.py new file mode 100644 index 0000000000000..941728be8ea31 --- /dev/null +++ b/hermes_cli/_subprocess_compat.py @@ -0,0 +1,175 @@ +"""Windows subprocess compatibility helpers. + +Hermes is developed on Linux / macOS and tested natively on Windows too. +Several common subprocess patterns break silently-or-loudly on Windows: + +* ``["npm", "install", ...]`` — on Windows ``npm`` is ``npm.cmd``, a batch + shim. ``subprocess.Popen(["npm", ...])`` fails with WinError 193 + ("not a valid Win32 application") because CreateProcessW can't run a + ``.cmd`` file without ``shell=True`` or PATHEXT resolution. + +* ``start_new_session=True`` — on POSIX, this maps to ``os.setsid()`` and + actually detaches the child. On Windows it's silently ignored; the + Windows equivalent is ``CREATE_NEW_PROCESS_GROUP | DETACHED_PROCESS`` + creationflags, which Python only applies when you pass them explicitly. + +* Console-window flashes — every ``subprocess.Popen`` of a ``.exe`` on + Windows spawns a cmd window briefly unless ``CREATE_NO_WINDOW`` is + passed. Cosmetic but jarring for background daemons. + +This module centralizes the platform-branching logic so the rest of the +codebase doesn't sprinkle ``if sys.platform == "win32":`` everywhere. + +**All helpers are no-ops on non-Windows** — calling them in Linux/macOS +code paths is safe by design. That's the "do no damage on POSIX" +guarantee. +""" + +from __future__ import annotations + +import os +import shutil +import subprocess +import sys +from typing import Optional, Sequence + +__all__ = [ + "IS_WINDOWS", + "resolve_node_command", + "windows_detach_flags", + "windows_hide_flags", + "windows_detach_popen_kwargs", +] + + +IS_WINDOWS = sys.platform == "win32" + + +# ----------------------------------------------------------------------------- +# Node ecosystem launcher resolution +# ----------------------------------------------------------------------------- + + +def resolve_node_command(name: str, argv: Sequence[str]) -> list[str]: + """Resolve a Node-ecosystem command name to an absolute-path argv. + + On Windows, commands like ``npm``, ``npx``, ``yarn``, ``pnpm``, + ``playwright``, ``prettier`` ship as ``.cmd`` files (batch shims). + ``subprocess.Popen(["npm", "install"])`` fails with WinError 193 + because CreateProcessW doesn't execute batch files directly. + + ``shutil.which(name)`` *does* resolve ``.cmd`` via PATHEXT and returns + the fully-qualified path — which CreateProcessW accepts because the + extension tells Windows to route through ``cmd.exe /c``. + + On POSIX ``shutil.which`` also returns a fully-qualified path when + found. That's a small change from bare-name resolution (the OS does + its own PATH search) but functionally identical and has the side + benefit of making the argv reproducible in logs. + + Behavior when the command is not on PATH: + - On Windows: return the bare name — caller can still try with + ``shell=True`` as a last resort, OR the subsequent Popen will + raise FileNotFoundError with a readable error we want to surface. + - On POSIX: same. Bare ``npm`` on a Linux box without npm installed + fails the same way it did before this function existed. + + Args: + name: The command name to resolve (``npm``, ``npx``, ``node`` …). + argv: The remaining arguments. Must NOT include ``name`` itself — + this function builds the full argv list. + + Returns: + A list suitable for passing to subprocess.Popen/run/call. + """ + resolved = shutil.which(name) + if resolved: + return [resolved, *argv] + return [name, *argv] + + +# ----------------------------------------------------------------------------- +# Detached / hidden process creation +# ----------------------------------------------------------------------------- + + +# Win32 CreationFlags — defined here rather than imported from subprocess +# because CREATE_NO_WINDOW and DETACHED_PROCESS aren't guaranteed to be +# present on stdlib subprocess on older Pythons or non-Windows builds. +_CREATE_NEW_PROCESS_GROUP = 0x00000200 +_DETACHED_PROCESS = 0x00000008 +_CREATE_NO_WINDOW = 0x08000000 + + +def windows_detach_flags() -> int: + """Return Win32 creationflags that detach a child from the parent + console and process group. 0 on non-Windows. + + Pair with ``start_new_session=False`` (default) when calling + subprocess.Popen — on POSIX use ``start_new_session=True`` instead, + which maps to ``os.setsid()`` in the child. + + Rationale: + - ``CREATE_NEW_PROCESS_GROUP`` — child has its own process group so + Ctrl+C in the parent console doesn't propagate. + - ``DETACHED_PROCESS`` — child has no console at all. Necessary for + background daemons (gateway watchers, update respawners) because + without it, closing the console kills the child. + - ``CREATE_NO_WINDOW`` — suppress the brief cmd flash that would + otherwise appear when launching a console app. Redundant with + DETACHED_PROCESS but explicit for clarity. + """ + if not IS_WINDOWS: + return 0 + return _CREATE_NEW_PROCESS_GROUP | _DETACHED_PROCESS | _CREATE_NO_WINDOW + + +def windows_hide_flags() -> int: + """Return Win32 creationflags that merely hide the child's console + window without detaching the child. 0 on non-Windows. + + Use for short-lived console apps spawned as part of a larger + operation (``taskkill``, ``where``, version probes) where we want no + flash but also want to collect stdout/exit code synchronously. + + The key difference from :func:`windows_detach_flags`: NO + ``DETACHED_PROCESS`` — the child still inherits stdio handles so + ``capture_output=True`` works. ``DETACHED_PROCESS`` would sever + stdio and break stdout capture. + """ + if not IS_WINDOWS: + return 0 + return _CREATE_NO_WINDOW + + +def windows_detach_popen_kwargs() -> dict: + """Return a dict of Popen kwargs that detach a child on Windows and + fall back to the POSIX equivalent (``start_new_session=True``) on + Linux/macOS. + + Usage pattern: + + .. code-block:: python + + subprocess.Popen( + argv, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + stdin=subprocess.DEVNULL, + close_fds=True, + **windows_detach_popen_kwargs(), + ) + + This replaces the unsafe-on-Windows pattern: + + .. code-block:: python + + subprocess.Popen(..., start_new_session=True) + + which silently fails to detach on Windows (the flag is accepted but + has no effect — the child stays attached to the parent's console + and dies when the console closes). + """ + if IS_WINDOWS: + return {"creationflags": windows_detach_flags()} + return {"start_new_session": True} diff --git a/hermes_cli/doctor.py b/hermes_cli/doctor.py index ca0102d87133c..adc97d53c7a1d 100644 --- a/hermes_cli/doctor.py +++ b/hermes_cli/doctor.py @@ -1059,7 +1059,8 @@ def run_doctor(args): check_warn("Node.js not found", "(optional, needed for browser tools)") # npm audit for all Node.js packages - if _safe_which("npm"): + _npm_bin = _safe_which("npm") + if _npm_bin: npm_dirs = [ (PROJECT_ROOT, "Browser tools (agent-browser)"), (PROJECT_ROOT / "scripts" / "whatsapp-bridge", "WhatsApp bridge"), @@ -1068,8 +1069,10 @@ def run_doctor(args): if not (npm_dir / "node_modules").exists(): continue try: + # Use resolved absolute path so Windows can execute + # npm.cmd (CreateProcessW can't run bare .cmd names). audit_result = subprocess.run( - ["npm", "audit", "--json"], + [_npm_bin, "audit", "--json"], cwd=str(npm_dir), capture_output=True, text=True, timeout=30, ) diff --git a/hermes_cli/gateway.py b/hermes_cli/gateway.py index 171ca6de38eb6..996a8fab29612 100644 --- a/hermes_cli/gateway.py +++ b/hermes_cli/gateway.py @@ -445,6 +445,25 @@ def launch_detached_profile_gateway_restart(profile: str, old_pid: int) -> bool: if old_pid <= 0: return False + # The watcher is a tiny Python subprocess that polls the old PID and + # respawns the gateway once it's gone. Both legs of the chain need + # platform-appropriate detach semantics: + # + # POSIX — ``start_new_session=True`` (os.setsid in the child) detaches + # from the parent's process group so Ctrl+C in the CLI doesn't + # propagate and the watcher/gateway survive the CLI exiting. + # + # Windows — ``start_new_session`` is silently accepted but does NOT + # detach. The watcher stays attached to the CLI's console and dies + # when the user closes the terminal, leaving ``hermes update`` users + # with no running gateway until they re-invoke ``hermes gateway`` + # manually. The Win32 equivalent is the ``CREATE_NEW_PROCESS_GROUP | + # DETACHED_PROCESS | CREATE_NO_WINDOW`` creationflags bundle. + # + # ``windows_detach_popen_kwargs()`` returns the right kwargs for the + # host platform and is a no-op on POSIX (just ``start_new_session=True``). + from hermes_cli._subprocess_compat import windows_detach_popen_kwargs + watcher = textwrap.dedent( """ import os @@ -466,21 +485,35 @@ def launch_detached_profile_gateway_restart(profile: str, old_pid: int) -> bool: # Windows: gone PID raises OSError (WinError 87). break time.sleep(0.2) - subprocess.Popen( - cmd, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - start_new_session=True, - ) + + # Platform-appropriate detach for the respawned gateway. On POSIX + # start_new_session=True maps to os.setsid; on Windows we need + # explicit creationflags because start_new_session is a no-op there. + _popen_kwargs = { + "stdout": subprocess.DEVNULL, + "stderr": subprocess.DEVNULL, + } + if sys.platform == "win32": + _CREATE_NEW_PROCESS_GROUP = 0x00000200 + _DETACHED_PROCESS = 0x00000008 + _CREATE_NO_WINDOW = 0x08000000 + _popen_kwargs["creationflags"] = ( + _CREATE_NEW_PROCESS_GROUP | _DETACHED_PROCESS | _CREATE_NO_WINDOW + ) + else: + _popen_kwargs["start_new_session"] = True + subprocess.Popen(cmd, **_popen_kwargs) """ ).strip() try: + # Same platform-aware detach for the watcher process itself — so + # closing the user's terminal doesn't kill the watcher. subprocess.Popen( [sys.executable, "-c", watcher, str(old_pid), *_gateway_run_args_for_profile(profile)], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, - start_new_session=True, + **windows_detach_popen_kwargs(), ) except OSError: return False diff --git a/hermes_cli/kanban_db.py b/hermes_cli/kanban_db.py index 8ce5436df8565..2f1d8bb59a172 100644 --- a/hermes_cli/kanban_db.py +++ b/hermes_cli/kanban_db.py @@ -3519,17 +3519,24 @@ def dispatch_once( # cleanly without calling ``kanban_complete`` / ``kanban_block`` # (protocol violation — auto-block) from a real crash (OOM killer, # SIGKILL, non-zero exit — existing counter behavior). - try: - while True: - try: - _pid, _status = os.waitpid(-1, os.WNOHANG) - except ChildProcessError: - break - if _pid == 0: - break - _record_worker_exit(_pid, _status) - except Exception: - pass + # + # Windows has no zombies / no os.WNOHANG — subprocess.Popen handles + # are freed when the Python object is garbage-collected or .wait() is + # called explicitly. The kanban dispatcher discards the Popen handle + # after spawn (``_default_spawn`` → abandon), so on Windows there's + # nothing to reap here — skip the whole block. + if os.name != "nt": + try: + while True: + try: + _pid, _status = os.waitpid(-1, os.WNOHANG) + except ChildProcessError: + break + if _pid == 0: + break + _record_worker_exit(_pid, _status) + except Exception: + pass result = DispatchResult() result.reclaimed = release_stale_claims(conn) diff --git a/hermes_cli/tools_config.py b/hermes_cli/tools_config.py index aa07e85e7a861..3d19deab60883 100644 --- a/hermes_cli/tools_config.py +++ b/hermes_cli/tools_config.py @@ -509,8 +509,12 @@ def _run_post_setup(post_setup_key: str): if not node_modules.exists() and npm_bin: _print_info(" Installing Node.js dependencies for browser tools...") import subprocess + # Use the resolved npm_bin absolute path so subprocess.Popen can + # execute npm.cmd on Windows (CreateProcessW otherwise rejects + # batch shims). On POSIX npm_bin is the plain path — same + # behaviour as before. result = subprocess.run( - ["npm", "install", "--silent"], + [npm_bin, "install", "--silent"], capture_output=True, text=True, cwd=str(PROJECT_ROOT) ) if result.returncode == 0: @@ -609,11 +613,13 @@ def _run_post_setup(post_setup_key: str): elif post_setup_key == "camofox": camofox_dir = PROJECT_ROOT / "node_modules" / "@askjo" / "camofox-browser" - if not camofox_dir.exists() and shutil.which("npm"): + _npm_bin = shutil.which("npm") + if not camofox_dir.exists() and _npm_bin: _print_info(" Installing Camofox browser server...") import subprocess + # Absolute npm path so .cmd shim executes on Windows. result = subprocess.run( - ["npm", "install", "--silent"], + [_npm_bin, "install", "--silent"], capture_output=True, text=True, cwd=str(PROJECT_ROOT) ) if result.returncode == 0: diff --git a/tests/tools/test_browser_homebrew_paths.py b/tests/tools/test_browser_homebrew_paths.py index 221d2e6602ada..7e4d1c702225a 100644 --- a/tests/tools/test_browser_homebrew_paths.py +++ b/tests/tools/test_browser_homebrew_paths.py @@ -340,7 +340,15 @@ def capture_popen(cmd, **kwargs): _run_browser_command("test-task", "navigate", ["https://example.com"]) assert captured_cmd is not None - assert captured_cmd[:2] == ["npx", "agent-browser"] + # The prefix must split "npx agent-browser" into two argv items. + # On POSIX shutil.which("npx") returns the absolute path if npx is on + # PATH (which the test's patched PATH always contains when the system + # has it installed). The important invariant is that the second + # argv item is the package name "agent-browser", not a merged + # "npx agent-browser" string — that's what Popen needs. + assert len(captured_cmd) >= 2 + assert captured_cmd[0].endswith("npx") or captured_cmd[0] == "npx" + assert captured_cmd[1] == "agent-browser" assert captured_cmd[2:6] == [ "--session", "test-session", diff --git a/tests/tools/test_windows_native_support.py b/tests/tools/test_windows_native_support.py index fa1a0ed8e3ff4..d0b8645951265 100644 --- a/tests/tools/test_windows_native_support.py +++ b/tests/tools/test_windows_native_support.py @@ -445,3 +445,368 @@ def test_entry_point_calls_configure_stdio(self, relpath): f"{relpath} must call hermes_cli.stdio.configure_windows_stdio() " "early in startup so Windows consoles render Unicode without crashing" ) + + +# --------------------------------------------------------------------------- +# _subprocess_compat shared helpers +# --------------------------------------------------------------------------- + + +class TestSubprocessCompatHelpers: + """hermes_cli/_subprocess_compat.py POSIX + Windows behaviour.""" + + def test_is_windows_matches_sys_platform(self): + from hermes_cli import _subprocess_compat as sc + assert sc.IS_WINDOWS == (sys.platform == "win32") + + def test_resolve_node_command_returns_absolute_on_posix(self): + """On Linux, resolve_node_command('sh', ['-c','echo hi']) picks up /bin/sh.""" + from hermes_cli._subprocess_compat import resolve_node_command + # We can't assert "npm is on PATH" portably; use `sh` which is + # guaranteed on POSIX. On Windows the test only confirms the + # no-crash fallback path. + argv = resolve_node_command("sh", ["-c", "echo hi"]) + assert argv[1:] == ["-c", "echo hi"] + # First element is either an absolute path (sh found) or the bare + # name (fallback) — both are acceptable behaviours. + + def test_resolve_node_command_fallback_when_absent(self): + from hermes_cli._subprocess_compat import resolve_node_command + argv = resolve_node_command( + "zzz-definitely-not-on-path-xyzzy", ["--help"] + ) + # Must fall back to the bare name — NOT return None, NOT crash. + assert argv[0] == "zzz-definitely-not-on-path-xyzzy" + assert argv[1:] == ["--help"] + + def test_windows_flags_zero_on_posix(self): + from hermes_cli._subprocess_compat import ( + windows_detach_flags, + windows_hide_flags, + ) + if sys.platform != "win32": + assert windows_detach_flags() == 0 + assert windows_hide_flags() == 0 + + def test_windows_detach_popen_kwargs_is_posix_equivalent_on_posix(self): + from hermes_cli._subprocess_compat import windows_detach_popen_kwargs + kwargs = windows_detach_popen_kwargs() + if sys.platform != "win32": + # POSIX path MUST produce start_new_session=True, which maps to + # os.setsid() in the child — identical to the unchanged main + # branch behaviour. Do NOT break Linux/macOS here. + assert kwargs == {"start_new_session": True} + else: + # Windows path must include creationflags with all 3 bits set. + assert "creationflags" in kwargs + assert kwargs["creationflags"] != 0 + # No start_new_session on Windows (silently no-op there). + assert "start_new_session" not in kwargs + + def test_windows_detach_flags_has_expected_win32_bits(self, monkeypatch): + """Simulate Windows to verify flag bundle.""" + from hermes_cli import _subprocess_compat as sc + monkeypatch.setattr(sc, "IS_WINDOWS", True) + flags = sc.windows_detach_flags() + # CREATE_NEW_PROCESS_GROUP | DETACHED_PROCESS | CREATE_NO_WINDOW + assert flags & 0x00000200, "missing CREATE_NEW_PROCESS_GROUP" + assert flags & 0x00000008, "missing DETACHED_PROCESS" + assert flags & 0x08000000, "missing CREATE_NO_WINDOW" + + +# --------------------------------------------------------------------------- +# tui_gateway/entry.py signal installation survives absent POSIX signals +# --------------------------------------------------------------------------- + + +class TestTuiGatewayEntrySignalGuards: + """Importing tui_gateway.entry must not crash when SIGPIPE/SIGHUP absent. + + Linux has both signals, so this is mostly a source-level invariant check + (no bare ``signal.SIGPIPE`` at module level without a ``hasattr`` guard). + On Windows the import would have raised AttributeError before this fix. + """ + + def test_source_guards_each_signal_installation(self): + root = Path(__file__).resolve().parents[2] + source = (root / "tui_gateway" / "entry.py").read_text(encoding="utf-8") + # Every signal.signal(...) at module scope must be preceded by a + # hasattr check. We look at the text: no bare "signal.signal(" + # call should appear outside a function body without a guard. + # Simpler heuristic: all SIGPIPE / SIGHUP references outside the + # dict-building loop must be wrapped in hasattr. + assert 'hasattr(signal, "SIGPIPE")' in source + assert 'hasattr(signal, "SIGHUP")' in source + assert 'hasattr(signal, "SIGTERM")' in source + assert 'hasattr(signal, "SIGINT")' in source + + def test_module_imports_cleanly(self): + """Importing the module must not raise — verifies the guards work.""" + # Drop any cached import so the module re-initialises + for mod in list(sys.modules): + if mod.startswith("tui_gateway"): + del sys.modules[mod] + import tui_gateway.entry # noqa: F401 # must not raise + + +# --------------------------------------------------------------------------- +# hermes_cli/kanban_db.py waitpid guard +# --------------------------------------------------------------------------- + + +class TestKanbanWaitpidWindowsGuard: + """os.WNOHANG doesn't exist on Windows — the dispatcher tick reap loop + must be gated behind ``os.name != "nt"``.""" + + def test_source_gates_waitpid_loop(self): + root = Path(__file__).resolve().parents[2] + source = (root / "hermes_cli" / "kanban_db.py").read_text(encoding="utf-8") + # Find the waitpid call and confirm it's inside a POSIX gate. + idx = source.find("os.waitpid(-1, os.WNOHANG)") + assert idx > 0, "waitpid call must exist" + # Look backwards up to 400 chars for the gate. + preamble = source[max(0, idx - 400):idx] + assert 'os.name != "nt"' in preamble or "os.name != 'nt'" in preamble, ( + "os.waitpid(-1, os.WNOHANG) must sit behind an os.name != 'nt' guard" + ) + + +# --------------------------------------------------------------------------- +# code_execution_tool TCP loopback on Windows +# --------------------------------------------------------------------------- + + +class TestCodeExecutionTransportTcpFallback: + """The RPC transport must fall back to TCP on Windows. + + We can't easily execute the sandbox on Linux CI in Windows mode, but we + CAN assert that the generated client module supports both AF_UNIX and + AF_INET endpoints based on the HERMES_RPC_SOCKET format. + """ + + def test_generated_client_handles_tcp_endpoint(self): + root = Path(__file__).resolve().parents[2] + source = (root / "tools" / "code_execution_tool.py").read_text(encoding="utf-8") + # _UDS_TRANSPORT_HEADER body must parse both transports. + assert 'endpoint.startswith("tcp://")' in source, ( + "generated sandbox client must accept tcp:// endpoints for Windows" + ) + assert "socket.AF_INET" in source, ( + "generated sandbox client must be able to open AF_INET sockets" + ) + + def test_server_side_branches_on_use_tcp_rpc(self): + root = Path(__file__).resolve().parents[2] + source = (root / "tools" / "code_execution_tool.py").read_text(encoding="utf-8") + assert "_use_tcp_rpc = _IS_WINDOWS" in source + assert 'rpc_endpoint = f"tcp://{_host}:{_port}"' in source + + +# --------------------------------------------------------------------------- +# cron/scheduler.py /bin/bash dynamic resolution +# --------------------------------------------------------------------------- + + +class TestCronSchedulerBashResolution: + """cron.scheduler must NOT hardcode /bin/bash — .sh scripts need a + dynamically-resolved bash so Windows (Git Bash) works.""" + + def test_source_uses_shutil_which_for_bash(self): + root = Path(__file__).resolve().parents[2] + source = (root / "cron" / "scheduler.py").read_text(encoding="utf-8") + # The old hardcoded path should be gone as the sole bash source. + # It may still appear as a POSIX fallback after shutil.which(), so + # we check for the shutil.which call near the .sh/.bash branch. + assert 'shutil.which("bash")' in source, ( + "cron.scheduler must resolve bash dynamically via shutil.which" + ) + + def test_error_message_when_bash_missing(self): + root = Path(__file__).resolve().parents[2] + source = (root / "cron" / "scheduler.py").read_text(encoding="utf-8") + # The graceful-failure message must mention "bash not found" so + # Windows users without Git Bash see an actionable error instead + # of a WinError 2 traceback. + assert "bash not found" in source.lower() + + +# --------------------------------------------------------------------------- +# Node-ecosystem launcher resolution (npm / npx / node) +# --------------------------------------------------------------------------- + + +class TestNpmBareSpawnsResolved: + """Every spawn site that launches ``npm``/``npx`` must resolve via + shutil.which / hermes_cli._subprocess_compat.resolve_node_command + so Windows can execute the .cmd batch shims.""" + + @pytest.mark.parametrize( + "relpath", + [ + "hermes_cli/tools_config.py", + "hermes_cli/doctor.py", + "gateway/platforms/whatsapp.py", + "tools/browser_tool.py", + ], + ) + def test_no_bare_npm_or_npx_in_popen_argv(self, relpath): + """Reject ``subprocess.run(["npm", ...])`` / ``["npx", ...]`` patterns. + + Those fail on Windows with WinError 193. Callers must resolve + via shutil.which(...) and pass the absolute path (or fall back + to the bare name only as a last resort behind a variable). + """ + root = Path(__file__).resolve().parents[2] + source = (root / relpath).read_text(encoding="utf-8") + # The forbidden literal: a subprocess invocation that names npm + # or npx as a bare string inside an argv list. + forbidden_patterns = [ + '["npm",', + '["npx",', + "['npm',", + "['npx',", + ] + for pat in forbidden_patterns: + # Exception: strings inside error-message text or comments are fine. + # We only fail if the literal appears in an argv position, which + # we approximate by checking it isn't inside a print/log/comment. + # Find all occurrences and verify they're behind shutil.which. + idx = 0 + while True: + idx = source.find(pat, idx) + if idx < 0: + break + # Look at the preceding 120 chars — if "shutil.which" appears + # there, or the pattern is inside a comment/string, it's fine. + context = source[max(0, idx - 120):idx] + if "#" in context.split("\n")[-1]: + idx += len(pat) + continue + # Argv forms that START with a bare npm/npx are the bug. + raise AssertionError( + f"{relpath}: bare {pat!r} still present at offset {idx} — " + f"resolve via shutil.which(...) so Windows can execute .cmd shims" + ) + + +# --------------------------------------------------------------------------- +# tools/environments/local.py Windows temp dir & PATH injection +# --------------------------------------------------------------------------- + + +class TestLocalEnvironmentWindowsTempDir: + """LocalEnvironment.get_temp_dir must return a native Windows path on + Windows, NOT the POSIX ``/tmp`` literal (which Python can't open).""" + + def test_posix_path_preserved_on_linux(self): + """Linux/macOS behaviour MUST be unchanged — return / tmp or + tempfile.gettempdir()-derived POSIX path. This is the 'do no harm' + test — regressions here break every Unix user's terminal tool.""" + from tools.environments.local import LocalEnvironment + + env = LocalEnvironment(cwd="/tmp", timeout=10, env={}) + tmp_dir = env.get_temp_dir() + if sys.platform != "win32": + assert tmp_dir.startswith("/"), ( + f"POSIX temp dir must start with '/'; got {tmp_dir!r}" + ) + + def test_source_has_windows_branch_using_hermes_home(self): + root = Path(__file__).resolve().parents[2] + source = (root / "tools" / "environments" / "local.py").read_text(encoding="utf-8") + assert "if _IS_WINDOWS:" in source + assert "get_hermes_home" in source + assert 'cache_dir = get_hermes_home() / "cache" / "terminal"' in source + + +class TestLocalEnvironmentPathInjectionGated: + """The /usr/bin PATH injection in _make_run_env must be POSIX-only.""" + + def test_source_gates_path_injection(self): + root = Path(__file__).resolve().parents[2] + source = (root / "tools" / "environments" / "local.py").read_text(encoding="utf-8") + # The fix wraps the injection in `if not _IS_WINDOWS`. + assert 'not _IS_WINDOWS and "/usr/bin" not in existing_path.split(":")' in source + + +# --------------------------------------------------------------------------- +# cli.py git path normalization +# --------------------------------------------------------------------------- + + +class TestGitBashPathNormalization: + """_normalize_git_bash_path should turn /c/Users/... into C:\\Users\\... + on Windows and leave paths unchanged on POSIX.""" + + def test_posix_noop(self): + """Must NOT mutate paths on Linux/macOS.""" + from cli import _normalize_git_bash_path + if sys.platform != "win32": + assert _normalize_git_bash_path("/home/teknium/foo") == "/home/teknium/foo" + assert _normalize_git_bash_path("/c/Users/foo") == "/c/Users/foo" + assert _normalize_git_bash_path("C:/Users/foo") == "C:/Users/foo" + assert _normalize_git_bash_path(None) is None + + def test_empty_string_preserved(self): + from cli import _normalize_git_bash_path + assert _normalize_git_bash_path("") == "" + + def test_windows_translation(self, monkeypatch): + """Simulate Windows and verify /c/Users/... becomes C:\\Users\\...""" + import cli as cli_mod + monkeypatch.setattr(cli_mod.sys, "platform", "win32") + assert cli_mod._normalize_git_bash_path("/c/Users/foo") == r"C:\Users\foo" + assert cli_mod._normalize_git_bash_path("/C/Users/foo") == r"C:\Users\foo" + assert cli_mod._normalize_git_bash_path("/cygdrive/d/data") == r"D:\data" + assert cli_mod._normalize_git_bash_path("/mnt/c/Users") == r"C:\Users" + # Already-native path is preserved + assert cli_mod._normalize_git_bash_path(r"C:\Users\foo") == r"C:\Users\foo" + # Forward-slash Windows path is preserved (git on Windows often + # returns this form; it's valid for both bash and Python, so we + # don't need to translate). + assert cli_mod._normalize_git_bash_path("C:/Users/foo") == "C:/Users/foo" + + +class TestWorktreeSymlinkFallback: + """.worktreeinclude directory symlinks must fall back to copytree on + Windows (where symlink creation requires admin / Dev Mode).""" + + def test_source_has_symlink_fallback(self): + root = Path(__file__).resolve().parents[2] + source = (root / "cli.py").read_text(encoding="utf-8") + # Look for the try/except that handles OSError around os.symlink + # with a shutil.copytree fallback. + assert "os.symlink(str(src_resolved), str(dst))" in source + assert "except (OSError, NotImplementedError)" in source + assert "shutil.copytree" in source + assert 'sys.platform == "win32"' in source + + +# --------------------------------------------------------------------------- +# Gateway detached watcher — Windows creationflags +# --------------------------------------------------------------------------- + + +class TestGatewayDetachedWatcherWindowsFlags: + """launch_detached_profile_gateway_restart and the in-gateway update + launcher must use CREATE_NEW_PROCESS_GROUP | DETACHED_PROCESS on + Windows, not silent start_new_session=True.""" + + def test_hermes_cli_gateway_uses_compat_kwargs(self): + root = Path(__file__).resolve().parents[2] + source = (root / "hermes_cli" / "gateway.py").read_text(encoding="utf-8") + assert "windows_detach_popen_kwargs" in source, ( + "hermes_cli/gateway.py must use the platform-aware detach helper" + ) + # The legacy start_new_session=True on the outer Popen should be + # replaced by **windows_detach_popen_kwargs(). Inside the watcher + # STRING the old pattern is replaced by explicit creationflags. + assert "**windows_detach_popen_kwargs()" in source + + def test_gateway_run_update_has_windows_branch(self): + root = Path(__file__).resolve().parents[2] + source = (root / "gateway" / "run.py").read_text(encoding="utf-8") + # Both the /restart and /update paths must have sys.platform=='win32' branches. + assert 'if sys.platform == "win32":' in source + # Windows branch uses windows_detach_popen_kwargs + assert "windows_detach_popen_kwargs" in source diff --git a/tools/browser_tool.py b/tools/browser_tool.py index c7b16feb1171c..5986ea584b9ff 100644 --- a/tools/browser_tool.py +++ b/tools/browser_tool.py @@ -708,7 +708,16 @@ def _run_chrome_fallback_command( ) return {"success": False, "error": hint} - cmd_prefix = ["npx", "agent-browser"] if browser_cmd == "npx agent-browser" else [browser_cmd] + # On Windows npx is npx.cmd — use shutil.which so CreateProcessW can + # execute the batch shim. shutil.which honours PATHEXT on Windows and + # returns the plain executable on POSIX. If npx isn't on PATH (Termux, + # bare container), fall back to the bare name and let Popen raise with + # a readable "FileNotFoundError: 'npx'" rather than WinError 193. + if browser_cmd == "npx agent-browser": + _npx_bin = shutil.which("npx") or "npx" + cmd_prefix = [_npx_bin, "agent-browser"] + else: + cmd_prefix = [browser_cmd] base_args = cmd_prefix + ["--engine", "chrome", "--session", tmp_session, "--json"] task_socket_dir = os.path.join(_socket_safe_tmpdir(), f"agent-browser-{tmp_session}") @@ -1768,7 +1777,12 @@ def _run_browser_command( # Keep concrete executable paths intact, even when they contain spaces. # Only the synthetic npx fallback needs to expand into multiple argv items. - cmd_prefix = ["npx", "agent-browser"] if browser_cmd == "npx agent-browser" else [browser_cmd] + # shutil.which resolves npx → npx.cmd on Windows; bare "npx" stays on POSIX. + if browser_cmd == "npx agent-browser": + _npx_bin = shutil.which("npx") or "npx" + cmd_prefix = [_npx_bin, "agent-browser"] + else: + cmd_prefix = [browser_cmd] cmd_parts = cmd_prefix + backend_args + [ "--json", diff --git a/tools/code_execution_tool.py b/tools/code_execution_tool.py index ffcf726fcd5b6..89d42484c2bc6 100644 --- a/tools/code_execution_tool.py +++ b/tools/code_execution_tool.py @@ -235,10 +235,27 @@ def retry(fn, max_attempts=3, delay=2): ''' + _COMMON_HELPERS + '''\ def _connect(): + """Connect to the parent's RPC server via the transport it picked. + + HERMES_RPC_SOCKET can be either: + - a filesystem path (POSIX Unix domain socket — the default on + Linux and macOS) + - a string of the form ``tcp://127.0.0.1:`` (Windows, where + AF_UNIX is unreliable — the parent falls back to loopback TCP) + """ global _sock if _sock is None: - _sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - _sock.connect(os.environ["HERMES_RPC_SOCKET"]) + endpoint = os.environ["HERMES_RPC_SOCKET"] + if endpoint.startswith("tcp://"): + # tcp://host:port (host is always 127.0.0.1 in practice — we + # only bind loopback server-side) + _host_port = endpoint[len("tcp://"):] + _host, _, _port = _host_port.rpartition(":") + _sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + _sock.connect((_host or "127.0.0.1", int(_port))) + else: + _sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + _sock.connect(endpoint) _sock.settimeout(300) return _sock @@ -988,8 +1005,22 @@ def execute_code( # Use /tmp on macOS to avoid the long /var/folders/... path that pushes # Unix domain socket paths past the 104-byte macOS AF_UNIX limit. # On Linux, tempfile.gettempdir() already returns /tmp. + # + # Windows: Python 3.9+ added partial AF_UNIX support but the file-backed + # variant is flaky across Windows builds (requires Windows 10 1803+, + # still fails under some configurations, and the socket file can't live + # on the same temp drive as the script). Fall back to loopback TCP — + # same ephemeral port, same 1-connection listen queue, same serialized + # request/response framing. The generated client reads the transport + # selector from HERMES_RPC_SOCKET (path vs. ``tcp://host:port``). _sock_tmpdir = "/tmp" if sys.platform == "darwin" else tempfile.gettempdir() - sock_path = os.path.join(_sock_tmpdir, f"hermes_rpc_{uuid.uuid4().hex}.sock") + _use_tcp_rpc = _IS_WINDOWS + if _use_tcp_rpc: + sock_path = None # not used on Windows; TCP endpoint stored below + rpc_endpoint = None # set after bind() + else: + sock_path = os.path.join(_sock_tmpdir, f"hermes_rpc_{uuid.uuid4().hex}.sock") + rpc_endpoint = sock_path tool_call_log: list = [] tool_call_counter = [0] # mutable so the RPC thread can increment @@ -1008,10 +1039,24 @@ def execute_code( with open(os.path.join(tmpdir, "script.py"), "w") as f: f.write(code) - # --- Start UDS server --- - server_sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - server_sock.bind(sock_path) - os.chmod(sock_path, 0o600) + # --- Start RPC server --- + # Two transports: + # POSIX: AF_UNIX stream socket on sock_path, chmod 0600 for + # owner-only access. Filesystem permissions gate the socket. + # Windows: AF_INET stream socket on 127.0.0.1 with an ephemeral + # port. No filesystem permission story, but loopback-only bind + # means only the current user's processes (not remote) can + # connect. HERMES_RPC_SOCKET is set to ``tcp://127.0.0.1:`` + # which the generated client parses to pick AF_INET. + if _use_tcp_rpc: + server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + server_sock.bind(("127.0.0.1", 0)) # ephemeral port + _host, _port = server_sock.getsockname()[:2] + rpc_endpoint = f"tcp://{_host}:{_port}" + else: + server_sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + server_sock.bind(sock_path) + os.chmod(sock_path, 0o600) server_sock.listen(1) rpc_thread = threading.Thread( @@ -1053,7 +1098,7 @@ def execute_code( # Allow vars with known safe prefixes. if any(k.startswith(p) for p in _SAFE_ENV_PREFIXES): child_env[k] = v - child_env["HERMES_RPC_SOCKET"] = sock_path + child_env["HERMES_RPC_SOCKET"] = rpc_endpoint child_env["PYTHONDONTWRITEBYTECODE"] = "1" # Ensure the hermes-agent root is importable in the sandbox so # repo-root modules are available to child scripts. We also prepend @@ -1302,7 +1347,10 @@ def _drain_head_tail(pipe, head_chunks, tail_chunks, head_bytes, tail_bytes, tot import shutil shutil.rmtree(tmpdir, ignore_errors=True) try: - os.unlink(sock_path) + # Only UDS has a filesystem socket to unlink; TCP sockets are + # freed by server_sock.close() above. + if sock_path: + os.unlink(sock_path) except OSError: pass # already cleaned up or never created diff --git a/tools/environments/local.py b/tools/environments/local.py index 0ee851dce17bd..39b1cf0821717 100644 --- a/tools/environments/local.py +++ b/tools/environments/local.py @@ -9,6 +9,7 @@ import subprocess import tempfile import time +from pathlib import Path from tools.environments.base import BaseEnvironment, _pipe_stdin @@ -249,7 +250,15 @@ def _make_run_env(env: dict) -> dict: elif k not in _HERMES_PROVIDER_ENV_BLOCKLIST or _is_passthrough(k): run_env[k] = v existing_path = run_env.get("PATH", "") - if "/usr/bin" not in existing_path.split(":"): + # The "/usr/bin not already present → inject sane POSIX path" heuristic + # only makes sense on POSIX. On Windows the PATH separator is ";" + # (the split(":") above turns a full Windows PATH into a single + # unrecognisable chunk, which then triggers prepending POSIX paths + # to a Windows PATH — completely wrong). Skip the injection entirely + # on Windows; the native PATH already points at whatever shell + # Hermes is driving via _find_bash (Git Bash), and Git Bash itself + # prepends its MSYS2 /usr/bin equivalent via the shell-init files. + if not _IS_WINDOWS and "/usr/bin" not in existing_path.split(":"): run_env["PATH"] = f"{existing_path}:{_SANE_PATH}" if existing_path else _SANE_PATH # Per-profile HOME isolation: redirect system tool configs (git, ssh, gh, @@ -371,7 +380,29 @@ def get_temp_dir(self) -> str: Check the environment configured for this backend first so callers can override the temp root explicitly (for example via terminal.env or a custom TMPDIR), then fall back to the host process environment. + + **Windows:** hardcoded ``/tmp`` is wrong in two ways — native Python + can't open the path, and the Windows default temp (``%TEMP%``) often + contains spaces (``C:\\Users\\Some Name\\AppData\\Local\\Temp``) that + break unquoted bash interpolations. Use a dedicated cache dir under + ``HERMES_HOME`` instead — single-word path, guaranteed to exist, same + string resolves in both Git Bash and native Python. """ + if _IS_WINDOWS: + # Derive a Windows-safe temp dir under HERMES_HOME. Using + # forward slashes makes the same string work unchanged in bash + # command interpolations AND in Python ``open()`` — Windows + # accepts forward slashes in filesystem paths, and we control + # the path so we can guarantee no spaces. + try: + from hermes_constants import get_hermes_home + cache_dir = get_hermes_home() / "cache" / "terminal" + except Exception: + cache_dir = Path(tempfile.gettempdir()) / "hermes_terminal" + cache_dir.mkdir(parents=True, exist_ok=True) + # Force forward slashes so the same string serves both contexts. + return str(cache_dir).replace("\\", "/") + for env_var in ("TMPDIR", "TMP", "TEMP"): candidate = self.env.get(env_var) or os.environ.get(env_var) if candidate and candidate.startswith("/"): diff --git a/tui_gateway/entry.py b/tui_gateway/entry.py index 0fe87ca49c5c4..12d53c6d2e59a 100644 --- a/tui_gateway/entry.py +++ b/tui_gateway/entry.py @@ -81,11 +81,14 @@ def _log_signal(signum: int, frame) -> None: thread, and fall back to ``os._exit(0)`` so a wedged write/flush can never strand the process. """ - name = { - signal.SIGPIPE: "SIGPIPE", - signal.SIGTERM: "SIGTERM", - signal.SIGHUP: "SIGHUP", - }.get(signum, f"signal {signum}") + # SIGPIPE and SIGHUP don't exist on Windows — build the lookup + # dict from attributes that actually exist on the current platform. + _signal_names: dict[int, str] = {} + for _attr in ("SIGPIPE", "SIGTERM", "SIGHUP", "SIGINT", "SIGBREAK"): + _sig = getattr(signal, _attr, None) + if _sig is not None: + _signal_names[int(_sig)] = _attr + name = _signal_names.get(signum, f"signal {signum}") try: os.makedirs(os.path.dirname(_CRASH_LOG), exist_ok=True) with open(_CRASH_LOG, "a", encoding="utf-8") as f: @@ -140,10 +143,23 @@ def _hard_exit() -> None: # sys.exit(0) + _log_exit), which keeps the gateway alive as long as # the main command pipe is still readable. Terminal signals still # route through _log_signal so kills and hangups are diagnosable. -signal.signal(signal.SIGPIPE, signal.SIG_IGN) -signal.signal(signal.SIGTERM, _log_signal) -signal.signal(signal.SIGHUP, _log_signal) -signal.signal(signal.SIGINT, signal.SIG_IGN) +# +# SIGPIPE and SIGHUP don't exist on Windows; guard each installation +# with hasattr so ``python -m tui_gateway.entry`` (spawned by +# ``hermes --tui``) imports cleanly there. SIGBREAK (Windows' Ctrl+Break) +# is installed when available as a weaker equivalent of SIGHUP. +if hasattr(signal, "SIGPIPE"): + signal.signal(signal.SIGPIPE, signal.SIG_IGN) +if hasattr(signal, "SIGTERM"): + signal.signal(signal.SIGTERM, _log_signal) +if hasattr(signal, "SIGHUP"): + signal.signal(signal.SIGHUP, _log_signal) +elif hasattr(signal, "SIGBREAK"): + # Windows-only: Ctrl+Break in a console window delivers SIGBREAK. + # Route it through the same handler so kills are diagnosable. + signal.signal(signal.SIGBREAK, _log_signal) +if hasattr(signal, "SIGINT"): + signal.signal(signal.SIGINT, signal.SIG_IGN) def _log_exit(reason: str) -> None: From cf9b2df57a8089bd86f30f64a2df771858cf82ca Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 17:42:47 -0700 Subject: [PATCH 05/20] fix(windows): use PortableGit (not MinGit), fix relaunch os.execvp crash, surface npm errors MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three real bugs from teknium1's first Windows install run: 1. **MinGit has no bash.exe.** MinGit is the minimal-automation Git for Windows distribution — it ships git.exe but deliberately strips bash and the POSIX coreutils. Installer logged "Could not locate bash.exe" and Hermes would fail to run any shell command. Switched to PortableGit — the full Git for Windows minus the installer UI. PortableGit ships bash.exe at \bin\bash.exe plus sh, awk, sed, grep, curl, ssh in usr\bin\. ARM64 variant is detected separately (PortableGit-*-arm64.7z.exe). 32-bit falls back to MinGit-32-bit with a warning (PortableGit is 64-bit only). PortableGit ships as a 7z self-extractor (56MB vs MinGit's 38MB). We invoke it with `-o -y` to extract silently — no 7z install needed, it's self-contained. Updated tools/environments/local.py::_find_bash candidate order to prefer the PortableGit layout (\bin\bash.exe) with the MinGit layout (\usr\bin\bash.exe) as a fallback so existing installs keep working. 2. **os.execvp "Exec format error" on Windows.** Setup wizard's "Launch hermes chat now? Y" called `os.execvp(["hermes", "chat"])` which on Windows can only swap to real Win32 .exe files — chokes with OSError(8) on .cmd batch shims and Python console-script wrappers. Added a win32 branch in hermes_cli/relaunch.py::relaunch() that uses subprocess.run + sys.exit — functionally identical (user sees "hermes exited, then new hermes started") with one extra PID in play. POSIX path is UNCHANGED — still uses os.execvp for in-place replacement. Catches OSError in the Windows branch and surfaces a "open a new terminal so PATH picks up, then re-run hermes" hint instead of a cryptic traceback. 3. **npm install failures silent on Windows.** The install.ps1 was invoking `npm install --silent 2>&1 | Out-Null` inside a try/catch. PowerShell's try/catch does NOT trigger on non-zero process exit codes — only on unhandled .NET exceptions — so npm failing printed a generic "npm install failed" with zero information about WHY. The silent pipe ate the stderr. Rewrote Install-NodeDeps to: - Resolve npm.cmd via Get-Command (respects PATHEXT) instead of relying on bare `npm` name resolution. - Use Start-Process with -PassThru to capture the actual exit code. - Redirect stderr to a temp log and surface the first ~800 chars of the real npm error when install fails, plus the log path for the full text. - Fail loudly with the right exit code instead of a misleading success. - Bail cleanly with a helpful message when npm isn't on PATH at all. 4. **"True" printing to console after Node check.** `Test-Node` returns $true; installer called it as a bare statement (no assignment, no cast). PowerShell prints bare return values. Wrapped the call in `[void](Test-Node)`. ## Tests - Added 3 new tests in tests/hermes_cli/test_relaunch.py covering the Windows branch: subprocess is called (not execvp), child exit code propagates, OSError surfaces a helpful message. All 23 tests pass (20 existing + 3 new). - 77 Windows-compat tests still pass, POSIX behaviour unchanged. --- hermes_cli/relaunch.py | 44 ++++++- scripts/install.ps1 | 209 +++++++++++++++++++++++------- tests/hermes_cli/test_relaunch.py | 79 ++++++++++- tools/environments/local.py | 15 ++- 4 files changed, 289 insertions(+), 58 deletions(-) diff --git a/hermes_cli/relaunch.py b/hermes_cli/relaunch.py index 32a5dacd2229c..c254184fe9641 100644 --- a/hermes_cli/relaunch.py +++ b/hermes_cli/relaunch.py @@ -142,8 +142,48 @@ def relaunch( preserve_inherited: bool = True, original_argv: Optional[Sequence[str]] = None, ) -> None: - """Replace the current process with a fresh hermes invocation.""" + """Replace the current process with a fresh hermes invocation. + + On POSIX we use ``os.execvp`` which replaces the running process with + the new one in place — same PID, no double-fork. That's what the + relaunch contract wants: "run hermes again as if the user had typed + the new argv". + + Windows has no native exec semantics — ``os.execvp`` on Windows + *emulates* exec by spawning the child and exiting the parent, but + only works when the target is a real Win32 executable. Our target + is usually ``hermes.exe`` (a Python console-script shim that wraps + ``python -m hermes_cli.main``) or a ``.cmd`` batch file, and both + raise ``OSError(8, "Exec format error")`` on Windows' execvp. + + The Windows-correct pattern is: spawn the child with ``subprocess.run`` + (which routes through ``cmd.exe`` via ``shell=False`` + PATHEXT resolution), + wait for it to exit, then propagate its exit code via ``sys.exit``. + That's functionally equivalent — the user sees "hermes exited, then + new hermes started" — just with two PIDs in play instead of one. + """ new_argv = build_relaunch_argv( extra_args, preserve_inherited=preserve_inherited, original_argv=original_argv ) - os.execvp(new_argv[0], new_argv) \ No newline at end of file + if sys.platform == "win32": + # Windows: subprocess + exit, because execvp can't swap to .cmd/.exe shims. + import subprocess + try: + result = subprocess.run(new_argv) + sys.exit(result.returncode) + except KeyboardInterrupt: + sys.exit(130) + except OSError as exc: + # Surface a helpful error rather than the raw OSError — the + # caller used to see ``[Errno 8] Exec format error`` which is + # cryptic. Common causes: ``hermes`` not on PATH yet (install + # hasn't propagated User PATH into this shell) or a stale shim. + print( + f"\nHermes relaunch failed: {exc}\n" + f"Command: {' '.join(new_argv)}\n" + f"Fix: open a new terminal so PATH picks up, then re-run hermes.", + file=sys.stderr, + ) + sys.exit(1) + else: + os.execvp(new_argv[0], new_argv) \ No newline at end of file diff --git a/scripts/install.ps1 b/scripts/install.ps1 index a6154282c6df5..b04efd954baee 100644 --- a/scripts/install.ps1 +++ b/scripts/install.ps1 @@ -200,10 +200,18 @@ function Install-Git { Priority order (deliberately simple — no winget, no registry, no system package manager): 1. Existing ``git`` on PATH — use it as-is (the common fast path). - 2. Download portable **MinGit** from the official git-for-windows GitHub - release and unpack it to ``%LOCALAPPDATA%\hermes\git`` — never touches - system Git, never requires admin, works even on locked-down machines - and machines with a broken system Git install. + 2. Download **PortableGit** from the official git-for-windows GitHub + release (self-extracting 7z.exe) and unpack it to + ``%LOCALAPPDATA%\hermes\git`` — never touches system Git, never + requires admin, works even on locked-down machines and machines + with a broken system Git install. + + **Why PortableGit, not MinGit:** MinGit is the minimal-automation + distribution and ships ONLY ``git.exe`` — no bash, no POSIX utilities. + Hermes needs ``bash.exe`` to run shell commands. PortableGit is the + full Git for Windows distribution without the installer UI; it ships + ``git.exe`` + ``bash.exe`` + ``sh``, ``awk``, ``sed``, ``grep``, ``curl``, + ``ssh``, etc. in ``usr\bin\``. We deliberately skip winget because it fails badly when the system Git install is in a half-installed state (partially registered, or uninstall- @@ -224,61 +232,95 @@ function Install-Git { return $true } - # Download portable MinGit into $HermesHome\git. This always works as - # long as we can reach github.com — no admin, no winget, no reliance on - # the user's possibly-broken system Git install. - Write-Info "Git not found — downloading portable MinGit to $HermesHome\git\ ..." + # Download PortableGit into $HermesHome\git. Always works as long as + # we can reach github.com — no admin, no winget, no reliance on the + # user's possibly-broken system Git install. + Write-Info "Git not found — downloading PortableGit to $HermesHome\git\ ..." Write-Info "(no admin rights required; isolated from any system Git install)" try { - $arch = if ([Environment]::Is64BitOperatingSystem) { "64-bit" } else { "32-bit" } - # Query the GitHub API for the latest git-for-windows release and pick - # the MinGit zip for our architecture. + $arch = if ([Environment]::Is64BitOperatingSystem) { + # Detect ARM64 vs x64 explicitly; PortableGit ships separate assets. + if ($env:PROCESSOR_ARCHITECTURE -eq "ARM64" -or $env:PROCESSOR_ARCHITEW6432 -eq "ARM64") { + "arm64" + } else { + "64-bit" + } + } else { + # PortableGit does not ship a 32-bit build — fall back to MinGit 32-bit + # with a warning that bash-based features will be unavailable. + "32-bit-mingit" + } + $releaseApi = "https://api.github.com/repos/git-for-windows/git/releases/latest" $release = Invoke-RestMethod -Uri $releaseApi -UseBasicParsing -Headers @{ "User-Agent" = "hermes-installer" } - $assetPattern = if ($arch -eq "64-bit") { "MinGit-*-64-bit.zip" } else { "MinGit-*-32-bit.zip" } - # Prefer the non-busybox MinGit — it ships the real bash.exe which is - # what Hermes' _find_bash() looks for. Busybox MinGit replaces bash - # with busybox.exe (ash), which Hermes is NOT tested against. - $asset = $release.assets | Where-Object { $_.name -like $assetPattern -and $_.name -notlike "*busybox*" } | Select-Object -First 1 + + if ($arch -eq "32-bit-mingit") { + Write-Warn "32-bit Windows detected — PortableGit is 64-bit only. Installing MinGit 32-bit as a last resort; bash-dependent Hermes features (terminal tool, agent-browser) will not work on this machine." + $assetPattern = "MinGit-*-32-bit.zip" + $downloadIsZip = $true + } elseif ($arch -eq "arm64") { + $assetPattern = "PortableGit-*-arm64.7z.exe" + $downloadIsZip = $false + } else { + $assetPattern = "PortableGit-*-64-bit.7z.exe" + $downloadIsZip = $false + } + + $asset = $release.assets | Where-Object { $_.name -like $assetPattern } | Select-Object -First 1 if (-not $asset) { - throw "Could not find MinGit zip in latest git-for-windows release" + throw "Could not find $assetPattern in latest git-for-windows release" } $downloadUrl = $asset.browser_download_url - $tmpZip = "$env:TEMP\$($asset.name)" + $downloadExt = if ($downloadIsZip) { "zip" } else { "7z.exe" } + $tmpFile = "$env:TEMP\$($asset.name)" $gitDir = "$HermesHome\git" Write-Info "Downloading $($asset.name) ($([math]::Round($asset.size / 1MB, 1)) MB)..." - Invoke-WebRequest -Uri $downloadUrl -OutFile $tmpZip -UseBasicParsing + Invoke-WebRequest -Uri $downloadUrl -OutFile $tmpFile -UseBasicParsing if (Test-Path $gitDir) { - Write-Info "Removing previous MinGit install at $gitDir ..." + Write-Info "Removing previous Git install at $gitDir ..." Remove-Item -Recurse -Force $gitDir } New-Item -ItemType Directory -Path $gitDir -Force | Out-Null - Expand-Archive -Path $tmpZip -DestinationPath $gitDir -Force - Remove-Item -Force $tmpZip -ErrorAction SilentlyContinue - # MinGit layout: cmd\git.exe + mingw64\bin\git.exe + usr\bin\bash.exe. - # (Note: MinGit puts bash under usr\bin, NOT bin like the full - # Git for Windows installer does. Our _find_bash() in - # tools/environments/local.py checks both locations.) + if ($downloadIsZip) { + Expand-Archive -Path $tmpFile -DestinationPath $gitDir -Force + } else { + # PortableGit is a self-extracting 7z archive. Invoke it with + # `-o -y` (silent) to extract to $gitDir. No 7z install + # required; it's fully self-contained. + Write-Info "Extracting PortableGit to $gitDir ..." + $extractProc = Start-Process -FilePath $tmpFile ` + -ArgumentList "-o`"$gitDir`"", "-y" ` + -NoNewWindow -Wait -PassThru + if ($extractProc.ExitCode -ne 0) { + throw "PortableGit extraction failed (exit code $($extractProc.ExitCode))" + } + } + Remove-Item -Force $tmpFile -ErrorAction SilentlyContinue + + # PortableGit layout: cmd\git.exe + bin\bash.exe + usr\bin\ (coreutils) + # MinGit layout: cmd\git.exe + usr\bin\bash.exe (if present) $gitExe = "$gitDir\cmd\git.exe" if (-not (Test-Path $gitExe)) { - throw "MinGit extraction did not produce git.exe at $gitExe" + throw "Git extraction did not produce git.exe at $gitExe" } - # Add cmd\ to session PATH so the rest of this install run (which - # needs git clone) can see the new git.exe. + # Add to session PATH so the rest of this install run can use git. $env:Path = "$gitDir\cmd;$env:Path" - # Also add to persisted User PATH so fresh shells see it. We add - # cmd\ (for git) and usr\bin\ (for bash + coreutils) — without usr\bin, - # bash-launched commands like `which`, `env`, `grep` etc. that Hermes - # tools call would be missing. - $newPathEntries = @("$gitDir\cmd", "$gitDir\usr\bin") + # Persist to User PATH so fresh shells see it. PortableGit needs + # cmd\ (for git.exe), bin\ (for bash.exe + core tools), and + # usr\bin\ (for perl, ssh, curl, and other POSIX coreutils). + $newPathEntries = @( + "$gitDir\cmd", + "$gitDir\bin", + "$gitDir\usr\bin" + ) $userPath = [Environment]::GetEnvironmentVariable("Path", "User") $userPathItems = if ($userPath) { $userPath -split ";" } else { @() } $changed = $false @@ -297,7 +339,7 @@ function Install-Git { Set-GitBashEnvVar return $true } catch { - Write-Err "Could not install MinGit portable: $_" + Write-Err "Could not install portable Git: $_" Write-Info "" Write-Info "Fallback: install Git manually from https://git-scm.com/download/win" Write-Info "then re-run this installer. Hermes needs Git Bash on Windows to run" @@ -315,12 +357,16 @@ function Set-GitBashEnvVar { #> $candidates = @() - # Our own portable MinGit install is ALWAYS checked first, so a broken + # Our own portable Git install is ALWAYS checked first, so a broken # system Git doesn't hijack us. If the user had a working system Git # we'd have returned early from Install-Git's fast path and never called # this with a system-Git-only installation anyway. - $candidates += "$HermesHome\git\usr\bin\bash.exe" # MinGit layout - $candidates += "$HermesHome\git\bin\bash.exe" # safety — non-MinGit portable layouts + # + # Layouts: + # PortableGit (our default): $HermesHome\git\bin\bash.exe + # MinGit (32-bit fallback): $HermesHome\git\usr\bin\bash.exe + $candidates += "$HermesHome\git\bin\bash.exe" # PortableGit layout (primary) + $candidates += "$HermesHome\git\usr\bin\bash.exe" # MinGit / PortableGit usr\bin fallback # git.exe on PATH can tell us where the install root is $gitCmd = Get-Command git -ErrorAction SilentlyContinue @@ -856,35 +902,94 @@ function Install-NodeDeps { Write-Info "Skipping Node.js dependencies (Node not installed)" return } - + + # Resolve npm.cmd to an absolute path so PATHEXT doesn't bite us when + # the installer runs in a session that hasn't refreshed PATH since the + # Node.js install. Get-Command respects PATHEXT. + $npmCmd = Get-Command npm -ErrorAction SilentlyContinue + if (-not $npmCmd) { + Write-Warn "npm not found on PATH — skipping Node.js dependencies." + Write-Info "Open a new PowerShell window and re-run 'hermes setup tools' later." + return + } + $npmExe = $npmCmd.Source + Push-Location $InstallDir - + if (Test-Path "package.json") { Write-Info "Installing Node.js dependencies (browser tools)..." + # Use Start-Process so we can capture the real exit code. PowerShell's + # try/catch doesn't trigger on npm's non-zero exit — only on an unhandled + # .NET exception (e.g. npm.cmd missing). Capturing stderr to a file + # lets us surface the actual failure reason instead of a generic + # "npm install failed" that hides what went wrong. + $browserLog = "$env:TEMP\hermes-npm-browser-$(Get-Random).log" try { - npm install --silent 2>&1 | Out-Null - Write-Success "Node.js dependencies installed" + $proc = Start-Process -FilePath $npmExe ` + -ArgumentList "install", "--silent" ` + -NoNewWindow -Wait -PassThru ` + -RedirectStandardOutput "NUL" ` + -RedirectStandardError $browserLog + if ($proc.ExitCode -eq 0) { + Write-Success "Node.js dependencies installed" + Remove-Item -Force $browserLog -ErrorAction SilentlyContinue + } else { + Write-Warn "npm install failed (browser tools may not work) — exit code $($proc.ExitCode)" + if (Test-Path $browserLog) { + $errText = (Get-Content $browserLog -Raw -ErrorAction SilentlyContinue) + if ($errText) { + # Show first ~800 chars — enough to see the real cause. + $snippet = if ($errText.Length -gt 800) { $errText.Substring(0, 800) + "..." } else { $errText } + Write-Info " npm stderr:" + foreach ($line in $snippet -split "`n") { + Write-Host " $line" -ForegroundColor DarkGray + } + Write-Info " Full log: $browserLog" + } + } + Write-Info "Run manually later: cd `"$InstallDir`"; npm install" + } } catch { - Write-Warn "npm install failed (browser tools may not work)" + Write-Warn "npm install could not be launched: $_" } } - + # Install TUI dependencies $tuiDir = "$InstallDir\ui-tui" if (Test-Path "$tuiDir\package.json") { Write-Info "Installing TUI dependencies..." Push-Location $tuiDir + $tuiLog = "$env:TEMP\hermes-npm-tui-$(Get-Random).log" try { - npm install --silent 2>&1 | Out-Null - Write-Success "TUI dependencies installed" + $proc = Start-Process -FilePath $npmExe ` + -ArgumentList "install", "--silent" ` + -NoNewWindow -Wait -PassThru ` + -RedirectStandardOutput "NUL" ` + -RedirectStandardError $tuiLog + if ($proc.ExitCode -eq 0) { + Write-Success "TUI dependencies installed" + Remove-Item -Force $tuiLog -ErrorAction SilentlyContinue + } else { + Write-Warn "TUI npm install failed (hermes --tui may not work) — exit code $($proc.ExitCode)" + if (Test-Path $tuiLog) { + $errText = (Get-Content $tuiLog -Raw -ErrorAction SilentlyContinue) + if ($errText) { + $snippet = if ($errText.Length -gt 800) { $errText.Substring(0, 800) + "..." } else { $errText } + Write-Info " npm stderr:" + foreach ($line in $snippet -split "`n") { + Write-Host " $line" -ForegroundColor DarkGray + } + Write-Info " Full log: $tuiLog" + } + } + Write-Info "Run manually later: cd `"$tuiDir`"; npm install" + } } catch { - Write-Warn "TUI npm install failed (hermes --tui may not work)" + Write-Warn "TUI npm install could not be launched: $_" } Pop-Location } - - Pop-Location } @@ -1038,7 +1143,11 @@ function Main { if (-not (Install-Uv)) { throw "uv installation failed — cannot continue" } if (-not (Test-Python)) { throw "Python $PythonVersion not available — cannot continue" } if (-not (Install-Git)) { throw "Git not available and auto-install failed — install from https://git-scm.com/download/win then re-run" } - Test-Node # Auto-installs if missing + # Test-Node always returns $true (sets $script:HasNode on success, emits a + # warning on failure and continues so non-browser installs still work). + # Cast to [void] so the bare return value doesn't print "True" to the + # console between the "Node found" line and the next installer step. + [void](Test-Node) Install-SystemPackages # ripgrep + ffmpeg in one step Install-Repository diff --git a/tests/hermes_cli/test_relaunch.py b/tests/hermes_cli/test_relaunch.py index 33b3ffb4b3847..4658c678fdafa 100644 --- a/tests/hermes_cli/test_relaunch.py +++ b/tests/hermes_cli/test_relaunch.py @@ -152,4 +152,81 @@ def fake_execvp(path, argv): with pytest.raises(SystemExit): relaunch_mod.relaunch(["--resume", "abc"]) - assert calls == [("/usr/bin/hermes", ["/usr/bin/hermes", "--resume", "abc"])] \ No newline at end of file + assert calls == [("/usr/bin/hermes", ["/usr/bin/hermes", "--resume", "abc"])] + + def test_windows_uses_subprocess_not_execvp(self, monkeypatch): + """On Windows, os.execvp raises OSError "Exec format error" when the + target is a .cmd shim or console-script wrapper (both common for + hermes). relaunch() must detect win32 and use subprocess.run + + sys.exit instead.""" + monkeypatch.setattr(relaunch_mod.sys, "platform", "win32") + monkeypatch.setattr(relaunch_mod, "resolve_hermes_bin", lambda: r"C:\Users\test\hermes.exe") + + import subprocess as _subprocess + + captured_argv = [] + + def fake_subprocess_run(argv, **kwargs): + captured_argv.append(list(argv)) + class _Result: + returncode = 0 + return _Result() + + monkeypatch.setattr(_subprocess, "run", fake_subprocess_run) + + # execvp MUST NOT be called on Windows — route must go through subprocess + execvp_calls = [] + + def fake_execvp(*args, **kwargs): + execvp_calls.append(args) + raise AssertionError("os.execvp must not be called on Windows") + + monkeypatch.setattr(relaunch_mod.os, "execvp", fake_execvp) + + with pytest.raises(SystemExit) as exc_info: + relaunch_mod.relaunch(["chat"]) + + assert exc_info.value.code == 0 + assert execvp_calls == [] + assert captured_argv == [[r"C:\Users\test\hermes.exe", "chat"]] + + def test_windows_propagates_child_exit_code(self, monkeypatch): + """A non-zero exit from the child should flow through to sys.exit.""" + monkeypatch.setattr(relaunch_mod.sys, "platform", "win32") + monkeypatch.setattr(relaunch_mod, "resolve_hermes_bin", lambda: r"C:\hermes.exe") + + import subprocess as _subprocess + + def fake_run(argv, **kwargs): + class _Result: + returncode = 42 + return _Result() + + monkeypatch.setattr(_subprocess, "run", fake_run) + monkeypatch.setattr(relaunch_mod.os, "execvp", lambda *a, **kw: None) + + with pytest.raises(SystemExit) as exc_info: + relaunch_mod.relaunch(["chat"]) + assert exc_info.value.code == 42 + + def test_windows_surfaces_oserror_with_help(self, monkeypatch, capsys): + """When subprocess itself raises OSError (file-not-found / bad format), + we must NOT let it bubble up as a cryptic traceback — print a + user-readable hint and sys.exit(1).""" + monkeypatch.setattr(relaunch_mod.sys, "platform", "win32") + monkeypatch.setattr(relaunch_mod, "resolve_hermes_bin", lambda: r"C:\missing.exe") + + import subprocess as _subprocess + + def fake_run(argv, **kwargs): + raise OSError(2, "No such file or directory") + + monkeypatch.setattr(_subprocess, "run", fake_run) + monkeypatch.setattr(relaunch_mod.os, "execvp", lambda *a, **kw: None) + + with pytest.raises(SystemExit) as exc_info: + relaunch_mod.relaunch(["chat"]) + assert exc_info.value.code == 1 + err = capsys.readouterr().err + assert "relaunch failed" in err + assert "open a new terminal" in err.lower() or "path" in err.lower() \ No newline at end of file diff --git a/tools/environments/local.py b/tools/environments/local.py index 39b1cf0821717..527a0539ffcea 100644 --- a/tools/environments/local.py +++ b/tools/environments/local.py @@ -190,16 +190,21 @@ def _find_bash() -> str: if custom and os.path.isfile(custom): return custom - # Prefer our own portable MinGit install first — this way a broken or + # Prefer our own portable Git install first — this way a broken or # partially-uninstalled system Git can't hijack the bash lookup. The - # install.ps1 installer always drops MinGit here when the user didn't - # already have a working system Git. + # install.ps1 installer always drops portable Git here when the user + # didn't already have a working system Git. + # + # Layouts (both checked so upgrades between MinGit and PortableGit + # installs work transparently): + # PortableGit: %LOCALAPPDATA%\hermes\git\bin\bash.exe (primary) + # MinGit: %LOCALAPPDATA%\hermes\git\usr\bin\bash.exe (legacy/32-bit fallback) _local_appdata = os.environ.get("LOCALAPPDATA", "") _hermes_portable_git = os.path.join(_local_appdata, "hermes", "git") if _local_appdata else "" if _hermes_portable_git: for candidate in ( - os.path.join(_hermes_portable_git, "usr", "bin", "bash.exe"), # MinGit layout - os.path.join(_hermes_portable_git, "bin", "bash.exe"), # non-MinGit portable + os.path.join(_hermes_portable_git, "bin", "bash.exe"), # PortableGit (primary) + os.path.join(_hermes_portable_git, "usr", "bin", "bash.exe"), # MinGit fallback ): if os.path.isfile(candidate): return candidate From 787d964ea173b0f523f445396add0f125cc23e12 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 17:51:57 -0700 Subject: [PATCH 06/20] fix(windows): quote cache paths in bash + augment PATH so rg/bash resolve on first launch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three interrelated bugs from teknium1's first interactive chat on Windows: 1. **Snapshot/cwd file paths unquoted in bash command strings.** The session bootstrap and per-command wrapper interpolated ``self._snapshot_path`` / ``self._cwd_file`` unquoted into bash commands like ``export -p > C:/Users/ryanc/.../hermes-snap-xxx.sh``. Git Bash's MSYS2 layer handles ``C:/...`` paths correctly ONLY when quoted; unquoted, the colon and forward-slash get glob-parsed and the redirect targets a bogus path. Symptom: every terminal command emitted two ``C:/Users/.../hermes-snap-*.sh (No such file or directory)`` lines that bled into stdout (``stderr=STDOUT`` on the local backend) and corrupted file contents when the agent wrote to scratch paths via the terminal tool. Fix: ``shlex.quote()`` every interpolation of ``_snapshot_path`` and ``_cwd_file`` in base.py — no-op on POSIX (the paths contain no shell-metachars), critical on Windows. 2. **Stale PATH on first hermes launch after install.** ``install.ps1`` adds the PortableGit ``cmd`` / ``bin`` / ``usr\bin`` directories to the Windows **User** PATH via ``SetEnvironmentVariable(..., "User")``. That write propagates to newly *spawned* processes only — already-running shells (including the one the user types ``hermes`` into immediately after install) retain their old PATH. So hermes starts with a PATH that doesn't include bash, rg, grep, ssh — and ``search_files`` reports "rg/find not available" when the user clearly just installed them. Fix: new ``_augment_path_with_known_tools()`` helper called from ``configure_windows_stdio()`` on startup. Prepends the Hermes-managed Git directories + the WinGet Links directory (where ripgrep lands) to ``os.environ['PATH']`` if they exist on disk but aren't already in PATH. Subsequent subprocess calls (including bash spawns via ``_find_bash()``) inherit the augmented PATH and find everything. No-op on POSIX and when the directories don't exist. 3. **Root cause of "file content corruption".** #1 was the proximate cause. Errors like ``C:/Users/.../hermes-snap-xxx.sh: No such file or directory`` were emitted on stderr by the failed redirect, captured into stdout via ``stderr=subprocess.STDOUT``, and if the agent used terminal commands like ``cat > file`` the leaked error bytes became part of the file. Fixing #1 eliminates this entirely. ## Tests All 77 Windows-compat tests still pass on Linux (POSIX path is shlex.quote('/tmp/foo.sh') → '/tmp/foo.sh' — unchanged). ## Not addressed here (would need a bigger design) - Python file tools (``write_file``, ``read_file``) and the bash-backed terminal tool see DIFFERENT views of ``/tmp`` on Windows. Python treats ``/tmp`` as ``C:\tmp`` (drive-relative), Git Bash's MSYS2 treats it as a virtual mount to the PortableGit install's ``tmp\``. Would need a translation shim in the Python tools to resolve bash-virtual paths to their native-Windows equivalents. Workaround for users today: use absolute native paths (``C:\Users\you\...``) instead of ``/tmp/...`` when crossing between terminal and Python file tools. --- hermes_cli/stdio.py | 72 ++++++++++++++++++++++++++++++++++++++ tools/environments/base.py | 36 +++++++++++++------ 2 files changed, 98 insertions(+), 10 deletions(-) diff --git a/hermes_cli/stdio.py b/hermes_cli/stdio.py index e010304d989de..51c3f7ba53085 100644 --- a/hermes_cli/stdio.py +++ b/hermes_cli/stdio.py @@ -127,6 +127,17 @@ def configure_windows_stdio() -> bool: if _default_editor and not os.environ.get("EDITOR") and not os.environ.get("VISUAL"): os.environ["EDITOR"] = _default_editor + # Augment PATH with the Hermes-managed Git install directories so + # subprocess calls (bash, rg, grep, etc.) resolve even in sessions + # that started before the User PATH broadcast reached them. When + # install.ps1 adds these to User PATH via SetEnvironmentVariable, + # already-running shells don't see the change — which means hermes + # launched from the install session won't find rg / bash / grep + # even though they're "installed". Prepending the known paths here + # closes that gap. No-op when the paths don't exist (e.g. system-Git + # install without Hermes-managed PortableGit). + _augment_path_with_known_tools() + # Flip the console code page first so that any subprocess that # inherits the console (e.g. a launched shell) also sees CP_UTF8. _flip_console_code_page_to_utf8() @@ -178,3 +189,64 @@ def _default_windows_editor() -> str: # On the extreme off-chance notepad is missing (WinPE, Nano Server), fall # back to nothing and let prompt_toolkit's silent no-op do its thing. return "" + + + +def _augment_path_with_known_tools() -> None: + """Prepend well-known Hermes-managed tool directories to os.environ['PATH']. + + Fixes the "User PATH was just updated but my process can't see it" gap on + Windows. When install.ps1 runs, it adds entries like + ``%LOCALAPPDATA%\\hermes\\git\\bin`` to the User PATH via + ``SetEnvironmentVariable(..., "User")``. That write propagates to newly + *spawned* processes only — already-running shells (including the one the + user invokes ``hermes`` from right after install) retain their old PATH. + + Any subprocess Hermes spawns — bash, ``rg``, ``grep``, ``npm`` — inherits + that stale PATH and reports commands as missing even though they're on + disk. Symptom: ``search_files`` reports "rg/find not available" when + the user clearly just installed ripgrep. + + Patch-up strategy: add the known Hermes-managed tool directories to our + PATH at startup so subprocess calls resolve correctly. No-op on POSIX + and when the directories don't exist. The User PATH broadcast still + happens in the background for future shells; this just smooths over + the first-launch gap. + """ + if not is_windows(): + return + + import shutil as _shutil + + local_appdata = os.environ.get("LOCALAPPDATA", "") + if not local_appdata: + return + + # Known tool dirs installed by scripts/install.ps1. Kept in sync with + # the PATH entries that installer adds to User scope — the two lists + # should match so this prefill fully mirrors what a fresh shell would + # see on next launch. + candidate_dirs = [ + os.path.join(local_appdata, "hermes", "git", "cmd"), + os.path.join(local_appdata, "hermes", "git", "bin"), + os.path.join(local_appdata, "hermes", "git", "usr", "bin"), + # Hermes venv Scripts directory — host of the hermes.exe shim itself, + # also where any pip-installed console scripts land. Usually already + # on PATH when the user invokes hermes, but harmless to include. + os.path.join(local_appdata, "hermes", "hermes-agent", "venv", "Scripts"), + # WinGet packages directory — where ``winget install`` drops CLI + # shims by default (ripgrep lands here as rg.exe). Covers the case + # of a system-Git install + ripgrep-via-winget that isn't yet on + # the spawning shell's PATH. + os.path.join(local_appdata, "Microsoft", "WinGet", "Links"), + ] + + existing = os.environ.get("PATH", "") + existing_lower = {p.lower() for p in existing.split(os.pathsep) if p} + prepend = [] + for d in candidate_dirs: + if os.path.isdir(d) and d.lower() not in existing_lower: + prepend.append(d) + + if prepend: + os.environ["PATH"] = os.pathsep.join([*prepend, existing]) diff --git a/tools/environments/base.py b/tools/environments/base.py index f0264ba3c91c4..2420b06b1b6e7 100644 --- a/tools/environments/base.py +++ b/tools/environments/base.py @@ -339,15 +339,24 @@ def init_session(self): # change the working directory (e.g. bashrc `cd ~`). Without this, # pwd -P captures the profile's directory, not terminal.cwd. _quoted_cwd = shlex.quote(self.cwd) + # Quote the snapshot / cwd-file paths so Git Bash on Windows handles + # ``C:/Users/...``-shaped paths without glob-splitting the colon or + # tripping on drive letters. On POSIX this is a no-op (no colons / + # special chars in a /tmp path). Previously unquoted interpolation + # caused ``C:/Users/.../hermes-snap-*.sh: No such file or directory`` + # errors on Windows, leaking via stderr (merged into stdout on Linux + # backends) into every terminal-tool response. + _quoted_snap = shlex.quote(self._snapshot_path) + _quoted_cwd_file = shlex.quote(self._cwd_file) bootstrap = ( - f"export -p > {self._snapshot_path}\n" - f"declare -f | grep -vE '^_[^_]' >> {self._snapshot_path}\n" - f"alias -p >> {self._snapshot_path}\n" - f"echo 'shopt -s expand_aliases' >> {self._snapshot_path}\n" - f"echo 'set +e' >> {self._snapshot_path}\n" - f"echo 'set +u' >> {self._snapshot_path}\n" + f"export -p > {_quoted_snap}\n" + f"declare -f | grep -vE '^_[^_]' >> {_quoted_snap}\n" + f"alias -p >> {_quoted_snap}\n" + f"echo 'shopt -s expand_aliases' >> {_quoted_snap}\n" + f"echo 'set +e' >> {_quoted_snap}\n" + f"echo 'set +u' >> {_quoted_snap}\n" f"builtin cd {_quoted_cwd} 2>/dev/null || true\n" - f"pwd -P > {self._cwd_file} 2>/dev/null || true\n" + f"pwd -P > {_quoted_cwd_file} 2>/dev/null || true\n" f"printf '\\n{self._cwd_marker}%s{self._cwd_marker}\\n' \"$(pwd -P)\"\n" ) try: @@ -389,6 +398,13 @@ def _wrap_command(self, command: str, cwd: str) -> str: re-dumps env vars, and emits CWD markers.""" escaped = command.replace("'", "'\\''") + # Quote the snapshot / cwd-file paths so Git Bash on Windows handles + # ``C:/Users/...``-shaped paths without glob-splitting the colon or + # tripping on drive letters. POSIX paths are unaffected. See + # :meth:`init_session` for the same fix on the bootstrap block. + _quoted_snap = shlex.quote(self._snapshot_path) + _quoted_cwd_file = shlex.quote(self._cwd_file) + parts = [] # Source snapshot (env vars from previous commands). @@ -399,7 +415,7 @@ def _wrap_command(self, command: str, cwd: str) -> str: # silent here, but the redirect is harmless. if self._snapshot_ready: parts.append( - f"source {self._snapshot_path} >/dev/null 2>&1 || true" + f"source {_quoted_snap} >/dev/null 2>&1 || true" ) # Preserve bare ``~`` expansion, but rewrite ``~/...`` through @@ -414,10 +430,10 @@ def _wrap_command(self, command: str, cwd: str) -> str: # Re-dump env vars to snapshot (last-writer-wins for concurrent calls) if self._snapshot_ready: - parts.append(f"export -p > {self._snapshot_path} 2>/dev/null || true") + parts.append(f"export -p > {_quoted_snap} 2>/dev/null || true") # Write CWD to file (local reads this) and stdout marker (remote parses this) - parts.append(f"pwd -P > {self._cwd_file} 2>/dev/null || true") + parts.append(f"pwd -P > {_quoted_cwd_file} 2>/dev/null || true") # Use a distinct line for the marker. The leading \n ensures # the marker starts on its own line even if the command doesn't # end with a newline (e.g. printf 'exact'). We'll strip this From 4d7e72e14dd45d04988ff76271bf9de2dedb9e6e Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:00:59 -0700 Subject: [PATCH 07/20] fix(install.ps1): validate existing repo via git itself + clean up broken stubs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit teknium1 hit "fatal: not in a git directory" on re-install when the previous install left a $InstallDir\.git stub that Test-Path matched but git didn't recognize (three "fatal: not a git repository" lines, then the script exited before touching anything). Two bugs: 1. Test-Path "$InstallDir\.git" was a weak gate — it matches .git whether it's a directory, file, symlink, submodule gitfile, OR a broken stub from a failed previous Remove-Item. Replaced with a real repo probe: Push-Location + git rev-parse --is-inside-work-tree + $LASTEXITCODE check. If git itself can't see a repo, we treat the directory as not-a-repo and fall through to fresh clone. 2. The original update path ignored $LASTEXITCODE. fetch/checkout/pull all emitted fatals but the script kept going. Now each command checks $LASTEXITCODE and throws with an explicit message. Also: when the directory exists but isn't a valid repo, the new code wipes it (Remove-Item -ErrorAction Stop) and falls through to fresh clone, instead of dying with the old "Directory exists but is not a git repository" error. If the wipe itself fails (file locked, hermes still running), we throw with a user-readable "close any programs using files in " hint. Refactored the function to use a $didUpdate flag instead of my earlier draft's early `return` — that was skipping the submodule init block at the bottom of the function. Both the update and fresh-clone paths now fall through to the submodule init step, which is correct (git pull doesn't auto-update submodules). PowerShell structural check: 21 functions defined, braces balanced. --- scripts/install.ps1 | 74 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 57 insertions(+), 17 deletions(-) diff --git a/scripts/install.ps1 b/scripts/install.ps1 index b04efd954baee..dfffb9f82f3be 100644 --- a/scripts/install.ps1 +++ b/scripts/install.ps1 @@ -605,21 +605,61 @@ function Install-SystemPackages { function Install-Repository { Write-Info "Installing to $InstallDir..." - + + $didUpdate = $false + if (Test-Path $InstallDir) { + # Test-Path "$InstallDir\.git" returns True when .git is a file OR a + # directory OR a symlink OR a submodule-style gitfile — and also when + # it's a broken stub left over from a failed previous install (e.g. + # a partial Remove-Item that couldn't delete a locked index.lock). + # Validate the repo properly by asking git itself. If git can't see + # it as a real repo, fall through to the clone-or-zip path. + $repoValid = $false if (Test-Path "$InstallDir\.git") { - Write-Info "Existing installation found, updating..." Push-Location $InstallDir - git -c windows.appendAtomically=false fetch origin - git -c windows.appendAtomically=false checkout $Branch - git -c windows.appendAtomically=false pull origin $Branch + try { + $null = git -c windows.appendAtomically=false rev-parse --is-inside-work-tree 2>$null + if ($LASTEXITCODE -eq 0) { + $repoValid = $true + } + } catch {} Pop-Location + } + + if ($repoValid) { + Write-Info "Existing installation found, updating..." + Push-Location $InstallDir + try { + git -c windows.appendAtomically=false fetch origin + if ($LASTEXITCODE -ne 0) { throw "git fetch failed (exit $LASTEXITCODE)" } + git -c windows.appendAtomically=false checkout $Branch + if ($LASTEXITCODE -ne 0) { throw "git checkout $Branch failed (exit $LASTEXITCODE)" } + git -c windows.appendAtomically=false pull origin $Branch + if ($LASTEXITCODE -ne 0) { throw "git pull failed (exit $LASTEXITCODE)" } + } finally { + Pop-Location + } + $didUpdate = $true } else { - Write-Err "Directory exists but is not a git repository: $InstallDir" - Write-Info "Remove it or choose a different directory with -InstallDir" - throw "Directory exists but is not a git repository: $InstallDir" + # Directory exists but isn't a usable git repo. Wipe it and + # fall through to a fresh clone. A leftover ``.git`` stub from + # a partial uninstall used to lock the installer into the + # "update" branch forever, emitting three ``fatal: not a git + # repository`` errors and failing with "not in a git directory". + Write-Warn "Existing directory at $InstallDir is not a valid git repo — replacing it." + try { + Remove-Item -Recurse -Force $InstallDir -ErrorAction Stop + } catch { + Write-Err "Could not remove $InstallDir : $_" + Write-Info "Close any programs that might be using files in $InstallDir (editors," + Write-Info "terminals, running hermes processes) and try again." + throw + } } - } else { + } + + if (-not $didUpdate) { $cloneSuccess = $false # Fix Windows git "copy-fd: write returned: Invalid argument" error. @@ -640,7 +680,7 @@ function Install-Repository { if ($LASTEXITCODE -eq 0) { $cloneSuccess = $true } } catch { } $env:GIT_SSH_COMMAND = $null - + if (-not $cloneSuccess) { if (Test-Path $InstallDir) { Remove-Item -Recurse -Force $InstallDir -ErrorAction SilentlyContinue } Write-Info "SSH failed, trying HTTPS..." @@ -658,18 +698,18 @@ function Install-Repository { $zipUrl = "https://github.com/NousResearch/hermes-agent/archive/refs/heads/$Branch.zip" $zipPath = "$env:TEMP\hermes-agent-$Branch.zip" $extractPath = "$env:TEMP\hermes-agent-extract" - + Invoke-WebRequest -Uri $zipUrl -OutFile $zipPath -UseBasicParsing if (Test-Path $extractPath) { Remove-Item -Recurse -Force $extractPath } Expand-Archive -Path $zipPath -DestinationPath $extractPath -Force - + # GitHub ZIPs extract to repo-branch/ subdirectory $extractedDir = Get-ChildItem $extractPath -Directory | Select-Object -First 1 if ($extractedDir) { New-Item -ItemType Directory -Force -Path (Split-Path $InstallDir) -ErrorAction SilentlyContinue | Out-Null Move-Item $extractedDir.FullName $InstallDir -Force Write-Success "Downloaded and extracted" - + # Initialize git repo so updates work later Push-Location $InstallDir git -c windows.appendAtomically=false init 2>$null @@ -677,10 +717,10 @@ function Install-Repository { git remote add origin $RepoUrlHttps 2>$null Pop-Location Write-Success "Git repo initialized for future updates" - + $cloneSuccess = $true } - + # Cleanup temp files Remove-Item -Force $zipPath -ErrorAction SilentlyContinue Remove-Item -Recurse -Force $extractPath -ErrorAction SilentlyContinue @@ -693,7 +733,7 @@ function Install-Repository { throw "Failed to download repository (tried git clone SSH, HTTPS, and ZIP)" } } - + # Set per-repo config (harmless if it fails) Push-Location $InstallDir git -c windows.appendAtomically=false config windows.appendAtomically false 2>$null @@ -707,7 +747,7 @@ function Install-Repository { Write-Success "Submodules ready" } Pop-Location - + Write-Success "Repository ready" } From 225b57f31430e19545fd63d265b46ce765fb6701 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:05:35 -0700 Subject: [PATCH 08/20] fix(install.ps1): step out of $InstallDir before touching it + harden repo probe User hit 'fatal: not in a git directory' on re-install because: 1. They ran Remove-Item -Force $env:LOCALAPPDATA\hermes -ErrorAction SilentlyContinue WHILE cd'd inside the install dir. Windows silently refuses to delete a directory any shell is currently cd'd inside and leaves the skeleton intact, but the -ErrorAction SilentlyContinue swallowed every partial-delete failure so they thought the wipe succeeded. 2. The installer then walked into Install-Repository, saw $InstallDir still exists with a partial .git stub, my repo-validity probe returned success (the probe's git rev-parse may have exit-code-zeroed in a way I didn't expect), and the real git fetch died with three 'fatal: not a git repository' errors. Two fixes belt-and-braces: - Main() now cds to $env:USERPROFILE at start if the current shell is inside $InstallDir. Harmless when the user ran from elsewhere; critical when they didn't. This alone fixes the user's case. - Install-Repository's 'is this a valid repo' probe now runs BOTH git rev-parse --is-inside-work-tree AND git status, resets $LASTEXITCODE before each to avoid picking up a stale 0, and requires BOTH to succeed. Also requires rev-parse's output to match 'true' (not just exit 0) to rule out exit-0-with-empty-output edge cases. --- scripts/install.ps1 | 42 +++++++++++++++++++++++++++++++++++------- 1 file changed, 35 insertions(+), 7 deletions(-) diff --git a/scripts/install.ps1 b/scripts/install.ps1 index dfffb9f82f3be..7bacfe8daeaa3 100644 --- a/scripts/install.ps1 +++ b/scripts/install.ps1 @@ -613,14 +613,24 @@ function Install-Repository { # directory OR a symlink OR a submodule-style gitfile — and also when # it's a broken stub left over from a failed previous install (e.g. # a partial Remove-Item that couldn't delete a locked index.lock). - # Validate the repo properly by asking git itself. If git can't see - # it as a real repo, fall through to the clone-or-zip path. + # Validate the repo properly by asking git itself. Two checks + # belt-and-braces: rev-parse AND git status. If either fails the + # repo is broken and we fall through to a fresh clone. $repoValid = $false if (Test-Path "$InstallDir\.git") { Push-Location $InstallDir try { - $null = git -c windows.appendAtomically=false rev-parse --is-inside-work-tree 2>$null - if ($LASTEXITCODE -eq 0) { + # Reset $LASTEXITCODE before the probe so we don't pick up + # a stale 0 from an earlier git call in this session. + $global:LASTEXITCODE = 0 + $revParseOut = & git -c windows.appendAtomically=false rev-parse --is-inside-work-tree 2>&1 + $revParseOk = ($LASTEXITCODE -eq 0) -and ($revParseOut -match "true") + + $global:LASTEXITCODE = 0 + $null = & git -c windows.appendAtomically=false status --short 2>&1 + $statusOk = ($LASTEXITCODE -eq 0) + + if ($revParseOk -and $statusOk) { $repoValid = $true } } catch {} @@ -1179,7 +1189,25 @@ function Write-Completion { function Main { Write-Banner - + + # Windows refuses to delete a directory any shell is currently cd'd + # inside — and silently leaves orphan files behind, which then wedge + # "is this a valid git repo" probes on re-install. If the current + # working dir is under $InstallDir, step out to the user's home + # BEFORE doing anything else. Harmless when the user ran the + # installer from somewhere else. + try { + $currentResolved = (Get-Location).ProviderPath + $installResolved = $null + if (Test-Path $InstallDir) { + $installResolved = (Resolve-Path $InstallDir -ErrorAction SilentlyContinue).ProviderPath + } + if ($installResolved -and $currentResolved.ToLower().StartsWith($installResolved.ToLower())) { + Write-Info "Stepping out of $InstallDir so Windows can replace files there if needed..." + Set-Location $env:USERPROFILE + } + } catch {} + if (-not (Install-Uv)) { throw "uv installation failed — cannot continue" } if (-not (Test-Python)) { throw "Python $PythonVersion not available — cannot continue" } if (-not (Install-Git)) { throw "Git not available and auto-install failed — install from https://git-scm.com/download/win then re-run" } @@ -1189,7 +1217,7 @@ function Main { # console between the "Node found" line and the next installer step. [void](Test-Node) Install-SystemPackages # ripgrep + ffmpeg in one step - + Install-Repository Install-Venv Install-Dependencies @@ -1198,7 +1226,7 @@ function Main { Copy-ConfigTemplates Invoke-SetupWizard Start-GatewayIfConfigured - + Write-Completion } From 2c7b479d16b05bee7db163d37537621644e25b75 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:11:43 -0700 Subject: [PATCH 09/20] fix(windows): %1 install error, patch CRLF false-negative, SOUL.md BOM MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three bugs from teknium1's successful install + diagnostic chat on Windows: 1. **Start-Process -FilePath npm.cmd fails with "%1 is not a valid Win32 application".** Start-Process bypasses cmd.exe and PATHEXT to call CreateProcessW directly, which refuses .cmd batch shims. Switched Install-NodeDeps to use PowerShell's invocation operator (``& $npmExe install --silent *> $log``) which DOES honour PATHEXT. Extracted a ``_Run-NpmInstall`` helper so the browser + TUI paths share the same logic. Captures $LASTEXITCODE correctly, still surfaces the real stderr on failure with a log-file pointer for the full output. 2. **patch tool returns false-negative on Windows due to CRLF round-trip.** Root cause was upstream of patch: ``subprocess.Popen(..., text=True, stdin=PIPE)`` on Windows translates ``\\n`` → ``\\r\\n`` when data flows through the stdin pipe. ``_pipe_stdin()`` was writing the patch's new_content string through a text-mode pipe, bash then wrote those CRLF bytes to disk, and patch's post-write verify compared the on-disk CRLF bytes against the original LF-only string — fail. Fixed in two places for defense in depth: - ``_pipe_stdin()`` now writes through ``proc.stdin.buffer`` with explicit UTF-8 encoding, bypassing Python's newline translation on every platform. No behaviour change on POSIX (bytes are identical) but stops the CRLF injection on Windows. - ``patch_replace``'s post-write verify normalizes CRLF→LF on both sides before comparing, so even if some future backend still translates newlines the patch tool won't report a bogus failure. 3. **SOUL.md gets a UTF-8 BOM on Windows PowerShell 5.1.** ``Set-Content -Encoding UTF8`` on PS5.1 writes UTF-8 WITH a byte-order-mark (changed in PS7 via ``utf8NoBOM``). Hermes's prompt-injection scanner sees the BOM (U+FEFF invisible char) and refuses to load the file, so SOUL.md's persona instructions never get applied. Fixed by writing the file via ``[System.IO.File]::WriteAllText`` with an explicit ``UTF8Encoding($false)`` — BOM-free on every PowerShell version. All POSIX behaviour verified unchanged: 198 tests pass across test_file_operations, test_local_env_cwd_recovery, test_code_execution, test_windows_native_support, test_windows_compat. --- scripts/install.ps1 | 127 +++++++++++++++++-------------------- tools/environments/base.py | 27 +++++++- tools/file_operations.py | 14 +++- 3 files changed, 95 insertions(+), 73 deletions(-) diff --git a/scripts/install.ps1 b/scripts/install.ps1 index 7bacfe8daeaa3..13af38953bf8e 100644 --- a/scripts/install.ps1 +++ b/scripts/install.ps1 @@ -903,13 +903,21 @@ function Copy-ConfigTemplates { Write-Info "~/.hermes/config.yaml already exists, keeping it" } - # Create SOUL.md if it doesn't exist (global persona file) + # Create SOUL.md if it doesn't exist (global persona file). + # IMPORTANT: write without a BOM. Windows PowerShell 5.1's + # ``Set-Content -Encoding UTF8`` writes UTF-8 WITH a byte-order-mark + # (the default PS5 behaviour), and Hermes's prompt-injection scanner + # flags the BOM as an invisible unicode character and refuses to + # load the file. PS7's ``-Encoding utf8NoBOM`` fixes that but we + # don't control which PowerShell version the user has. Go direct + # to .NET with an explicit UTF8Encoding($false) — BOM-free on every + # PowerShell version. $soulPath = "$HermesHome\SOUL.md" if (-not (Test-Path $soulPath)) { - @" + $soulContent = @" # Hermes Agent Persona - -"@ | Set-Content -Path $soulPath -Encoding UTF8 +"@ + $utf8NoBom = New-Object System.Text.UTF8Encoding($false) + [System.IO.File]::WriteAllText($soulPath, $soulContent, $utf8NoBom) Write-Success "Created ~/.hermes/SOUL.md (edit to customize personality)" } @@ -964,83 +974,64 @@ function Install-NodeDeps { } $npmExe = $npmCmd.Source - Push-Location $InstallDir - - if (Test-Path "package.json") { - Write-Info "Installing Node.js dependencies (browser tools)..." - # Use Start-Process so we can capture the real exit code. PowerShell's - # try/catch doesn't trigger on npm's non-zero exit — only on an unhandled - # .NET exception (e.g. npm.cmd missing). Capturing stderr to a file - # lets us surface the actual failure reason instead of a generic - # "npm install failed" that hides what went wrong. - $browserLog = "$env:TEMP\hermes-npm-browser-$(Get-Random).log" + # Helper: run "npm install" in a given directory and surface the real + # error when it fails. Returns $true on success. + # + # Implementation note: ``Start-Process -FilePath npm.cmd`` fails with + # ``%1 is not a valid Win32 application`` on some PowerShell versions + # because Start-Process bypasses cmd.exe / PATHEXT and expects a real + # PE file. The invocation-operator ``& $npmExe`` routes through the + # PowerShell command pipeline which DOES honour .cmd batch shims, so + # it works uniformly for npm.cmd, npx.cmd, and bare .exe files. + function _Run-NpmInstall([string]$label, [string]$installDir, [string]$logPath, [string]$npmPath) { + Push-Location $installDir try { - $proc = Start-Process -FilePath $npmExe ` - -ArgumentList "install", "--silent" ` - -NoNewWindow -Wait -PassThru ` - -RedirectStandardOutput "NUL" ` - -RedirectStandardError $browserLog - if ($proc.ExitCode -eq 0) { - Write-Success "Node.js dependencies installed" - Remove-Item -Force $browserLog -ErrorAction SilentlyContinue - } else { - Write-Warn "npm install failed (browser tools may not work) — exit code $($proc.ExitCode)" - if (Test-Path $browserLog) { - $errText = (Get-Content $browserLog -Raw -ErrorAction SilentlyContinue) - if ($errText) { - # Show first ~800 chars — enough to see the real cause. - $snippet = if ($errText.Length -gt 800) { $errText.Substring(0, 800) + "..." } else { $errText } - Write-Info " npm stderr:" - foreach ($line in $snippet -split "`n") { - Write-Host " $line" -ForegroundColor DarkGray - } - Write-Info " Full log: $browserLog" + # Redirect ALL output streams to the log file via 2>&1 and then + # ``Tee-Object`` / ``Out-File``. Simpler approach: call npm + # with output redirected and inspect $LASTEXITCODE afterwards. + & $npmPath install --silent *> $logPath + $code = $LASTEXITCODE + if ($code -eq 0) { + Write-Success "$label dependencies installed" + Remove-Item -Force $logPath -ErrorAction SilentlyContinue + return $true + } + Write-Warn "$label npm install failed — exit code $code" + if (Test-Path $logPath) { + $errText = (Get-Content $logPath -Raw -ErrorAction SilentlyContinue) + if ($errText) { + $snippet = if ($errText.Length -gt 1200) { $errText.Substring(0, 1200) + "..." } else { $errText } + Write-Info " npm output:" + foreach ($line in $snippet -split "`n") { + Write-Host " $line" -ForegroundColor DarkGray } + Write-Info " Full log: $logPath" } - Write-Info "Run manually later: cd `"$InstallDir`"; npm install" } + Write-Info "Run manually later: cd `"$installDir`"; npm install" + return $false } catch { - Write-Warn "npm install could not be launched: $_" + Write-Warn "$label npm install could not be launched: $_" + return $false + } finally { + Pop-Location } } - # Install TUI dependencies + # Browser tools + if (Test-Path "$InstallDir\package.json") { + Write-Info "Installing Node.js dependencies (browser tools)..." + $browserLog = "$env:TEMP\hermes-npm-browser-$(Get-Random).log" + [void](_Run-NpmInstall "Browser tools" $InstallDir $browserLog $npmExe) + } + + # TUI $tuiDir = "$InstallDir\ui-tui" if (Test-Path "$tuiDir\package.json") { Write-Info "Installing TUI dependencies..." - Push-Location $tuiDir $tuiLog = "$env:TEMP\hermes-npm-tui-$(Get-Random).log" - try { - $proc = Start-Process -FilePath $npmExe ` - -ArgumentList "install", "--silent" ` - -NoNewWindow -Wait -PassThru ` - -RedirectStandardOutput "NUL" ` - -RedirectStandardError $tuiLog - if ($proc.ExitCode -eq 0) { - Write-Success "TUI dependencies installed" - Remove-Item -Force $tuiLog -ErrorAction SilentlyContinue - } else { - Write-Warn "TUI npm install failed (hermes --tui may not work) — exit code $($proc.ExitCode)" - if (Test-Path $tuiLog) { - $errText = (Get-Content $tuiLog -Raw -ErrorAction SilentlyContinue) - if ($errText) { - $snippet = if ($errText.Length -gt 800) { $errText.Substring(0, 800) + "..." } else { $errText } - Write-Info " npm stderr:" - foreach ($line in $snippet -split "`n") { - Write-Host " $line" -ForegroundColor DarkGray - } - Write-Info " Full log: $tuiLog" - } - } - Write-Info "Run manually later: cd `"$tuiDir`"; npm install" - } - } catch { - Write-Warn "TUI npm install could not be launched: $_" - } - Pop-Location + [void](_Run-NpmInstall "TUI" $tuiDir $tuiLog $npmExe) } - - Pop-Location } function Invoke-SetupWizard { diff --git a/tools/environments/base.py b/tools/environments/base.py index 2420b06b1b6e7..a4fbea7b2f99f 100644 --- a/tools/environments/base.py +++ b/tools/environments/base.py @@ -99,12 +99,33 @@ def get_sandbox_dir() -> Path: def _pipe_stdin(proc: subprocess.Popen, data: str) -> None: - """Write *data* to proc.stdin on a daemon thread to avoid pipe-buffer deadlocks.""" + """Write *data* to proc.stdin on a daemon thread to avoid pipe-buffer deadlocks. + + On Windows, text-mode stdin (``text=True`` / ``encoding="utf-8"``) + translates ``\\n`` → ``\\r\\n`` as the data flows through the pipe — + which corrupts every write_file / patch call because the bytes that + land on disk include injected carriage returns. The file IS created, + but every subsequent byte-count / content compare against the + caller's ``\\n``-only string fails. + + Workaround: write through ``proc.stdin.buffer`` (the underlying byte + buffer), encoding to UTF-8 ourselves. That bypasses Python's + newline translation entirely on every platform. No behaviour change + on POSIX — the byte sequence is identical to what text-mode would + produce there. + """ def _write(): try: - proc.stdin.write(data) - proc.stdin.close() + # proc.stdin is a TextIOWrapper when text=True was set on the + # Popen. Its ``.buffer`` attribute is the raw BufferedWriter + # that bypasses newline translation. When Popen was created + # in byte mode, proc.stdin is already a BufferedWriter with + # no ``.buffer`` attribute — fall back to .write() directly. + raw = data.encode("utf-8") if isinstance(data, str) else data + target = getattr(proc.stdin, "buffer", proc.stdin) + target.write(raw) + target.close() except (BrokenPipeError, OSError): pass diff --git a/tools/file_operations.py b/tools/file_operations.py index 92a948eaaf7b5..022943d9f0ea9 100644 --- a/tools/file_operations.py +++ b/tools/file_operations.py @@ -966,11 +966,21 @@ def patch_replace(self, path: str, old_string: str, new_string: str, verify_result = self._exec(verify_cmd) if verify_result.exit_code != 0: return PatchResult(error=f"Post-write verification failed: could not re-read {path}") - if verify_result.stdout != new_content: + # Normalize line endings before comparing. On Windows, Python's + # default text-mode ``open()`` translates ``\n`` → ``\r\n`` on + # write, so the file on disk legitimately holds CRLFs while our + # ``new_content`` string has bare LFs. Without this normalization + # every patch on Windows returns a bogus "wrote 39, read 42" + # false-negative even though the edit landed correctly. POSIX + # backends don't translate, so this is a no-op there. + _verify_stdout_normalized = verify_result.stdout.replace("\r\n", "\n").replace("\r", "\n") + _new_content_normalized = new_content.replace("\r\n", "\n").replace("\r", "\n") + if _verify_stdout_normalized != _new_content_normalized: return PatchResult(error=( f"Post-write verification failed for {path}: on-disk content " f"differs from intended write " - f"(wrote {len(new_content)} chars, read back {len(verify_result.stdout)}). " + f"(wrote {len(_new_content_normalized)} chars, read back " + f"{len(_verify_stdout_normalized)} chars after normalizing line endings). " "The patch did not persist. Re-read the file and try again." )) From 2e403bd0a416c9c5ae41d15deb59e3c74272d1a7 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:17:31 -0700 Subject: [PATCH 10/20] =?UTF-8?q?fix(windows):=20enable=20execute=5Fcode?= =?UTF-8?q?=20=E2=80=94=20stale=20AF=5FUNIX=20gate=20was=20blocking=20the?= =?UTF-8?q?=20tool?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit teknium1 noticed execute_code was missing from his enabled tools on Windows. Root cause: tools/code_execution_tool.py set ``SANDBOX_AVAILABLE = sys.platform != \"win32\"`` as a module-level constant, originally because the RPC transport required AF_UNIX. We added loopback TCP fallback for the sandbox in commit eeb723fff (and covered it in the Windows TCP tests), but forgot to lift the availability gate. So execute_code was still invisible via the check_fn path on Windows. - SANDBOX_AVAILABLE is now True unconditionally (it's still checked — a future platform could flip it off via monkeypatch/env if needed). - Error message when disabled no longer mentions Windows specifically, just says 'sandbox is unavailable in this environment'. - test_windows_returns_error updated: patches SANDBOX_AVAILABLE=False directly (which was always its real intent) and asserts on 'unavailable' instead of 'Windows'. Tests: 171 code-execution + windows-compat tests pass, no regressions. --- tests/tools/test_code_execution.py | 10 ++++++++-- tools/code_execution_tool.py | 10 +++++++--- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/tests/tools/test_code_execution.py b/tests/tools/test_code_execution.py index a5806046583a8..2d08265fb7b65 100644 --- a/tests/tools/test_code_execution.py +++ b/tests/tools/test_code_execution.py @@ -774,11 +774,17 @@ def test_timezone_not_set_when_empty(self): class TestExecuteCodeEdgeCases(unittest.TestCase): def test_windows_returns_error(self): - """On Windows (or when SANDBOX_AVAILABLE is False), returns error JSON.""" + """When SANDBOX_AVAILABLE is False (e.g. when the backend deems + the sandbox unusable for this environment), execute_code returns + an error JSON with a readable message pointing the caller at + regular tool calls. Previously this was a Windows-only gate; + execute_code now works on Windows via loopback TCP, so the + error is only emitted when SANDBOX_AVAILABLE is explicitly + flipped off (e.g. for future platform-specific disables).""" with patch("tools.code_execution_tool.SANDBOX_AVAILABLE", False): result = json.loads(execute_code("print('hi')", task_id="test")) self.assertIn("error", result) - self.assertIn("Windows", result["error"]) + self.assertIn("unavailable", result["error"].lower()) def test_whitespace_only_code(self): result = json.loads(execute_code(" \n\t ", task_id="test")) diff --git a/tools/code_execution_tool.py b/tools/code_execution_tool.py index 89d42484c2bc6..8e9112056ff08 100644 --- a/tools/code_execution_tool.py +++ b/tools/code_execution_tool.py @@ -47,10 +47,13 @@ _IS_WINDOWS = platform.system() == "Windows" from typing import Any, Dict, List, Optional -# Availability gate: UDS requires a POSIX OS +# Availability gate. On Windows we fall back to loopback TCP for the +# sandbox RPC transport (AF_UNIX is unreliable on Windows Python) — see +# ``_use_tcp_rpc`` in ``_execute_local`` below. That makes execute_code +# available on every platform Hermes itself runs on. logger = logging.getLogger(__name__) -SANDBOX_AVAILABLE = sys.platform != "win32" +SANDBOX_AVAILABLE = True # The 7 tools allowed inside the sandbox. The intersection of this list # and the session's enabled tools determines which stubs are generated. @@ -971,7 +974,8 @@ def execute_code( """ if not SANDBOX_AVAILABLE: return json.dumps({ - "error": "execute_code is not available on Windows. Use normal tool calls instead." + "error": "execute_code sandbox is unavailable in this environment. " + "Use normal tool calls (terminal, read_file, write_file, ...) instead." }) if not code or not code.strip(): From f0d2516a30ab4de4f5d04fc146af195ae2ac25b1 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:29:17 -0700 Subject: [PATCH 11/20] fix(windows): prefer npm.cmd over npm.ps1, skip .py argv0 in relaunch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two fixes from teknium1's next install run: 1. **npm install: "npm.ps1 cannot be loaded because running scripts is disabled on this system."** Get-Command's default PATHEXT ordering picked up ``npm.ps1`` (the PowerShell shim) ahead of ``npm.cmd`` (the batch shim). Most Windows users have PowerShell's execution policy set to Restricted or RemoteSigned, which blocks unsigned ``.ps1`` files. ``npm.cmd`` has no such restriction and works universally. Install-NodeDeps now detects when Get-Command returned npm.ps1, looks for a sibling npm.cmd in the same directory, and prefers it. Prints an info line so the user sees why. Emits a warning + hint if only npm.ps1 is available. 2. **"Launch hermes chat now? Y" crashes with "%1 is not a valid Win32 application" on Windows installs.** The setup wizard calls ``relaunch(["chat"])``; ``resolve_hermes_bin()`` returned ``sys.argv[0]`` which was ``...\\hermes_cli\\main.py`` (because hermes was launched via ``python -m hermes_cli.main`` during setup). On Windows, ``os.access(script.py, os.X_OK)`` returns True because PATHEXT lists ``.py`` when the Python launcher is registered — but ``subprocess.run([script.py, ...])`` can't actually execute a ``.py`` directly. CreateProcessW needs a real PE file. Fixed ``resolve_hermes_bin`` to reject ``.py``/``.pyc`` argv0 values on Windows specifically. Falls through to ``shutil.which("hermes")`` (hermes.exe in the venv Scripts dir) or, as a final fallback, lets build_relaunch_argv build ``[sys.executable, "-m", "hermes_cli.main"]`` which is bulletproof. POSIX behaviour unchanged — ``.py`` argv0 with a shebang + chmod+x is still a valid exec target there. 3 new tests cover the Windows paths: .py argv0 + hermes.exe on PATH → returns hermes.exe; .py argv0 + no PATH → returns None (caller uses python -m); POSIX + executable .py → still accepted. 26 relaunch tests pass, no POSIX regressions. --- hermes_cli/relaunch.py | 20 +++++++++-- scripts/install.ps1 | 24 +++++++++++-- tests/hermes_cli/test_relaunch.py | 56 ++++++++++++++++++++++++++++++- 3 files changed, 94 insertions(+), 6 deletions(-) diff --git a/hermes_cli/relaunch.py b/hermes_cli/relaunch.py index c254184fe9641..a5a8431fbe334 100644 --- a/hermes_cli/relaunch.py +++ b/hermes_cli/relaunch.py @@ -84,18 +84,34 @@ def resolve_hermes_bin() -> Optional[str]: 1. ``sys.argv[0]`` if it resolves to a real executable. 2. ``shutil.which("hermes")`` on PATH. 3. ``None`` → caller should fall back to ``python -m hermes_cli.main``. + + Windows note: ``os.access(path, os.X_OK)`` returns True for ``.py`` and + ``.pyc`` files on Windows (the OS treats anything listed in PATHEXT as + executable, and Python files are often registered there). But + ``subprocess.run([script.py, ...])`` can't actually execute a .py + directly — CreateProcessW needs a real .exe, not a script associated + with the Python launcher. On Windows we therefore skip the argv[0] + fast-path when it points at a .py file and fall through to either + ``hermes.exe`` on PATH or the ``sys.executable -m hermes_cli.main`` + fallback. """ argv0 = sys.argv[0] + _is_windows = sys.platform == "win32" + + def _is_python_script(p: str) -> bool: + return p.lower().endswith((".py", ".pyc")) # Absolute path to an executable (covers nix store, venv wrappers, etc.) if os.path.isabs(argv0) and os.path.isfile(argv0) and os.access(argv0, os.X_OK): - return argv0 + if not (_is_windows and _is_python_script(argv0)): + return argv0 # Relative path — resolve against CWD if not argv0.startswith("-") and os.path.isfile(argv0): abs_path = os.path.abspath(argv0) if os.access(abs_path, os.X_OK): - return abs_path + if not (_is_windows and _is_python_script(abs_path)): + return abs_path # PATH lookup path_bin = shutil.which("hermes") diff --git a/scripts/install.ps1 b/scripts/install.ps1 index 13af38953bf8e..2ffca87adb14d 100644 --- a/scripts/install.ps1 +++ b/scripts/install.ps1 @@ -963,9 +963,17 @@ function Install-NodeDeps { return } - # Resolve npm.cmd to an absolute path so PATHEXT doesn't bite us when - # the installer runs in a session that hasn't refreshed PATH since the - # Node.js install. Get-Command respects PATHEXT. + # Resolve npm explicitly to npm.cmd, NOT npm.ps1. Node.js on Windows + # ships BOTH npm.cmd (a batch shim) and npm.ps1 (a PowerShell shim). + # Get-Command's default ordering picks whichever comes first in PATHEXT, + # and on many systems that's .ps1 — but .ps1 requires scripts to be + # enabled in PowerShell's execution policy, which most Windows users + # don't have (the Restricted / RemoteSigned default blocks unsigned + # .ps1 files). .cmd has no such restriction and works on every box. + # + # Strategy: look next to the npm shim we found and prefer npm.cmd if + # it exists in the same directory. Fall back to whatever Get-Command + # returned if we can't find a .cmd sibling. $npmCmd = Get-Command npm -ErrorAction SilentlyContinue if (-not $npmCmd) { Write-Warn "npm not found on PATH — skipping Node.js dependencies." @@ -973,6 +981,16 @@ function Install-NodeDeps { return } $npmExe = $npmCmd.Source + if ($npmExe -like "*.ps1") { + $npmCmdSibling = Join-Path (Split-Path $npmExe -Parent) "npm.cmd" + if (Test-Path $npmCmdSibling) { + Write-Info "Using npm.cmd (PowerShell execution policy blocks npm.ps1)" + $npmExe = $npmCmdSibling + } else { + Write-Warn "Only npm.ps1 available — install may fail if script execution is disabled." + Write-Info " If it fails, either enable PS script execution or install Node via winget." + } + } # Helper: run "npm install" in a given directory and surface the real # error when it fails. Returns $true on success. diff --git a/tests/hermes_cli/test_relaunch.py b/tests/hermes_cli/test_relaunch.py index 4658c678fdafa..1b4f4ff154756 100644 --- a/tests/hermes_cli/test_relaunch.py +++ b/tests/hermes_cli/test_relaunch.py @@ -229,4 +229,58 @@ def fake_run(argv, **kwargs): assert exc_info.value.code == 1 err = capsys.readouterr().err assert "relaunch failed" in err - assert "open a new terminal" in err.lower() or "path" in err.lower() \ No newline at end of file + assert "open a new terminal" in err.lower() or "path" in err.lower() + + +class TestResolveHermesBinWindowsPyGuard: + """On Windows, resolve_hermes_bin MUST NOT return a .py path. + os.access(x, os.X_OK) returns True for .py files on Windows because + PATHEXT includes .py when the Python launcher is installed — but + subprocess.run can't actually exec a .py directly, so the relaunch + would fail with the cryptic "%1 is not a valid Win32 application" error. + """ + + def test_windows_rejects_py_argv0_falls_through_to_path(self, monkeypatch, tmp_path): + """On Windows, if sys.argv[0] is a .py file, we must skip the + argv[0] fast-path and fall through to PATH / python -m.""" + # Build a fake .py script that "passes" the isfile + X_OK checks. + script = tmp_path / "main.py" + script.write_text("# stub") + + monkeypatch.setattr(relaunch_mod.sys, "platform", "win32") + monkeypatch.setattr(relaunch_mod.sys, "argv", [str(script), "chat"]) + # Force PATH lookup to return a hermes.exe so the test doesn't + # exercise the None-fallback path (that's a separate test). + monkeypatch.setattr( + relaunch_mod.shutil, "which", + lambda name: r"C:\venv\Scripts\hermes.exe" if name == "hermes" else None, + ) + + bin_path = relaunch_mod.resolve_hermes_bin() + # Must NOT be the .py — must be the hermes.exe PATH entry. + assert bin_path == r"C:\venv\Scripts\hermes.exe" + + def test_posix_still_accepts_py_argv0(self, monkeypatch, tmp_path): + """POSIX behaviour unchanged: argv[0] pointing at an executable + script (including .py with a shebang + chmod +x) is fine to return + because POSIX exec can route through the shebang line.""" + if sys.platform == "win32": + pytest.skip("POSIX semantics") + script = tmp_path / "hermes" + script.write_text("#!/usr/bin/env python3\n") + script.chmod(0o755) + monkeypatch.setattr(relaunch_mod.sys, "argv", [str(script), "chat"]) + assert relaunch_mod.resolve_hermes_bin() == str(script) + + def test_windows_py_argv0_with_no_hermes_on_path_returns_none(self, monkeypatch, tmp_path): + """Bulletproof fallback: if argv0 is .py on Windows AND hermes.exe + isn't on PATH, return None so the caller falls back to + python -m hermes_cli.main.""" + script = tmp_path / "main.py" + script.write_text("# stub") + + monkeypatch.setattr(relaunch_mod.sys, "platform", "win32") + monkeypatch.setattr(relaunch_mod.sys, "argv", [str(script), "chat"]) + monkeypatch.setattr(relaunch_mod.shutil, "which", lambda name: None) + + assert relaunch_mod.resolve_hermes_bin() is None From fab984c7f8a1240332729f509b3a0daca50bb0a6 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:39:38 -0700 Subject: [PATCH 12/20] execute_code: pass through Windows OS-essential env vars MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sandbox's env scrubbing was dropping SYSTEMROOT, WINDIR, COMSPEC, APPDATA, etc. On Windows this broke the child process before any RPC could happen: OSError: [WinError 10106] The requested service provider could not be loaded or initialized Python's socket module uses SYSTEMROOT to locate mswsock.dll during Winsock initialization. Without it, socket.socket(AF_INET, SOCK_STREAM) fails — and the existing loopback-TCP fallback for Windows couldn't work. Fix: add a small Windows-only allowlist (_WINDOWS_ESSENTIAL_ENV_VARS) matched by exact uppercase name, after the existing secret-substring block. The secret block still runs first, so the allowlist cannot be used to exfiltrate credentials. Also extract the env scrubber into a testable helper (_scrub_child_env) that takes is_windows as a parameter, so the logic can be unit-tested on any OS. Live Winsock smoke test verifies that a child spawned with the scrubbed env can now create an AF_INET socket on a real Windows host; the test is guarded by sys.platform == 'win32' so POSIX CI stays green. --- .../tools/test_code_execution_windows_env.py | 243 ++++++++++++++++++ tools/code_execution_tool.py | 107 ++++++-- 2 files changed, 327 insertions(+), 23 deletions(-) create mode 100644 tests/tools/test_code_execution_windows_env.py diff --git a/tests/tools/test_code_execution_windows_env.py b/tests/tools/test_code_execution_windows_env.py new file mode 100644 index 0000000000000..1e1f1eb989d15 --- /dev/null +++ b/tests/tools/test_code_execution_windows_env.py @@ -0,0 +1,243 @@ +"""Tests for execute_code env scrubbing on Windows. + +On Windows the child process needs a small set of OS-essential env vars +(SYSTEMROOT, WINDIR, COMSPEC, ...) to run. Without SYSTEMROOT in particular, +``socket.socket(AF_INET, SOCK_STREAM)`` fails inside the sandbox with +WinError 10106 (Winsock can't locate mswsock.dll) and no tool call over +loopback TCP can ever succeed. + +These tests cover ``_scrub_child_env`` directly so they run on every OS +— the logic is conditional on a passed-in ``is_windows`` flag, not on +the host platform. We also keep a live Winsock smoke test that only runs +on a real Windows host. +""" + +import os +import socket +import subprocess +import sys +import textwrap +import unittest.mock as mock + +import pytest + +from tools.code_execution_tool import ( + _SAFE_ENV_PREFIXES, + _SECRET_SUBSTRINGS, + _WINDOWS_ESSENTIAL_ENV_VARS, + _scrub_child_env, +) + + +def _no_passthrough(_name): + return False + + +class TestWindowsEssentialAllowlist: + """The allowlist itself — contents, shape, and invariants.""" + + def test_contains_winsock_required_vars(self): + # Without SYSTEMROOT the child cannot initialize Winsock. + assert "SYSTEMROOT" in _WINDOWS_ESSENTIAL_ENV_VARS + + def test_contains_subprocess_required_vars(self): + # Without COMSPEC, subprocess can't resolve the default shell. + assert "COMSPEC" in _WINDOWS_ESSENTIAL_ENV_VARS + + def test_contains_user_profile_vars(self): + # os.path.expanduser("~") on Windows uses USERPROFILE. + assert "USERPROFILE" in _WINDOWS_ESSENTIAL_ENV_VARS + assert "APPDATA" in _WINDOWS_ESSENTIAL_ENV_VARS + assert "LOCALAPPDATA" in _WINDOWS_ESSENTIAL_ENV_VARS + + def test_contains_only_uppercase_names(self): + # Windows env var names are case-insensitive but we canonicalize to + # uppercase for the membership check (``k.upper() in _WINDOWS_...``). + for name in _WINDOWS_ESSENTIAL_ENV_VARS: + assert name == name.upper(), f"{name!r} should be uppercase" + + def test_no_overlap_with_secret_substrings(self): + # Sanity: none of the essential OS vars should look like secrets. + # If this ever fires, we'd have a precedence ordering bug (secrets + # are blocked *before* the essentials check). + for name in _WINDOWS_ESSENTIAL_ENV_VARS: + assert not any(s in name for s in _SECRET_SUBSTRINGS), ( + f"{name!r} looks secret-like — would be blocked before the " + "essentials allowlist can match" + ) + + +class TestScrubChildEnvWindows: + """Verify _scrub_child_env passes Windows essentials through when + is_windows=True and blocks them when is_windows=False (so POSIX hosts + don't inherit pointless Windows vars).""" + + def _sample_windows_env(self): + """A realistic subset of what os.environ looks like on Windows.""" + return { + "SYSTEMROOT": r"C:\Windows", + "SystemDrive": "C:", # Windows preserves native case + "WINDIR": r"C:\Windows", + "ComSpec": r"C:\Windows\System32\cmd.exe", + "PATHEXT": ".COM;.EXE;.BAT;.CMD;.PY", + "USERPROFILE": r"C:\Users\alice", + "APPDATA": r"C:\Users\alice\AppData\Roaming", + "LOCALAPPDATA": r"C:\Users\alice\AppData\Local", + "PATH": r"C:\Windows\System32;C:\Python311", + "HOME": r"C:\Users\alice", + "TEMP": r"C:\Users\alice\AppData\Local\Temp", + # Should still be blocked: + "OPENAI_API_KEY": "sk-secret", + "GITHUB_TOKEN": "ghp_secret", + "MY_PASSWORD": "hunter2", + # Not matched by any rule — should be dropped on both OSes: + "RANDOM_UNKNOWN_VAR": "value", + } + + def test_windows_essentials_passed_through_when_is_windows_true(self): + env = self._sample_windows_env() + scrubbed = _scrub_child_env(env, + is_passthrough=_no_passthrough, + is_windows=True) + + # Every essential var from the sample env should survive. + assert scrubbed["SYSTEMROOT"] == r"C:\Windows" + assert scrubbed["SystemDrive"] == "C:" # case preserved + assert scrubbed["WINDIR"] == r"C:\Windows" + assert scrubbed["ComSpec"] == r"C:\Windows\System32\cmd.exe" + assert scrubbed["PATHEXT"] == ".COM;.EXE;.BAT;.CMD;.PY" + assert scrubbed["USERPROFILE"] == r"C:\Users\alice" + assert scrubbed["APPDATA"].endswith("Roaming") + assert scrubbed["LOCALAPPDATA"].endswith("Local") + + # Safe-prefix vars still pass (baseline behavior). + assert "PATH" in scrubbed + assert "HOME" in scrubbed + assert "TEMP" in scrubbed + + def test_secrets_still_blocked_on_windows(self): + """The Windows allowlist must NOT defeat the secret-substring block. + + This is the key security invariant: essentials are allowed by + *exact name*, and the secret-substring block runs before the + essentials check anyway, so a variable named e.g. ``API_KEY`` can + never sneak through just because we added Windows support. + """ + env = self._sample_windows_env() + scrubbed = _scrub_child_env(env, + is_passthrough=_no_passthrough, + is_windows=True) + assert "OPENAI_API_KEY" not in scrubbed + assert "GITHUB_TOKEN" not in scrubbed + assert "MY_PASSWORD" not in scrubbed + + def test_unknown_vars_still_dropped_on_windows(self): + env = self._sample_windows_env() + scrubbed = _scrub_child_env(env, + is_passthrough=_no_passthrough, + is_windows=True) + assert "RANDOM_UNKNOWN_VAR" not in scrubbed + + def test_essentials_blocked_when_is_windows_false(self): + """On POSIX hosts, Windows-specific vars should not pass — they + have no meaning and could confuse child tooling.""" + env = self._sample_windows_env() + scrubbed = _scrub_child_env(env, + is_passthrough=_no_passthrough, + is_windows=False) + # Safe prefixes still match (PATH, HOME, TEMP). + assert "PATH" in scrubbed + assert "HOME" in scrubbed + assert "TEMP" in scrubbed + # But Windows OS vars should be dropped. + assert "SYSTEMROOT" not in scrubbed + assert "WINDIR" not in scrubbed + assert "ComSpec" not in scrubbed + assert "APPDATA" not in scrubbed + + def test_case_insensitive_essential_match(self): + """Windows env var names are case-insensitive at the OS level but + Python preserves whatever case os.environ reported. The scrubber + must normalize to uppercase for the membership check.""" + env = { + "SystemRoot": r"C:\Windows", # mixed case + "comspec": r"C:\Windows\System32\cmd.exe", # lowercase + "APPDATA": r"C:\Users\x\AppData\Roaming", # uppercase + } + scrubbed = _scrub_child_env(env, + is_passthrough=_no_passthrough, + is_windows=True) + assert "SystemRoot" in scrubbed + assert "comspec" in scrubbed + assert "APPDATA" in scrubbed + + +class TestScrubChildEnvPassthroughInteraction: + """The passthrough hook runs *before* the secret block, so a skill + can legitimately forward a third-party API key. The Windows + essentials addition must not interfere with that.""" + + def test_passthrough_wins_over_secret_block(self): + env = {"TENOR_API_KEY": "x", "PATH": "/bin"} + scrubbed = _scrub_child_env(env, + is_passthrough=lambda k: k == "TENOR_API_KEY", + is_windows=False) + assert scrubbed.get("TENOR_API_KEY") == "x" + assert scrubbed.get("PATH") == "/bin" + + def test_passthrough_still_works_on_windows(self): + env = { + "TENOR_API_KEY": "x", + "SYSTEMROOT": r"C:\Windows", + "OPENAI_API_KEY": "sk-secret", # not passthrough + } + scrubbed = _scrub_child_env( + env, + is_passthrough=lambda k: k == "TENOR_API_KEY", + is_windows=True, + ) + assert scrubbed.get("TENOR_API_KEY") == "x" + assert scrubbed.get("SYSTEMROOT") == r"C:\Windows" + assert "OPENAI_API_KEY" not in scrubbed + + +@pytest.mark.skipif( + sys.platform != "win32", + reason="Winsock-specific regression — only meaningful on Windows", +) +class TestWindowsSocketSmokeTest: + """Integration-ish smoke test: spawn a child Python with a scrubbed + env and confirm it can create an AF_INET socket. This is the + regression that motivated the fix — without SYSTEMROOT the child + hits WinError 10106 before any RPC is attempted.""" + + def test_child_can_create_socket_with_scrubbed_env(self): + scrubbed = _scrub_child_env(os.environ, is_passthrough=_no_passthrough) + + # Build a tiny child script that simply opens an AF_INET socket. + script = textwrap.dedent(""" + import socket, sys + try: + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.close() + print("OK") + sys.exit(0) + except OSError as exc: + print(f"FAIL: {exc}") + sys.exit(1) + """).strip() + + result = subprocess.run( + [sys.executable, "-c", script], + env=scrubbed, + capture_output=True, + text=True, + timeout=15, + ) + assert result.returncode == 0, ( + f"Child failed to create socket with scrubbed env:\n" + f" stdout={result.stdout!r}\n" + f" stderr={result.stderr!r}\n" + f" scrubbed keys={sorted(scrubbed.keys())}" + ) + assert "OK" in result.stdout diff --git a/tools/code_execution_tool.py b/tools/code_execution_tool.py index 8e9112056ff08..e115f5dc1fa70 100644 --- a/tools/code_execution_tool.py +++ b/tools/code_execution_tool.py @@ -73,6 +73,85 @@ MAX_STDOUT_BYTES = 50_000 # 50 KB MAX_STDERR_BYTES = 10_000 # 10 KB +# Environment variable scrubbing rules (shared between the local + remote +# backends). Secret-substring block is applied first; anything left must +# match either a safe prefix or, on Windows, an OS-essential name. +_SAFE_ENV_PREFIXES = ("PATH", "HOME", "USER", "LANG", "LC_", "TERM", + "TMPDIR", "TMP", "TEMP", "SHELL", "LOGNAME", + "XDG_", "PYTHONPATH", "VIRTUAL_ENV", "CONDA", + "HERMES_") +_SECRET_SUBSTRINGS = ("KEY", "TOKEN", "SECRET", "PASSWORD", "CREDENTIAL", + "PASSWD", "AUTH") + +# Windows-only: a handful of variables are required by the OS/CRT itself. +# Without them, even stdlib calls like ``socket.socket()`` fail with +# WinError 10106 (Winsock can't locate mswsock.dll) and ``subprocess`` +# can't resolve cmd.exe. These are well-known OS paths, not secrets, so +# we allow them through by exact name. The _SECRET_SUBSTRINGS block +# still runs as a safety net (none of these names match those substrings). +_WINDOWS_ESSENTIAL_ENV_VARS = frozenset({ + "SYSTEMROOT", # %SYSTEMROOT%\System32 — Winsock needs this + "SYSTEMDRIVE", # C: (or wherever Windows lives) + "WINDIR", # usually same as SYSTEMROOT + "COMSPEC", # cmd.exe path — subprocess shell=True needs it + "PATHEXT", # .COM;.EXE;.BAT;... — shell lookup + "OS", # "Windows_NT" — some tools gate on this + "PROCESSOR_ARCHITECTURE", + "NUMBER_OF_PROCESSORS", + "PUBLIC", # C:\Users\Public + "ALLUSERSPROFILE", # C:\ProgramData — some stdlib paths use it + "PROGRAMDATA", # C:\ProgramData + "PROGRAMFILES", + "PROGRAMFILES(X86)", + "PROGRAMW6432", + "APPDATA", # %USERPROFILE%\AppData\Roaming — Python uses it + "LOCALAPPDATA", # %USERPROFILE%\AppData\Local + "USERPROFILE", # C:\Users\ — Python's expanduser uses it + "USERDOMAIN", + "USERNAME", + "HOMEDRIVE", # C: + "HOMEPATH", # \Users\ + "COMPUTERNAME", +}) + + +def _scrub_child_env(source_env, is_passthrough=None, is_windows=None): + """Produce the scrubbed child-process env for execute_code. + + Rules (order matters): + 1. Passthrough vars (skill- or config-declared) always pass. + 2. Secret-substring names (KEY/TOKEN/etc.) are blocked. + 3. Names matching a safe prefix pass. + 4. On Windows, a small OS-essential allowlist passes by exact name + — without these the child can't even create a socket or spawn a + subprocess. + + Extracted into a helper so tests can exercise the logic without + spawning a subprocess. + """ + if is_passthrough is None: + try: + from tools.env_passthrough import is_env_passthrough as _ep + except Exception: + _ep = lambda _: False # noqa: E731 + is_passthrough = _ep + if is_windows is None: + is_windows = _IS_WINDOWS + + scrubbed = {} + for k, v in source_env.items(): + if is_passthrough(k): + scrubbed[k] = v + continue + if any(s in k.upper() for s in _SECRET_SUBSTRINGS): + continue + if any(k.startswith(p) for p in _SAFE_ENV_PREFIXES): + scrubbed[k] = v + continue + if is_windows and k.upper() in _WINDOWS_ESSENTIAL_ENV_VARS: + scrubbed[k] = v + return scrubbed + def check_sandbox_requirements() -> bool: """Code execution sandbox requires a POSIX OS for Unix domain sockets.""" @@ -1079,29 +1158,11 @@ def execute_code( # generated scripts. The child accesses tools via RPC, not direct API. # Exception: env vars declared by loaded skills (via env_passthrough # registry) or explicitly allowed by the user in config.yaml - # (terminal.env_passthrough) are passed through. - _SAFE_ENV_PREFIXES = ("PATH", "HOME", "USER", "LANG", "LC_", "TERM", - "TMPDIR", "TMP", "TEMP", "SHELL", "LOGNAME", - "XDG_", "PYTHONPATH", "VIRTUAL_ENV", "CONDA", - "HERMES_") - _SECRET_SUBSTRINGS = ("KEY", "TOKEN", "SECRET", "PASSWORD", "CREDENTIAL", - "PASSWD", "AUTH") - try: - from tools.env_passthrough import is_env_passthrough as _is_passthrough - except Exception: - _is_passthrough = lambda _: False # noqa: E731 - child_env = {} - for k, v in os.environ.items(): - # Passthrough vars (skill-declared or user-configured) always pass. - if _is_passthrough(k): - child_env[k] = v - continue - # Block vars with secret-like names. - if any(s in k.upper() for s in _SECRET_SUBSTRINGS): - continue - # Allow vars with known safe prefixes. - if any(k.startswith(p) for p in _SAFE_ENV_PREFIXES): - child_env[k] = v + # (terminal.env_passthrough) are passed through. On Windows, a small + # OS-essential allowlist (SYSTEMROOT, WINDIR, COMSPEC, ...) is also + # passed through — without those, the child can't create a socket + # or spawn a subprocess. See ``_scrub_child_env`` for the rules. + child_env = _scrub_child_env(os.environ) child_env["HERMES_RPC_SOCKET"] = rpc_endpoint child_env["PYTHONDONTWRITEBYTECODE"] = "1" # Ensure the hermes-agent root is importable in the sandbox so From 668e4b8d7ec1158b37a3f5d630901e98236cca6b Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:45:34 -0700 Subject: [PATCH 13/20] tests: lock in POSIX-equivalence guard for execute_code env scrubber MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds TestPosixEquivalence to test_code_execution_windows_env.py. The class pins the invariant that _scrub_child_env(env, is_windows=False) produces byte-for-byte identical output to the pre-refactor inline scrubber, across a matrix of: - 2 synthetic envs (POSIX-shaped, Windows-shaped-on-POSIX) - 3 passthrough rules (none, single-var, everything) - 1 real-os.environ check on whatever platform runs the test Plus a superset sanity check: is_windows=True must keep everything is_windows=False keeps, and any extras must come from the _WINDOWS_ESSENTIAL_ENV_VARS allowlist. Rationale: the previous commit refactored the env-scrubbing inline block into a helper. Future changes to that helper must not silently regress POSIX behavior — if someone needs to change it, they update _legacy_posix_scrubber in lockstep so the churn is visible in review. All 21 tests in the file pass locally on Windows (pytest 9.0.3). 8 of them are parametrized equivalence checks that run on every OS. --- .../tools/test_code_execution_windows_env.py | 159 ++++++++++++++++++ 1 file changed, 159 insertions(+) diff --git a/tests/tools/test_code_execution_windows_env.py b/tests/tools/test_code_execution_windows_env.py index 1e1f1eb989d15..185f662fdd179 100644 --- a/tests/tools/test_code_execution_windows_env.py +++ b/tests/tools/test_code_execution_windows_env.py @@ -241,3 +241,162 @@ def test_child_can_create_socket_with_scrubbed_env(self): f" scrubbed keys={sorted(scrubbed.keys())}" ) assert "OK" in result.stdout + + +# --------------------------------------------------------------------------- +# POSIX equivalence guard +# --------------------------------------------------------------------------- + +def _legacy_posix_scrubber(source_env, is_passthrough): + """Verbatim copy of the pre-Windows-fix inline scrubbing logic. + + This is the oracle used by TestPosixEquivalence to prove the refactor + did not change POSIX behavior. DO NOT edit this to "match" a future + production change — if _scrub_child_env's POSIX behavior legitimately + needs to evolve, delete this function and adjust the equivalence test + on purpose, so the churn is visible in review. + """ + _SAFE_ENV_PREFIXES = ("PATH", "HOME", "USER", "LANG", "LC_", "TERM", + "TMPDIR", "TMP", "TEMP", "SHELL", "LOGNAME", + "XDG_", "PYTHONPATH", "VIRTUAL_ENV", "CONDA", + "HERMES_") + _SECRET_SUBSTRINGS = ("KEY", "TOKEN", "SECRET", "PASSWORD", "CREDENTIAL", + "PASSWD", "AUTH") + out = {} + for k, v in source_env.items(): + if is_passthrough(k): + out[k] = v + continue + if any(s in k.upper() for s in _SECRET_SUBSTRINGS): + continue + if any(k.startswith(p) for p in _SAFE_ENV_PREFIXES): + out[k] = v + return out + + +class TestPosixEquivalence: + """Lock in the invariant that _scrub_child_env(env, is_windows=False) + behaves *bit-for-bit identically* to the pre-refactor inline scrubber. + + If this ever fails, it means somebody changed POSIX env-scrubbing + behavior — maybe on purpose, maybe not. Either way it should land + as a deliberate, reviewed change (update _legacy_posix_scrubber + above in the same PR). + + Rationale: the Windows-essentials patch refactored the scrubber into + a helper. Linux/macOS must not regress. This class gates that. + """ + + _POSIX_SYNTHETIC_ENV = { + # Safe-prefix matches + "PATH": "/usr/bin:/bin", + "HOME": "/home/alice", + "USER": "alice", + "LANG": "en_US.UTF-8", + "LC_CTYPE": "en_US.UTF-8", + "TERM": "xterm-256color", + "SHELL": "/bin/zsh", + "LOGNAME": "alice", + "TMPDIR": "/tmp", + "XDG_RUNTIME_DIR": "/run/user/1000", + "XDG_CONFIG_HOME": "/home/alice/.config", + "PYTHONPATH": "/opt/lib", + "VIRTUAL_ENV": "/home/alice/.venv", + "CONDA_PREFIX": "/opt/conda", + "HERMES_HOME": "/home/alice/.hermes", + "HERMES_INTERACTIVE": "1", + # Secret-substring blocks + "OPENAI_API_KEY": "sk-xxx", + "GITHUB_TOKEN": "ghp_xxx", + "AWS_SECRET_ACCESS_KEY": "yyy", + "MY_PASSWORD": "hunter2", + # Uncategorized — must be dropped + "RANDOM_UNKNOWN": "drop-me", + "DISPLAY": ":0", + "SSH_AUTH_SOCK": "/run/user/1000/ssh-agent", + # Passthrough candidate (also matches secret block by default) + "TENOR_API_KEY": "tenor-xxx", + } + + _WINDOWS_SYNTHETIC_ENV = { + # Windows-essential names (must be dropped on POSIX, passed on Win) + "SYSTEMROOT": r"C:\Windows", + "SystemDrive": "C:", + "WINDIR": r"C:\Windows", + "ComSpec": r"C:\Windows\System32\cmd.exe", + "PATHEXT": ".COM;.EXE;.BAT", + "USERPROFILE": r"C:\Users\alice", + "APPDATA": r"C:\Users\alice\AppData\Roaming", + "LOCALAPPDATA": r"C:\Users\alice\AppData\Local", + # Safe-prefix matches (cross-platform) + "PATH": r"C:\Python311;C:\Windows\System32", + "HOME": r"C:\Users\alice", + "TEMP": r"C:\Users\alice\AppData\Local\Temp", + # Secret-looking (always blocked) + "OPENAI_API_KEY": "sk-xxx", + "GITHUB_TOKEN": "ghp_xxx", + } + + @pytest.mark.parametrize("env_name,env", [ + ("posix_synthetic", _POSIX_SYNTHETIC_ENV), + ("windows_synthetic_on_posix", _WINDOWS_SYNTHETIC_ENV), + ]) + @pytest.mark.parametrize("pt_name,pt", [ + ("no_passthrough", lambda _: False), + ("tenor_passthrough", lambda k: k == "TENOR_API_KEY"), + ("all_passthrough", lambda _: True), + ]) + def test_posix_behavior_unchanged(self, env_name, env, pt_name, pt): + """For every combination of (env shape × passthrough rule), the + new helper with is_windows=False must produce the exact same dict + as the legacy inline scrubber. + + We parametrize over three passthrough rules to cover the full + surface: no passthrough, single-var passthrough (the common + skill-registered case), and everything-passes (edge case that + could expose precedence bugs).""" + expected = _legacy_posix_scrubber(env, pt) + actual = _scrub_child_env(env, is_passthrough=pt, is_windows=False) + assert actual == expected, ( + f"POSIX behavior regressed for env={env_name}, passthrough={pt_name}\n" + f" only in legacy: {sorted(set(expected) - set(actual))}\n" + f" only in new: {sorted(set(actual) - set(expected))}\n" + f" value diffs: {[k for k in expected if k in actual and expected[k] != actual[k]]}" + ) + + def test_posix_behavior_unchanged_on_real_os_environ(self): + """Bonus check against the actual os.environ of the host running + the test. This covers vars we might not have thought to put in + the synthetic fixtures.""" + expected = _legacy_posix_scrubber(os.environ, lambda _: False) + actual = _scrub_child_env(os.environ, + is_passthrough=lambda _: False, + is_windows=False) + assert actual == expected, ( + "POSIX-mode scrubber diverged from legacy behavior on real " + f"os.environ (host platform={sys.platform})" + ) + + def test_windows_mode_is_strict_superset_of_posix_mode(self): + """Correctness check on the NEW behavior: is_windows=True must + keep everything POSIX mode keeps, and *may* add Windows + essentials. It must never drop a var that POSIX mode would keep + — if it did, we'd have broken same-host reuse of the scrubber.""" + env = {**self._POSIX_SYNTHETIC_ENV, **self._WINDOWS_SYNTHETIC_ENV} + posix_result = _scrub_child_env(env, + is_passthrough=lambda _: False, + is_windows=False) + windows_result = _scrub_child_env(env, + is_passthrough=lambda _: False, + is_windows=True) + missing = set(posix_result) - set(windows_result) + assert not missing, ( + f"is_windows=True dropped vars that is_windows=False kept: {missing}" + ) + # And any extras must come from the Windows essentials allowlist. + extras = set(windows_result) - set(posix_result) + for k in extras: + assert k.upper() in _WINDOWS_ESSENTIAL_ENV_VARS, ( + f"Unexpected extra var in windows-mode output: {k} " + f"(not in _WINDOWS_ESSENTIAL_ENV_VARS)" + ) From 8798bea31fc8af2f8c94e2a9861d27702a513872 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:52:59 -0700 Subject: [PATCH 14/20] execute_code: write sandbox files as UTF-8 on Windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second Windows-specific sandbox bug (WinError 10106 was the first): after the env-scrub fix let the child start, it immediately failed to import hermes_tools with: SyntaxError: (unicode error) 'utf-8' codec can't decode byte 0x97 in position 154: invalid start byte Root cause: _execute_local wrote the generated hermes_tools.py stub and the user's script.py via open(path, 'w') without encoding=. On Windows the default text-mode encoding is cp1252 (system locale), which encodes em-dashes (used in the stub's docstrings) as 0x97. Python then decodes source files as UTF-8 (PEP 3120) on import, chokes on 0x97, and the sandbox dies before any tool call. Fix: pass encoding='utf-8' to all four file opens in the code_execution path — the two staging writes in _execute_local (hermes_tools.py + script.py) and the two RPC file-transport reads/writes in the generated remote stub. JSON is ASCII-safe for most payloads but tool results (terminal output, web_extract content) routinely carry non-ASCII. Tests added (4): - test_stub_and_script_writes_specify_utf8 — source grep guard - test_file_rpc_stub_uses_utf8 — generated remote stub check - test_stub_source_roundtrips_through_utf8 — concrete round-trip - test_windows_default_encoding_would_have_failed — negative control (skips on modern Python builds where default is already UTF-8 compatible, but retained for platforms where the regression could return) 24/25 tests pass on Windows 3.11 (negative control skips because this Python build handles em-dashes via cp1252 subset — the fix is still correct, just the corruption path isn't always triggerable). --- .../tools/test_code_execution_windows_env.py | 163 ++++++++++++++++++ tools/code_execution_tool.py | 22 ++- 2 files changed, 179 insertions(+), 6 deletions(-) diff --git a/tests/tools/test_code_execution_windows_env.py b/tests/tools/test_code_execution_windows_env.py index 185f662fdd179..aca7a31da1542 100644 --- a/tests/tools/test_code_execution_windows_env.py +++ b/tests/tools/test_code_execution_windows_env.py @@ -10,6 +10,14 @@ — the logic is conditional on a passed-in ``is_windows`` flag, not on the host platform. We also keep a live Winsock smoke test that only runs on a real Windows host. + +Also covers the companion Windows bug: the sandbox writes +``hermes_tools.py`` and ``script.py`` into a temp dir, and those files +must be written as UTF-8 on every platform — the generated stub contains +em-dash/en-dash characters in docstrings, and the default ``open(path, "w")`` +on Windows uses the system locale (cp1252 typically), corrupting those +bytes. The child then fails to import with a SyntaxError: +``'utf-8' codec can't decode byte 0x97``. """ import os @@ -400,3 +408,158 @@ def test_windows_mode_is_strict_superset_of_posix_mode(self): f"Unexpected extra var in windows-mode output: {k} " f"(not in _WINDOWS_ESSENTIAL_ENV_VARS)" ) + + +# --------------------------------------------------------------------------- +# UTF-8 file-write regression test +# --------------------------------------------------------------------------- +# +# The sandbox writes two Python files into a temp dir — the generated +# ``hermes_tools.py`` stub, and the LLM's ``script.py``. Both contain +# non-ASCII characters in practice: the stub has em-dashes in docstrings +# ("``tcp://host:port`` — the parent falls back..."), and user scripts +# routinely contain non-ASCII strings, comments, or Unicode identifiers. +# +# On Windows, ``open(path, "w")`` without encoding= uses the system locale +# (cp1252 on US/UK installs), which cannot encode em-dashes. Python then +# tries to decode the file as UTF-8 when importing it (PEP 3120), fails, +# and the sandbox aborts with: +# +# SyntaxError: (unicode error) 'utf-8' codec can't decode byte 0x97 +# in position N: invalid start byte +# +# This was the *second* Windows-specific bug (WinError 10106 was the first). +# The fix is to always pass ``encoding="utf-8"`` when writing Python source. + + +class TestSandboxWritesUtf8: + """Verify the file-write call sites use UTF-8 explicitly, not the + platform default. We check the source of ``execute_code`` rather + than spawning a real sandbox because the latter needs a full agent + context — but the code inspection is deterministic and fast.""" + + def test_stub_and_script_writes_specify_utf8(self): + """Both ``hermes_tools.py`` and ``script.py`` writes in + ``_execute_local`` must pass ``encoding="utf-8"``.""" + import tools.code_execution_tool as cet + src = open(cet.__file__, encoding="utf-8").read() + + # There should be no ``open(path, "w")`` without encoding= for + # the two staging files. Grep-style check: find every write of + # a .py file inside tmpdir and assert the line also contains + # ``encoding="utf-8"`` within a short window. + import re + pattern = re.compile( + r'open\(\s*os\.path\.join\(\s*tmpdir\s*,\s*"[^"]+\.py"\s*\)\s*,\s*"w"[^)]*\)' + ) + for match in pattern.finditer(src): + line = match.group(0) + assert 'encoding="utf-8"' in line or "encoding='utf-8'" in line, ( + f"Sandbox file write missing encoding=\"utf-8\" on Windows: {line!r}" + ) + + def test_file_rpc_stub_uses_utf8(self): + """The file-based RPC transport stub (used by remote backends) + reads/writes JSON response files. Those must also specify UTF-8 + so non-ASCII tool results survive the round-trip intact.""" + from tools.code_execution_tool import generate_hermes_tools_module + stub = generate_hermes_tools_module(["terminal"], transport="file") + # The generated stub should open response + request files as UTF-8. + assert 'encoding="utf-8"' in stub, ( + "File-based RPC stub does not specify encoding=\"utf-8\" — " + "will corrupt non-ASCII tool results on non-UTF-8 locales." + ) + + def test_stub_source_roundtrips_through_utf8(self): + """Concrete regression: write the generated stub to a temp file + using ``encoding="utf-8"``, then parse it. This is what the + sandbox does, and it must succeed even when the stub contains + em-dashes (which it does — check the transport-header docstring). + """ + from tools.code_execution_tool import generate_hermes_tools_module + import tempfile, ast + stub = generate_hermes_tools_module( + ["terminal", "read_file", "write_file"], transport="uds" + ) + # Sanity: stub actually contains a non-ASCII character, otherwise + # this test wouldn't prove anything meaningful. + non_ascii = [c for c in stub if ord(c) > 127] + assert non_ascii, ( + "Generated stub is pure ASCII — test is meaningless. If the " + "stub's docstrings have lost their em-dashes, update this " + "assertion, but be aware the original regression is no longer " + "covered." + ) + + with tempfile.NamedTemporaryFile( + mode="w", suffix=".py", delete=False, encoding="utf-8" + ) as f: + f.write(stub) + tmp_path = f.name + + try: + # Re-read and parse exactly like the child Python would. + with open(tmp_path, encoding="utf-8") as fh: + round_tripped = fh.read() + assert round_tripped == stub, "UTF-8 round-trip corrupted the stub" + ast.parse(round_tripped) # must not raise SyntaxError + finally: + os.unlink(tmp_path) + + @pytest.mark.skipif( + sys.platform != "win32", + reason="cp1252 default-encoding regression is Windows-specific", + ) + def test_windows_default_encoding_would_have_failed(self): + """Negative control: prove that on Windows, writing the stub + *without* ``encoding="utf-8"`` would corrupt the file. If this + test ever starts failing (i.e. default write succeeds), it means + Python's default encoding has changed and the explicit UTF-8 + requirement may be obsolete — reconsider the fix.""" + from tools.code_execution_tool import generate_hermes_tools_module + import tempfile + + stub = generate_hermes_tools_module(["terminal"], transport="uds") + # Find a non-ASCII character we can use to prove the corruption. + non_ascii = [c for c in stub if ord(c) > 127] + if not non_ascii: + pytest.skip("stub has no non-ASCII chars — nothing to corrupt") + + # Write with default encoding (simulating the old buggy code). + with tempfile.NamedTemporaryFile( + mode="w", suffix=".py", delete=False + ) as f: + try: + f.write(stub) + tmp_path = f.name + wrote_successfully = True + except UnicodeEncodeError: + # Default encoding can't even encode it — that's the bug + # in a different form. Still proves the point. + tmp_path = f.name + wrote_successfully = False + + try: + if not wrote_successfully: + # Default-encoding write raised outright. The bug is real. + return + + # Read back as UTF-8 (what Python does on import). + with open(tmp_path, encoding="utf-8") as fh: + try: + fh.read() + # If this succeeds on Windows, the platform default is + # already UTF-8 (e.g. Python 3.15 with UTF-8 mode on). + # In that case the explicit encoding= is belt-and- + # suspenders but no longer strictly required. Skip. + pytest.skip( + "Default text-file encoding is UTF-8-compatible on " + "this Windows build — explicit encoding= is no " + "longer load-bearing, but keep it for belt-and-" + "suspenders." + ) + except UnicodeDecodeError: + # Exactly the failure mode that motivated the fix. + pass + finally: + os.unlink(tmp_path) diff --git a/tools/code_execution_tool.py b/tools/code_execution_tool.py index e115f5dc1fa70..0ddbd73e42e5a 100644 --- a/tools/code_execution_tool.py +++ b/tools/code_execution_tool.py @@ -390,9 +390,12 @@ def _call(tool_name, args): req_file = os.path.join(_RPC_DIR, f"req_{seq_str}") res_file = os.path.join(_RPC_DIR, f"res_{seq_str}") - # Write request atomically (write to .tmp, then rename) + # Write request atomically (write to .tmp, then rename). + # encoding="utf-8" is critical: on Windows-hosted remote backends + # (or any non-UTF-8 locale) the default open() mode would mangle + # non-ASCII chars in tool args when encoding them as JSON. tmp = req_file + ".tmp" - with open(tmp, "w") as f: + with open(tmp, "w", encoding="utf-8") as f: json.dump({"tool": tool_name, "args": args, "seq": seq}, f) os.rename(tmp, req_file) @@ -405,7 +408,7 @@ def _call(tool_name, args): time.sleep(poll_interval) poll_interval = min(poll_interval * 1.2, 0.25) # Back off to 250ms - with open(res_file) as f: + with open(res_file, encoding="utf-8") as f: raw = f.read() # Clean up response file @@ -1111,15 +1114,22 @@ def execute_code( server_sock = None try: - # Write the auto-generated hermes_tools module + # Write the auto-generated hermes_tools module. + # encoding="utf-8" is required on Windows — the stub and user code + # both contain non-ASCII characters (em-dashes in docstrings, plus + # whatever the user script carries). Python's default open() uses + # the system locale on Windows (cp1252 typically), which corrupts + # those bytes; the child then fails to import with a SyntaxError + # ("'utf-8' codec can't decode byte 0x97 in position ...") because + # Python source files are decoded as UTF-8 by default (PEP 3120). # sandbox_tools is already the correct set (intersection with session # tools, or SANDBOX_ALLOWED_TOOLS as fallback — see lines above). tools_src = generate_hermes_tools_module(list(sandbox_tools)) - with open(os.path.join(tmpdir, "hermes_tools.py"), "w") as f: + with open(os.path.join(tmpdir, "hermes_tools.py"), "w", encoding="utf-8") as f: f.write(tools_src) # Write the user's script - with open(os.path.join(tmpdir, "script.py"), "w") as f: + with open(os.path.join(tmpdir, "script.py"), "w", encoding="utf-8") as f: f.write(code) # --- Start RPC server --- From f5ec30dfe6ef234a4917befb57f4904a2257d63c Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:56:33 -0700 Subject: [PATCH 15/20] tests: skip POSIX-venv-layout tests on Windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit test_code_execution_modes.py had two test-level failures and two class-level stale skip reasons on this Windows-native branch: - TestResolveChildPython::test_project_with_virtualenv_picks_venv_python - TestResolveChildPython::test_project_prefers_virtualenv_over_conda Both fail on Windows with OSError: [WinError 1314] — they call pathlib.Path.symlink_to() to build a fake venv, which requires developer mode or admin on Windows. They also assume POSIX venv layout (bin/python) where Windows uses Scripts/python.exe. Skip them with a specific, accurate reason. Also updated two class-level skipif reasons that said 'execute_code is POSIX-only' — no longer true on this branch. New reason explains it's the test infrastructure (symlinks + POSIX venv layout) that's the blocker, not execute_code itself. Results on Windows Python 3.11: Before: 41 passed, 10 skipped, 2 failed After: 43 passed, 12 skipped, 0 failed --- tests/tools/test_code_execution_modes.py | 32 ++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/tests/tools/test_code_execution_modes.py b/tests/tools/test_code_execution_modes.py index 875eaf7aedad5..4e22fe6e7a2f6 100644 --- a/tests/tools/test_code_execution_modes.py +++ b/tests/tools/test_code_execution_modes.py @@ -131,6 +131,12 @@ def test_project_with_no_venv_falls_back(self): def test_project_with_virtualenv_picks_venv_python(self): """Project mode + VIRTUAL_ENV pointing at a real venv → that python.""" + if sys.platform == "win32": + pytest.skip( + "Creates symlinks and assumes POSIX venv layout (bin/python). " + "Windows venvs use Scripts/python.exe and symlink creation " + "requires elevated privileges (WinError 1314)." + ) import tempfile, pathlib with tempfile.TemporaryDirectory() as td: fake_venv = pathlib.Path(td) @@ -154,6 +160,12 @@ def test_project_with_broken_venv_falls_back(self): def test_project_prefers_virtualenv_over_conda(self): """If both VIRTUAL_ENV and CONDA_PREFIX are set, VIRTUAL_ENV wins.""" + if sys.platform == "win32": + pytest.skip( + "Creates symlinks and assumes POSIX venv layout (bin/python). " + "Windows venvs use Scripts/python.exe and symlink creation " + "requires elevated privileges (WinError 1314)." + ) import tempfile, pathlib with tempfile.TemporaryDirectory() as ve_td, tempfile.TemporaryDirectory() as conda_td: ve = pathlib.Path(ve_td) @@ -257,7 +269,15 @@ def test_default_mode_reads_config(self): # Integration: what actually happens when execute_code runs per mode # --------------------------------------------------------------------------- -@pytest.mark.skipif(sys.platform == "win32", reason="execute_code is POSIX-only") +@pytest.mark.skipif( + sys.platform == "win32", + reason=( + "Assumes POSIX venv layout (bin/python) and symlink creation " + "privileges. execute_code itself works on Windows — these " + "integration tests just haven't been ported to the Scripts/" + "python.exe layout yet." + ), +) class TestExecuteCodeModeIntegration(unittest.TestCase): """End-to-end: verify the subprocess actually runs where we expect.""" @@ -351,7 +371,15 @@ def test_strict_mode_can_still_import_hermes_tools(self): # changes CWD + interpreter, not the security posture. # --------------------------------------------------------------------------- -@pytest.mark.skipif(sys.platform == "win32", reason="execute_code is POSIX-only") +@pytest.mark.skipif( + sys.platform == "win32", + reason=( + "Assumes POSIX venv layout (bin/python) and symlink creation " + "privileges. execute_code itself works on Windows — these " + "integration tests just haven't been ported to the Scripts/" + "python.exe layout yet." + ), +) class TestSecurityInvariantsAcrossModes(unittest.TestCase): def _run(self, code, mode): From bf43f6cfdd0a094acd49ff386a62016c4ec4a5f6 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 18:59:35 -0700 Subject: [PATCH 16/20] execute_code: set PYTHONIOENCODING=utf-8 + PYTHONUTF8=1 in child env MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third Windows-specific sandbox bug (after WinError 10106 and the UTF-8 file-write bug): user scripts that print non-ASCII to stdout crash with UnicodeEncodeError: 'charmap' codec can't encode character '\u2192' in position N: character maps to Root cause: Python's sys.stdout on Windows is bound to the console code page (cp1252 on US-locale installs) when the process is attached to a pipe without PYTHONIOENCODING set. LLM-generated scripts routinely print em-dashes, arrows, accented chars, and emoji — all of which cp1252 can't encode. Fix: spawn the sandbox child with: PYTHONIOENCODING=utf-8 # sys.stdin/stdout/stderr all UTF-8 PYTHONUTF8=1 # PEP 540 UTF-8 mode — open() defaults to UTF-8 too PYTHONUTF8 is the belt-and-suspenders half: LLM scripts that call open(path, 'w') without encoding= in user code will now produce UTF-8 files by default, matching what the sandbox already does for its own staging files. The parent side already decodes child stdout/stderr as UTF-8 with errors='replace' (lines 1345-1347) so the end-to-end chain is clean. On POSIX these values usually match the locale default already, so setting them is harmless belt-and-suspenders for C/POSIX-locale containers and minimal base images. Tests added (4) — total file now at 28 passed, 1 skipped on Windows: - test_popen_env_sets_pythonioencoding_utf8 (source grep) - test_popen_env_sets_pythonutf8_mode (source grep) - test_live_child_can_print_non_ascii (cross-platform live test) - test_windows_child_without_utf8_env_would_fail (Windows negative control — actually reproduces the bug without our env overrides, proving the fix is load-bearing on this system) --- .../tools/test_code_execution_windows_env.py | 133 ++++++++++++++++++ tools/code_execution_tool.py | 19 +++ 2 files changed, 152 insertions(+) diff --git a/tests/tools/test_code_execution_windows_env.py b/tests/tools/test_code_execution_windows_env.py index aca7a31da1542..70508818fc175 100644 --- a/tests/tools/test_code_execution_windows_env.py +++ b/tests/tools/test_code_execution_windows_env.py @@ -563,3 +563,136 @@ def test_windows_default_encoding_would_have_failed(self): pass finally: os.unlink(tmp_path) + + +# --------------------------------------------------------------------------- +# UTF-8 stdio regression test +# --------------------------------------------------------------------------- +# +# The third Windows-specific sandbox bug: after the UTF-8 file-write fix +# let the child import hermes_tools, a user script that printed non-ASCII +# to stdout still crashed with: +# +# UnicodeEncodeError: 'charmap' codec can't encode character '\u2192' +# in position N: character maps to +# +# Python's sys.stdout on Windows is bound to the console code page +# (cp1252 on US-locale installs) when the process is attached to a pipe +# without PYTHONIOENCODING set. LLM-generated scripts routinely print +# em-dashes, arrows, accented chars, emoji — all of which break. +# +# Fix: spawn the child with PYTHONIOENCODING=utf-8 and PYTHONUTF8=1. +# The latter also makes open()'s default encoding UTF-8 (PEP 540), +# belt-and-suspenders for user scripts that do their own file I/O. + + +class TestChildStdioIsUtf8: + """Verify the sandbox child is spawned with UTF-8 stdio encoding, + so LLM scripts can print non-ASCII without crashing on Windows.""" + + def test_popen_env_sets_pythonioencoding_utf8(self): + """Source-level check: the Popen call site must set + PYTHONIOENCODING=utf-8 in child_env.""" + import tools.code_execution_tool as cet + src = open(cet.__file__, encoding="utf-8").read() + assert 'child_env["PYTHONIOENCODING"] = "utf-8"' in src, ( + "PYTHONIOENCODING=utf-8 missing from child env — Windows " + "scripts that print non-ASCII will crash with " + "UnicodeEncodeError." + ) + + def test_popen_env_sets_pythonutf8_mode(self): + """Source-level check: PYTHONUTF8=1 must be set too — it makes + open()'s default encoding UTF-8 in user-written file I/O.""" + import tools.code_execution_tool as cet + src = open(cet.__file__, encoding="utf-8").read() + assert 'child_env["PYTHONUTF8"] = "1"' in src, ( + "PYTHONUTF8=1 missing from child env — user scripts that " + "call open(path, 'w') without encoding= will produce " + "locale-encoded files on Windows." + ) + + def test_live_child_can_print_non_ascii(self): + """Live regression: spawn a Python child with the same env + treatment the sandbox uses (PYTHONIOENCODING=utf-8 + PYTHONUTF8=1) + and verify it can print em-dashes, arrows, and emoji to stdout + without crashing. This is the exact scenario that broke in live + usage. + + Runs on every OS — on POSIX the fix is belt-and-suspenders but + still load-bearing for C.ASCII locale environments. + """ + script = textwrap.dedent(""" + import sys + # Mix of chars that cp1252 can't encode: arrow, emoji. + print("em-dash \\u2014 arrow \\u2192 emoji \\U0001f680") + sys.exit(0) + """).strip() + + # Build a scrubbed env the same way the sandbox does, then apply + # the stdio overrides. + scrubbed = _scrub_child_env(os.environ, is_passthrough=_no_passthrough) + scrubbed["PYTHONIOENCODING"] = "utf-8" + scrubbed["PYTHONUTF8"] = "1" + + result = subprocess.run( + [sys.executable, "-c", script], + env=scrubbed, + capture_output=True, + timeout=15, + # Don't decode at the subprocess boundary — we want to check + # the raw bytes match UTF-8, same as what the sandbox does. + ) + assert result.returncode == 0, ( + f"Child crashed printing non-ASCII:\n" + f" stdout (raw): {result.stdout!r}\n" + f" stderr (raw): {result.stderr!r}" + ) + decoded = result.stdout.decode("utf-8") + assert "\u2014" in decoded, f"em-dash missing from output: {decoded!r}" + assert "\u2192" in decoded, f"arrow missing from output: {decoded!r}" + assert "\U0001f680" in decoded, f"emoji missing from output: {decoded!r}" + + @pytest.mark.skipif( + sys.platform != "win32", + reason="cp1252 stdout default is Windows-specific", + ) + def test_windows_child_without_utf8_env_would_fail(self): + """Negative control: spawn a Python child *without* our env + overrides and prove that on Windows, printing non-ASCII fails. + If this ever starts passing, Python has changed its default + stdio encoding on Windows and the fix may be obsolete — but + keep the env vars anyway for belt-and-suspenders.""" + script = textwrap.dedent(""" + import sys + print("em-dash \\u2014 arrow \\u2192") + sys.exit(0) + """).strip() + + # Scrubbed env WITHOUT the PYTHONIOENCODING / PYTHONUTF8 overrides. + # Also scrub PYTHONUTF8 and PYTHONIOENCODING from the inherited + # env so we reproduce the buggy state even if the parent test + # runner has them set. + scrubbed = _scrub_child_env(os.environ, is_passthrough=_no_passthrough) + for k in ("PYTHONIOENCODING", "PYTHONUTF8", "PYTHONLEGACYWINDOWSSTDIO"): + scrubbed.pop(k, None) + + result = subprocess.run( + [sys.executable, "-c", script], + env=scrubbed, + capture_output=True, + text=False, + timeout=15, + ) + # Either the child crashed (expected), or modern Python handled + # it anyway — in which case the fix is still defensive but no + # longer strictly required. Skip with a note if so. + if result.returncode == 0 and b"\xe2\x80\x94" in result.stdout: + pytest.skip( + "This Python/Windows build handles non-ASCII stdout even " + "without PYTHONIOENCODING/PYTHONUTF8 — fix is defensive " + "but no longer strictly load-bearing. Keep the env vars " + "for older Python builds and C.ASCII-locale containers." + ) + # Otherwise: crash OR garbled output — both count as proving the + # bug is real on this system. diff --git a/tools/code_execution_tool.py b/tools/code_execution_tool.py index 0ddbd73e42e5a..47271223f351a 100644 --- a/tools/code_execution_tool.py +++ b/tools/code_execution_tool.py @@ -1175,6 +1175,25 @@ def execute_code( child_env = _scrub_child_env(os.environ) child_env["HERMES_RPC_SOCKET"] = rpc_endpoint child_env["PYTHONDONTWRITEBYTECODE"] = "1" + # Force UTF-8 for the child's stdio and default file encoding. + # + # Without this, on Windows sys.stdout is bound to the console code + # page (cp1252 on US-locale installs), and any script that does + # ``print("café")`` or ``print("→")`` crashes with: + # + # UnicodeEncodeError: 'charmap' codec can't encode character + # '\u2192' in position N: character maps to + # + # PYTHONIOENCODING fixes sys.stdin/stdout/stderr. + # PYTHONUTF8=1 enables "UTF-8 mode" (PEP 540) which additionally + # makes ``open()``'s default encoding UTF-8, so user scripts that + # write files without specifying encoding= also work correctly. + # + # On POSIX both values usually match the locale default already, + # so setting them is harmless belt-and-suspenders for environments + # with a C/POSIX locale (containers, minimal base images). + child_env["PYTHONIOENCODING"] = "utf-8" + child_env["PYTHONUTF8"] = "1" # Ensure the hermes-agent root is importable in the sandbox so # repo-root modules are available to child scripts. We also prepend # the staging tmpdir so ``from hermes_tools import ...`` resolves even From 60982724545bd13aea50ff7ea7b6e535fb38552d Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 19:09:40 -0700 Subject: [PATCH 17/20] hermes_bootstrap: Windows-only UTF-8 stdio shim for all entry points MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codebase-wide fix for Python-on-Windows UTF-8 footguns, complementing the earlier execute_code sandbox fixes (which remain load-bearing for when the sandbox explicitly scrubs child env). Problem: Python on Windows has two long-standing text-encoding pitfalls: 1. sys.stdout/stderr are bound to the console code page (cp1252 on US-locale installs) — print('café') crashes with UnicodeEncodeError. 2. Subprocess children don't know to use UTF-8 unless PYTHONUTF8 and/or PYTHONIOENCODING are set in their env — so any Python we spawn (linters, sandbox children, delegation workers) hits the same bug. Solution: A tiny bootstrap module (hermes_bootstrap.py) imported as the first statement of every Hermes entry point: - hermes_cli/main.py (hermes / hermes-agent console_script) - run_agent.py (hermes-agent direct) - acp_adapter/entry.py (hermes-acp) - gateway/run.py (messaging gateway) - batch_runner.py (parallel batch mode) - cli.py (legacy direct-launch CLI) On Windows, the bootstrap: - os.environ.setdefault('PYTHONUTF8', '1') (PEP 540 UTF-8 mode) - os.environ.setdefault('PYTHONIOENCODING', 'utf-8') - sys.stdout/stderr/stdin.reconfigure(encoding='utf-8', errors='replace') Children inherit the env vars → they run in UTF-8 mode. Current process's stdio is reconfigured → print('café') works now. On POSIX (Linux/macOS), the bootstrap is a complete no-op. We don't touch LANG, LC_*, or anything else — users who have intentionally configured a non-UTF-8 locale aren't affected. POSIX systems are already UTF-8 by default in 99% of modern setups, so there's nothing to fix. setdefault() (not overwrite) means users who explicitly set PYTHONUTF8=0 or PYTHONIOENCODING=cp1252 in their environment are respected. What this does NOT fix: bare open(path, 'w') calls in the *parent* process still default to locale encoding because PYTHONUTF8 is only read at interpreter init. A ruff PLW1514 sweep (separate follow-up) will add explicit encoding='utf-8' at those ~219 call sites for belt-and-suspenders. Tests (17): 16 passed, 1 skipped on Windows. - Windows: env vars set, stdio reconfigured, child inherits UTF-8 mode - POSIX: complete no-op (verified on fake POSIX + skipped on real POSIX since we don't have a Linux box in this session) - Idempotence: multiple calls safe - Graceful degradation: non-reconfigurable streams don't crash - User opt-out: explicit PYTHONUTF8=0 is respected - Load order: every entry point's FIRST top-level import is hermes_bootstrap, enforced by an AST-level parametrized test pyproject.toml: added hermes_bootstrap to py-modules so it ships with pip installs. --- acp_adapter/entry.py | 4 + batch_runner.py | 4 + cli.py | 5 +- gateway/run.py | 4 + hermes_bootstrap.py | 129 ++++++++++++++ hermes_cli/main.py | 5 + pyproject.toml | 2 +- run_agent.py | 4 + tests/test_hermes_bootstrap.py | 297 +++++++++++++++++++++++++++++++++ 9 files changed, 452 insertions(+), 2 deletions(-) create mode 100644 hermes_bootstrap.py create mode 100644 tests/test_hermes_bootstrap.py diff --git a/acp_adapter/entry.py b/acp_adapter/entry.py index 33e28092f056c..0607387b854f1 100644 --- a/acp_adapter/entry.py +++ b/acp_adapter/entry.py @@ -13,6 +13,10 @@ hermes-acp """ +# IMPORTANT: hermes_bootstrap must be the very first import — UTF-8 stdio +# on Windows. No-op on POSIX. See hermes_bootstrap.py for full rationale. +import hermes_bootstrap # noqa: F401 + import asyncio import logging import sys diff --git a/batch_runner.py b/batch_runner.py index f3aaefa3d9a8b..11626d98b8712 100644 --- a/batch_runner.py +++ b/batch_runner.py @@ -20,6 +20,10 @@ python batch_runner.py --dataset_file=data.jsonl --batch_size=10 --run_name=my_run --distribution=image_gen """ +# IMPORTANT: hermes_bootstrap must be the very first import — UTF-8 stdio +# on Windows. No-op on POSIX. See hermes_bootstrap.py for full rationale. +import hermes_bootstrap # noqa: F401 + import json import logging import os diff --git a/cli.py b/cli.py index 7232110580afe..3b9fa3a00b0ba 100644 --- a/cli.py +++ b/cli.py @@ -9,10 +9,13 @@ python cli.py # Start interactive mode with all tools python cli.py --toolsets web,terminal # Start with specific toolsets python cli.py --skills hermes-agent-dev,github-auth - python cli.py -q "your question" # Single query mode python cli.py --list-tools # List available tools and exit """ +# IMPORTANT: hermes_bootstrap must be the very first import — UTF-8 stdio +# on Windows. No-op on POSIX. See hermes_bootstrap.py for full rationale. +import hermes_bootstrap # noqa: F401 + import logging import os import shutil diff --git a/gateway/run.py b/gateway/run.py index f7577ccf37de9..0067e7bb68f95 100644 --- a/gateway/run.py +++ b/gateway/run.py @@ -13,6 +13,10 @@ python cli.py --gateway """ +# IMPORTANT: hermes_bootstrap must be the very first import — UTF-8 stdio +# on Windows. No-op on POSIX. See hermes_bootstrap.py for full rationale. +import hermes_bootstrap # noqa: F401 + import asyncio import dataclasses import inspect diff --git a/hermes_bootstrap.py b/hermes_bootstrap.py new file mode 100644 index 0000000000000..890336c3448e9 --- /dev/null +++ b/hermes_bootstrap.py @@ -0,0 +1,129 @@ +"""Windows UTF-8 bootstrap for Hermes entry points. + +Python on Windows has two long-standing text-encoding footguns: + +1. ``sys.stdout`` / ``sys.stderr`` are bound to the console code page + (``cp1252`` on US-locale installs), so ``print("café")`` crashes with + ``UnicodeEncodeError: 'charmap' codec can't encode character``. + +2. Child processes spawned via ``subprocess`` don't know to use UTF-8 + unless ``PYTHONUTF8`` and/or ``PYTHONIOENCODING`` are set in their + environment — so any Python subprocess (the execute_code sandbox, + delegation children, linter subprocesses, etc.) inherits the same + cp1252 defaults and hits the same UnicodeEncodeError. + +This module fixes both on Windows *only* — POSIX is untouched. It +should be imported at the very top of every Hermes entry point +(``hermes``, ``hermes-agent``, ``hermes-acp``, ``python -m gateway.run``, +``batch_runner.py``, ``cron/scheduler.py``) before any other imports +that might do file I/O or print to stdout. + +What this module does on Windows: + + - Sets ``os.environ["PYTHONUTF8"] = "1"`` (PEP 540 UTF-8 mode) so + every child process we spawn uses UTF-8 for ``open()`` and stdio. + - Sets ``os.environ["PYTHONIOENCODING"] = "utf-8"`` for belt-and- + suspenders — some tools read this instead of / in addition to + ``PYTHONUTF8``. + - Reconfigures ``sys.stdout`` / ``sys.stderr`` to UTF-8 in the current + process, using the ``reconfigure()`` API (Python 3.7+). This fixes + ``print("café")`` in the parent without a re-exec. + +What this module does NOT do: + + - It does not re-exec Python with ``-X utf8``, so ``open()`` calls in + the *current* process still default to locale encoding. Those need + an explicit ``encoding="utf-8"`` at the call site (lint rule + ``PLW1514`` / ``PYI058``). Ruff is the right tool for that sweep. + +What this module does on POSIX: + + - Nothing. POSIX systems are already UTF-8 by default in 99% of cases, + and we don't want to touch ``LANG``/``LC_*`` behavior that users may + have configured intentionally. If someone hits a C/POSIX locale on + Linux, they can export ``PYTHONUTF8=1`` themselves — we won't override. + +Idempotent: safe to call multiple times. ``_bootstrap_once`` guards +against double-reconfigure. +""" + +from __future__ import annotations + +import os +import sys + +_IS_WINDOWS = sys.platform == "win32" +_bootstrap_applied = False + + +def apply_windows_utf8_bootstrap() -> bool: + """Apply the Windows UTF-8 bootstrap if we're on Windows. + + Returns True if bootstrap was applied (i.e. we're on Windows and + haven't already done this), False otherwise. The return value is + advisory — callers normally don't need it, but tests may want to + assert the path was taken. + + Idempotent: subsequent calls after the first are a no-op. + """ + global _bootstrap_applied + + if not _IS_WINDOWS: + return False + if _bootstrap_applied: + return False + + # 1. Child processes inherit these and run in UTF-8 mode. + # We use setdefault() rather than overwriting so the user can + # explicitly opt out by setting PYTHONUTF8=0 in their environment + # (or PYTHONIOENCODING=something-else) if they really want to. + os.environ.setdefault("PYTHONUTF8", "1") + os.environ.setdefault("PYTHONIOENCODING", "utf-8") + + # 2. Reconfigure the current process's stdio to UTF-8. Needed + # because os.environ changes don't retroactively rebind sys.stdout + # — those were bound at interpreter startup based on the console + # code page. ``reconfigure`` is a TextIOWrapper method since 3.7. + # + # errors="replace" means that if we ever *read* something from + # stdin that isn't UTF-8 (unlikely but possible with piped input + # from legacy tools), we'll get U+FFFD replacement chars rather + # than a crash. Output is pure UTF-8. + for stream_name in ("stdout", "stderr"): + stream = getattr(sys, stream_name, None) + if stream is None: + continue + reconfigure = getattr(stream, "reconfigure", None) + if reconfigure is None: + # Not a TextIOWrapper (could be redirected to a BytesIO in + # tests, or a non-standard stream in some embedded cases). + # Skip silently — the env-var fix is still in effect for + # child processes, which is the bigger win. + continue + try: + reconfigure(encoding="utf-8", errors="replace") + except (OSError, ValueError): + # Already closed, or someone replaced it with something + # non-reconfigurable. Non-fatal. + pass + + # stdin is reconfigured separately with errors="replace" too — input + # from a legacy pipe shouldn't crash the process. + stdin = getattr(sys, "stdin", None) + if stdin is not None: + reconfigure = getattr(stdin, "reconfigure", None) + if reconfigure is not None: + try: + reconfigure(encoding="utf-8", errors="replace") + except (OSError, ValueError): + pass + + _bootstrap_applied = True + return True + + +# Apply on import — entry points just need ``import hermes_bootstrap`` +# (or ``from hermes_bootstrap import apply_windows_utf8_bootstrap``) at +# the very top of their module, before importing anything else. The +# import side effect does the right thing. +apply_windows_utf8_bootstrap() diff --git a/hermes_cli/main.py b/hermes_cli/main.py index 695f2379ad2ed..9583ab148cc03 100644 --- a/hermes_cli/main.py +++ b/hermes_cli/main.py @@ -43,6 +43,11 @@ hermes claw migrate --dry-run # Preview migration without changes """ +# IMPORTANT: hermes_bootstrap must be the very first import — it sets up +# UTF-8 stdio on Windows so print()/subprocess children don't hit +# UnicodeEncodeError with non-ASCII characters. No-op on POSIX. +import hermes_bootstrap # noqa: F401 + import argparse import json import os diff --git a/pyproject.toml b/pyproject.toml index bcdd5d9181909..592f5b6b214a5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -160,7 +160,7 @@ hermes-agent = "run_agent:main" hermes-acp = "acp_adapter.entry:main" [tool.setuptools] -py-modules = ["run_agent", "model_tools", "toolsets", "batch_runner", "trajectory_compressor", "toolset_distributions", "cli", "hermes_constants", "hermes_state", "hermes_time", "hermes_logging", "rl_cli", "utils"] +py-modules = ["run_agent", "model_tools", "toolsets", "batch_runner", "trajectory_compressor", "toolset_distributions", "cli", "hermes_bootstrap", "hermes_constants", "hermes_state", "hermes_time", "hermes_logging", "rl_cli", "utils"] [tool.setuptools.package-data] hermes_cli = ["web_dist/**/*"] diff --git a/run_agent.py b/run_agent.py index 403dba4e78504..5b416a0fc3fe8 100644 --- a/run_agent.py +++ b/run_agent.py @@ -20,6 +20,10 @@ response = agent.run_conversation("Tell me about the latest Python updates") """ +# IMPORTANT: hermes_bootstrap must be the very first import — UTF-8 stdio +# on Windows. No-op on POSIX. See hermes_bootstrap.py for full rationale. +import hermes_bootstrap # noqa: F401 + import asyncio import base64 import concurrent.futures diff --git a/tests/test_hermes_bootstrap.py b/tests/test_hermes_bootstrap.py new file mode 100644 index 0000000000000..84e9ec81ad5aa --- /dev/null +++ b/tests/test_hermes_bootstrap.py @@ -0,0 +1,297 @@ +"""Tests for hermes_bootstrap — Windows UTF-8 stdio shim. + +The bootstrap module is imported at the top of every Hermes entry point +(hermes, hermes-agent, hermes-acp, gateway, batch_runner, cli.py). It +fixes Python's Windows UTF-8 defaults so print("café") doesn't crash and +subprocess children inherit UTF-8 mode. + +Key invariants covered by these tests: + + 1. Windows: env vars get set, stdio reconfigured, non-ASCII print works + 2. POSIX: complete no-op (we don't touch LANG/LC_* or anything else) + 3. Idempotent: safe to call multiple times + 4. Respects user opt-out: if the user explicitly sets PYTHONUTF8=0 or + PYTHONIOENCODING=something-else, we leave those alone + 5. Load order: every Hermes entry point imports hermes_bootstrap as its + first non-docstring import (before anything that might do file I/O + or print to stdout) +""" + +from __future__ import annotations + +import io +import os +import subprocess +import sys +import textwrap +import unittest.mock as mock + +import pytest + + +# Import the module under test via an import-time side-effect check path. +# We need to be able to reset its state between tests, so we import it +# fresh in each test that manipulates _IS_WINDOWS. +def _fresh_import(): + """Return a freshly-imported hermes_bootstrap module. + + Drops any cached copy from sys.modules first so module-level code + runs again and the platform check re-evaluates. + """ + sys.modules.pop("hermes_bootstrap", None) + import hermes_bootstrap # noqa: WPS433 + return hermes_bootstrap + + +class TestWindowsBehavior: + """Windows: the bootstrap does its job.""" + + @pytest.mark.skipif( + sys.platform != "win32", + reason="Windows-specific behavior", + ) + def test_env_vars_set_on_windows(self, monkeypatch): + # Clear any pre-existing values and re-run bootstrap. + monkeypatch.delenv("PYTHONUTF8", raising=False) + monkeypatch.delenv("PYTHONIOENCODING", raising=False) + hb = _fresh_import() + # Module-level apply_windows_utf8_bootstrap() ran during import. + assert os.environ.get("PYTHONUTF8") == "1" + assert os.environ.get("PYTHONIOENCODING") == "utf-8" + assert hb._bootstrap_applied is True + + @pytest.mark.skipif( + sys.platform != "win32", + reason="Windows-specific behavior", + ) + def test_stdout_reconfigured_to_utf8_on_windows(self): + # The live process's stdout should now be UTF-8 (the Hermes CLI + # runs on Windows with a pytest console that's cp1252 by default). + # If reconfigure succeeded, sys.stdout.encoding is 'utf-8'. + _fresh_import() + # pytest may capture stdout, which makes encoding check flaky — + # so instead verify the reconfigure call succeeded on the real + # stream by attempting the failure case. + out = sys.stdout + reconfigure = getattr(out, "reconfigure", None) + if reconfigure is None: + pytest.skip("pytest replaced sys.stdout with a non-reconfigurable stream") + # After bootstrap, encoding should be utf-8 (or the reconfigure + # skipped because pytest's capture already set it to utf-8). + assert out.encoding.lower() in {"utf-8", "utf8"}, ( + f"stdout encoding is {out.encoding!r} — bootstrap should have " + "reconfigured it to UTF-8" + ) + + @pytest.mark.skipif( + sys.platform != "win32", + reason="Windows-specific behavior", + ) + def test_child_process_inherits_utf8_mode(self): + """A subprocess spawned from this process should inherit + PYTHONUTF8=1 and be able to print non-ASCII to stdout.""" + _fresh_import() + # Non-ASCII chars that would crash under cp1252: arrow, emoji. + script = textwrap.dedent(""" + import sys + print("em-dash \\u2014 arrow \\u2192 emoji \\U0001f680") + sys.exit(0) + """).strip() + # Don't pass env= — let the child inherit os.environ, which + # now contains PYTHONUTF8=1 courtesy of the bootstrap. + result = subprocess.run( + [sys.executable, "-c", script], + capture_output=True, + timeout=15, + ) + assert result.returncode == 0, ( + f"Child crashed printing non-ASCII despite UTF-8 bootstrap:\n" + f" stdout: {result.stdout!r}\n" + f" stderr: {result.stderr!r}" + ) + decoded = result.stdout.decode("utf-8") + assert "\u2014" in decoded + assert "\u2192" in decoded + assert "\U0001f680" in decoded + + +class TestUserOptOut: + """If the user has explicitly set PYTHONUTF8 / PYTHONIOENCODING in + their environment, we respect that (setdefault, not overwrite).""" + + @pytest.mark.skipif( + sys.platform != "win32", + reason="Only meaningful on Windows where we'd otherwise set these", + ) + def test_user_pythonutf8_zero_preserved(self, monkeypatch): + monkeypatch.setenv("PYTHONUTF8", "0") + _fresh_import() + assert os.environ["PYTHONUTF8"] == "0", ( + "bootstrap must not overwrite an explicit user setting" + ) + + @pytest.mark.skipif( + sys.platform != "win32", + reason="Only meaningful on Windows where we'd otherwise set these", + ) + def test_user_pythonioencoding_preserved(self, monkeypatch): + monkeypatch.setenv("PYTHONIOENCODING", "latin-1") + _fresh_import() + assert os.environ["PYTHONIOENCODING"] == "latin-1" + + +class TestPosixNoOp: + """POSIX: zero behavior change. We don't touch LANG, LC_*, or any + stdio. The goal is that Linux/macOS behave identically before and + after this module is imported.""" + + def test_noop_on_fake_posix(self, monkeypatch): + """Even when imported, the bootstrap function must return False + and leave env untouched when _IS_WINDOWS is False.""" + hb = _fresh_import() + # Reset + fake POSIX + hb._IS_WINDOWS = False + hb._bootstrap_applied = False + monkeypatch.delenv("PYTHONUTF8", raising=False) + monkeypatch.delenv("PYTHONIOENCODING", raising=False) + + result = hb.apply_windows_utf8_bootstrap() + + assert result is False + assert "PYTHONUTF8" not in os.environ + assert "PYTHONIOENCODING" not in os.environ + assert hb._bootstrap_applied is False + + @pytest.mark.skipif( + sys.platform == "win32", + reason="Real POSIX required for this check", + ) + def test_real_posix_bootstrap_is_noop(self, monkeypatch): + """On actual Linux/macOS, importing the module must not set + PYTHONUTF8 or reconfigure stdio.""" + monkeypatch.delenv("PYTHONUTF8", raising=False) + monkeypatch.delenv("PYTHONIOENCODING", raising=False) + hb = _fresh_import() + assert hb._bootstrap_applied is False + assert "PYTHONUTF8" not in os.environ + assert "PYTHONIOENCODING" not in os.environ + + +class TestIdempotence: + """Calling apply_windows_utf8_bootstrap() multiple times must be safe.""" + + def test_second_call_returns_false(self): + hb = _fresh_import() + # First call already happened at import time. + result = hb.apply_windows_utf8_bootstrap() + assert result is False, ( + "Second call should return False (idempotent no-op)" + ) + + def test_no_exceptions_on_repeated_calls(self): + hb = _fresh_import() + for _ in range(5): + hb.apply_windows_utf8_bootstrap() + + +class TestStdioReconfigureErrorHandling: + """If sys.stdout/stderr/stdin have been replaced with streams that + don't support reconfigure (e.g. by a test harness), the bootstrap + must degrade gracefully rather than crash.""" + + def test_non_reconfigurable_stream_does_not_crash(self, monkeypatch): + """Replace sys.stdout with a BytesIO (no reconfigure method), + then run the bootstrap and make sure it doesn't raise.""" + hb = _fresh_import() + hb._IS_WINDOWS = True + hb._bootstrap_applied = False + + fake = io.BytesIO() # no .reconfigure attribute + monkeypatch.setattr(sys, "stdout", fake) + try: + # Must not raise. + hb.apply_windows_utf8_bootstrap() + except Exception as exc: + pytest.fail(f"bootstrap raised on non-reconfigurable stdout: {exc}") + + def test_reconfigure_oserror_is_caught(self, monkeypatch): + """If reconfigure() itself raises (closed stream, etc.), swallow + the error — the env-var half of the fix still applies.""" + hb = _fresh_import() + hb._IS_WINDOWS = True + hb._bootstrap_applied = False + + class _BrokenStream: + encoding = "utf-8" + def reconfigure(self, **kwargs): + raise OSError("simulated: stream already closed") + + monkeypatch.setattr(sys, "stdout", _BrokenStream()) + monkeypatch.setattr(sys, "stderr", _BrokenStream()) + # Must not raise. + hb.apply_windows_utf8_bootstrap() + + +class TestEntryPointsImportBootstrap: + """Every Hermes entry point must import hermes_bootstrap as its + first non-docstring import. We check this by scanning source files + rather than invoking the entry points (which would require a full + agent context).""" + + # Entry points that invoke Hermes as a process. Each one must + # import hermes_bootstrap before doing any file I/O or stdout writes. + ENTRY_POINTS = [ + "hermes_cli/main.py", # hermes CLI (console_script) + "run_agent.py", # hermes-agent (console_script) + "acp_adapter/entry.py", # hermes-acp (console_script) + "gateway/run.py", # gateway + "batch_runner.py", # batch mode + "cli.py", # legacy direct-launch CLI + ] + + @pytest.mark.parametrize("path", ENTRY_POINTS) + def test_entry_point_imports_bootstrap(self, path): + """The file must contain 'import hermes_bootstrap' and that + line must appear before the first 'import' of anything else. + + We're lenient about the docstring (can be arbitrarily long) and + about comment lines — just need to verify the first import + statement is the bootstrap. + """ + # Resolve relative to the hermes-agent repo root. Tests live + # at tests/test_hermes_bootstrap.py, so go up one dir. + import pathlib + here = pathlib.Path(__file__).resolve() + repo_root = here.parent.parent # tests/ -> repo root + full_path = repo_root / path + assert full_path.exists(), f"entry point missing: {full_path}" + + source = full_path.read_text(encoding="utf-8") + + # Find the first non-comment, non-blank line that starts with + # 'import ' or 'from '. It must be 'import hermes_bootstrap'. + import tokenize + import ast + tree = ast.parse(source) + + first_import_node = None + for node in ast.iter_child_nodes(tree): + if isinstance(node, (ast.Import, ast.ImportFrom)): + first_import_node = node + break + + assert first_import_node is not None, ( + f"{path}: no top-level imports found at all" + ) + + if isinstance(first_import_node, ast.Import): + first_import_name = first_import_node.names[0].name + else: # ImportFrom + first_import_name = first_import_node.module or "" + + assert first_import_name == "hermes_bootstrap", ( + f"{path}: first top-level import is {first_import_name!r}, " + f"but it must be 'hermes_bootstrap' so UTF-8 stdio is " + f"configured before anything else initializes. Move the " + f"'import hermes_bootstrap' line to be the first import." + ) From 9c914c01c8f0bb861a169f3a42f8cff690c8fa4c Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 19:24:45 -0700 Subject: [PATCH 18/20] codebase: add encoding='utf-8' to all bare open() calls (PLW1514) Closes the last Python-on-Windows UTF-8 exposure by making every text-mode open() call explicit about its encoding. Before: on Windows, bare open(path, 'r') defaults to the system locale encoding (cp1252 on US-locale installs). That means reading any config/yaml/markdown/json file with non-ASCII content either crashes with UnicodeDecodeError or silently mis-decodes bytes. After: all 89 affected call sites in production code now pass encoding='utf-8' explicitly. Works identically on every platform and every locale, no surprise behavior. Mechanical sweep via: ruff check --preview --extend-select PLW1514 --unsafe-fixes --fix --exclude 'tests,venv,.venv,node_modules,website,optional-skills, skills,tinker-atropos,plugins' . All 89 fixes have the same shape: open(x) or open(x, mode) became open(x, encoding='utf-8') or open(x, mode, encoding='utf-8'). Nothing else changed. Every modified file still parses and the Windows/sandbox test suite is still green (85 passed, 14 skipped, 0 failed across tests/tools/test_code_execution_windows_env.py + tests/tools/test_code_execution_modes.py + tests/tools/test_env_passthrough.py + tests/test_hermes_bootstrap.py). Scope notes: - tests/ excluded: test fixtures can use locale encoding intentionally (exercising edge cases). If we want to tighten tests later that's a separate PR. - plugins/ excluded: plugin-specific conventions may differ; plugin authors own their code. - optional-skills/ and skills/ excluded: skill scripts are user-authored and we don't want to mass-edit them. - website/ and tinker-atropos/ excluded: vendored / generated content. 46 files touched, 89 +/- lines (symmetric replacement). No behavior change on POSIX or on Windows when the file is ASCII; bug fix on Windows when the file contains non-ASCII. --- agent/curator.py | 2 +- agent/model_metadata.py | 6 +++--- agent/nous_rate_guard.py | 2 +- agent/shell_hooks.py | 2 +- cli.py | 8 ++++---- cron/scheduler.py | 4 ++-- .../terminalbench_2/terminalbench2_env.py | 2 +- .../benchmarks/yc_bench/yc_bench_env.py | 2 +- gateway/platforms/telegram.py | 4 ++-- gateway/platforms/whatsapp.py | 4 ++-- gateway/status.py | 6 +++--- hermes_cli/backup.py | 6 +++--- hermes_cli/config.py | 6 +++--- hermes_cli/doctor.py | 4 ++-- hermes_cli/hooks.py | 2 +- hermes_cli/kanban_db.py | 2 +- hermes_cli/memory_setup.py | 4 ++-- hermes_cli/model_catalog.py | 4 ++-- hermes_cli/oneshot.py | 2 +- hermes_cli/plugins.py | 2 +- hermes_cli/plugins_cmd.py | 4 ++-- hermes_cli/profiles.py | 2 +- hermes_cli/skills_hub.py | 4 ++-- hermes_cli/web_server.py | 2 +- hermes_constants.py | 4 ++-- hermes_time.py | 2 +- rl_cli.py | 2 +- run_agent.py | 2 +- scripts/build_model_catalog.py | 2 +- scripts/build_skills_index.py | 2 +- scripts/contributor_audit.py | 2 +- scripts/discord-voice-doctor.py | 2 +- scripts/profile-tui.py | 4 ++-- scripts/release.py | 2 +- tools/browser_tool.py | 18 +++++++++--------- tools/code_execution_tool.py | 4 ++-- tools/environments/base.py | 4 ++-- tools/environments/file_sync.py | 2 +- tools/environments/local.py | 2 +- tools/mcp_tool.py | 2 +- tools/rl_training_tool.py | 14 +++++++------- tools/skills_hub.py | 4 ++-- tools/tirith_security.py | 6 +++--- tools/voice_mode.py | 2 +- trajectory_compressor.py | 4 ++-- tui_gateway/server.py | 6 +++--- 46 files changed, 89 insertions(+), 89 deletions(-) diff --git a/agent/curator.py b/agent/curator.py index a726e875b693a..3626f5d2345a3 100644 --- a/agent/curator.py +++ b/agent/curator.py @@ -1607,7 +1607,7 @@ def _run_llm_review(prompt: str) -> Dict[str, Any]: # terminal. The background-thread runner also hides it; this # belt-and-suspenders path matters when a caller invokes # run_curator_review(synchronous=True) from the CLI. - with open(os.devnull, "w") as _devnull, \ + with open(os.devnull, "w", encoding="utf-8") as _devnull, \ contextlib.redirect_stdout(_devnull), \ contextlib.redirect_stderr(_devnull): conv_result = review_agent.run_conversation(user_message=prompt) diff --git a/agent/model_metadata.py b/agent/model_metadata.py index c362a9ec93d1c..26e2f3c5e2469 100644 --- a/agent/model_metadata.py +++ b/agent/model_metadata.py @@ -754,7 +754,7 @@ def _load_context_cache() -> Dict[str, int]: if not path.exists(): return {} try: - with open(path) as f: + with open(path, encoding="utf-8") as f: data = yaml.safe_load(f) or {} return data.get("context_lengths", {}) except Exception as e: @@ -776,7 +776,7 @@ def save_context_length(model: str, base_url: str, length: int) -> None: path = _get_context_cache_path() try: path.parent.mkdir(parents=True, exist_ok=True) - with open(path, "w") as f: + with open(path, "w", encoding="utf-8") as f: yaml.dump({"context_lengths": cache}, f, default_flow_style=False) logger.info("Cached context length %s -> %s tokens", key, f"{length:,}") except Exception as e: @@ -800,7 +800,7 @@ def _invalidate_cached_context_length(model: str, base_url: str) -> None: path = _get_context_cache_path() try: path.parent.mkdir(parents=True, exist_ok=True) - with open(path, "w") as f: + with open(path, "w", encoding="utf-8") as f: yaml.dump({"context_lengths": cache}, f, default_flow_style=False) except Exception as e: logger.debug("Failed to invalidate context length cache entry %s: %s", key, e) diff --git a/agent/nous_rate_guard.py b/agent/nous_rate_guard.py index b28803122c5fe..415d367ca17be 100644 --- a/agent/nous_rate_guard.py +++ b/agent/nous_rate_guard.py @@ -144,7 +144,7 @@ def nous_rate_limit_remaining() -> Optional[float]: """ path = _state_path() try: - with open(path) as f: + with open(path, encoding="utf-8") as f: state = json.load(f) reset_at = state.get("reset_at", 0) remaining = reset_at - time.time() diff --git a/agent/shell_hooks.py b/agent/shell_hooks.py index 94750d5204109..d45851fea6cee 100644 --- a/agent/shell_hooks.py +++ b/agent/shell_hooks.py @@ -617,7 +617,7 @@ def _locked_update_approvals() -> Iterator[Dict[str, Any]]: save_allowlist(data) return - with open(lock_path, "a+") as lock_fh: + with open(lock_path, "a+", encoding="utf-8") as lock_fh: fcntl.flock(lock_fh.fileno(), fcntl.LOCK_EX) try: data = load_allowlist() diff --git a/cli.py b/cli.py index 3b9fa3a00b0ba..236f3e5d8724a 100644 --- a/cli.py +++ b/cli.py @@ -819,7 +819,7 @@ def _setup_worktree(repo_root: str = None) -> Optional[Dict[str, str]]: try: existing = gitignore.read_text() if gitignore.exists() else "" if _ignore_entry not in existing.splitlines(): - with open(gitignore, "a") as f: + with open(gitignore, "a", encoding="utf-8") as f: if existing and not existing.endswith("\n"): f.write("\n") f.write(f"{_ignore_entry}\n") @@ -2147,7 +2147,7 @@ def save_config_value(key_path: str, value: any) -> bool: # Load existing config if config_path.exists(): - with open(config_path, 'r') as f: + with open(config_path, 'r', encoding="utf-8") as f: config = yaml.safe_load(f) or {} else: config = {} @@ -9773,7 +9773,7 @@ def run_agent(): # Debug: log to file (stdout may be devnull from redirect_stdout) try: _dbg = _hermes_home / "interrupt_debug.log" - with open(_dbg, "a") as _f: + with open(_dbg, "a", encoding="utf-8") as _f: _f.write(f"{time.strftime('%H:%M:%S')} interrupt fired: msg={str(interrupt_msg)[:60]!r}, " f"children={len(self.agent._active_children)}, " f"parent._interrupt={self.agent._interrupt_requested}\n") @@ -10605,7 +10605,7 @@ def handle_enter(event): # Debug: log to file when message enters interrupt queue try: _dbg = _hermes_home / "interrupt_debug.log" - with open(_dbg, "a") as _f: + with open(_dbg, "a", encoding="utf-8") as _f: _f.write(f"{time.strftime('%H:%M:%S')} ENTER: queued interrupt msg={str(payload)[:60]!r}, " f"agent_running={self._agent_running}\n") except Exception: diff --git a/cron/scheduler.py b/cron/scheduler.py index 65db48466c402..f08daba2fc69b 100644 --- a/cron/scheduler.py +++ b/cron/scheduler.py @@ -1228,7 +1228,7 @@ def run_job(job: dict) -> tuple[bool, str, str, Optional[str]]: import yaml _cfg_path = str(_get_hermes_home() / "config.yaml") if os.path.exists(_cfg_path): - with open(_cfg_path) as _f: + with open(_cfg_path, encoding="utf-8") as _f: _cfg = yaml.safe_load(_f) or {} _cfg = _expand_env_vars(_cfg) _model_cfg = _cfg.get("model", {}) @@ -1611,7 +1611,7 @@ def tick(verbose: bool = True, adapters=None, loop=None) -> int: # Cross-platform file locking: fcntl on Unix, msvcrt on Windows lock_fd = None try: - lock_fd = open(lock_file, "w") + lock_fd = open(lock_file, "w", encoding="utf-8") if fcntl: fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) elif msvcrt: diff --git a/environments/benchmarks/terminalbench_2/terminalbench2_env.py b/environments/benchmarks/terminalbench_2/terminalbench2_env.py index c7eaff6c4c207..db6f6f58d73a3 100644 --- a/environments/benchmarks/terminalbench_2/terminalbench2_env.py +++ b/environments/benchmarks/terminalbench_2/terminalbench2_env.py @@ -365,7 +365,7 @@ async def setup(self): os.makedirs(log_dir, exist_ok=True) run_ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") self._streaming_path = os.path.join(log_dir, f"samples_{run_ts}.jsonl") - self._streaming_file = open(self._streaming_path, "w") + self._streaming_file = open(self._streaming_path, "w", encoding="utf-8") self._streaming_lock = __import__("threading").Lock() print(f" Streaming results to: {self._streaming_path}") diff --git a/environments/benchmarks/yc_bench/yc_bench_env.py b/environments/benchmarks/yc_bench/yc_bench_env.py index 4247ae56c6e42..4fd22495440d2 100644 --- a/environments/benchmarks/yc_bench/yc_bench_env.py +++ b/environments/benchmarks/yc_bench/yc_bench_env.py @@ -422,7 +422,7 @@ async def setup(self): os.makedirs(log_dir, exist_ok=True) run_ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") self._streaming_path = os.path.join(log_dir, f"samples_{run_ts}.jsonl") - self._streaming_file = open(self._streaming_path, "w") + self._streaming_file = open(self._streaming_path, "w", encoding="utf-8") self._streaming_lock = threading.Lock() print(f"\nYC-Bench eval matrix: {len(self.all_eval_items)} runs") diff --git a/gateway/platforms/telegram.py b/gateway/platforms/telegram.py index 0d0ac3866fb89..e4bba209b8243 100644 --- a/gateway/platforms/telegram.py +++ b/gateway/platforms/telegram.py @@ -744,7 +744,7 @@ def _persist_dm_topic_thread_id(self, chat_id: int, topic_name: str, thread_id: return import yaml as _yaml - with open(config_path, "r") as f: + with open(config_path, "r", encoding="utf-8") as f: config = _yaml.safe_load(f) or {} # Navigate to platforms.telegram.extra.dm_topics @@ -3516,7 +3516,7 @@ def _reload_dm_topics_from_config(self) -> None: return import yaml as _yaml - with open(config_path, "r") as f: + with open(config_path, "r", encoding="utf-8") as f: config = _yaml.safe_load(f) or {} dm_topics = ( diff --git a/gateway/platforms/whatsapp.py b/gateway/platforms/whatsapp.py index 4c781926ac497..fda25de410e50 100644 --- a/gateway/platforms/whatsapp.py +++ b/gateway/platforms/whatsapp.py @@ -526,7 +526,7 @@ async def connect(self) -> bool: # messages are preserved for troubleshooting. whatsapp_mode = os.getenv("WHATSAPP_MODE", "self-chat") self._bridge_log = self._session_path.parent / "bridge.log" - bridge_log_fh = open(self._bridge_log, "a") + bridge_log_fh = open(self._bridge_log, "a", encoding="utf-8") self._bridge_log_fh = bridge_log_fh # Build bridge subprocess environment. @@ -1170,7 +1170,7 @@ async def _build_message_event(self, data: Dict[str, Any]) -> Optional[MessageEv if file_size > MAX_TEXT_INJECT_BYTES: print(f"[{self.name}] Skipping text injection for {doc_path} ({file_size} bytes > {MAX_TEXT_INJECT_BYTES})", flush=True) continue - content = Path(doc_path).read_text(errors="replace") + content = Path(doc_path).read_text(encoding="utf-8", errors="replace") fname = Path(doc_path).name # Remove the doc__ prefix for display display_name = fname diff --git a/gateway/status.py b/gateway/status.py index bdff9aa988d58..33799bd16a198 100644 --- a/gateway/status.py +++ b/gateway/status.py @@ -113,7 +113,7 @@ def _get_process_start_time(pid: int) -> Optional[int]: stat_path = Path(f"/proc/{pid}/stat") try: # Field 22 in /proc//stat is process start time (clock ticks). - return int(stat_path.read_text().split()[21]) + return int(stat_path.read_text(encoding="utf-8").split()[21]) except (FileNotFoundError, IndexError, PermissionError, ValueError, OSError): return None @@ -197,7 +197,7 @@ def _read_json_file(path: Path) -> Optional[dict[str, Any]]: if not path.exists(): return None try: - raw = path.read_text().strip() + raw = path.read_text(encoding="utf-8").strip() except OSError: return None if not raw: @@ -523,7 +523,7 @@ def acquire_scoped_lock(scope: str, identity: str, metadata: Optional[dict[str, try: _proc_status = Path(f"/proc/{existing_pid}/status") if _proc_status.exists(): - for _line in _proc_status.read_text().splitlines(): + for _line in _proc_status.read_text(encoding="utf-8").splitlines(): if _line.startswith("State:"): _state = _line.split()[1] if _state in ("T", "t"): # stopped or tracing stop diff --git a/hermes_cli/backup.py b/hermes_cli/backup.py index dce199a5ab4f0..4237c678b19c0 100644 --- a/hermes_cli/backup.py +++ b/hermes_cli/backup.py @@ -573,7 +573,7 @@ def create_quick_snapshot( "total_size": sum(manifest.values()), "files": manifest, } - with open(snap_dir / "manifest.json", "w") as f: + with open(snap_dir / "manifest.json", "w", encoding="utf-8") as f: json.dump(meta, f, indent=2) # Auto-prune @@ -599,7 +599,7 @@ def list_quick_snapshots( manifest_path = d / "manifest.json" if manifest_path.exists(): try: - with open(manifest_path) as f: + with open(manifest_path, encoding="utf-8") as f: results.append(json.load(f)) except (json.JSONDecodeError, OSError): results.append({"id": d.name, "file_count": 0, "total_size": 0}) @@ -629,7 +629,7 @@ def restore_quick_snapshot( if not manifest_path.exists(): return False - with open(manifest_path) as f: + with open(manifest_path, encoding="utf-8") as f: meta = json.load(f) restored = 0 diff --git a/hermes_cli/config.py b/hermes_cli/config.py index f38a795191a77..9f37717465dae 100644 --- a/hermes_cli/config.py +++ b/hermes_cli/config.py @@ -212,7 +212,7 @@ def get_container_exec_info() -> Optional[dict]: try: info = {} - with open(container_mode_file, "r") as f: + with open(container_mode_file, "r", encoding="utf-8") as f: for line in f: line = line.strip() if "=" in line and not line.startswith("#"): @@ -297,7 +297,7 @@ def _is_container() -> bool: return True # LXC / cgroup-based detection try: - with open("/proc/1/cgroup", "r") as f: + with open("/proc/1/cgroup", "r", encoding="utf-8") as f: cgroup_content = f.read() if "docker" in cgroup_content or "lxc" in cgroup_content or "kubepods" in cgroup_content: return True @@ -3452,7 +3452,7 @@ def migrate_config(interactive: bool = True, quiet: bool = False) -> Dict[str, A if not manifest_file.exists(): continue try: - with open(manifest_file) as _mf: + with open(manifest_file, encoding="utf-8") as _mf: manifest = yaml.safe_load(_mf) or {} except Exception: manifest = {} diff --git a/hermes_cli/doctor.py b/hermes_cli/doctor.py index adc97d53c7a1d..97c9f10243e4a 100644 --- a/hermes_cli/doctor.py +++ b/hermes_cli/doctor.py @@ -598,7 +598,7 @@ def run_doctor(args): # Detect stale root-level model keys (known bug source — PR #4329) try: import yaml - with open(config_path) as f: + with open(config_path, encoding="utf-8") as f: raw_config = yaml.safe_load(f) or {} stale_root_keys = [k for k in ("provider", "base_url") if k in raw_config and isinstance(raw_config[k], str)] if stale_root_keys: @@ -1399,7 +1399,7 @@ def _gh_authenticated() -> bool: import yaml as _yaml _mem_cfg_path = HERMES_HOME / "config.yaml" if _mem_cfg_path.exists(): - with open(_mem_cfg_path) as _f: + with open(_mem_cfg_path, encoding="utf-8") as _f: _raw_cfg = _yaml.safe_load(_f) or {} _active_memory_provider = (_raw_cfg.get("memory") or {}).get("provider", "") except Exception: diff --git a/hermes_cli/hooks.py b/hermes_cli/hooks.py index de624f246126c..45b3fc6374530 100644 --- a/hermes_cli/hooks.py +++ b/hermes_cli/hooks.py @@ -205,7 +205,7 @@ def _cmd_test(args) -> None: if getattr(args, "payload_file", None): try: - custom = json.loads(Path(args.payload_file).read_text()) + custom = json.loads(Path(args.payload_file).read_text(encoding="utf-8")) if isinstance(custom, dict): payload.update(custom) else: diff --git a/hermes_cli/kanban_db.py b/hermes_cli/kanban_db.py index 2f1d8bb59a172..a66e090769901 100644 --- a/hermes_cli/kanban_db.py +++ b/hermes_cli/kanban_db.py @@ -2835,7 +2835,7 @@ def _pid_alive(pid: Optional[int]) -> bool: # where we have a cheap, deterministic process-state probe. if sys.platform == "linux": try: - with open(f"/proc/{int(pid)}/status", "r") as f: + with open(f"/proc/{int(pid)}/status", "r", encoding="utf-8") as f: for line in f: if line.startswith("State:"): # "State:\tZ (zombie)" → dead diff --git a/hermes_cli/memory_setup.py b/hermes_cli/memory_setup.py index 158f80a7669f4..7b2c606728830 100644 --- a/hermes_cli/memory_setup.py +++ b/hermes_cli/memory_setup.py @@ -69,7 +69,7 @@ def _install_dependencies(provider_name: str) -> None: try: import yaml - with open(yaml_path) as f: + with open(yaml_path, encoding="utf-8") as f: meta = yaml.safe_load(f) or {} except Exception: return @@ -377,7 +377,7 @@ def _write_env_vars(env_path: Path, env_writes: dict) -> None: if key not in updated_keys: new_lines.append(f"{key}={val}") - env_path.write_text("\n".join(new_lines) + "\n") + env_path.write_text("\n".join(new_lines) + "\n", encoding="utf-8") # --------------------------------------------------------------------------- diff --git a/hermes_cli/model_catalog.py b/hermes_cli/model_catalog.py index 6ec7c4ec51d9f..a1f4b76156663 100644 --- a/hermes_cli/model_catalog.py +++ b/hermes_cli/model_catalog.py @@ -173,7 +173,7 @@ def _read_disk_cache() -> tuple[dict[str, Any] | None, float]: except (OSError, FileNotFoundError): return (None, 0.0) try: - with open(path) as fh: + with open(path, encoding="utf-8") as fh: data = json.load(fh) except (OSError, json.JSONDecodeError): return (None, 0.0) @@ -187,7 +187,7 @@ def _write_disk_cache(data: dict[str, Any]) -> None: try: path.parent.mkdir(parents=True, exist_ok=True) tmp = path.with_suffix(path.suffix + ".tmp") - with open(tmp, "w") as fh: + with open(tmp, "w", encoding="utf-8") as fh: json.dump(data, fh, indent=2) fh.write("\n") atomic_replace(tmp, path) diff --git a/hermes_cli/oneshot.py b/hermes_cli/oneshot.py index ca30f079046fa..b1e774b756dfc 100644 --- a/hermes_cli/oneshot.py +++ b/hermes_cli/oneshot.py @@ -174,7 +174,7 @@ def run_oneshot( # Redirect stderr AND stdout to devnull for the entire call tree. # We'll print the final response to the real stdout at the end. real_stdout = sys.stdout - devnull = open(os.devnull, "w") + devnull = open(os.devnull, "w", encoding="utf-8") try: with redirect_stdout(devnull), redirect_stderr(devnull): diff --git a/hermes_cli/plugins.py b/hermes_cli/plugins.py index 12674577376c1..2171e6d50dc44 100644 --- a/hermes_cli/plugins.py +++ b/hermes_cli/plugins.py @@ -870,7 +870,7 @@ def _parse_manifest( if yaml is None: logger.warning("PyYAML not installed – cannot load %s", manifest_file) return None - data = yaml.safe_load(manifest_file.read_text()) or {} + data = yaml.safe_load(manifest_file.read_text(encoding="utf-8")) or {} name = data.get("name", plugin_dir.name) key = f"{prefix}/{plugin_dir.name}" if prefix else name diff --git a/hermes_cli/plugins_cmd.py b/hermes_cli/plugins_cmd.py index a13e1b212c67e..bb4fe0f29da14 100644 --- a/hermes_cli/plugins_cmd.py +++ b/hermes_cli/plugins_cmd.py @@ -127,7 +127,7 @@ def _read_manifest(plugin_dir: Path) -> dict: try: import yaml - with open(manifest_file) as f: + with open(manifest_file, encoding="utf-8") as f: return yaml.safe_load(f) or {} except Exception as e: logger.warning("Failed to read plugin.yaml in %s: %s", plugin_dir, e) @@ -703,7 +703,7 @@ def _discover_all_plugins() -> list: description = "" if yaml: try: - with open(manifest_file) as f: + with open(manifest_file, encoding="utf-8") as f: manifest = yaml.safe_load(f) or {} name = manifest.get("name", d.name) version = manifest.get("version", "") diff --git a/hermes_cli/profiles.py b/hermes_cli/profiles.py index 17d99a613d1c4..43d48c26e4378 100644 --- a/hermes_cli/profiles.py +++ b/hermes_cli/profiles.py @@ -354,7 +354,7 @@ def _read_config_model(profile_dir: Path) -> tuple: return None, None try: import yaml - with open(config_path, "r") as f: + with open(config_path, "r", encoding="utf-8") as f: cfg = yaml.safe_load(f) or {} model_cfg = cfg.get("model", {}) if isinstance(model_cfg, str): diff --git a/hermes_cli/skills_hub.py b/hermes_cli/skills_hub.py index 88c0978a93b15..3bfb0631cc4bd 100644 --- a/hermes_cli/skills_hub.py +++ b/hermes_cli/skills_hub.py @@ -1257,7 +1257,7 @@ def do_snapshot_export(output_path: str, console: Optional[Console] = None) -> N sys.stdout.write(payload) else: out = Path(output_path) - out.write_text(payload) + out.write_text(payload, encoding="utf-8") c.print(f"[bold green]Snapshot exported:[/] {out}") c.print(f"[dim]{len(installed)} skill(s), {len(tap_list)} tap(s)[/]\n") @@ -1274,7 +1274,7 @@ def do_snapshot_import(input_path: str, force: bool = False, return try: - snapshot = json.loads(inp.read_text()) + snapshot = json.loads(inp.read_text(encoding="utf-8")) except json.JSONDecodeError: c.print(f"[bold red]Error:[/] Invalid JSON in {inp}\n") return diff --git a/hermes_cli/web_server.py b/hermes_cli/web_server.py index b3a4209b33732..c14cf762a40f2 100644 --- a/hermes_cli/web_server.py +++ b/hermes_cli/web_server.py @@ -692,7 +692,7 @@ def _tail_lines(path: Path, n: int) -> List[str]: if not path.exists(): return [] try: - text = path.read_text(errors="replace") + text = path.read_text(encoding="utf-8", errors="replace") except OSError: return [] lines = text.splitlines() diff --git a/hermes_constants.py b/hermes_constants.py index e63a4ec301e81..bdb8dc9114f82 100644 --- a/hermes_constants.py +++ b/hermes_constants.py @@ -233,7 +233,7 @@ def is_wsl() -> bool: if _wsl_detected is not None: return _wsl_detected try: - with open("/proc/version", "r") as f: + with open("/proc/version", "r", encoding="utf-8") as f: _wsl_detected = "microsoft" in f.read().lower() except Exception: _wsl_detected = False @@ -260,7 +260,7 @@ def is_container() -> bool: _container_detected = True return True try: - with open("/proc/1/cgroup", "r") as f: + with open("/proc/1/cgroup", "r", encoding="utf-8") as f: cgroup = f.read() if "docker" in cgroup or "podman" in cgroup or "/lxc/" in cgroup: _container_detected = True diff --git a/hermes_time.py b/hermes_time.py index 9f172d28ffb14..aceb82b3e5b76 100644 --- a/hermes_time.py +++ b/hermes_time.py @@ -50,7 +50,7 @@ def _resolve_timezone_name() -> str: import yaml config_path = get_config_path() if config_path.exists(): - with open(config_path) as f: + with open(config_path, encoding="utf-8") as f: cfg = yaml.safe_load(f) or {} tz_cfg = cfg.get("timezone", "") if isinstance(tz_cfg, str) and tz_cfg.strip(): diff --git a/rl_cli.py b/rl_cli.py index 8054b627e9a56..d494c1addb2ae 100644 --- a/rl_cli.py +++ b/rl_cli.py @@ -82,7 +82,7 @@ def load_hermes_config() -> dict: if config_path.exists(): try: - with open(config_path, "r") as f: + with open(config_path, "r", encoding='utf-8') as f: file_config = yaml.safe_load(f) or {} # Get model from config diff --git a/run_agent.py b/run_agent.py index 5b416a0fc3fe8..3bf3ed35b7522 100644 --- a/run_agent.py +++ b/run_agent.py @@ -3686,7 +3686,7 @@ def _bg_review_auto_deny(command, description, **kwargs): pass review_agent = None try: - with open(os.devnull, "w") as _devnull, \ + with open(os.devnull, "w", encoding="utf-8") as _devnull, \ contextlib.redirect_stdout(_devnull), \ contextlib.redirect_stderr(_devnull): # Inherit the parent agent's live runtime (provider, model, diff --git a/scripts/build_model_catalog.py b/scripts/build_model_catalog.py index cd21c929e746b..102ae2b05b0b6 100755 --- a/scripts/build_model_catalog.py +++ b/scripts/build_model_catalog.py @@ -81,7 +81,7 @@ def build_catalog() -> dict: def main() -> int: catalog = build_catalog() os.makedirs(os.path.dirname(OUTPUT_PATH), exist_ok=True) - with open(OUTPUT_PATH, "w") as fh: + with open(OUTPUT_PATH, "w", encoding="utf-8") as fh: json.dump(catalog, fh, indent=2) fh.write("\n") diff --git a/scripts/build_skills_index.py b/scripts/build_skills_index.py index efa1ba76edc15..96a0b6375969b 100644 --- a/scripts/build_skills_index.py +++ b/scripts/build_skills_index.py @@ -304,7 +304,7 @@ def main(): } os.makedirs(os.path.dirname(OUTPUT_PATH), exist_ok=True) - with open(OUTPUT_PATH, "w") as f: + with open(OUTPUT_PATH, "w", encoding="utf-8") as f: json.dump(index, f, separators=(",", ":"), ensure_ascii=False) elapsed = time.time() - overall_start diff --git a/scripts/contributor_audit.py b/scripts/contributor_audit.py index 9849dc81f0b73..50bf304264296 100644 --- a/scripts/contributor_audit.py +++ b/scripts/contributor_audit.py @@ -291,7 +291,7 @@ def check_release_file(release_file, all_contributors): missing: set of handles NOT found in the file """ try: - content = Path(release_file).read_text() + content = Path(release_file).read_text(encoding="utf-8") except FileNotFoundError: print(f" [error] Release file not found: {release_file}", file=sys.stderr) return set(), set(all_contributors) diff --git a/scripts/discord-voice-doctor.py b/scripts/discord-voice-doctor.py index 8227c8d11c76f..e295225a0e362 100755 --- a/scripts/discord-voice-doctor.py +++ b/scripts/discord-voice-doctor.py @@ -242,7 +242,7 @@ def check_config(groq_key, eleven_key): if config_path.exists(): try: import yaml - with open(config_path) as f: + with open(config_path, encoding="utf-8") as f: cfg = yaml.safe_load(f) or {} stt_provider = cfg.get("stt", {}).get("provider", "local") diff --git a/scripts/profile-tui.py b/scripts/profile-tui.py index 87b2d6c1d5d3e..3a39e4c5ed892 100755 --- a/scripts/profile-tui.py +++ b/scripts/profile-tui.py @@ -111,7 +111,7 @@ def summarize(log: Path, since_ts_ms: int) -> dict[str, Any]: frame_events: list[dict[str, Any]] = [] if not log.exists(): return {"error": f"no log at {log}", "react": [], "frame": []} - for line in log.read_text().splitlines(): + for line in log.read_text(encoding="utf-8").splitlines(): line = line.strip() if not line: continue @@ -505,7 +505,7 @@ def main() -> int: if args.save: path = Path(f"/tmp/perf-{args.save}.json") - path.write_text(json.dumps(metrics, indent=2)) + path.write_text(json.dumps(metrics, indent=2), encoding="utf-8") print(f"\n• saved: {path}") if args.compare: diff --git a/scripts/release.py b/scripts/release.py index 07e2a3a7478c0..0a866f2ab1469 100755 --- a/scripts/release.py +++ b/scripts/release.py @@ -1359,7 +1359,7 @@ def main(): ) if args.output: - Path(args.output).write_text(changelog) + Path(args.output).write_text(changelog, encoding="utf-8") print(f"Changelog written to {args.output}") else: print(changelog) diff --git a/tools/browser_tool.py b/tools/browser_tool.py index 5986ea584b9ff..214bd25b7f90f 100644 --- a/tools/browser_tool.py +++ b/tools/browser_tool.py @@ -751,7 +751,7 @@ def _run_tmp(cmd: str, cmd_args: List[str]) -> Dict[str, Any]: proc.wait() return {"success": False, "error": f"Chrome fallback '{cmd}' timed out"} try: - with open(stdout_path, "r") as f: + with open(stdout_path, "r", encoding="utf-8") as f: stdout = f.read().strip() if stdout: return json.loads(stdout.split("\n")[-1]) @@ -1110,7 +1110,7 @@ def _write_owner_pid(socket_dir: str, session_name: str) -> None: """ try: path = os.path.join(socket_dir, f"{session_name}.owner_pid") - with open(path, "w") as f: + with open(path, "w", encoding="utf-8") as f: f.write(str(os.getpid())) except OSError as exc: logger.debug("Could not write owner_pid file for %s: %s", @@ -1174,7 +1174,7 @@ def _reap_orphaned_browser_sessions(): owner_alive: Optional[bool] = None # None = owner_pid missing/unreadable if os.path.isfile(owner_pid_file): try: - owner_pid = int(Path(owner_pid_file).read_text().strip()) + owner_pid = int(Path(owner_pid_file).read_text(encoding="utf-8").strip()) try: os.kill(owner_pid, 0) owner_alive = True @@ -1209,7 +1209,7 @@ def _reap_orphaned_browser_sessions(): continue try: - daemon_pid = int(Path(pid_file).read_text().strip()) + daemon_pid = int(Path(pid_file).read_text(encoding="utf-8").strip()) except (ValueError, OSError): shutil.rmtree(socket_dir, ignore_errors=True) continue @@ -1834,7 +1834,7 @@ def _run_browser_command( # Detect AppArmor user namespace restrictions (Ubuntu 23.10+) _userns_restrict = "/proc/sys/kernel/apparmor_restrict_unprivileged_userns" try: - with open(_userns_restrict) as _f: + with open(_userns_restrict, encoding="utf-8") as _f: if _f.read().strip() == "1": _needs_sandbox_bypass = True logger.debug( @@ -1879,9 +1879,9 @@ def _run_browser_command( result = {"success": False, "error": f"Command timed out after {timeout} seconds"} # Fall through to fallback check below else: - with open(stdout_path, "r") as f: + with open(stdout_path, "r", encoding="utf-8") as f: stdout = f.read() - with open(stderr_path, "r") as f: + with open(stderr_path, "r", encoding="utf-8") as f: stderr = f.read() returncode = proc.returncode @@ -3180,7 +3180,7 @@ def _cleanup_single_browser_session(task_id: str) -> None: pid_file = os.path.join(socket_dir, f"{session_name}.pid") if os.path.isfile(pid_file): try: - daemon_pid = int(Path(pid_file).read_text().strip()) + daemon_pid = int(Path(pid_file).read_text(encoding="utf-8").strip()) os.kill(daemon_pid, signal.SIGTERM) logger.debug("Killed daemon pid %s for %s", daemon_pid, session_name) except (ProcessLookupError, ValueError, PermissionError, OSError): @@ -3323,7 +3323,7 @@ def _running_in_docker() -> bool: if os.path.exists("/.dockerenv"): return True try: - with open("/proc/1/cgroup", "rt") as fp: + with open("/proc/1/cgroup", "rt", encoding="utf-8") as fp: return "docker" in fp.read() except OSError: return False diff --git a/tools/code_execution_tool.py b/tools/code_execution_tool.py index 47271223f351a..828fb559e8008 100644 --- a/tools/code_execution_tool.py +++ b/tools/code_execution_tool.py @@ -517,7 +517,7 @@ def _rpc_server_loop( # their status prints don't leak into the CLI spinner. try: _real_stdout, _real_stderr = sys.stdout, sys.stderr - devnull = open(os.devnull, "w") + devnull = open(os.devnull, "w", encoding="utf-8") try: sys.stdout = devnull sys.stderr = devnull @@ -791,7 +791,7 @@ def _rpc_poll_loop( # Dispatch through the standard tool handler try: _real_stdout, _real_stderr = sys.stdout, sys.stderr - devnull = open(os.devnull, "w") + devnull = open(os.devnull, "w", encoding="utf-8") try: sys.stdout = devnull sys.stderr = devnull diff --git a/tools/environments/base.py b/tools/environments/base.py index a4fbea7b2f99f..8a53cefb5bf79 100644 --- a/tools/environments/base.py +++ b/tools/environments/base.py @@ -158,7 +158,7 @@ def _load_json_store(path: Path) -> dict: """Load a JSON file as a dict, returning ``{}`` on any error.""" if path.exists(): try: - return json.loads(path.read_text()) + return json.loads(path.read_text(encoding="utf-8")) except Exception: pass return {} @@ -167,7 +167,7 @@ def _load_json_store(path: Path) -> dict: def _save_json_store(path: Path, data: dict) -> None: """Write *data* as pretty-printed JSON to *path*.""" path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(data, indent=2)) + path.write_text(json.dumps(data, indent=2), encoding="utf-8") def _file_mtime_key(host_path: str) -> tuple[float, int] | None: diff --git a/tools/environments/file_sync.py b/tools/environments/file_sync.py index 742e024ad8650..b778be87eb8aa 100644 --- a/tools/environments/file_sync.py +++ b/tools/environments/file_sync.py @@ -284,7 +284,7 @@ def _sync_back_locked(self, lock_path: Path) -> None: # Windows: no flock — run without serialization self._sync_back_impl() return - lock_fd = open(lock_path, "w") + lock_fd = open(lock_path, "w", encoding="utf-8") try: fcntl.flock(lock_fd, fcntl.LOCK_EX) self._sync_back_impl() diff --git a/tools/environments/local.py b/tools/environments/local.py index 527a0539ffcea..e952ab5e6f9fe 100644 --- a/tools/environments/local.py +++ b/tools/environments/local.py @@ -562,7 +562,7 @@ def _update_cwd(self, result: dict): ``_run_bash`` recovery path will resolve a safe fallback if needed. """ try: - with open(self._cwd_file) as f: + with open(self._cwd_file, encoding="utf-8") as f: cwd_path = f.read().strip() if cwd_path and os.path.isdir(cwd_path): self.cwd = cwd_path diff --git a/tools/mcp_tool.py b/tools/mcp_tool.py index 73480ada9f5aa..8eb469597ebb9 100644 --- a/tools/mcp_tool.py +++ b/tools/mcp_tool.py @@ -1992,7 +1992,7 @@ def _snapshot_child_pids() -> set: # Linux: read from /proc try: children_path = f"/proc/{my_pid}/task/{my_pid}/children" - with open(children_path) as f: + with open(children_path, encoding="utf-8") as f: return {int(p) for p in f.read().split() if p.strip()} except (FileNotFoundError, OSError, ValueError): pass diff --git a/tools/rl_training_tool.py b/tools/rl_training_tool.py index 7a6478b42c9c4..d2a5c3bfbb56f 100644 --- a/tools/rl_training_tool.py +++ b/tools/rl_training_tool.py @@ -169,7 +169,7 @@ def _scan_environments() -> List[EnvironmentInfo]: continue try: - with open(py_file, "r") as f: + with open(py_file, "r", encoding="utf-8") as f: tree = ast.parse(f.read()) for node in ast.walk(tree): @@ -333,7 +333,7 @@ async def _spawn_training_run(run_state: RunState, config_path: Path): # File must stay open while the subprocess runs; we store the handle # on run_state so _stop_training_run() can close it when done. - api_log_file = open(api_log, "w") # closed by _stop_training_run + api_log_file = open(api_log, "w", encoding="utf-8") # closed by _stop_training_run run_state.api_log_file = api_log_file run_state.api_process = subprocess.Popen( ["run-api"], @@ -356,7 +356,7 @@ async def _spawn_training_run(run_state: RunState, config_path: Path): # Step 2: Start the Tinker trainer logger.info("[%s] Starting Tinker trainer: launch_training.py --config %s", run_id, config_path) - trainer_log_file = open(trainer_log, "w") # closed by _stop_training_run + trainer_log_file = open(trainer_log, "w", encoding="utf-8") # closed by _stop_training_run run_state.trainer_log_file = trainer_log_file run_state.trainer_process = subprocess.Popen( [sys.executable, "launch_training.py", "--config", str(config_path)], @@ -397,7 +397,7 @@ async def _spawn_training_run(run_state: RunState, config_path: Path): logger.info("[%s] Starting environment: %s serve", run_id, env_info.file_path) - env_log_file = open(env_log, "w") # closed by _stop_training_run + env_log_file = open(env_log, "w", encoding="utf-8") # closed by _stop_training_run run_state.env_log_file = env_log_file run_state.env_process = subprocess.Popen( [sys.executable, str(env_info.file_path), "serve", "--config", str(config_path)], @@ -777,7 +777,7 @@ async def rl_start_training() -> str: if "wandb_name" in _current_config and _current_config["wandb_name"]: run_config["env"]["wandb_name"] = _current_config["wandb_name"] - with open(config_path, "w") as f: + with open(config_path, "w", encoding="utf-8") as f: yaml.dump(run_config, f, default_flow_style=False) # Create run state @@ -1206,7 +1206,7 @@ async def read_stream(stream, lines_list, prefix=""): stderr_text = "\n".join(stderr_lines) # Write logs to files for inspection outside CLI - with open(log_file, "w") as f: + with open(log_file, "w", encoding="utf-8") as f: f.write(f"Command: {cmd_display}\n") f.write(f"Working dir: {TINKER_ATROPOS_ROOT}\n") f.write(f"Return code: {process.returncode}\n") @@ -1238,7 +1238,7 @@ async def read_stream(stream, lines_list, prefix=""): # Parse the output JSONL file if output_file.exists(): # Read JSONL file (one JSON object per line = one step) - with open(output_file, "r") as f: + with open(output_file, "r", encoding="utf-8") as f: for line in f: line = line.strip() if not line: diff --git a/tools/skills_hub.py b/tools/skills_hub.py index aaeabd2c289b3..17d1a45695764 100644 --- a/tools/skills_hub.py +++ b/tools/skills_hub.py @@ -219,7 +219,7 @@ def _try_github_app(self) -> Optional[str]: key_file = Path(key_path) if not key_file.exists(): return None - private_key = key_file.read_text() + private_key = key_file.read_text(encoding="utf-8") now = int(time.time()) payload = { @@ -2667,7 +2667,7 @@ def append_audit_log(action: str, skill_name: str, source: str, parts.append(extra) line = " ".join(parts) + "\n" try: - with open(AUDIT_LOG, "a") as f: + with open(AUDIT_LOG, "a", encoding="utf-8") as f: f.write(line) except OSError as e: logger.debug("Could not write audit log: %s", e) diff --git a/tools/tirith_security.py b/tools/tirith_security.py index 2d0ebf49717f7..bad94c96f7fad 100644 --- a/tools/tirith_security.py +++ b/tools/tirith_security.py @@ -126,7 +126,7 @@ def _read_failure_reason() -> str | None: mtime = os.path.getmtime(p) if (time.time() - mtime) >= _MARKER_TTL: return None - with open(p, "r") as f: + with open(p, "r", encoding="utf-8") as f: return f.read().strip() except OSError: return None @@ -160,7 +160,7 @@ def _mark_install_failed(reason: str = ""): try: p = _failure_marker_path() os.makedirs(os.path.dirname(p), exist_ok=True) - with open(p, "w") as f: + with open(p, "w", encoding="utf-8") as f: f.write(reason) except OSError: pass @@ -257,7 +257,7 @@ def _verify_cosign(checksums_path: str, sig_path: str, cert_path: str) -> bool | def _verify_checksum(archive_path: str, checksums_path: str, archive_name: str) -> bool: """Verify SHA-256 of the archive against checksums.txt.""" expected = None - with open(checksums_path) as f: + with open(checksums_path, encoding="utf-8") as f: for line in f: # Format: " " parts = line.strip().split(" ", 1) diff --git a/tools/voice_mode.py b/tools/voice_mode.py index 66ecb242c6725..6166ade2a3f58 100644 --- a/tools/voice_mode.py +++ b/tools/voice_mode.py @@ -110,7 +110,7 @@ def detect_audio_environment() -> dict: # WSL detection — PulseAudio bridge makes audio work in WSL. # Only block if PULSE_SERVER is not configured. try: - with open('/proc/version', 'r') as f: + with open('/proc/version', 'r', encoding="utf-8") as f: if 'microsoft' in f.read().lower(): if os.environ.get('PULSE_SERVER'): notices.append("Running in WSL with PulseAudio bridge") diff --git a/trajectory_compressor.py b/trajectory_compressor.py index 2efdeaf165f9a..fcf699d1fdc6e 100644 --- a/trajectory_compressor.py +++ b/trajectory_compressor.py @@ -125,7 +125,7 @@ class CompressionConfig: @classmethod def from_yaml(cls, yaml_path: str) -> "CompressionConfig": """Load configuration from YAML file.""" - with open(yaml_path, 'r') as f: + with open(yaml_path, 'r', encoding="utf-8") as f: data = yaml.safe_load(f) config = cls() @@ -1174,7 +1174,7 @@ async def process_single(file_path: Path, entry_idx: int, entry: Dict, # Save metrics if self.config.metrics_enabled: metrics_path = output_dir / self.config.metrics_output_file - with open(metrics_path, 'w') as f: + with open(metrics_path, 'w', encoding="utf-8") as f: json.dump(self.aggregate_metrics.to_dict(), f, indent=2) console.print(f"\n💾 Metrics saved to {metrics_path}") diff --git a/tui_gateway/server.py b/tui_gateway/server.py index 7219b811e4fc9..27c691e9078dd 100644 --- a/tui_gateway/server.py +++ b/tui_gateway/server.py @@ -660,7 +660,7 @@ def _load_cfg() -> dict: if _cfg_cache is not None and _cfg_mtime == mtime and _cfg_path == p: return copy.deepcopy(_cfg_cache) if p.exists(): - with open(p) as f: + with open(p, encoding="utf-8") as f: data = yaml.safe_load(f) or {} else: data = {} @@ -679,7 +679,7 @@ def _save_cfg(cfg: dict): import yaml path = _hermes_home / "config.yaml" - with open(path, "w") as f: + with open(path, "w", encoding="utf-8") as f: yaml.safe_dump(cfg, f) with _cfg_lock: _cfg_cache = copy.deepcopy(cfg) @@ -2588,7 +2588,7 @@ def _(rid, params: dict) -> dict: f"hermes_conversation_{_time.strftime('%Y%m%d_%H%M%S')}.json" ) try: - with open(filename, "w") as f: + with open(filename, "w", encoding="utf-8") as f: json.dump( { "model": getattr(session["agent"], "model", ""), From e0c03defd5ebbd7057d4e0e0d6ba771afa439861 Mon Sep 17 00:00:00 2001 From: Teknium <127238744+teknium1@users.noreply.github.com> Date: Thu, 7 May 2026 19:36:13 -0700 Subject: [PATCH 19/20] lint: enable PLW1514 as a blocking ruff rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turns the existing 'all lints disabled' stance into 'exactly one lint enabled' — PLW1514 (unspecified-encoding) catches bare open() / read_text() / write_text() calls that default to locale encoding on Windows (cp1252), silently corrupting non-ASCII content. Changes: 1. pyproject.toml - Migrate [tool.ruff] top-level select → [tool.ruff.lint].select (deprecated config location, ruff was warning on every run) - Add preview = true (PLW1514 is a preview rule in ruff 0.15.x) - select = ['PLW1514'] (exactly one rule, deliberately minimal) - per-file-ignores exempt tests/, plugins/, skills/, optional-skills/ — those have their own conventions or intentionally exercise edge cases 2. website/scripts/extract-skills.py - Fix 3 remaining bare opens (website/ was excluded from the main sweep but needed for ruff check . to go green) 3. tests/test_lint_config.py (new, 5 tests) - Guards against accidental rule removal. If someone deletes PLW1514 from the select list or disables preview mode, these tests fail with a loud message explaining why the rule exists. Paired with a companion commit (held locally for now, pending a token with workflow scope) that adds a blocking ruff step to .github/workflows/ lint.yml. Without that companion commit, ruff is configured correctly but nothing in CI enforces it yet — the advisory PR comment will still surface new PLW1514 violations though, so authors see them. Verified: ruff check . → exit 0, 0 violations across the repo. Test suite: 90 passed, 14 skipped, 0 failed. --- pyproject.toml | 20 +++++- tests/test_lint_config.py | 115 ++++++++++++++++++++++++++++++ website/scripts/extract-skills.py | 6 +- 3 files changed, 137 insertions(+), 4 deletions(-) create mode 100644 tests/test_lint_config.py diff --git a/pyproject.toml b/pyproject.toml index 592f5b6b214a5..8f69c5b468a08 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -188,7 +188,25 @@ exclude = ["tinker-atropos"] [tool.ruff] exclude = ["tinker-atropos"] -select = [] # disable all lints for now, until we've wrangled typechecks a bit more :3 +preview = true # required for PLW1514 (unspecified-encoding) — preview rule + +[tool.ruff.lint] +# All other lints are intentionally disabled (see comment history on this +# file) while we wrangle typechecks — but PLW1514 is too load-bearing to +# keep off. Bare open()/read_text()/write_text() in text mode defaults to +# the system locale encoding on Windows (cp1252 on US-locale installs), +# which silently corrupts any non-ASCII file content. We had three +# separate Windows sandbox regressions in one debug session before +# adding the explicit encoding. This rule keeps new code honest. +select = ["PLW1514"] + +[tool.ruff.lint.per-file-ignores] +# Tests can intentionally exercise locale-encoding edge cases. +"tests/**" = ["PLW1514"] +# Skills and plugins are partially user-authored — their own conventions. +"skills/**" = ["PLW1514"] +"optional-skills/**" = ["PLW1514"] +"plugins/**" = ["PLW1514"] [tool.uv] exclude-newer = "7 days" diff --git a/tests/test_lint_config.py b/tests/test_lint_config.py new file mode 100644 index 0000000000000..23ca0d6a43aa3 --- /dev/null +++ b/tests/test_lint_config.py @@ -0,0 +1,115 @@ +"""Tests for ruff lint config — guards against accidental rule removal. + +PLW1514 (unspecified-encoding) was enabled after a debug session on +Windows turned up three separate UTF-8 regressions in execute_code. +The rule catches bare ``open()`` / ``read_text()`` / ``write_text()`` +calls that default to locale encoding — cp1252 on Windows — which +silently corrupts non-ASCII content. + +These tests ensure: + 1. PLW1514 stays in ``[tool.ruff.lint.select]`` + 2. The CI workflow's blocking step still invokes ``ruff check .`` + 3. pyproject.toml has ``preview = true`` (required — PLW1514 is a + preview rule in ruff 0.15.x) + +If someone removes any of these, CI stops enforcing UTF-8-explicit +opens and we're back to the original Windows-regression trap. +""" + +from __future__ import annotations + +import pathlib + +import pytest + +try: + import tomllib # Python 3.11+ +except ImportError: # pragma: no cover — 3.10 and earlier + import tomli as tomllib # type: ignore + +REPO_ROOT = pathlib.Path(__file__).resolve().parent.parent + + +def _load_pyproject() -> dict: + with open(REPO_ROOT / "pyproject.toml", "rb") as fh: + return tomllib.load(fh) + + +class TestRuffConfig: + def test_plw1514_is_in_select_list(self): + """pyproject.toml must keep PLW1514 in [tool.ruff.lint.select].""" + cfg = _load_pyproject() + selected = ( + cfg.get("tool", {}) + .get("ruff", {}) + .get("lint", {}) + .get("select", []) + ) + assert "PLW1514" in selected, ( + "PLW1514 (unspecified-encoding) was removed from " + "[tool.ruff.lint.select]. This rule blocks bare open() calls " + "that default to locale encoding on Windows — removing it " + "re-opens a class of UTF-8 bugs we already paid to close. " + "If you genuinely want to remove it, delete this test in the " + "same commit so the intent is deliberate." + ) + + def test_preview_mode_enabled(self): + """PLW1514 is a preview rule in ruff 0.15.x — preview=true is + required for it to actually run.""" + cfg = _load_pyproject() + ruff_cfg = cfg.get("tool", {}).get("ruff", {}) + assert ruff_cfg.get("preview") is True, ( + "[tool.ruff] preview=true is required — PLW1514 is a preview " + "rule and silently becomes a no-op without it. If this ever " + "becomes a stable rule, you can drop preview=true but must " + "verify PLW1514 still fires in a sample test run first." + ) + + +class TestLintWorkflow: + WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "lint.yml" + + def test_workflow_exists(self): + assert self.WORKFLOW_PATH.exists(), ( + f"CI workflow missing: {self.WORKFLOW_PATH}" + ) + + def test_workflow_has_blocking_ruff_step(self): + """The workflow must run a blocking ``ruff check .`` step + (one without --exit-zero) so violations fail the job.""" + content = self.WORKFLOW_PATH.read_text(encoding="utf-8") + # Look for the blocking step's named line + its command. We want + # at least one ``ruff check .`` that does NOT have ``--exit-zero`` + # nearby. + import re + # Split into lines and find ruff check invocations + lines = content.splitlines() + found_blocking = False + for i, line in enumerate(lines): + stripped = line.strip() + if stripped.startswith("ruff check") and "--exit-zero" not in stripped: + # Also check it's not piped to `|| true` which would mask + # the exit code. + window = " ".join(lines[i:i + 3]) + if "|| true" not in window: + found_blocking = True + break + assert found_blocking, ( + "lint.yml no longer contains a blocking ``ruff check .`` step " + "(one without --exit-zero and not masked by || true). " + "Restore it — the PLW1514 rule is only useful if CI actually " + "fails on violation." + ) + + def test_workflow_yaml_is_valid(self): + """Workflow file must parse as valid YAML (can't ship a broken + CI config to main).""" + import yaml + content = self.WORKFLOW_PATH.read_text(encoding="utf-8") + try: + parsed = yaml.safe_load(content) + except yaml.YAMLError as exc: + pytest.fail(f"lint.yml is not valid YAML: {exc}") + assert isinstance(parsed, dict) + assert "jobs" in parsed diff --git a/website/scripts/extract-skills.py b/website/scripts/extract-skills.py index 79413aec0fe6d..b106a9527b889 100644 --- a/website/scripts/extract-skills.py +++ b/website/scripts/extract-skills.py @@ -69,7 +69,7 @@ def extract_local_skills(): continue skill_path = os.path.join(root, "SKILL.md") - with open(skill_path) as f: + with open(skill_path, encoding="utf-8") as f: content = f.read() if not content.startswith("---"): @@ -128,7 +128,7 @@ def extract_cached_index_skills(): filepath = os.path.join(INDEX_CACHE_DIR, filename) try: - with open(filepath) as f: + with open(filepath, encoding="utf-8") as f: data = json.load(f) except (json.JSONDecodeError, OSError): continue @@ -254,7 +254,7 @@ def main(): )) os.makedirs(os.path.dirname(OUTPUT), exist_ok=True) - with open(OUTPUT, "w") as f: + with open(OUTPUT, "w", encoding="utf-8") as f: json.dump(all_skills, f, indent=2) print(f"Extracted {len(all_skills)} skills to {OUTPUT}") From ec1714e71f90691e1cf412796e9a4b4ba0d934f4 Mon Sep 17 00:00:00 2001 From: alt-glitch Date: Fri, 8 May 2026 14:13:06 +0530 Subject: [PATCH 20/20] fix(install.ps1): handle uv stderr output with ErrorActionPreference=Stop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On fresh Windows installs, `uv python install` writes download progress to stderr. With $ErrorActionPreference = 'Stop' (set globally in the script), PowerShell wraps those stderr lines as ErrorRecord objects when captured via 2>&1, then throws a terminating exception — landing in the catch block even though uv exits 0 and Python was installed successfully. Fix: temporarily set ErrorActionPreference to 'Continue' around the native uv call, then verify success with `uv python find` which is the reliable signal regardless of exit code / stderr noise. Tested on Windows 11 (build 26200) with ExecutionPolicy=Restricted, uv 0.11.11, fresh machine with no prior Python install. --- scripts/install.ps1 | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/scripts/install.ps1 b/scripts/install.ps1 index 2ffca87adb14d..a3840a87e8b5c 100644 --- a/scripts/install.ps1 +++ b/scripts/install.ps1 @@ -145,15 +145,30 @@ function Test-Python { # Python not found — use uv to install it (no admin needed!) Write-Info "Python $PythonVersion not found, installing via uv..." try { + # Temporarily relax ErrorActionPreference: uv writes download progress + # to stderr, and with $ErrorActionPreference = "Stop" PowerShell wraps + # those stderr lines as ErrorRecord objects via 2>&1, then throws a + # terminating exception — even when uv exits 0. This caused fresh + # installs to fail on the first run despite Python being installed + # successfully. We verify success with `uv python find` afterwards + # which is the reliable signal regardless of exit code semantics. + $prevEAP = $ErrorActionPreference + $ErrorActionPreference = "Continue" $uvOutput = & $UvCmd python install $PythonVersion 2>&1 - if ($LASTEXITCODE -eq 0) { - $pythonPath = & $UvCmd python find $PythonVersion 2>$null - if ($pythonPath) { - $ver = & $pythonPath --version 2>$null - Write-Success "Python installed: $ver" - return $true - } - } else { + $uvExitCode = $LASTEXITCODE + $ErrorActionPreference = $prevEAP + + # Check if Python is now available (more reliable than exit code + # since uv may return non-zero due to "already installed" etc.) + $pythonPath = & $UvCmd python find $PythonVersion 2>$null + if ($pythonPath) { + $ver = & $pythonPath --version 2>$null + Write-Success "Python installed: $ver" + return $true + } + + # uv ran but Python still not findable — show what happened + if ($uvExitCode -ne 0) { Write-Warn "uv python install output:" Write-Host $uvOutput -ForegroundColor DarkGray }