Skip to content

fix: use UTF-8 with error-replace for all subprocess.run calls (Windows robustness) - #43494

Closed
Colinchiu007 wants to merge 1 commit into
NousResearch:mainfrom
Colinchiu007:fix/subprocess-encoding-utf8
Closed

fix: use UTF-8 with error-replace for all subprocess.run calls (Windows robustness)#43494
Colinchiu007 wants to merge 1 commit into
NousResearch:mainfrom
Colinchiu007:fix/subprocess-encoding-utf8

Conversation

@Colinchiu007

Copy link
Copy Markdown

Background

On Chinese-Windows the gateway crashes repeatedly with:

UnicodeDecodeError: 'gbk' codec can't decode byte 0xa2 in position 29: illegal multibyte sequence

The traceback points to subprocess.py line 1599 in _readerthread. The root cause: many subprocess.run(..., text=True, ...) calls use the system default encoding (GBK on Chinese Windows). When a subprocess writes any byte sequence that is invalid GBK (UTF-8 JSON, model responses, URLs, etc.), the reader thread raises UnicodeDecodeError, the gateway process exits, and the Desktop UI shows offline.

Setting PYTHONUTF8=1 only changes the Python interpreter's default encoding — it does not affect subprocess.run(text=True), which still falls back to the system code page. After PYTHONUTF8=1 was set, the error appeared as a UTF-8 decode error instead of GBK.

Fix

Replaced every text=True in hermes_cli/gateway.py and hermes_cli/gateway_windows.py with an explicit UTF-8 decode that never raises:

subprocess.run(...,
    capture_output=True,
    encoding='utf-8',
    errors='replace',
    ...)

This forces UTF-8 decoding and replaces invalid byte sequences with the Unicode replacement character, preventing the reader thread from crashing.

Impact

No runtime behaviour changes other than robust subprocess output handling. Works on every platform; completely eliminates the GBK-related crash on Windows.

Testing

  1. Run hermes gateway run on a Chinese-Windows machine.
  2. Verify gateway-crash.log no longer contains UnicodeDecodeError.
  3. Confirm normal operation (model calls, tool usage) works as before.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard labels Jun 10, 2026

@austinpickett austinpickett left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — the core objective (no more bare text=True, explicit UTF-8 on all subprocess calls) is achieved and the Windows robustness goal is met.

Coverage in gateway.py: 13 calls converted from text=Trueencoding='utf-8', errors='replace'
Coverage in gateway_windows.py: 1 call converted ✓

Two calls not normalised (non-blocking, flagging for awareness):
Lines ~387 and ~411 in gateway.py (WMIC and PowerShell process-list calls) already had encoding="utf-8", errors="ignore" before this PR. The diff correctly removes the now-redundant text=True from both, but leaves errors="ignore" rather than normalising to errors="replace". These calls still cannot raise UnicodeDecodeError, so the primary bug is fixed either way. However the PR title claims "ALL subprocess.run calls" use error-replace; these two are a minor discrepancy worth a follow-up if uniform behaviour is desired.

gateway_windows.py comment: The existing comment notes schtasks output uses the console code page, not UTF-8; hard-coding encoding='utf-8' means non-ASCII characters on non-English Windows will get replacement chars (\ufffd). This is acceptable for crash-safety but worth a TODO for a future locale-aware decode.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for identifying the Windows decoding failure mode. Automated hermes-sweeper review found that current main already implements the requested crash-safety behavior.

  • hermes_cli/gateway_windows.py:160-175 decodes schtasks.exe with the locale-aware _schtasks_encoding() and errors="replace"; this was added in 973decc05048f1d6fe990d5d6457e80389d5f1c5 specifically for localized Windows output.
  • hermes_cli/gateway.py:411-450 already gives the Windows WMIC and PowerShell process-scan paths explicit UTF-8 decoding with non-throwing error handling, from cf83982da0fd33a7030e88b65c298167d9abdbc9.
  • tests/hermes_cli/test_gateway_windows.py:45-72 covers the explicit decoder and replacement-error behavior.

The submitted hard-coded UTF-8 change for schtasks would replace the current locale-aware decoder even though schtasks.exe emits the console code page, so the existing main implementation is the stronger fix.

@teknium1 teknium1 closed this Jul 14, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants