fix(tui-gateway): decode subprocess output as UTF-8 to stop cp950 crashes - #52700
fix(tui-gateway): decode subprocess output as UTF-8 to stop cp950 crashes#52700devorun wants to merge 1 commit into
Conversation
…shes On non-UTF-8 Windows consoles (e.g. cp950 / zh-TW) the text-mode subprocess I/O in tui_gateway decoded child output with the OS locale codepage. Any non-ASCII byte (Hermes emits UTF-8) then raised UnicodeDecodeError inside the reader threads — the `_drain_stdout` / `_drain_stderr` slash-worker threads and the `_readerthread` threads behind the periodic `git branch` / `git rev-parse` status lookups — killing them and stalling the gateway on a fixed cadence, which the watchdog then respawns (NousResearch#52649). Pass `encoding="utf-8", errors="replace"` to the slash-worker `Popen` and both `_git_branch_for_cwd` `subprocess.run` calls. A stray byte can no longer crash the stream; Hermes/git output decodes normally. Adds a test asserting the git-branch lookups decode as UTF-8 with replacement.
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating the locale-dependent decoder failure. The slash-worker premise still holds on current main: tui_gateway/server.py:314-325 uses text=True without encoding or errors, so adding the proposed Popen arguments is sound.
Problems
- The patch leaves same-class current-main captures unprotected at
tui_gateway/server.py:9723-9726,11796-11806,11865-11873, and14416-14419. Each usestext=Trueand captures child output without a decoding policy. - The new regression test only covers git lookup arguments. That path is already hardened in
tui_gateway/git_probe.py:52-62and covered bytests/test_windows_subprocess_no_window_flags.py:38-68; it does not verify the slash-worker Popen change.
Suggested changes
- Carry the explicit UTF-8/replacement policy through the remaining text-mode captures above.
- Add a Popen-focused test for
_SlashWorker, then cover each additionally hardened capture.
This is an automated hermes-sweeper review.
| """ | ||
|
|
||
| from unittest.mock import MagicMock, patch | ||
|
|
There was a problem hiding this comment.
This only exercises the git path, which current main already hardens and tests through tui_gateway.git_probe. Please add a focused assertion for _SlashWorker's subprocess.Popen kwargs, because that is the still-unfixed production path this PR changes.
|
Closing with credit: this fix shipped in PR #70850 (merged, commit ead23aa). You were the earliest submitter of the Your second site ( |
Summary
Addresses #52649. On non-UTF-8 Windows consoles (e.g. cp950 / zh-TW) the text-mode subprocess I/O in
tui_gateway/server.pydecoded child output with the OS locale codepage, not UTF-8. Since Hermes (and git) emit UTF-8, any non-ASCII byte raisedUnicodeDecodeErrorinside the subprocess reader threads — exactly the threads named in the report:_drain_stdout/_drain_stderr— the slash-workerPopen(..., text=True)(noencoding)_readerthread— thesubprocess.run(..., capture_output=True, text=True)calls in_git_branch_for_cwd, which run on the periodic session-status refresh (hence the fixed ~580 s crash cadence)A crashed reader thread stalls the worker → the watchdog sees a stale log → kills and respawns (with a brief console popup), 10–20×/hour.
Fix
Pass
encoding="utf-8", errors="replace"to:subprocess.Popensubprocess.runcalls in_git_branch_for_cwderrors="replace"means a stray byte can never crash the stream, and UTF-8 is what these children actually emit. No behavior change on UTF-8 consoles.Scope note: this fixes the subprocess sites in
tui_gateway/server.pythat match the reported crash threads. Other text-mode subprocess reads elsewhere could warrant the same hardening, but are out of scope here.Tests
tests/tui_gateway/test_subprocess_encoding.py: the_git_branch_for_cwdgit lookups are invoked withencoding="utf-8"anderrors="replace". Runs on any platform (no cp950 console required).