Skip to content

fix(tui): decode slash worker streams as UTF-8 - #63716

Closed
frizikk wants to merge 1 commit into
NousResearch:mainfrom
frizikk:fix/63686-slash-worker-utf8
Closed

fix(tui): decode slash worker streams as UTF-8#63716
frizikk wants to merge 1 commit into
NousResearch:mainfrom
frizikk:fix/63686-slash-worker-utf8

Conversation

@frizikk

@frizikk frizikk commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #63686.

On Windows systems using a CJK locale, subprocess.Popen(..., text=True) defaults to the system code page (for example GBK). _SlashWorker reads UTF-8 output from the slash-worker subprocess without an explicit encoding, so undecodable bytes can raise UnicodeDecodeError in the stderr-draining daemon thread and take down the gateway.

The worker now explicitly decodes stdout/stderr as UTF-8 and replaces malformed bytes, preserving the gateway process while retaining diagnostic output.

Related Issue

Fixes #63686

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • tui_gateway/server.py
    • Set encoding="utf-8" and errors="replace" on the persistent _SlashWorker subprocess.
  • tests/test_windows_subprocess_no_window_flags.py
    • Added a regression test that locks the subprocess stream decoding contract.

How to Test

  1. The reported failure mechanism was reproduced on Linux by simulating the Windows GBK locale: decoding UTF-8 subprocess bytes containing 0xAA with gbk raised UnicodeDecodeError.
  2. Run the focused Windows subprocess contract tests:
scripts/run_tests.sh tests/test_windows_subprocess_no_window_flags.py -q
16 passed
  1. Run the gateway server regression suite:
scripts/run_tests.sh tests/test_tui_gateway_server.py -q
322 passed

Checklist

Code

Documentation & Housekeeping

  • I've updated relevant documentation — N/A; this is an internal subprocess encoding fix
  • I've updated cli-config.yaml.example — N/A; no config keys changed
  • I've updated CONTRIBUTING.md or AGENTS.md — N/A; architecture/workflow unchanged
  • I've considered cross-platform impact; UTF-8 is explicit on all platforms and errors="replace" prevents malformed diagnostics from terminating the reader
  • I've updated tool descriptions/schemas — N/A; no tool behavior/schema changed

For New Skills

N/A — this PR does not add a skill.

Screenshots / Logs

Not applicable. The focused tests and gateway regression suite pass; the simulated Windows-locale reproduction produced the reported UnicodeDecodeError before the fix path was applied.

@frizikk
frizikk force-pushed the fix/63686-slash-worker-utf8 branch from e19b27e to 105d077 Compare July 13, 2026 10:47
@frizikk
frizikk force-pushed the fix/63686-slash-worker-utf8 branch from 105d077 to 6a5afa3 Compare July 13, 2026 10:56
@alt-glitch alt-glitch added type/bug Something isn't working comp/tui Terminal UI (ui-tui/ + tui_gateway/) platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows P2 Medium — degraded but workaround exists labels Jul 13, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused Windows compatibility fix. The current _SlashWorker spawn uses text=True without an explicit codec at tui_gateway/server.py:314-325, while its daemon readers consume both pipe streams at tui_gateway/server.py:329-340. The worker emits JSON-line responses on stdout at tui_gateway/slash_worker.py:169; specifying UTF-8 with replacement handling at the parent pipe boundary is correct and preserves diagnostics.

The only text-mode Popen stream-reader in tui_gateway/ is this _SlashWorker path. The added regression assertion covers the exact Popen contract. No problems found.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform area/streaming Streaming responses: gateway delivery, provider wire labels Jul 16, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Closing as resolved by PR #70850 (merged, commit ead23aa), which hardened the slash-worker Popen (and every other text=True capture site in the desktop/TUI backend) with encoding="utf-8", errors="replace". Credit for the earliest version of this fix goes to @devorun (#52700) and @Sahil-SS9 (#61978, cherry-picked as the base). Thanks for the fix and test.

@teknium1 teknium1 closed this Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/streaming Streaming responses: gateway delivery, provider wire comp/tui Terminal UI (ui-tui/ + tui_gateway/) P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows CJK locale: UnicodeDecodeError in _SlashWorker crashes gateway (text=True defaults to GBK)

3 participants