Skip to content

fix(tests): read and write test files as UTF-8 so the suite runs on Windows - #80895

Closed
Adolanium wants to merge 1 commit into
NousResearch:mainfrom
Adolanium:fix/test-utf8-encoding-windows
Closed

fix(tests): read and write test files as UTF-8 so the suite runs on Windows#80895
Adolanium wants to merge 1 commit into
NousResearch:mainfrom
Adolanium:fix/test-utf8-encoding-windows

Conversation

@Adolanium

Copy link
Copy Markdown
Contributor

What does this PR do?

tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin fails on every native Windows machine:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 47744: character maps to <undefined>

The test reads run_agent.py back as text to assert a removed comment is gone, but calls open() with no encoding=. Python falls back to locale.getpreferredencoding(False), which is UTF-8 on Linux and macOS but cp1252 on a default US Windows install. run_agent.py is UTF-8 and contains nine bytes cp1252 leaves undefined, so the decode raises before the assertion is ever reached. It passes everywhere else, which is why CI stayed green.

That one line is the only active failure, and fixing it takes test_plugins_cmd.py from 38 passed with 1 failure to 39 passing on Windows.

Why the rest of the diff. scripts/check-windows-footguns.py flags eleven more bare encoding sites across the same three files, and the #71014 read_text campaign has been closing this class elsewhere in the tree. Two of them matter beyond tidiness: tests/tools/test_web_tools_truncate.py:52 reads stored extracted web text, which is arbitrary content from the internet and therefore one non-ASCII page away from the same crash, and tests/stress/test_atypical_scenarios.py:708 is the write that pairs with the read at 739, so fixing only the read would leave an asymmetric round trip. All three files are clean under check-windows-footguns.py after this change.

Reads go through Path.read_text(encoding="utf-8") rather than open(...).read(), which also closes the handle instead of leaving it to the garbage collector. A live handle blocks tmpdir cleanup on Windows, so that part is not cosmetic either.

Prior art: #37423 covered the gateway/run.py instances of this and is closed, #71014 carried the campaign forward, and #79490 is the same fix applied to tests/scripts/test_contributor_map.py. This is the tests/hermes_cli, tests/tools and tests/stress corner of it.

Related Issue

Fixes #80894

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • tests/hermes_cli/test_plugins_cmd.py: the failing read now uses Path.read_text(encoding="utf-8"). Nine bare write_text calls writing YAML manifests, config and plugin sources get an explicit encoding="utf-8".
  • tests/tools/test_web_tools_truncate.py: read the stored full text with an explicit encoding.
  • tests/stress/test_atypical_scenarios.py: explicit encoding on the two barrier and result writes and the matching read.

How to Test

On Windows, before this change:

pytest tests/hermes_cli/test_plugins_cmd.py -q
38 passed, 1 failed

After:

pytest tests/hermes_cli/test_plugins_cmd.py -q
39 passed

Full set touched by this PR:

pytest tests/hermes_cli/test_plugins_cmd.py tests/tools/test_web_tools_truncate.py -q
python scripts/check-windows-footguns.py --diff main

The footgun check reports no findings on all three files after the change.

To see the underlying cause on any platform:

import locale
print(locale.getpreferredencoding(False))
data = open("run_agent.py", "rb").read()
print([(i, b) for i, b in enumerate(data) if b in (0x8f, 0x90, 0x9d, 0x81)][:3])

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 11 Pro 10.0.26200, Python 3.13.13

On the tests checkbox, to be straight about it: no new test was added, because the thing being fixed is a test. It fails on Windows before this change and passes after, so it is its own regression coverage. scripts/check-windows-footguns.py is the mechanical guard against the pattern coming back, and it is clean on these files now.

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Before:

tests\hermes_cli\test_plugins_cmd.py:532: in test_compressor_default_ignores_plugin
    source = open(ra_module.__file__).read()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Lib\encodings\cp1252.py:23: in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 47744: character maps to <undefined>
=========================== short test summary info ===========================
FAILED tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin
1 failed

…indows

`tests/hermes_cli/test_plugins_cmd.py::TestNoAutoActivation::test_compressor_default_ignores_plugin`
fails on every Windows machine:

    UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in
    position 47744: character maps to <undefined>

The test reads `run_agent.py` back as text to assert a removed comment is
gone, but called `open()` with no `encoding=`. Python then falls back to
the locale preferred encoding, which is cp1252 on a default Windows
install rather than UTF-8. `run_agent.py` contains nine bytes cp1252
leaves undefined, so the read raises before the assertion is reached. On
Linux and macOS the preferred encoding is UTF-8 and the same line is
fine, which is why CI never caught it.

That one line is the only active failure. The rest of this change closes
the same gap in the files it touches, which `scripts/check-windows-footguns.py`
flags and which the NousResearch#71014 read_text campaign has been working through
elsewhere in the tree:

- `tests/hermes_cli/test_plugins_cmd.py`: nine bare `write_text`/`read_text`
  calls writing YAML manifests, config and plugin sources
- `tests/tools/test_web_tools_truncate.py`: reads stored extracted web text,
  which is arbitrary content from the internet
- `tests/stress/test_atypical_scenarios.py`: writes and reads worker task
  ids and a barrier file

All three files are now clean under `check-windows-footguns.py`.

Reads go through `Path.read_text(encoding="utf-8")` rather than
`open(...).read()`, which also closes the handle instead of leaving it to
the garbage collector. On Windows a live handle blocks tmpdir cleanup, so
that part is not cosmetic either.

No new test. The repaired test is the regression coverage: it fails
before this change and passes after, on Windows.
@alt-glitch alt-glitch added type/test Test coverage or test infrastructure P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows labels Aug 7, 2026
@teknium1

teknium1 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Merged in #81965 — clean cherry-pick with authorship (test_plugins_cmd + stress + web_tools UTF-8). Fixes #80894. Thanks!

@teknium1 teknium1 closed this Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have platform/windows Native Windows-specific behavior or breakage sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/test Test coverage or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: test_plugins_cmd fails on Windows with UnicodeDecodeError reading run_agent.py

3 participants