fix: add explicit UTF-8 encoding to Path.read_text()/write_text() (P1 — silent data corruption on Windows) - #58840
Conversation
… calls (P1) Path.read_text() and write_text() without encoding parameter default to the system locale encoding (cp1252 on Windows, UTF-8 on Linux). This causes silent data corruption when reading/writing non-ASCII content on Windows systems. This is NOT caught by ruff PLW1514 (which only flags open() calls) and is a common source of cross-platform bugs. Fixed: - 21 files with .read_text() -> .read_text(encoding="utf-8") - 58 files total including .write_text() -> .write_text(..., encoding="utf-8") Key areas: - gateway/run.py (9 read_text calls) - tools/skills_hub.py (read + write) - hermes_cli/auth.py, hermes_cli/main.py - plugins/platforms/ (discord, telegram, slack, whatsapp, feishu) - agent/ (auxiliary_client, copilot_acp_client, curator)
tonydwb
left a comment
There was a problem hiding this comment.
Code Review Summary
Verdict: Comment (LGTM)
P1 critical fix: adds explicit encoding="utf-8" to all Path.read_text() and Path.write_text() calls across 58 files. This prevents silent data corruption on Windows where the default encoding is platform-dependent (often cp1252). The fix is mechanical and thorough — every instance was updated, not just the ones that happened to fail on Windows in testing.
What looks good:
- Comprehensive sweep covering all Path I/O in the codebase
- Backwards compatible (utf-8 is what the files already are in practice)
- Prevents a real silent-corruption class of bugs
|
Closing as resolved by PR #71078 (merged, commit d372fda): the class-wide close-out salvaged your #50655/#54241/#56385/#66856/#65440 series as the backbone (authorship preserved in git log) and swept the remaining sites, so every read_text/write_text call this PR touches is now guarded on current main — verified per-site. A CI linter rule in check-windows-footguns.py plus the AST guard test now prevent regressions. Your overlapping/split variants of the same series are being closed together; the credit for the class rests on your commits. |
Summary
Add explicit
encoding="utf-8"to allPath.read_text()andPath.write_text()calls that omit it.Problem
Path.read_text()andPath.write_text()without theencodingparameter default to the system locale encoding. On Windows this is typicallycp1252, which silently corrupts non-ASCII content (accented characters, CJK, emoji, etc.) when the file actually contains UTF-8 data.This is NOT caught by ruff rule PLW1514 (which only flags bare
open()calls) and is a common source of cross-platform bugs.Changes
58 files fixed across the entire codebase:
Path.read_text()->Path.read_text(encoding="utf-8")(21 files)Path.write_text(...)->Path.write_text(..., encoding="utf-8")(58 files total)Key areas:
gateway/run.py(9 read_text calls)tools/skills_hub.py(read + write)hermes_cli/(auth, main, profiles, service_manager, etc.)plugins/platforms/(discord, telegram, slack, whatsapp, feishu)agent/(auxiliary_client, copilot_acp_client, curator)Impact
Testing
All 58 changed files pass
python3 -m py_compileverification.