read_file: transcode UTF-16 text files to UTF-8 instead of refusing as binary (port from kimi-code)

[teknium1]

Summary

read_file now reads UTF-16 text files by transcoding them to UTF-8 instead of refusing them as binary. Windows Notepad .txt files and PowerShell > redirects are UTF-16 by default; the terminal env decodes stdout as UTF-8 with errors="replace", so their content arrived as U+FFFD mojibake and tripped the binary guard.

Ported from MoonshotAI/kimi-code#2647 (detection derived from VS Code's encoding sniffer), adapted to Hermes' shell-backend architecture: the probe runs via python3 -c through the backend's own shell, so it works identically on local, docker, and ssh backends.

Changes

• tools/file_operations.py: new ShellFileOperations._try_read_utf16() invoked when the binary guard fires — trusts a BOM first, then the zero-byte parity heuristic (zeros clustered at odd indices = UTF-16 LE, even = BE; placement-only so mixed Latin/CJK content still detects). Transcodes to UTF-8, strips BOM, normalizes CRLF, supports pagination, and appends a "Transcoded from UTF-16-LE" hint so edits re-encoding as UTF-8 aren't a surprise. Real binaries (zeros at both parities), binary extensions, >10 MiB files, and legacy 8-bit encodings (GBK/Big5) still refuse — a wrong silent guess is worse than a clear refusal.
• tests/tools/test_utf16_read.py: 9 E2E tests against a real LocalEnvironment (BOM LE/BE, BOM-less, mixed CJK, CRLF, pagination, real-binary refusal, binary-extension skip, UTF-8 untouched).

Validation

	Before	After
UTF-16 LE/BE text file	"Binary file - cannot display as text"	numbered lines, transcoded hint
Real binary / GBK	refused	still refused
tests/tools/test_utf16_read.py	6/9 fail (sabotage run, fix disabled)	9/9 pass
Adjacent suites	—	test_file_ops_cwd_tracking + test_file_write_safety: 32/32 pass

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 8cfd852 — Port from MoonshotAI/kimi-code#2647: read UTF-16 text files}

⚠️ Warnings

OSV vulnerability scan · View job

5 known vulnerabilities found in pinned dependencies.

• CVE-2026-67213 in package-lock.json
• CVE-2026-67213 in website/package-lock.json
• CVE-2026-70608 in package-lock.json
• CVE-2026-56876 in package-lock.json
• CVE-2026-70606 in package-lock.json

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.

---

debug info

CI timings

CI timings · View report · View job

Wall time 20m48s vs 35m26s (-41.3%). 7 job(s) slower, 14 faster, 3 unchanged.

• Python tests / Run tests slice 6/12: +31.0s
• Python tests / Run tests slice 10/12: +28.0s
• Python tests / Run tests slice 4/12: -21.0s
• Python tests / Run tests slice 3/12: -16.0s
• Python tests / Run tests slice 8/12: -12.0s