fix(file-ops): tolerate truncated UTF-8 sample boundary

[loker174]

Fix truncated UTF-8 sample boundary in file operations

Bug Description

read_file can misclassify valid UTF-8 text as binary when its binary-detection sample is produced with head -c 1000 and the 1000-byte boundary falls inside a multi-byte UTF-8 code point. The terminal output layer decodes the truncated sample with errors="replace", producing a synthetic trailing U+FFFD; the detector then treats that replacement character as proof of binary content.

This is especially visible with long CJK Markdown files. The path encoding is not the cause: a Chinese path containing long ASCII content does not reproduce the failure.

Related upstream issue/PRs previously observed during investigation (their
current state should be rechecked by maintainers; this change is a potentially
overlapping alternative, not a claim that the upstream maintainers have
accepted this particular implementation):

• read_file reports valid UTF-8 text as binary when the 1000-byte sample cuts a multibyte character (regression in 0.19.1) #76886
• read_file falsely reports UTF-8/GBK Chinese text files as binary (byte-truncated sample + U+FFFD heuristic) #77842
• fix: stop misclassifying multibyte UTF-8 text files as binary in read_file #80250
• fix(read_file): stop misjudging UTF-8 CJK files as binary #80188

These reports and PRs appear to concern the same broad sample-boundary/U+FFFD
failure mode. Their proposed implementations differ from this commit: this
draft keeps the existing non-UTF-8 safety guard and makes the sample-truncated
condition explicit before ignoring a single trailing replacement character.
The commit should therefore be evaluated as a potentially complementary or
alternative implementation, not as an unrelated new bug or a claim that the
other proposals are insufficient.

Root Cause

ShellFileOperations.read_file() and read_file_raw() sample the first 1000 bytes before reading the requested content. The terminal environment decodes command output as UTF-8 with errors="replace". A byte-truncated multi-byte character therefore becomes a trailing U+FFFD, and _is_likely_binary() previously rejected any sample containing that character.

Fix

• Pass an explicit sample_truncated fact to _is_likely_binary() based on the measured file size.
• When the file is larger than the 1000-byte sample and the replacement character occurs only at the sample end, treat that one character as a possible UTF-8 boundary artifact.
• Continue treating U+FFFD in the middle of the sample, or in an untruncated sample, as binary evidence. This preserves the existing protection against lossy reads of non-UTF-8 files.
• Apply the same logic to both paginated read_file() and full-content read_file_raw().

How to Verify

1. Create or use a UTF-8 Markdown file larger than 1000 bytes whose first 1000 bytes end inside a CJK or other multi-byte character.
2. Call read_file and confirm it returns text rather than is_binary=true.
3. Call read_file_raw on the same file and confirm it returns text.
4. Confirm a sample with U+FFFD in the middle remains classified as binary.
5. Confirm a known .bin file containing NUL bytes remains classified as binary; this exercises the known-binary-extension guard as well as the real-file path, not a universal claim about every binary format.

Test Plan

• Added regression tests for truncated-sample trailing U+FFFD.
• Added regression test that an untruncated trailing U+FFFD remains binary.
• Existing tests/tools/test_file_operations.py passes: 48 passed (run on the clean contribution worktree).
• python3 -m py_compile tools/file_operations.py tests/tools/test_file_operations.py passes (clean contribution worktree).
• git diff --check passes (clean contribution worktree).
• Manual verification against two real long Chinese Markdown files through read_file and read_file_raw passes; the files were copied from the Hermes repository/docs into Chinese-named paths before testing.
• Re-ran the final verification on the exact PR head immediately before preparing this draft for submission; see the commands and results recorded below.

Final exact-HEAD verification (a1fd18985, clean contribution worktree):

• pytest -q tests/tools/test_file_operations.py → 48 passed.
• python3 -m py_compile tools/file_operations.py tests/tools/test_file_operations.py → exit 0.
• git diff --check origin/main...HEAD → exit 0.
• Real long Chinese Markdown files copied from the Hermes repository/docs into Chinese-named paths: both read_file and read_file_raw returned text; the negative U+FFFD and known .bin checks retained binary classification.

Risk Assessment

Low to medium — the change is limited to binary-sample classification in the two file-reading paths. The main trade-off is a narrow residual ambiguity for a genuinely invalid byte sequence that appears only as the final replacement character of a truncated sample. Replacement characters in the sample body remain binary evidence, and the change does not remove extension-based binary detection or the existing non-printable-byte heuristic.

Notes for Maintainers

This draft uses a small local fix rather than changing all terminal backends or redesigning sampling around raw bytes/base64. If maintainers prefer the more rigorous raw-byte approach, this commit can be adapted or superseded by that design.

[alt-glitch]

This was generated by AI during triage.

Related to #76925 and #79408: this is a competing UTF-8 sample-boundary repair. It uses an explicit truncation signal, so a real trailing replacement character in an untruncated sample remains binary evidence. Maintainer selection is needed.

[teknium1]

Closing — superseded by #81961, which fixed this bug class (1000-byte sample cutting a multibyte char → binary false positive) at the byte layer across read_file, patch, and search_files, with regression tests for truncated-CJK/BOM/UTF-16/NUL cases. ~24 PRs raced on this one; earliest diagnosis credit to @webtecnica (#76924), merged implementation from @ayushnangia (#80440). Thanks for jumping on it.