fix: stop misclassifying multibyte UTF-8 text files as binary in read_file

[0809android]

Summary

read_file (and read_file_raw) classify valid UTF-8 text files as binary when the 1000-byte sampling boundary (head -c 1000) cuts through a multibyte character. Japanese, Chinese, Korean, and emoji-heavy text files are affected frequently (3-byte UTF-8 chars make a mid-character cut highly likely).

Reproduction

Any UTF-8 text file whose byte 1000 lands inside a multibyte char, e.g. a Japanese markdown file > 1000 bytes. read_file returns:

Binary file - cannot display as text. Use appropriate tools to handle this file type.

Verified against a real 10 KB Japanese file: head -c 1000 cut "自" (\xe5\xbe\xaa) after its 2nd byte, the incremental decoder (errors="replace") produced one trailing U+FFFD at sample position 386, and _is_likely_binary returned True.

Root cause

1. read_file samples the first 1000 bytes (head -c 1000) for binary detection.
2. The terminal backend decodes child output with codecs.getincrementaldecoder("utf-8")(errors="replace") (tools/environments/base.py), so the dangling bytes of a cut multibyte char arrive as U+FFFD at the end of the sample.
3. _is_likely_binary treats any U+FFFD in the sample as evidence of mojibake → binary. That guard exists to prevent read→edit→write corruption of genuinely non-UTF-8 files, but it does not distinguish a truncation artifact (1–3 trailing U+FFFD) from genuine mojibake (U+FFFD throughout).

Fix

In _is_likely_binary, strip a trailing run of ≤3 U+FFFD from the sample before judging. A 4-byte UTF-8 char split across the cut yields at most 3 dangling bytes, so a longer trailing run (≥4) still counts as binary (safe direction). Genuine mojibake (U+FFFD in the middle of the sample) is still flagged binary, preserving the corruption guard.

sample = content_sample[:1000]
stripped = sample.rstrip("\ufffd")
if len(sample) - len(stripped) <= 3:
    sample = stripped
if "\ufffd" in sample:
    return True
non_printable = sum(1 for c in sample if ord(c) < 32 and c not in '\n\r\t')
return non_printable / min(len(sample), 1000) > 0.30

Tests

Added 3 regression tests to tests/tools/test_file_operations.py (TestReadNonUtf8IsBinary):

• trailing U+FFFD (cut artifact) → text (not binary)
• mid-sample U+FFFD (genuine mojibake) → binary (guard preserved)
• trailing run of 5 → binary / trailing run of 3 → text (safe direction)

Full file: 49 passed.

[0809android]

Fixes #80251

[monerostar]

Ubuntu 26.04 on linux-5800x (kernel 7.0.0-28-generic).

Live ShellFileOperations.read_file on a 1200-byte Japanese hiragana file (あ x 400). Byte 1000 lands mid-character so head -c 1000 yields a trailing U+FFFD in the sample.

fixture	main	this PR
jp.txt (multibyte cut)	is_binary=True	is_binary=False, content_len=402
ascii.txt	text OK	text OK
nul.bin	binary	binary
bytes(range(256))*4	binary	binary

Also: _is_likely_binary on that sample is True on main, False on PR. tests/tools/test_file_operations.py 49 passed here.

Looks good. The trailing ≤3 FFFD carve-out matches the real head -c cut; real binary still blocked.

[kyssta-exe]

Triage note: this appears to duplicate three other PRs that fix the same issue — UTF-8 files whose 1000-byte read_file sample lands mid-multibyte-char (trailing U+FFFD) get misclassified as binary. Same files (tools/file_operations.py + tests), same fix. The cluster: #80186, #80188, #80261. Recommend picking one to champion and closing the rest as duplicates.

[teknium1]

Closing — superseded by #81961, which fixed this bug class (1000-byte sample cutting a multibyte char → binary false positive) at the byte layer across read_file, patch, and search_files, with regression tests for truncated-CJK/BOM/UTF-16/NUL cases. ~24 PRs raced on this one; earliest diagnosis credit to @webtecnica (#76924), merged implementation from @ayushnangia (#80440). Thanks for jumping on it.