fix(read): don't flag UTF-8 text as binary when byte 1000 cuts a multibyte char

[luntion]

Summary

read_file / read_file_raw misclassify legitimate UTF-8 text files as binary
when the 1000-byte sampling window lands mid-way through a multi-byte character.

Symptoms

A UTF-8 text file whose byte 1000 falls inside a multi-byte char (CJK, Cyrillic,
emoji — anything 2–4 bytes) gets flagged is_binary: true and refuses to read.
Concretely on a Chinese-language markdown file:

read_file → error: "Binary file - cannot display as text."

Same file reads fine on a retry because the byte position of the truncation
depends on the file's exact byte length — it is a ~50% lottery per file, not a
content property.

Root cause

tools/file_operations.py samples the first 1000 bytes with
head -c 1000, then decodes that sample with errors="replace". When byte
1000 cuts a multi-byte UTF-8 character in half, the truncated tail decodes to
U+FFFD (replacement char). _is_likely_binary sees U+FFFD and — correctly, for
genuine mojibake — treats the file as binary and blocks the read. The sample
boundary is byte-aligned, not character-aligned, so any UTF-8 text can trip it.

Verified: for the affected files, head -c 1000 ends with a continuation byte
(0x80–0xBF); decoding that sample yields exactly one U+FFFD at the tail;
_is_likely_binary then returns True.

Changes

tools/file_operations.py (both call sites — read_file and read_file_raw):

Replace the raw head -c 1000 sample with a small python probe that reads the
first 1000 bytes and extends the window to the next UTF-8 character
boundary before writing the sample:

• while n < len(d) and (d[n] & 0xC0) == 0x80: n += 1 — skip continuation
  bytes so the sample always ends on a complete character.
• Output is written via sys.stdout.buffer.write (raw bytes, no re-encoding),
  matching how the terminal env already decodes stdout as UTF-8.
• sys.executable is shell-escaped via the existing _escape_shell_arg.

Why this approach (not errors="ignore"): decoding the sample with
errors="ignore" would drop the U+FFFD signal entirely, so genuinely
undecodable content (latin-1 blobs, raw binary without a known extension) would
pass _is_likely_binary and get returned as lossy text — enabling a
read→edit→write round-trip to silently corrupt the original bytes. That
protection is the entire point of the U+FFFD branch (see its docstring). The
boundary-extension fix removes only the artifact of the byte-aligned cut;
real mojibake still produces U+FFFD mid-sample and is still caught.

Testing

• New TestReadUtf8BoundaryCut class (4 tests, real subprocess via
  LocalEnvironment):
  • 3-byte CJK char straddling byte 1000 → read_file returns text, not binary
  • same for read_file_raw
  • 4-byte emoji spanning the window tail stays intact
  • genuine binary (NUL + high-bit garbage) still flagged binary
• Updated the existing test_read_file_uses_bash_safe_windows_paths command
  assertions to the new python sampler command shape.
• tests/tools/test_file_operations.py + test_file_operations_edge_cases.py:
  63 passed, 8 failed — all 8 failures are pre-existing on clean main on this
  Windows box (TestSearchFilesFallbackHiddenPaths ×2, atomic-write umask
  permissions ×4, symlink writes ×2 — local-environment limitations, unrelated
  to this change; verified by stashing the change and re-running).

Related

• Mirrors the same root cause as fix(read): stop misclassifying UTF-8 text files as binary #79641 (which proposed errors="ignore" and
  was rejected in review for weakening the mojibake guard; this PR fixes the
  same bug without that regression).

Risks

• The sample command is now python -c … instead of head -c 1000. It runs
  once per read call, reads at most ~1003 bytes, and exits immediately — cost
  is negligible. sys.executable is escaped like every other shell arg in the
  file. Non-UTF-8 text files (e.g. latin-1) keep the existing behavior: sample
  decodes with replacement chars → flagged binary → read-only, preventing
  silent corruption. No config, schema, or env changes.

[kyssta-exe]

Triage note: this appears to duplicate three other PRs that fix the same issue — UTF-8 files whose 1000-byte read_file sample lands mid-multibyte-char (trailing U+FFFD) get misclassified as binary. Same files (tools/file_operations.py + tests), same fix. The cluster: #80186, #80188, #80250. Recommend picking one to champion and closing the rest as duplicates.

[teknium1]

Closing — superseded by #81961, which fixed this bug class (1000-byte sample cutting a multibyte char → binary false positive) at the byte layer across read_file, patch, and search_files, with regression tests for truncated-CJK/BOM/UTF-16/NUL cases. ~24 PRs raced on this one; earliest diagnosis credit to @webtecnica (#76924), merged implementation from @ayushnangia (#80440). Thanks for jumping on it.

[webtecnica]

Thanks @teknium1 for the credit — glad the diagnosis held up as the earliest of the ~24. The byte-layer detection in #81961 covering both file-ops sites is the right contract, and the truncated-CJK/BOM/UTF-16/NUL regressions make it solid. Appreciate the recognition!