fix(file-ops): stop reading truncated UTF-8 samples as binary

[diesdaas]

The bug

read_file and read_file_raw probe a file with head -c 1000 before deciding whether it is text. That is a byte cut, so a multibyte codepoint straddling the limit decodes (errors="replace") to a trailing U+FFFD — an artifact the probe itself created.

_is_likely_binary treated any U+FFFD in the sample as proof of undecodable bytes, so those files came back as Binary file — cannot display as text from both read_file and patch. The file was then neither readable nor editable, with no way for the agent to recover.

Non-ASCII text hits this constantly — byte 1000 only has to land inside an umlaut. It surfaced on a German screenplay, where the agent worked around the block by stripping every en dash from the prose.

Reproducer against the current code, no fixtures needed:

from tools.environments.local import LocalEnvironment
# any UTF-8 file whose byte 1000 falls inside a multibyte codepoint
s = LocalEnvironment().execute(f"head -c 1000 '{path}'").get("output", "")
assert "�" in s   # -> the file is rejected as binary

The fix

A marker on the cut itself is ambiguous, and the decoded sample cannot resolve it: a split codepoint and a genuinely undecodable byte sitting on the boundary both leave exactly one trailing U+FFFD. Stripping the tail unconditionally would let real corruption through, so the check looks past the boundary instead of guessing.

Four more bytes is the most a UTF-8 codepoint can need:

• a split codepoint completes and its marker vanishes → text;
• a bad byte moves into the body, where it is unmistakable → binary;
• a read that returns nothing new means the file ended there, so nothing was cut off and the marker was never an artifact → binary.

Only that second read can clear a file; anything unresolved stays read-only. The extra head runs solely in the ambiguous case.

The corruption guard this check exists for is therefore intact — lossy content still makes a file read-only, so a read/edit/write round-trip cannot replace the original bytes with mojibake. The >30% non-printable ratio that catches real binaries is unchanged.

The other head -c probes were checked and are unaffected: _detect_file_line_ending only looks for \r\n vs \n, _file_has_bom compares an exact 3-byte prefix, and file_tools.py's guard is extension-only.

Tests

tests/tools/test_file_operations.py gains coverage for the split-codepoint case and for both ways a marker on the cut can be genuine (a bad byte followed by more content, and a file that ends on one). make_real_subprocess_env gained a lossy_utf8 flag so the fixture decodes the way LocalEnvironment actually does (encoding="utf-8", errors="replace").

Green: every test file that touches file_operations — 207 tests.

Pre-existing failures in tests/tools/ are unrelated and reproduce on a clean tree (test_approval.py, test_clipboard.py — prompt_toolkit is not installed in this environment).

🤖 Generated with Claude Code

[teknium1]

Closing — superseded by #81961, which fixed this bug class (1000-byte sample cutting a multibyte char → binary false positive) at the byte layer across read_file, patch, and search_files, with regression tests for truncated-CJK/BOM/UTF-16/NUL cases. ~24 PRs raced on this one; earliest diagnosis credit to @webtecnica (#76924), merged implementation from @ayushnangia (#80440). Thanks for jumping on it.