fix: CJK UTF-8 source files misclassified as binary when head -c 1000 truncates mid-character

[MISAKIGA]

Summary

read_file misclassifies valid UTF-8 source files containing CJK (or any multi-byte) characters as binary, refusing to display them.

Root cause

read_file samples a file with head -c 1000 and passes the decoded output to _is_likely_binary(). The content-analysis path treats any occurrence of U+FFFD (replacement char) as binary — a guard against lossy read→edit→write round-trips on genuinely non-UTF-8 files.

The guard's assumption ("legitimate UTF-8 text effectively never contains U+FFFD") holds for a complete file, but not for a truncated sample: when byte 1000 lands inside a 3- or 4-byte CJK character, the incomplete sequence decodes as exactly one trailing U+FFFD — a sampling artifact.

Impact

A real-world case: hermes-shine/front/src/pages/chat/store.ts (24,892 bytes, valid UTF-8 with Chinese comments) became unreadable via read_file because its byte 1000 splits the 3-byte character 会 (E4 BC 9A).

Fix

Strip at most one U+FFFD from the tail of the sample before judging. This precisely targets the truncation artifact:

• Genuine non-UTF-8 files have replacement chars throughout the sample, not just at the tail → still flagged binary (safety net preserved).
• Valid UTF-8 text truncated mid-character ends with exactly one U+FFFD → artifact removed → readable as text.

Tests

Added 4 unit tests in tests/tools/test_file_operations_edge_cases.py:

Test	Verifies
test_truncated_multibyte_utf8_not_flagged	CJK text with trailing truncation artifact → NOT binary
test_trailing_fffd_from_real_cjk_file	999 ASCII + orphaned UTF-8 lead byte → NOT binary
test_internal_replacement_chars_still_binary	U+FFFD spread through sample (real corruption) → STILL binary
test_only_trailing_replacement_char_stripped	Middle corruption → binary; lone trailing artifact → text

Verification

• tests/tools/test_file_operations.py tests/tools/test_file_operations_edge_cases.py: 71 passed (incl. the 4 new + pre-existing U+FFFD guard tests)
• Full tests/tools/ suite: 5112 passed, 228 pre-existing failures in unrelated modules (approval/delegation/daytona/browser — environment-dependent, zero in file ops)
• End-to-end with real LocalEnvironment: the previously-unreadable chat/store.ts now returns is_binary: False and full content

[alt-glitch]

This was generated by AI during triage.

Duplicate of #79408: both implement the same trailing-U+FFFD exemption for a valid UTF-8 byte-sample boundary cut. Please consolidate review and tests there.

[teknium1]

Closing — superseded by #81961, which fixed this bug class (1000-byte sample cutting a multibyte char → binary false positive) at the byte layer across read_file, patch, and search_files, with regression tests for truncated-CJK/BOM/UTF-16/NUL cases. ~24 PRs raced on this one; earliest diagnosis credit to @webtecnica (#76924), merged implementation from @ayushnangia (#80440). Thanks for jumping on it.