fix: verify NUL bytes before flagging U+FFFD samples as binary (#80308)

[JonthanaHanh]

Summary

read_file misclassifies valid UTF-8 files containing CJK characters (Chinese, Japanese, Korean, emoji) as binary and refuses to display them. The root cause is in _is_likely_binary() in tools/file_operations.py.

Root Cause

The function reads the first 1000 bytes via head -c 1000 and checks for U+FFFD (replacement character). However, CJK characters are 3 bytes each in UTF-8 (emoji = 4 bytes), so the 1000-byte boundary frequently cuts a character in half. The terminal decoder (using errors="replace") produces U+FFFD from the truncated sequence, which the function incorrectly treats as evidence of binary content.

Key insight: head -c 1000 is a byte-level operation that doesn't respect UTF-8 character boundaries. A file of 1002 CJK bytes will always trigger this false positive.

Fix

When U+FFFD is detected in the sample, instead of immediately returning True (binary), verify by checking the raw file bytes for NUL (0x00) via Python. NUL is the reliable binary indicator:

• Genuine binary files almost always contain NUL bytes
• Text files (regardless of encoding) effectively never contain NUL

If no NUL is found, the U+FFFD is a truncation artifact from head -c splitting a multibyte character, and the file should be treated as text.

Changes

• tools/file_operations.py: _is_likely_binary() — add NUL byte verification when U+FFFD is detected before flagging as binary

Test Plan

• Syntax check passes (ast.parse)
• CJK file > 1000 bytes: correctly identified as TEXT (not binary)
• Binary file with NUL bytes: correctly identified as BINARY
• ASCII file: correctly identified as TEXT
• Existing test test_replacement_char_sample_flagged_binary still passes (non-printable ratio check catches genuine encoding errors)
• Existing test test_plain_utf8_text_not_flagged still passes

Fixes #80308

[alt-glitch]

This was generated by AI during triage.

Duplicate of #76925: this addresses the same UTF-8 sample-boundary false-binary mechanism. Please consolidate the raw-NUL validation idea and tests with the canonical PR.

[teknium1]

Closing — the NUL-only verification here would pass non-UTF-8 latin-1 files as text, which turns a read→edit→write cycle into mojibake corruption; it also shells out to python which isn't guaranteed present. The merged fix (#81961) validates UTF-8 at the byte layer instead. Thanks for digging into the sample-boundary root cause — the diagnosis was right.