fix(file): classify UTF-8 samples before lossy decoding

[milnerrad]

Summary

Fix false binary classification when the 1,000-byte content sample splits a valid multibyte UTF-8 sequence.

ShellFileOperations currently sends head -c 1000 output through environment APIs that decode stdout with errors="replace". A valid character beginning before byte 1,000 and ending after it therefore arrives as U+FFFD and is classified as binary.

This change:

• transports a bounded 1,003-byte sample as validated ASCII hex before backend text decoding;
• strictly validates the 1,000-byte logical UTF-8 prefix with up to three completion bytes;
• accepts valid 2-, 3-, and 4-byte characters split at every boundary position;
• treats literal valid U+FFFD as text;
• rejects malformed and incomplete UTF-8 samples;
• preserves existing extension and control-character heuristics;
• consolidates read_file() and read_file_raw() through the byte-safe sampler;
• fails closed on malformed, truncated, or incomplete sample transport.

patch_replace() is intentionally unchanged. Prefix classification does not prove an entire editable file is valid UTF-8, and the existing display-read helper performs lossy terminal-fence cleanup; edit-path hardening requires a separate lossless full-file snapshot design.

The size probe, bounded sample, and final display read remain separate opens. Exact-length validation fails closed on many growth/shrink races, but it cannot detect same-size replacement or mutation after sampling. This PR fixes the byte-boundary classification defect; it does not claim stable-snapshot or full-file edit validation.

Relationship to existing PRs

Several open PRs address the same symptom, but this implementation differs at the safety boundary:

• fix(file-ops): base64-encode binary sample to avoid UTF-8 truncation false positive #80864 transports only the 1,000-byte prefix and peels trailing bytes until strict decoding succeeds; that cannot prove whether discarded trailing bytes begin a valid character. This PR fetches continuation lookahead and accepts the boundary only when that pending character actually completes.
• fix(file-ops): byte-layer binary detection — stop flagging boundary-cut UTF-8 as binary (#80308 class) #80440 uses a byte transport but falls back to the legacy lossy heuristic when transport fails and introduces an any-NUL rule. This PR fails closed on transport errors and preserves the existing strict >30% low-control threshold.
• fix(tools): keep invalid UTF-8 boundaries binary #76934 performs a bounded raw probe only after replacement-decoded text ends in U+FFFD. This PR never bases classification on replacement-decoded sample bytes: it reconstructs the bounded raw sample first, so literal U+FFFD and every legal split are handled directly.

The regression matrix covers every split position for valid 2-, 3-, and 4-byte UTF-8 characters, malformed continuations at those positions, literal U+FFFD, true-EOF incompleteness, and invalid bytes beginning immediately after the logical boundary.

Reproduction

A valid 4,109-byte UTF-8 file with E2 at byte 1,000 and 80 9D immediately after it:

• current main: is_binary=True with “Binary file” error;
• this branch: is_binary=False, no error, and the closing quote is preserved.

Verification

scripts/run_tests.sh tests/tools/test_file*.py tests/tools/test_read_extract.py -q --tb=short
# 346 passed, 0 failed

scripts/run_tests.sh \
  tests/tools/test_file_operations.py \
  tests/tools/test_file_operations_edge_cases.py \
  tests/tools/test_patch_parser.py \
  tests/tools/test_file_ops_cwd_tracking.py \
  -q --tb=short
# 121 passed, 0 failed

ruff check \
  tools/file_operations.py \
  tests/tools/test_file_operations.py \
  tests/tools/test_file_operations_edge_cases.py
# All checks passed

python -m py_compile \
  tools/file_operations.py \
  tests/tools/test_file_operations.py \
  tests/tools/test_file_operations_edge_cases.py

git diff --check

The exact od -An -v -tx1 -N form was also exercised with GNU coreutils and BusyBox. CI will provide the repository-wide test result.

[alt-glitch]

This was generated by AI during triage.

Related: this is another implementation for #76886 alongside #76924 and #76925. It uses byte-safe hex transport with continuation lookahead rather than lossy sample handling; maintainer choice is needed on the desired boundary semantics.

[teknium1]

Closing — superseded by #81961 (salvaged from #80440). Classify-before-lossy-decode was the right instinct; the merged fix does the same at the byte layer with a lighter transport. Thanks.