Skip to content

fix: preserve UTF-8 text at sample boundaries - #79834

Closed
cgordoncarroll wants to merge 1 commit into
NousResearch:mainfrom
cgordoncarroll:fix/read-file-text-detection
Closed

fix: preserve UTF-8 text at sample boundaries#79834
cgordoncarroll wants to merge 1 commit into
NousResearch:mainfrom
cgordoncarroll:fix/read-file-text-detection

Conversation

@cgordoncarroll

Copy link
Copy Markdown

Summary

  • read binary-detection samples through an ASCII-safe base64 transport
  • inspect four bytes beyond the 1,000-byte classification boundary so valid UTF-8 code points split at the boundary remain text
  • retain invalid UTF-8 and binary rejection, plus the existing direct-sample fallback when base64 is unavailable
  • apply the raw-byte classification path consistently to read_file and read_file_raw

Root cause

The terminal backend decodes subprocess output with replacement semantics. When head -c 1000 ended inside a multibyte UTF-8 code point, the orphaned lead byte became U+FFFD; _is_likely_binary then rejected an otherwise valid UTF-8 Markdown file as binary.

Verification

  • 170 passed across:
    • tests/tools/test_file_operations.py
    • tests/tools/test_file_operations_edge_cases.py
    • tests/tools/test_file_ops_cwd_tracking.py
    • tests/tools/test_file_read_guards.py
    • tests/tools/test_file_tools.py
    • tests/tools/test_file_tools_cwd_resolution.py
    • tests/tools/test_file_tools_tilde_profile.py
  • independent exact-diff review: APPROVE
  • existing terminal-fence fixtures remain byte-identical to main

@teknium1

teknium1 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Closing — superseded by #81961, which fixed this bug class (1000-byte sample cutting a multibyte char → binary false positive) at the byte layer across read_file, patch, and search_files, with regression tests for truncated-CJK/BOM/UTF-16/NUL cases. ~24 PRs raced on this one; earliest diagnosis credit to @webtecnica (#76924), merged implementation from @ayushnangia (#80440). Thanks for jumping on it.

@teknium1 teknium1 closed this Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists tool/file File tools (read, write, patch, search) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants