fix(file_operations): U+FFFD density threshold in binary detection - #82494
Open
zhuyuhao0612 wants to merge 2 commits into
Open
zhuyuhao0612 wants to merge 2 commits into
zhuyuhao0612 wants to merge 2 commits into
Conversation
…ary cut not binary, UTF-16 still binary)
_is_likely_binary sampled the first 1000 bytes via head -c 1000; when the 1000th byte landed inside a multi-byte UTF-8 char (e.g. CJK), the truncated tail decoded to U+FFFD. Old logic flagged ANY U+FFFD as binary, so normal Chinese text files crossing the 1000-byte boundary were refused as binary. Fix: only flag binary when U+FFFD density > 2% (real binary/GBK >10%).
szzhoujiarui
reviewed
Aug 9, 2026
szzhoujiarui
left a comment
Contributor
There was a problem hiding this comment.
This is now superseded by main commit e40315d, which fixes the same boundary-cut issue at the byte layer by transporting the sample through base64.
The 2% replacement-character threshold still classifies sparse invalid UTF-8 as text. For example, a 1000-byte mostly-ASCII file with one invalid byte in the middle decodes to one U+FFFD (0.1% density), so this patch allows a lossy read and a later write can silently corrupt the original byte.
The byte-layer implementation on main correctly accepts an incomplete UTF-8 sequence at the sample boundary while rejecting mid-stream invalid UTF-8. Given the existing conflict and broader main coverage, I recommend closing this PR as superseded.
19 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_is_likely_binarysamples the first 1000 bytes (viahead -c 1000). When the 1000th byte lands inside a multi-byte UTF-8 character (e.g. a 3-byte CJK char), the truncated tail decodes to U+FFFD replacement chars. The old logic flagged any U+FFFD as binary, so ordinary Chinese-language text files crossing the 1000-byte boundary were refused as "Binary file" and became unreadable via read_file.Fix
Flag binary only when U+FFFD density exceeds 2%. Real binary / GBK-encoded content produces >10% density, so the protection against garbled writes is preserved; a boundary cut produces at most ~0.3-0.5%, so legitimate text files pass.
Tests
Adds
TestUtf8BoundaryCutNotBinary(3 cases) intests/tools/test_file_operations.py:Impact
Reads previously failing on Chinese/CJK files between 1000 and ~50000 bytes now work. No behavior change for genuine binary content.