fix(tools): don't false-flag CJK files as binary in read_file - #80186
Open
fanxiong wants to merge 1 commit into
Open
fix(tools): don't false-flag CJK files as binary in read_file#80186fanxiong wants to merge 1 commit into
fanxiong wants to merge 1 commit into
Conversation
_is_likely_binary samples the first 1000 bytes via head -c 1000. When the byte cut lands mid-way through a multi-byte UTF-8 char, the terminal's errors=replace decode turns the partial char into a trailing U+FFFD, which the guard treated as evidence of binary content. Every CJK-heavy file (73-98% non-ASCII bytes) tripped it, so read_file refused to read them. Scan everything except the sample tail: genuine undecodable bytes still produce U+FFFD mid-sample; only the head -c truncation artifact sits at the very end. Regression test added: trailing U+FFFD not flagged, mid-sample U+FFFD still flagged as binary.
alt-glitch
added
type/bug
Collaborator
Contributor
|
Triage note: this appears to duplicate three other PRs that fix the same issue — UTF-8 files whose 1000-byte read_file sample lands mid-multibyte-char (trailing U+FFFD) get misclassified as binary. Same files (tools/file_operations.py + tests), same fix. The cluster: #80188, #80250, #80261. Recommend picking one to champion and closing the rest as duplicates. |
19 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
read_file会把 CJK 密集的文本文件误判为二进制并拒绝读取（报 "Binary file - cannot display as text"）。_is_likely_binary用head -c 1000采样前 1000 字节。当截断点正好落在一个多字节 UTF-8 字符中间时，终端以errors="replace"解码，半个字符变成尾部 U+FFFD；而该函数把样本里出现的任何 U+FFFD 都当作"存在不可解码字节"的证据。中文文本的非 ASCII 字节占比高达 73%~98%，第 1000 字节落在字符中间的概率极高——实测这些文件几乎 100% 被误判。修复
U+FFFD 出现在样本末尾是
head -c 1000的截断伪影，不是二进制证据；真正的乱码（不可解码字节）会出现在样本中部。因此改为扫描content_sample[:-1]（排除尾部截断伪影），其余判定逻辑不变：测试
test_trailing_replacement_char_is_truncation_artifact：scripts/run_tests.sh tests/tools/test_file_operations.py→ 47 passed, 0 failed