Repository navigation
Conversation
The text tools' read->modify->write round-trip re-encodes binary bytes
lossily: the terminal transport decodes stdout with errors=replace, so
a patch on a SQLite database reads mojibake and writes it back, silently
destroying the file (observed in the wild: a kanban board DB lost 550k
bytes / 26 days of history this way on 2026-09-15).
read_file already refuses to display binary files (extension + byte
sniffing), but write_file/patch had no write-side equivalent, and
patch_replace reads via plain cat so the read-side guard never fires
for it.
Guard rules (extends _check_binary_document_write, the existing
binary-document write guard):
- .db/.sqlite/.sqlite3 and their -wal/-shm/-journal sidecars: ALWAYS
refused (even new-file creation) — plain text can never be a valid
database, mirroring the opaque-document rule. The sidecars need a
basename regex because os.path.splitext('x.db-wal') yields '.db-wal',
which is not in BINARY_EXTENSIONS.
- Other BINARY_EXTENSIONS (.png, .zip, ...): refused when OVERWRITING an
existing file, mirroring the existing .pdf-overwrite rule — the model
can only have read mojibake, so writing back destroys it. Creating a
NEW file with such an extension stays allowed.
Tests: new test_binary_db_write_guard.py covers the regex (main names,
sidecars, non-db names), the guard function, and the write_file/patch
tool paths against real SQLite files (integrity_check still ok, bytes
untouched). 29 new+existing guard tests pass; file-tools/patch suites
green (196 tests).
Related: merged #86362 introduced binary-document write refusal; this extends the same guard to SQLite files and sidecars. |
|
Ran the PR head ( Reproduced on main first. With a WAL-mode database whose sidecars exist:
Two cases the name-based rule does not reach, both reproduced on the PR head:
Nothing needed for the merge decision: the write-side fix stands on its own, and the sidecars are the live hole, since For (2) I have a one-function change to |
|
Follow-up on case (2) from my review above — the standalone PR is up: #114159 It is one function in First preference: lift it as one small commit keeping authorship — |
|
Landed on The salvage PR body records what was kept, what was trimmed and why; the merged tests are the invariant subset. If anything in the landed shape misses your intent, comment there and I will follow up. |
Problem
write_fileandpatchaccept database and other binary files as text targets. The tools read such files as lossy text (errors=replaceon stdout decode) and write the mojibake back — silently destroying the file. This is not hypothetical: it destroyed this fleet's live kanban.db (550k bytes → U+FFFD, unrecoverable; SQLitedatabase disk image is malformed).The corruption path is structural: a text tool can never round-trip binary content, so accepting the target at all is the defect. Four sibling read paths already sniff binaries; the write paths did not.
Fix
Extends the existing
_check_binary_document_writeguard intools/file_tools_write_guards.py(the established write-guard home, following the repo's own PDF precedent):-wal/-shm/-journalsidecars —os.path.splitext("x.db-wal")yields.db-wal, which is not inBINARY_EXTENSIONS, so a plain extension check misses every WAL sidecar; matched via basename regex). Error message names the right tool (sqlite3 CLI / SQLite library).Tests
tests/tools/test_binary_db_write_guard.py— 29/29 new tests (db main + sidecars, overwrite-refused/new-file-allowed matrix, NUL-content refusal, text files unaffected). Full guard + patch + file-tools suites green (196 tests); the one failing test in the tree (test_approvalsmart-approval) fails identically on cleanorigin/main(verified via stash).Verified end-to-end against a live SQLite database: refusal fires before any byte is read or written; the DB stays byte-identical with
integrity_check ok.