fix(repair-encoding-cli): reconfigure stdio to UTF-8 like the other entry points - #2194
Merged
igorls merged 1 commit intoAug 11, 2026
Conversation
…ntry points mempalace/_stdio.py states the rule in its own docstring -- "every console entry point that touches stdio needs to fix this on Windows" -- and cli.py and fact_checker.py both apply it. scripts/mempalace_repair_encoding.py, added later, does not. That matters more here than anywhere else: this tool exists for Windows users whose palace carries legacy mojibake, and it prints a before/after preview for every proposed change. Under the console codepage it was written for, the preview cannot be encoded -- the lead bytes of the corruption it detects are exactly the characters the codepage rejects -- so the run dies with UnicodeEncodeError before repairing a single drawer. Reproduced with stdout on cp936: UnicodeEncodeError: 'gbk' codec can't encode character '\xc3' in position 14: illegal multibyte sequence U+00C3 is the lead character of the à family this tool repairs. Uses replace on stdout/stderr, matching cli.py and fact_checker.py, because the preview carries verbatim drawer text that may hold surrogate halves round-tripped from filenames; strict would crash mid-preview. Related to MemPalace#1122 (same exception class on the main CLI's help output) but a different entry point and a different fix -- that one can use ASCII-safe static strings, this one prints arbitrary user content.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and Why
mempalace/_stdio.pystates the rule in its own docstring:cli.py:2160andfact_checker.py:321both apply it.scripts/mempalace_repair_encoding.py, added in #2148, does not (grep -c _stdio= 0).That gap lands hardest on this particular entry point. The tool exists for Windows users whose palace carries legacy mojibake, and
_print_changeemits a before/after preview for every proposed change. Under the console codepage the tool was written for, that preview cannot be encoded — the lead bytes of the corruption it detects are precisely the characters the codepage rejects — so the run dies before repairing a single drawer.Reproduction
With stdout on a legacy codepage, printing the same preview the tool emits:
U+00C3is× the lead character of the family this tool repairs.Expected: the dry-run lists proposed changes and exits 0.
Actual: exits 1 on the first change containing a non-ASCII preview.
Change Summary
scripts/mempalace_repair_encoding.py— add_reconfigure_stdio_utf8_on_windows()(thin wrapper over the shared helper, mirroringcli.py/fact_checker.py) and call it first thing inmain().tests/test_repair_encoding_cli.py— new file, loading the script by path the waytests/test_backfill_authored_at.pydoes.replaceon stdout/stderr, matching the other two console entry points: the preview carries verbatim drawer text that may hold surrogate halves round-tripped from filenames, wherestrictwould crash mid-preview.Test Plan
test_reconfigures_stdio_to_utf8_on_windows— mirrors the existing assertions intest_cli.pyandtest_fact_checker.py, pinning the per-stream errors policy.test_change_preview_survives_a_legacy_console_codepage— reproduces the actual failure rather than asserting a call happened: a child process gets a legacy codepage for stdout and is asked to print a real mojibake preview.UnicodeEncodeError: 'gbk' codec can't encode character '\xc3', so it is pinning the behaviour and not the wiring.ruff check .andruff format --check .clean (213 files).Notes
Related to #1122 (same exception class, main CLI's help output) but not a duplicate: different entry point, and a different fix — that one can be solved with ASCII-safe static strings, whereas this one prints arbitrary user content and needs the stdio reconfigure.
Found while auditing the encoding-repair surface. I also filed #2193 for a separate, more serious defect in the same module — that one is a correctness question in
_HIGH_CONFIDENCE_RUNand needs a design call, so I kept it out of this PR to keep this one trivially reviewable.