Skip to content

fix(repair-encoding-cli): reconfigure stdio to UTF-8 like the other entry points - #2194

Merged
igorls merged 1 commit into
MemPalace:developfrom
arnoldwender:fix/repair-encoding-cli-stdio-utf8
Aug 11, 2026
Merged

fix(repair-encoding-cli): reconfigure stdio to UTF-8 like the other entry points#2194
igorls merged 1 commit into
MemPalace:developfrom
arnoldwender:fix/repair-encoding-cli-stdio-utf8

Conversation

@arnoldwender

Copy link
Copy Markdown
Contributor

What and Why

mempalace/_stdio.py states the rule in its own docstring:

Every console entry point that touches stdio needs to fix this on Windows -- the MCP server, the CLI, the fact_checker --stdin mode -- so the reconfigure code lives here in one place.

cli.py:2160 and fact_checker.py:321 both apply it. scripts/mempalace_repair_encoding.py, added in #2148, does not (grep -c _stdio = 0).

That gap lands hardest on this particular entry point. The tool exists for Windows users whose palace carries legacy mojibake, and _print_change emits a before/after preview for every proposed change. Under the console codepage the tool was written for, that preview cannot be encoded — the lead bytes of the corruption it detects are precisely the characters the codepage rejects — so the run dies before repairing a single drawer.

Reproduction

With stdout on a legacy codepage, printing the same preview the tool emits:

UnicodeEncodeError: 'gbk' codec can't encode character '\xc3'
in position 14: illegal multibyte sequence

U+00C3 is à — the lead character of the family this tool repairs.

Expected: the dry-run lists proposed changes and exits 0.
Actual: exits 1 on the first change containing a non-ASCII preview.

Change Summary

  • scripts/mempalace_repair_encoding.py — add _reconfigure_stdio_utf8_on_windows() (thin wrapper over the shared helper, mirroring cli.py / fact_checker.py) and call it first thing in main().
  • tests/test_repair_encoding_cli.py — new file, loading the script by path the way tests/test_backfill_authored_at.py does.

replace on stdout/stderr, matching the other two console entry points: the preview carries verbatim drawer text that may hold surrogate halves round-tripped from filenames, where strict would crash mid-preview.

Test Plan

  • test_reconfigures_stdio_to_utf8_on_windows — mirrors the existing assertions in test_cli.py and test_fact_checker.py, pinning the per-stream errors policy.
  • test_change_preview_survives_a_legacy_console_codepage — reproduces the actual failure rather than asserting a call happened: a child process gets a legacy codepage for stdout and is asked to print a real mojibake preview.
  • Mutant-checked. Neutralising the reconfigure call turns the second test red with the genuine UnicodeEncodeError: 'gbk' codec can't encode character '\xc3', so it is pinning the behaviour and not the wiring.
  • Full suite: 3861 passed, 31 skipped.
  • ruff check . and ruff format --check . clean (213 files).

Notes

Related to #1122 (same exception class, main CLI's help output) but not a duplicate: different entry point, and a different fix — that one can be solved with ASCII-safe static strings, whereas this one prints arbitrary user content and needs the stdio reconfigure.

Found while auditing the encoding-repair surface. I also filed #2193 for a separate, more serious defect in the same module — that one is a correctness question in _HIGH_CONFIDENCE_RUN and needs a design call, so I kept it out of this PR to keep this one trivially reviewable.

…ntry points

mempalace/_stdio.py states the rule in its own docstring -- "every console
entry point that touches stdio needs to fix this on Windows" -- and cli.py
and fact_checker.py both apply it. scripts/mempalace_repair_encoding.py,
added later, does not.

That matters more here than anywhere else: this tool exists for Windows users
whose palace carries legacy mojibake, and it prints a before/after preview for
every proposed change. Under the console codepage it was written for, the
preview cannot be encoded -- the lead bytes of the corruption it detects are
exactly the characters the codepage rejects -- so the run dies with
UnicodeEncodeError before repairing a single drawer.

Reproduced with stdout on cp936:

  UnicodeEncodeError: 'gbk' codec can't encode character '\xc3'
  in position 14: illegal multibyte sequence

U+00C3 is the lead character of the à family this tool repairs.

Uses replace on stdout/stderr, matching cli.py and fact_checker.py, because
the preview carries verbatim drawer text that may hold surrogate halves
round-tripped from filenames; strict would crash mid-preview.

Related to MemPalace#1122 (same exception class on the main CLI's help output) but a
different entry point and a different fix -- that one can use ASCII-safe
static strings, this one prints arbitrary user content.

@igorls igorls left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wave 1 for 3.7.0: LGTM.

@igorls
igorls merged commit ffb5559 into MemPalace:develop Aug 11, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants