fix(windows): safely repair legacy mojibake with backups

[fatkobra]

What does this PR do?

• add a supported repair path for legacy Windows mojibake
• automatically repair only high-confidence UTF-8-as-Windows-1252 signatures
• deliberately leave ambiguous Ä... and Å... sequences unchanged
• preserve the multilingual clean-text cases raised during review
• print bounded before/after previews for every proposed change
• default to a read-only dry-run
• create a private, non-overwriting backup before any update
• flush original documents to disk before overwriting live rows
• bind each backup to its collection name
• provide a validated restore operation
• hold the palace writer lock during apply and restore

Detection contract

The original implementation treated any short Windows-1252 character window whose bytes formed valid UTF-8 as damaged.

That was unsafe because legitimate sequences such as:

Ų
É®
É 

can also form valid UTF-8 byte sequences.

The revised automatic mode recognizes only the common visible lead families produced by UTF-8-as-Windows-1252 corruption:

Â...
Ã...
â...
ð...
ï...

The Ä... and Å... families are intentionally excluded from automatic repair because they are ambiguous with legitimate Central European and scientific text.

This deliberately favors false negatives over destructive false positives.

The old Windows corruption path can also preserve five byte values that Windows-1252 leaves undefined:

0x81
0x8D
0x8F
0x90
0x9D

Those values appear in stored Python strings as invisible C1 control characters.

The revised decoder accepts them only as continuation positions inside an already-recognized high-confidence mojibake sequence. It maps each control code point back to its original raw byte before UTF-8 decoding.

This repairs characters such as Á, Í, Ï, Ð, Ý, and the closing curly quote ” without enabling the ambiguous visible Ä... or Å... lead families.

Standalone control characters are not rewritten.

Dry-run behavior

Dry-run is the default.

It scans the collection without writing and prints every proposed drawer ID with a bounded before/after preview.

Users can therefore inspect the actual proposed changes before choosing --apply.

Apply and backup safety

Apply requires a backup path internally. When no path is supplied on the command line, a timestamped path beside the palace is generated.

The backup:

• is created with exclusive creation;
• never overwrites an existing file;
• uses owner-only permissions where supported;
• records the target collection name;
• stores each changed drawer ID and its complete original document;
• is flushed and fsynced before the corresponding collection update.

The collection name is resolved through MemPalace backend wrappers, including ChromaCollection.

Restore safety

The same command restores original documents with:

python3 scripts/mempalace_repair_encoding.py \
  --palace /path/to/palace \
  --restore-backup /path/to/backup.jsonl

Before the first restore write, the implementation:

• validates the backup header;
• validates every JSONL record;
• rejects a backup belonging to a different collection.

Locking

Apply and restore hold the normal palace writer lock for the complete operation.

The underlying backend write lock remains re-entrant, so calls through ChromaCollection.update() do not deadlock.

Tests

Regression coverage includes:

• 28 clean multilingual strings;
• the exact PERÚ, Ų, CAFÉ®, and RÉSUMÉ : review cases;
• 12 high-confidence damaged strings;
• double-encoded text;
• ambiguous sequences that must remain unchanged;
• idempotence;
• paginated dry-run behavior;
• backup-before-update behavior;
• backup overwrite refusal;
• wrapped Chroma collection-name resolution;
• rejection of a backup for another collection;
• complete-backup validation before restoration;
• a real ChromaDB collection dry-run, apply, verification, and restore path.

The actual command was also exercised manually across separate processes against a disposable persistent palace:

1. seed four clean and two damaged drawers;
2. run the real dry-run;
3. verify that only the damaged IDs are proposed;
4. apply with an explicit backup;
5. validate the collection-bound backup and original documents;
6. reopen the palace in a fresh process;
7. verify every clean and repaired document exactly;
8. restore the backup through the command;
9. reopen again and verify the original damaged documents.

Additional regression coverage verifies:

• all five undefined Windows-1252 continuation-byte values;
• Spanish uppercase Á and Í;
• Ï, Ð, and Ý;
• both opening and closing curly quotes;
• a mixed row containing visible and invisible corruption;
• one-pass completion through repair_collection();
• a real ChromaDB apply where all selected rows are completely repaired;
• an immediate second dry-run reporting zero remaining changes;
• backup restoration of the original invisible-byte rows.

Closes #1055

How to test

• python3 -m pytest tests/test_encoding_repair.py -q
• python3 -m ruff format --check .
• python3 -m ruff check .
• python -m pytest tests/ -v

Checklist

• Tests pass (python -m pytest tests/ -v)
• No hardcoded paths
• Linter passes (ruff check .)

[mvalentsev]

Hey again @fatkobra! Ran into a few hard ones in your latest batch. Could you run them through e2e in mempalace itself? They all show up on the first real run.

This one: the dry-run flags text that was already fine and --apply overwrites it, because _decode_candidate reads "É" + "®" as 0xC9 0xAE, a valid two-byte sequence, and collapses the pair.

La canción «PERÚ» abre el disco.  ->  La canción «PERڻ abre el disco.
Buried surface area 1250 Ų.      ->  Buried surface area 1250 Ų.
CAFÉ® is a registered mark.       ->  CAFɮ is a registered mark.

French typography hits it too, since the capital É lands right before the non-breaking space in RÉSUMÉ :. That's 6 of 26 natural es/fr/de/pt/pl/ca strings coming back changed, and collection.update() keeps no copy, so those are gone for good.

#2141: against a live chromadb the collection still comes back carrying DefaultEmbeddingFunction, so the blocker it closes isn't closed.

#2143: the real chunk_text emits no byte-identical chunks on ordinary code or prose, which is why the test has to monkeypatch it to produce any.

Worth putting the earlier PRs through e2e as well, the problem and the fix, not just the fix.

[fatkobra]

Thanks @mvalentsev — this is valid and blocking feedback.

The current detector is fundamentally too broad: it accepts any short CP1252 character window whose bytes happen to decode as one non-ASCII UTF-8 character. That means already-correct text can be changed even when it contains no mojibake marker.

The examples with É®, Ų, and French typography demonstrate that this is not safe enough for an in-place repair operation. A few special-case exclusions would not be a responsible fix because other valid character combinations could still collide.

I am converting this PR to draft.

Before marking it ready again, I will require:

• a broad multilingual clean-text corpus that must remain byte-identical;
• real corrupted strings representing the pre-3.1 Windows failure;
• a live ChromaDB dry-run/apply/reopen test;
• exact changed-ID and before/after reporting;
• a backup or rollback path before any document update;
• proof that the dry-run selects damaged documents and zero clean documents.

I will also revisit #2141 and #2143 with real end-to-end reproductions rather than mocked versions of the suspected failure.

Thank you for running these against the real application.

[fatkobra]

Thanks again — the initial E2E run found one additional wrapper issue before I pushed the revision: the backup records were correct, but the collection field was null because the MemPalace ChromaCollection wrapper does not expose a public name attribute.

That is now fixed and covered as well.

The revised PR now:

• removes the broad sliding-window decoder;
• repairs only the high-confidence Â, Ã, â, ð, and ï lead families;
• deliberately leaves ambiguous Ä... and Å... families unchanged;
• preserves all 28 multilingual clean-text cases;
• includes the exact PERÚ, Ų, CAFÉ®, and RÉSUMÉ : review examples;
• creates a private, non-overwriting backup before apply;
• records and validates the target collection name through backend wrappers;
• flushes and fsyncs originals before collection updates;
• validates the complete backup before restoration;
• rejects restoration into a different collection;
• provides a tested restore operation;
• holds the palace writer lock during apply and restore.

I reran the complete command against a newly created disposable palace across separate processes:

1. dry-run scanned six rows and proposed only the two damaged IDs;
2. apply updated exactly those two rows;
3. the backup contained the correct collection name and two complete originals;
4. a fresh process read all four clean review strings unchanged and both damaged strings repaired;
5. restore validated and restored two records;
6. another fresh process verified the original damaged strings.

The PR remains draft pending review of the revised safety contract.

[mvalentsev]

One thing is still open, from the same place. cp1252 leaves five bytes undefined (0x81 0x8D 0x8F 0x90 0x9D) and _CONTINUATION_CHARS drops them, so a character whose UTF-8 continuation byte lands there can never match: Á Í Ï Ð Ý, and the closing quote ” while the opening “ repairs.

On a palace damaged the way a pre-3.1.0 Windows install damaged it, invisible bytes escaped:

in the palace   Ã\x81LVARO vive en PARÃ\x8dS. Ã\x8dNDICE: página 12.
after --apply   Ã\x81LVARO vive en PARÃ\x8dS. Ã\x8dNDICE: página 12.

in the palace   Dijo “holaâ€\x9d y se fue.
after --apply   Dijo “holaâ€\x9d y se fue.

Rows scanned: 4 / Documents needing repair: 4 / Documents updated: 4

Three of those four rows get rewritten, counted as updated, and are still damaged. A second run does not finish them.

#1055 is a Spanish palace, and the reporter writes there that DIRECT_FIXES covers the uppercase accented characters the sliding window cannot resolve. Á and Í are ordinary Spanish capitals.

The six strings from the last round do survive a real --apply now.

[fatkobra]

Thanks again @mvalentsev for excellent feedback — this was valid and blocking.

The previous revision deliberately omitted the five byte values that Windows-1252 leaves undefined:

0x81, 0x8D, 0x8F, 0x90, and 0x9D.

That preserved the earlier clean-text safety, but it also meant that old palaces containing those values as invisible C1 controls could only be partially repaired. The row was counted as updated when one visible portion changed even though another damaged sequence remained.

The revised decoder now:

• includes those five invisible controls as valid continuation positions;
• maps them back to their original raw byte values before UTF-8 decoding;
• still recognizes only the existing high-confidence Â, Ã, â, ð, and ï lead families;
• does not enable the ambiguous visible Ä... or Å... lead families;
• leaves the previous PERÚ, Ų, CAFÉ®, and RÉSUMÉ : clean cases unchanged.

New coverage includes:

• Á, Í, Ï, Ð, and Ý;
• the exact Ã\x81LVARO ... PARÃ\x8dS ... Ã\x8dNDICE example;
• the exact Dijo “holaâ€\x9d example;
• a four-row fake-collection apply;
• a four-row real ChromaDB apply;
• exact post-apply document verification;
• an immediate second dry-run that must report zero remaining changes;
• backup restoration of all four original damaged rows.

I also exercised the actual command against a disposable persistent palace with four clean and four damaged rows:

1. dry-run selected exactly the four damaged rows;
2. apply updated exactly four rows;
3. a fresh process verified the complete repaired text and no remaining C1 controls;
4. a second real dry-run reported zero repairs;
5. restore recovered all four original damaged documents.

[fatkobra]

The current Windows failure is unrelated to this PR's encoding changes.

The only failing test is:

tests/test_closets.py::TestDrawerGrepExpansion::test_hybrid_search_enrichment_isolates_chunks_across_drawers_sharing_source_file

It failed inside Chroma with:

Error creating hnsw segment reader: Nothing found on disk

The run otherwise completed with 3,357 passed tests, and none of the tests/test_encoding_repair.py tests failed.

PR #2148 changes only the encoding-repair module, its command-line script, and its tests. It does not modify the closet/search/HNSW path. The same test and exact HNSW error have also appeared in an earlier Windows CI run.

The workflow currently reruns only failures matching:

Failed to apply logs to the hnsw segment writer

so this Nothing found on disk variant was not retried.

@igorls could you please rerun the failed Windows job?

[igorls]

The current Windows failure is unrelated to this PR's encoding changes.

The only failing test is:

tests/test_closets.py::TestDrawerGrepExpansion::test_hybrid_search_enrichment_isolates_chunks_across_drawers_sharing_source_file

It failed inside Chroma with:

Error creating hnsw segment reader: Nothing found on disk

The run otherwise completed with 3,357 passed tests, and none of the tests/test_encoding_repair.py tests failed.

PR #2148 changes only the encoding-repair module, its command-line script, and its tests. It does not modify the closet/search/HNSW path. The same test and exact HNSW error have also appeared in an earlier Windows CI run.

The workflow currently reruns only failures matching:

Failed to apply logs to the hnsw segment writer

so this Nothing found on disk variant was not retried.

@igorls could you please rerun the failed Windows job?

I've been testing from mac recently, will switch to windows to test this better