fix(normalize): handle UTF-8 BOM in transcript files

[arnoldwender]

What and Why

normalize() opens transcript files with encoding='utf-8'. When a file has a UTF-8 BOM prefix (\xef\xbb\xbf) — common from Windows exports of Claude Code JSONL sessions — json.loads() raises JSONDecodeError on every line because the first line starts with {. _try_claude_code_jsonl silently skips all lines and the file falls through as raw unstructured text, discarding all message content.

Root Cause

normalize.py:124 — encoding="utf-8" does not strip the BOM; encoding="utf-8-sig" does.

Fix

# Before
with open(filepath, "r", encoding="utf-8", errors="replace") as f:

# After
with open(filepath, "r", encoding="utf-8-sig", errors="replace") as f:

Python's utf-8-sig codec strips the BOM on read and is fully backward-compatible with BOM-free files. No behavioral change on Linux/macOS files.

Reproduction

import tempfile, os
from mempalace.normalize import normalize

# BOM-prefixed JSONL (Windows Claude Code export)
bom_file = tempfile.mktemp(suffix=".jsonl")
with open(bom_file, "wb") as f:
    f.write(b"\xef\xbb\xbf")  # UTF-8 BOM
    f.write(b'{"type":"human","message":{"content":"hello"}}\n')

result_before = normalize(bom_file)  # returns raw JSON lines, not a transcript
# After fix: returns properly formatted transcript with "hello"

Tests

All 107 test_normalize.py tests pass. All 1066 tests pass.

Closes #1034 (partial — encoding fix for normalize.py; other files addressed in separate PRs)

[arnoldwender]

Friendly ping — this one's been quiet since I opened it. CI is green across all 6 checks (linux 3.9/3.11/3.13, windows, macos, lint). Happy to adjust scope or rework the approach if a different fix shape works better here.

[arnoldwender]

Second ping after a week. CI is green across all 6 checks, no conflicts, the change is a single-character fix (utf-8 → utf-8-sig in normalize.py) plus the regression test that catches BOM-prefixed transcripts.

@igorls — pinging you given the recent encoding/durability cluster activity (#1214/#1215 merged, #1408/#1409 follow-ups in flight). Happy to adjust scope or approach if a different fix shape would be preferred.

[arnoldwender]

Rebased on upstream/develop (latest 1247e17, post-3.3.5 release). No logical changes — conflict-free rebase, only commit replay onto current develop. CI green pre-rebase; awaiting fresh CI run.

[arnoldwender]

Friendly ping — post-3.3.6 release this is still mergeable on upstream/develop @ f5ea021 (CI green, no conflicts). Happy to adjust anything if it would help land it.