fix(memory): log and degrade gracefully on invalid UTF-8 in memory files

[JoaoMarcos44]

Fixes #57754

Problem

MemoryStore._read_file() (tools/memory_tool.py:691-694) reads MEMORY.md/USER.md with path.read_text(encoding="utf-8") and only catches (OSError, IOError) around it.

UnicodeDecodeError is a ValueError subclass, not an OSError/IOError, so invalid UTF-8 bytes in either file raise past this catch. The exception then propagates into agent_init.py:1239-1253's broad try/except Exception: pass, which silently swallows it. Net effect: memory is disabled for the whole session (agent._memory_store stays None), while agent._memory_enabled/agent._user_profile_enabled remain True from config — and no log line is ever emitted. There is nothing for an operator to go on when debugging "why did memory stop loading."

Fix

Catch UnicodeDecodeError explicitly in _read_file(), log a clear error (file path + underlying decode error + remediation hint), and degrade to an empty entry list — same graceful-degradation behavior as before, but now visible.

try:
    raw = path.read_text(encoding="utf-8")
except (OSError, IOError):
    return []
except UnicodeDecodeError as e:
    logger.error(
        "Memory file %s has invalid UTF-8 (%s) -- treating as empty. "
        "Fix or remove the file to restore memory.",
        path, e,
    )
    return []

This is a minimal, targeted fix at the exact point the exception was escaping — no changes to control flow, call sites, or the broader except Exception: pass in agent_init.py (that catch-all is intentionally permissive by design for other truly-optional memory init steps; narrowing it is a separate, larger change out of scope here).

Verification

Reproduced the bug against main first (confirmed silent failure), then verified the fix on this branch:

from pathlib import Path
from tools.memory_tool import MemoryStore

p = Path(tempfile.mkdtemp()) / "MEMORY.md"
p.write_bytes(b"\xff\xfe invalid utf8 bytes")

result = MemoryStore._read_file(p)
# -> ERROR:tools.memory_tool:Memory file ...MEMORY.md has invalid UTF-8
#    ('utf-8' codec can't decode byte 0xff in position 0: invalid start byte)
#    -- treating as empty. Fix or remove the file to restore memory.
# result: []

Test plan

• PoC reproduces silent failure on unpatched code (no log emitted, _read_file raises uncaught)
• After fix: _read_file returns [] and emits a logger.error with file path + decode error
• No behavior change for valid UTF-8 files or missing files

[alt-glitch]

This was generated by AI during triage.

Related to the memory non-UTF-8 fix cluster for the same bug (#49508 / #10879 family, and the issue this Fixes, #57754). Note the mechanisms differ: #49516 and #10888 recover content (retry with errors='replace' / utf-8-sig), while this PR degrades to an empty entry list with a log line. Competing fixes — maintainer should pick a degradation strategy (recover-with-replacement vs. fail-visible-empty).

[teknium1]

Thanks for isolating the decode failure. The premise is confirmed on current main: tools/memory_tool.py:692 performs a strict UTF-8 read without catching UnicodeDecodeError, and the initialization path swallows that failure at agent/agent_init.py:1389-1391.

Problems

• The same uncaught strict read remains in _detect_external_drift() at tools/memory_tool.py:732. _reload_target() calls it at tools/memory_tool.py:303 before replace, remove, and apply_batch, so those mutation paths still fail on invalid bytes.
• Returning an empty list can lose the corrupt file's contents. add() deliberately skips the drift guard at tools/memory_tool.py:354, then saves its in-memory list at tools/memory_tool.py:382-384; after this fallback that writes over the original invalid bytes.
• No regression test is added. Existing persistence/load tests at tests/tools/test_memory_tool.py:481-503 and :843-917 cover valid UTF-8 only.

Suggested changes

• Select one data-preserving corrupt-file strategy and apply it to both strict read sites; the existing discussion correctly notes the trade-off between recovery and fail-visible handling.
• Add invalid-byte coverage for initialization and subsequent mutation paths, including protection against overwrite.

Automated hermes-sweeper review.