fix(normalize): strip noise blocks with multi-paragraph bodies and indented tags - #2064
fix(normalize): strip noise blocks with multi-paragraph bodies and indented tags#2064mazurd-acre wants to merge 2 commits into
Conversation
task-notification and system-reminder blocks carrying multi-paragraph payloads (subagent results, recalled memories) leaked wholesale into drawers because the tag body refused to cross blank lines. Replace the blank-line guard with a same-tag-reopen guard: bodies may now span paragraphs, but a dangling open tag still cannot merge with a later block and eat the content between them.
Claude Code emits slash-command chrome with indented tags (<command-message>, <command-args> after <command-name>), which the line-start anchor missed. Allow leading whitespace before a noise tag and add command-args to the noise list.
Introduces the strip_noise layer this fork never carried, reconciling three overlapping upstream PRs into one pass applied at every normalize() exit: - the full 10-tag Claude Code envelope (system-reminder, task- notification, the six slash-command chrome tags, user-prompt-submit- hook, hook_output) and ECMA-48 ANSI CSI/OSC escapes from Bash-tool output, per PR MemPalace#1909 (issue MemPalace#1333; PR MemPalace#1958 covers the same ground) - indent-tolerant line anchors and multi-paragraph tag bodies that halt at the next same-tag opening instead of the first blank line, per PR MemPalace#2064 — subagent results and recalled memories inside a block are stripped whole, while a dangling open tag can never merge with a later block and eat the real content between them Verbatim is sacred: patterns are line-anchored, ESC-byte-anchored, or narrow chrome shapes; prose that merely names "[1m" survives. Already- mined drawers keep their chrome until their transcript next changes — session logs are append-heavy, so re-mines pick this up naturally. Co-authored-by: KeilerHirsch <KeilerHirsch@users.noreply.github.com> Co-authored-by: mazurd-acre <mazurd-acre@users.noreply.github.com> Co-authored-by: jrzmurray <jrzmurray@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fatkobra
left a comment
There was a problem hiding this comment.
Blocking: allowing the noise-block body to cross blank lines reintroduces a destructive dangling-tag case.
The negative lookahead prevents an unclosed opener from merging with a later opening of the same tag, but it does not prevent that opener from consuming arbitrary transcript content until a later standalone or literal closing tag.
For example, an unclosed <system-reminder> followed by legitimate conversation text and a later user/code example containing </system-reminder> can cause all intervening content to be removed.
That contradicts the conservative normalization invariant: when pairing is uncertain, original transcript content should survive.
Please bound matching using actual transcript/message boundaries, or replace the cross-paragraph regex with a small parser that strips only demonstrably paired noise blocks.
Please add a regression containing:
- a dangling opening tag;
- blank lines;
- legitimate user/assistant content; and
- a later closing-tag token without another opening tag.
The legitimate intervening content must remain unchanged.
Problem
Two escape paths let Claude Code harness chrome leak into mined drawers and pollute search/recall:
<task-notification>and<system-reminder>blocks carrying multi-paragraph payloads (subagent results, recalled memories) were never stripped: the tag-body pattern refused to cross blank lines, so exactly the largest noise blocks survived wholesale.<command-message>,<command-args>after<command-name>), which the line-start anchor missed.command-argswas also missing from the noise list entirely.Found by mining ~90 real Claude Code sessions: thousands of drawers contained raw task-notification payloads, and semantic recall surfaced harness boilerplate instead of content.
Fix
>blockquote marker.command-argsadded to_NOISE_TAGS.Testing