Skip to content

fix(normalize): strip noise blocks with multi-paragraph bodies and indented tags - #2064

Open
mazurd-acre wants to merge 2 commits into
MemPalace:developfrom
mazurd-acre:fix/noise-tags-cross-blank-lines
Open

fix(normalize): strip noise blocks with multi-paragraph bodies and indented tags#2064
mazurd-acre wants to merge 2 commits into
MemPalace:developfrom
mazurd-acre:fix/noise-tags-cross-blank-lines

Conversation

@mazurd-acre

Copy link
Copy Markdown

Problem

Two escape paths let Claude Code harness chrome leak into mined drawers and pollute search/recall:

  1. <task-notification> and <system-reminder> blocks carrying multi-paragraph payloads (subagent results, recalled memories) were never stripped: the tag-body pattern refused to cross blank lines, so exactly the largest noise blocks survived wholesale.
  2. Claude Code emits slash-command chrome with indented tags (<command-message>, <command-args> after <command-name>), which the line-start anchor missed. command-args was also missing from the noise list entirely.

Found by mining ~90 real Claude Code sessions: thousands of drawers contained raw task-notification payloads, and semantic recall surfaced harness boilerplate instead of content.

Fix

  • Tag bodies may now span blank lines, guarded by a same-tag-reopen lookahead instead of the blank-line rule: a dangling open tag still cannot merge with a later block and eat the real content between them (the original span-eating protection this rule existed for).
  • The line anchor tolerates leading whitespace after the optional > blockquote marker.
  • command-args added to _NOISE_TAGS.

Testing

  • 3 new regression tests (multi-paragraph block, dangling-tag safety, indented slash-command chrome).
  • Full test suite green.
  • Re-mined the same ~90 real sessions (20k+ drawers): zero noise blocks survive, inline prose mentioning the tags is still preserved.

task-notification and system-reminder blocks carrying multi-paragraph
payloads (subagent results, recalled memories) leaked wholesale into
drawers because the tag body refused to cross blank lines. Replace the
blank-line guard with a same-tag-reopen guard: bodies may now span
paragraphs, but a dangling open tag still cannot merge with a later
block and eat the content between them.
Claude Code emits slash-command chrome with indented tags
(<command-message>, <command-args> after <command-name>), which the
line-start anchor missed. Allow leading whitespace before a noise tag
and add command-args to the noise list.
gnusam pushed a commit to gnusam/mempalace-pgsql that referenced this pull request Jul 29, 2026
Introduces the strip_noise layer this fork never carried, reconciling
three overlapping upstream PRs into one pass applied at every
normalize() exit:

- the full 10-tag Claude Code envelope (system-reminder, task-
  notification, the six slash-command chrome tags, user-prompt-submit-
  hook, hook_output) and ECMA-48 ANSI CSI/OSC escapes from Bash-tool
  output, per PR MemPalace#1909 (issue MemPalace#1333; PR MemPalace#1958 covers the same ground)
- indent-tolerant line anchors and multi-paragraph tag bodies that halt
  at the next same-tag opening instead of the first blank line, per
  PR MemPalace#2064 — subagent results and recalled memories inside a block are
  stripped whole, while a dangling open tag can never merge with a
  later block and eat the real content between them

Verbatim is sacred: patterns are line-anchored, ESC-byte-anchored, or
narrow chrome shapes; prose that merely names "[1m" survives. Already-
mined drawers keep their chrome until their transcript next changes —
session logs are append-heavy, so re-mines pick this up naturally.

Co-authored-by: KeilerHirsch <KeilerHirsch@users.noreply.github.com>
Co-authored-by: mazurd-acre <mazurd-acre@users.noreply.github.com>
Co-authored-by: jrzmurray <jrzmurray@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@fatkobra fatkobra left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: allowing the noise-block body to cross blank lines reintroduces a destructive dangling-tag case.

The negative lookahead prevents an unclosed opener from merging with a later opening of the same tag, but it does not prevent that opener from consuming arbitrary transcript content until a later standalone or literal closing tag.

For example, an unclosed <system-reminder> followed by legitimate conversation text and a later user/code example containing </system-reminder> can cause all intervening content to be removed.

That contradicts the conservative normalization invariant: when pairing is uncertain, original transcript content should survive.

Please bound matching using actual transcript/message boundaries, or replace the cross-paragraph regex with a small parser that strips only demonstrably paired noise blocks.

Please add a regression containing:

  1. a dangling opening tag;
  2. blank lines;
  3. legitimate user/assistant content; and
  4. a later closing-tag token without another opening tag.

The legitimate intervening content must remain unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants