Skip to content

fix(background_review): add system disclaimer to prevent curation prompt leakage into user memory (#32858) - #53167

Open
Tranquil-Flow wants to merge 1 commit into
NousResearch:mainfrom
Tranquil-Flow:fix/32858-bg-review-leak
Open

fix(background_review): add system disclaimer to prevent curation prompt leakage into user memory (#32858)#53167
Tranquil-Flow wants to merge 1 commit into
NousResearch:mainfrom
Tranquil-Flow:fix/32858-bg-review-leak

Conversation

@Tranquil-Flow

Copy link
Copy Markdown
Contributor

Summary

Fixes #32858.

When the background self-improvement curation loop runs, the review agent receives its operational guidelines (_SKILL_REVIEW_PROMPT / _MEMORY_REVIEW_PROMPT / _COMBINED_REVIEW_PROMPT) as a user_message. Because these directives travel in the user role — and they themselves contain language like "Be ACTIVE" and detailed preference hierarchies — the LLM misinterprets them as explicit user preferences and writes them to USER.md. Those false entries are then uploaded to Honcho and permanently digested as user traits.

Fix

Prepend an explicit [System Note] disclaimer to all three review prompt constants, telling the LLM that the instructions are system-generated (not user input) and must not be saved as user preferences/expectations in memory.

  • Introduces a shared _BG_REVIEW_SYSTEM_DISCLAIMER constant (single source of truth, avoids duplication).
  • Applied to _MEMORY_REVIEW_PROMPT, _SKILL_REVIEW_PROMPT, and _COMBINED_REVIEW_PROMPT.
  • Exported via __all__ for back-compat (AIAgent exposes these as class attributes).

This matches the fix the issue author proposed, with the added completeness of fixing all three prompts (not just two).

Why not just change the message role?

The review prompts are intentionally passed as user_message because run_conversation() treats the user role as the active turn to act on. Injecting them as a system message would change the agent's turn-taking semantics. The disclaimer approach is the minimal, lowest-risk fix.

Verification

  • 8 new regression tests in tests/run_agent/test_background_review_prompt_leak.py:
    • 3 tests assert each prompt constant contains the disclaimer.
    • 1 test asserts the disclaimer appears at the start of each prompt.
    • 1 test asserts all 3 prompts are distinct and all fixed (guard against the competitor gap).
    • 3 tests exercise the real spawn_background_review_thread() path for memory-only, skill-only, and combined modes, verifying the returned prompt includes the disclaimer.
  • Fail-without-fix proven: all 8 tests fail on upstream/main without the disclaimer.
  • No regressions: all 25 existing test_background_review* tests still pass.
  • Focused test command: python3 -m pytest tests/run_agent/test_background_review_prompt_leak.py -v -o addopts=

Competing PRs

Auto-published by Moonsong via Path B automated pipeline.

…mpt leakage into user memory (NousResearch#32858)

The background review fork passes its operational prompts
(_MEMORY_REVIEW_PROMPT, _SKILL_REVIEW_PROMPT, _COMBINED_REVIEW_PROMPT)
as user_message, making the LLM misinterpret system-generated
guidelines (e.g. 'Be ACTIVE', preference hierarchies) as explicit
user preferences. These false observations get written to USER.md
and ingested by Honcho as permanent user traits.

Fix: introduce _BG_REVIEW_SYSTEM_DISCLAIMER and prepend it to all
three review prompts. The disclaimer explicitly tells the LLM that
the instruction is system-generated and must not be saved as user
preferences. This covers all three prompts (competitor PR NousResearch#32862
only fixed two, missing _MEMORY_REVIEW_PROMPT).

Regression tests verify all three prompts carry the disclaimer
and spawn_background_review_thread selects the correct prompt.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/memory Memory tool and memory providers sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state P2 Medium — degraded but workaround exists labels Jun 26, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Competing fix cluster for #32858: this PR prepends a [System Note] disclaimer to all three review-prompt constants (_MEMORY_/_SKILL_/_COMBINED_REVIEW_PROMPT); #32862 takes the same disclaimer approach on two of three; #25934 instead switches the prompts to role:system. Flagging the cluster so a maintainer can pick disclaimer-vs-role:system. Not a duplicate (broadest variant here).

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for covering all three review prompts; the current-main premise is present because agent/background_review.py:838-845 passes the selected operational prompt as user_message, and the prompts include behavior/preference language at agent/background_review.py:175-176, 181-185, and 286-294.

Problems

  • The new common prefix makes every produced harness start with [System Note], but hermes_state.py:228-250 only recognizes two unprefixed Review the conversation... openings. SessionDB.get_messages_as_conversation() relies on that matcher at hermes_state.py:4426-4436 to remove a mistakenly persisted review harness and its curator reply. The combined opening is also not represented in the current prefix set. A persisted prefixed harness would therefore bypass this defense-in-depth recovery.

Suggested changes

  • Extend the harness detector and its tests to recognize each of the three actual prefixed review prompts, including combined mode, before changing their opening text.

Automated hermes-sweeper review.


_MEMORY_REVIEW_PROMPT = (
"Review the conversation above and consider saving to memory if appropriate.\n\n"
_BG_REVIEW_SYSTEM_DISCLAIMER

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new leading prefix invalidates the recovery matcher in hermes_state.py:228-250, which only accepts prompts beginning with Review the conversation...; get_messages_as_conversation() depends on it to strip accidentally persisted review turns. Please update that matcher and add coverage for all three prefixed prompt variants, including combined mode.

@teknium1 teknium1 added the sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users label Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/memory Memory tool and memory providers type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Background Curation Prompts Leak into User Memory & Honcho Representations

3 participants