fix(background_review): add system disclaimer to prevent curation prompt leakage into user memory (#32858) - #53167
Conversation
…mpt leakage into user memory (NousResearch#32858) The background review fork passes its operational prompts (_MEMORY_REVIEW_PROMPT, _SKILL_REVIEW_PROMPT, _COMBINED_REVIEW_PROMPT) as user_message, making the LLM misinterpret system-generated guidelines (e.g. 'Be ACTIVE', preference hierarchies) as explicit user preferences. These false observations get written to USER.md and ingested by Honcho as permanent user traits. Fix: introduce _BG_REVIEW_SYSTEM_DISCLAIMER and prepend it to all three review prompts. The disclaimer explicitly tells the LLM that the instruction is system-generated and must not be saved as user preferences. This covers all three prompts (competitor PR NousResearch#32862 only fixed two, missing _MEMORY_REVIEW_PROMPT). Regression tests verify all three prompts carry the disclaimer and spawn_background_review_thread selects the correct prompt.
Competing fix cluster for #32858: this PR prepends a |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for covering all three review prompts; the current-main premise is present because agent/background_review.py:838-845 passes the selected operational prompt as user_message, and the prompts include behavior/preference language at agent/background_review.py:175-176, 181-185, and 286-294.
Problems
- The new common prefix makes every produced harness start with
[System Note], buthermes_state.py:228-250only recognizes two unprefixedReview the conversation...openings.SessionDB.get_messages_as_conversation()relies on that matcher athermes_state.py:4426-4436to remove a mistakenly persisted review harness and its curator reply. The combined opening is also not represented in the current prefix set. A persisted prefixed harness would therefore bypass this defense-in-depth recovery.
Suggested changes
- Extend the harness detector and its tests to recognize each of the three actual prefixed review prompts, including combined mode, before changing their opening text.
Automated hermes-sweeper review.
|
|
||
| _MEMORY_REVIEW_PROMPT = ( | ||
| "Review the conversation above and consider saving to memory if appropriate.\n\n" | ||
| _BG_REVIEW_SYSTEM_DISCLAIMER |
There was a problem hiding this comment.
This new leading prefix invalidates the recovery matcher in hermes_state.py:228-250, which only accepts prompts beginning with Review the conversation...; get_messages_as_conversation() depends on it to strip accidentally persisted review turns. Please update that matcher and add coverage for all three prefixed prompt variants, including combined mode.
Summary
Fixes #32858.
When the background self-improvement curation loop runs, the review agent receives its operational guidelines (
_SKILL_REVIEW_PROMPT/_MEMORY_REVIEW_PROMPT/_COMBINED_REVIEW_PROMPT) as auser_message. Because these directives travel in theuserrole — and they themselves contain language like "Be ACTIVE" and detailed preference hierarchies — the LLM misinterprets them as explicit user preferences and writes them toUSER.md. Those false entries are then uploaded to Honcho and permanently digested as user traits.Fix
Prepend an explicit
[System Note]disclaimer to all three review prompt constants, telling the LLM that the instructions are system-generated (not user input) and must not be saved as user preferences/expectations in memory._BG_REVIEW_SYSTEM_DISCLAIMERconstant (single source of truth, avoids duplication)._MEMORY_REVIEW_PROMPT,_SKILL_REVIEW_PROMPT, and_COMBINED_REVIEW_PROMPT.__all__for back-compat (AIAgent exposes these as class attributes).This matches the fix the issue author proposed, with the added completeness of fixing all three prompts (not just two).
Why not just change the message role?
The review prompts are intentionally passed as
user_messagebecauserun_conversation()treats the user role as the active turn to act on. Injecting them as asystemmessage would change the agent's turn-taking semantics. The disclaimer approach is the minimal, lowest-risk fix.Verification
tests/run_agent/test_background_review_prompt_leak.py:spawn_background_review_thread()path for memory-only, skill-only, and combined modes, verifying the returned prompt includes the disclaimer.upstream/mainwithout the disclaimer.test_background_review*tests still pass.python3 -m pytest tests/run_agent/test_background_review_prompt_leak.py -v -o addopts=Competing PRs
_MEMORY_REVIEW_PROMPT), inline duplication, no tests. This PR completes the coverage and adds tests._persist_disabled). Addresses session-takeover, not the prompt-leak-into-memory root cause described in the issue.Auto-published by Moonsong via Path B automated pipeline.