fix(background-review): add relevance gate to skill review prompt

[Kailigithub]

Summary

The per-session background review fork could write unrelated content into a skill via skill_manage(write_file) and skill_manage(patch) when the conversation produced research or notes that did not match the skill's documented purpose. The "Be ACTIVE" pressure in the prompt encouraged false positives: the fork saw research output and grafted it onto the nearest plausible skill.

This PR adds a mandatory relevance gate sentence near the top of both _SKILL_REVIEW_PROMPT and _COMBINED_REVIEW_PROMPT. The gate explicitly forbids writing when the proposed content is not directly about the skill's documented purpose, and frames "do not write" as a valid outcome rather than a missed learning opportunity.

Diff

+ "Relevance gate (mandatory): before any skill_manage(write_file), "
+ "skill_manage(patch), or skill_manage(create), verify that the "
+ "content is directly about the skill's documented purpose and scope. "
+ "A skill about X should not receive references about Y, even if Y is "
+ "interesting research. When in doubt, do not write — 'nothing to "
+ "save' is the correct outcome for sessions that did not involve the "
+ "skill's domain. Topical mismatch is the most common false positive; "
+ "refusing to write is a valid, valuable action.\n\n"

The gate is placed before the existing preference order (1–4) so it frames every action that follows, not just one path. The "Be ACTIVE" language is preserved — the gate does not contradict the activity pressure, it qualifies it with a relevance precondition.

Scope

This is the prompt-only half of the issue's proposal. The code-gate half (gating _should_review_skills on a _skills_loaded_this_session flag tracked in turn_finalizer.py) is a larger change that touches the review trigger across multiple call sites and the session-finalizer state machine; it is intentionally out of scope for a 17-line prompt fix. A follow-up PR can land that change once we agree on the flag-tracking semantics.

Verification

• python3 -m py_compile agent/background_review.py → OK
• python3 -m pytest tests/run_agent/test_background_review*.py → 38 passed
• New test: /tmp/test_relevance_gate.py (4 assertions covering both prompts, gate positioning, and gate text). Confirmed it fails on main (no relevance gate) and passes with this fix via the git-stash three-step dance.
• No new imports. No new dependencies. No new config flags.

Closes #66350

[teknium1]

Thanks for the focused mitigation. Current main still triggers skill review from cadence plus tool availability (agent/turn_finalizer.py:493-499), while both prompts combine an active-update bias with instructions to add research/domain notes under existing skills (agent/background_review.py:181-235, 286-323). The existing read-before-write guard verifies only that the target was loaded, not that the content belongs there (tools/skill_manager_tool.py:382-409).

Problems

• The new gate says “before any” mutation but omits skill_manage(action="edit"), which can fully rewrite SKILL.md (tools/skill_manager_tool.py:864-890). That leaves the same unrelated-content failure mode available through a sibling action.
• No in-repo regression test accompanies the prompt change. tests/run_agent/test_review_prompt_class_first.py:24-222 is the existing behavior-test home for both prompt variants.

Suggested changes

• Cover all mutating actions, preferably by phrasing the gate as applying before any skill mutation; at minimum include edit.
• Add invariant-style tests for both prompts requiring purpose/scope relevance before the support-file guidance.

Automated hermes-sweeper review.

[teknium1]

This says the gate applies before “any” mutation, but skill_manage(action="edit") is also a supported full SKILL.md rewrite (tools/skill_manager_tool.py:864). Please cover edit too, preferably by phrasing this as every mutating skill action.

[GottZ]

This was generated by AI during triage.

Summary

Nine PRs address this background-review skill-mutation complex: #55657/#55906 add the read-before-write guard, #55665 adds a shrink heuristic, #60331/#62414/#62428/#62664 teach the prompt-side handshake, #66472 adds a topical-relevance gate, and #73975 repairs read-mark propagation across copied tool contexts. The diffs cover distinct layers of the reported failures, with substantial duplication within the guard and prompt-handshake groups.

Related pull requests

• fix(skills): require review forks to read before writing skills #55657 [closed] fixes — (+201/-2) — n/a: Adds exact-file read tracking through skill_view, guards patch/edit/existing-file overwrite/removal, and resets marks for each background review, directly addressing transcript-inferred corruption. Closed because its implementation was cherry-picked with attribution into fix(skills): require background-review forks to read before writing skills (#55647) #55906; the visible diff does not guard the delete path flagged by a non-contributor review.
• fix(tools): guard against catastrophic content loss in background review patches #55665 [closed] partial — (+26/-0) — n/a: Adds a >50% shrink rejection for background-review patches, limiting catastrophic truncation but not preventing smaller unrelated or hallucinated rewrites. Closed as superseded by fix(skills): require background-review forks to read before writing skills (#55647) #55906's structural read-before-write invariant.
• fix(skills): require background-review forks to read before writing skills (#55647) #55906 [merged] best fix — (+202/-2) — n/a: Merged reference implementation for background-review: skill_manage patches hallucinate existing skill content; review fork writes without read-before-write invariant #55647, carrying fix(skills): require review forks to read before writing skills #55657's exact-target skill_view authorization and per-review reset into the skill mutation paths. It remains relevant as the guard on which the later prompt-handshake and context-propagation fixes depend.
• fix(review): teach skill-review prompts the v2026.7.1 read-before-write guard #60331 [closed] best fix — (+27/-4) — n/a: Updates both review prompts to reload targets before mutation, addressing the prompt/guard mismatch, but its wording incorrectly includes delete and all support-file writes and adds no contract tests. The earlier keep_open review rated salvageability high and identified the narrower existing-target wording and tests needed; the PR is now closed but remains the recorded best-fix reference for Background review fork can't patch skills: prompt never calls skill_view, guard refuses the write (read-before-write mismatch) #62397.
• fix(review): teach skill-review prompts the v2026.7.1 read-before-write guard #62414 fixes — (+28/-6) — n/a: Makes a byte-near-identical two-prompt handshake change to fix(review): teach skill-review prompts the v2026.7.1 read-before-write guard #60331 and therefore addresses the same mismatch, while likewise overclaiming delete coverage and requiring pre-read of new support files. Despite its keep_open review, the complete diff provides no distinct salvage over fix(review): teach skill-review prompts the v2026.7.1 read-before-write guard #60331 and lacks the requested behavioral tests.
• fix(agent): instruct background review fork to call skill_view before patching skills (#62397) #62428 [closed] partial — (+27/-3) — n/a: Adds handshake and retry instructions to both prompts plus clearer tool-denial guidance, but uses skill_view(name) for every mutation, which does not authorize existing support-file paths. Closed as a duplicate of fix(review): teach skill-review prompts the v2026.7.1 read-before-write guard #60331; its keep_open review specifically required file_path-aware support-file handling and tests.
• fix(agent): document read-before-write handshake in background-review prompt (#62397) #62664 [closed] best fix — (+156/-6) — n/a: Adds a shared handshake to both prompt variants and tests both routes, including exact-path support-file guidance and retry handling, but the visible prompt still claims delete is guarded. Although its keep_open review initially identified the combined-prompt omission, the visible diff covers both prompts; it remains closed and is a tested best-fix reference for Background review fork can't patch skills: prompt never calls skill_view, guard refuses the write (read-before-write mismatch) #62397.
• fix(background-review): add relevance gate to skill review prompt #66472 best fix — (+17/-2) — n/a: Adds the relevance gate to both prompts, directly addressing unrelated research being attached to an existing skill while preserving a valid nothing-to-save outcome. The keep_open review is supported by the diff but requires salvage work: extend the invariant to every skill mutation, including edit, and add regression coverage for both prompt variants.
• fix(skills): preserve review read marks across tool contexts #73975 best fix — (+85/-9) — n/a: Replaces immutable ContextVar values with a lock-protected per-review mark store shared across copied tool contexts, with regressions for cross-context success and review isolation. This directly addresses bg-review: _background_review_read_before_write_guard uses turn-scoped ContextVar, preventing multi-turn view→patch workflow #73965; the maintainer-bot keep_open verdict found no substantive correctness issue and asks that salvage preserve main's f57cb2e atomic-write refactor.

Duplicates

#55657 was cherry-picked into and superseded by #55906; #55665 targets the same #55647 failure with a weaker size heuristic. #62414 is byte-near-identical to #60331, while #62428 and #62664 are additional prompt-handshake variants for #62397, with #62664 providing broader visible test coverage.

Suggested consolidation

Keep #66472 open with a salvage path: apply its relevance invariant to every skill mutation, including edit, and add two-prompt regression tests, as required by the visible keep_open review. Close #62414 as duplicate of #60331 despite its keep_open review because the complete diff is byte-near-identical and retains the same new-file/delete scope errors; leave the already closed prompt variants as references, and keep #73975 open on the maintainer-bot's high-salvageability path while preserving f57cb2e.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I66350(["issue #66350 (open)"])
    P66472["PR #66472 (open)"]
    P66472 -->|best fix| I66350
    class I66350 open
    class P66472 open
    class P66472 best
    class P66472 target
    click I66350 "https://github.com/NousResearch/hermes-agent/issues/66350"
    click P66472 "https://github.com/NousResearch/hermes-agent/pull/66472"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 9 pull requests and 4 issues in this complex. Each diff was read against this issue; Assessment working set: 59 kB of PR diffs, 39 kB of issue/PR text, 29 kB of discussion (23 comments), 20 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.