feat: trigger-wording check — detect skills that should have loaded but didn't - #55965
feat: trigger-wording check — detect skills that should have loaded but didn't#55965pinchy-SA wants to merge 1 commit into
Conversation
The background review agent already replays every conversation to check
if skills/memory should be updated. But it had a blind spot: skills that
SHOULD have loaded but DIDN'T.
Skills fire on-demand via description match ('Use when...' trigger line).
If the description doesn't match the task phrasing, the skill never loads.
The miss is invisible by definition — the self-repair loop only fires
after a successful load, so a badly-worded trigger creates a permanent
blind spot that never surfaces.
This adds a TRIGGER-WORDING CHECK to both _SKILL_REVIEW_PROMPT and
_COMBINED_REVIEW_PROMPT. After every review pass, the reviewer:
1. Lists all skills (skills_list)
2. Compares descriptions against the conversation
3. If a skill covers the territory but was never loaded, patches its
trigger description to match the actual task phrasing
4. Logs the miss as 'TRIGGER FIX: <skill> — missed for <task>'
This closes the gap between lazy-loaded skills (context-efficient) and
always-injected doctrines (guaranteed recall). The miss is now detectable
and fixable instead of silent and permanent.
Tests follow the existing behavior-assertion pattern (not snapshot tests).
b02765d to
f579b10
Compare
teknium1
left a comment
There was a problem hiding this comment.
Thanks for targeting an otherwise hard-to-observe skill-selection gap. The current prompt has no explicit unloaded-skill comparison pass, but this implementation needs adjustment for current review-fork safety and reporting behavior.
Problems
- The new block asks the review fork to patch after
skills_list, without first requiringskill_view(agent/background_review.py:251, mirrored at:337). Current main rejects background-review patches until that exact SKILL.md was loaded:tools/skill_manager_tool.py:382-409and:956-963; see the regression contract intests/tools/test_skill_manager_tool.py:1289-1317. TRIGGER FIXis requested only in the fork's final reply (agent/background_review.py:267), but current main exposes only successful tool actions viasummarize_background_review_actions()(agent/background_review.py:869-909). That reply is not surfaced to the user.
Suggested changes
- Require
skill_view(name)before patching each candidate, then add coverage for a successful view → patch review-fork path. - Either wire a structured trigger-fix event into the existing action summary or remove the promised visible log.
Automated hermes-sweeper review.
| " 2. Compare them against the conversation. Was there a task where " | ||
| "a skill's description COVERS the territory, but the skill was never " | ||
| "loaded via skill_view?\n" | ||
| " 3. If yes, the skill's description (the 'Use when...' trigger " |
There was a problem hiding this comment.
A background-review patch will be rejected here unless this prompt first requires skill_view(name) for the candidate SKILL.md. Current main enforces that read-before-write invariant in tools/skill_manager_tool.py:382-409 and :956-963; skills_list returns only metadata.
| "the user actually used. This is a description edit, not a content " | ||
| "edit.\n" | ||
| " 4. Log the miss: in your reply, note " | ||
| "'TRIGGER FIX: <skill-name> — missed for <task summary>'. This " |
There was a problem hiding this comment.
The review fork's final reply is not user-visible: current main only summarizes successful memory and skill_manage tool actions in agent/background_review.py:869-909. Please either emit this as structured summary data or avoid promising a visible TRIGGER FIX log.
What does this PR do?
Skills load on-demand via description match (the
Use when...trigger line). If a skill description does not match the task phrasing, the skill never fires. This miss is invisible by definition — the self-repair loop only fires after a successful load, so a badly-worded trigger creates a permanent blind spot that never surfaces.This is the core weakness of lazy-loaded skills vs always-injected rules: guaranteed recall is traded for context efficiency, but the cost is silent misses on trigger-wording bugs.
This PR adds a TRIGGER-WORDING CHECK to the background review prompts. After the existing review pass, the reviewer now also:
skills_list)skill_view, patches its trigger description to match the actual task phrasingTRIGGER FIX: <skill-name> — missed for <task summary>This surfaces the previously-invisible miss and fixes it at the source (the description), not the content. No new hooks, tools, env vars, or config keys — the background review fork already runs after every turn with
memory+skillstoolsets. Adding a new review dimension to the prompt is the smallest-footprint change that closes the gap.Related Issue
Fixes #429
Type of Change
Changes Made
agent/background_review.py— +21 lines to_SKILL_REVIEW_PROMPT, +21 lines to_COMBINED_REVIEW_PROMPT(identical block added to both prompts)tests/run_agent/test_review_prompt_class_first.py— +45 lines: 2 new behavior tests + shared assertion helperHow to Test
Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/A