fix(skill-manager): consult known-false-positives.json before blocking (#57954) - #58011
webtecnica wants to merge 2 commits into
Conversation
9262d80 to
6fec81d
Compare
|
Rebaseei no
Diff continua limpo — só |
6fec81d to
ac5e273
Compare
|
Rebased on latest |
|
@teknium1 Rebased on latest main. Skill whitelist fix — ready for re-review whenever you have a moment. |
ac5e273 to
8fe1d87
Compare
8fe1d87 to
18e8404
Compare
|
/update |
|
@teknium1 This PR is ready for review:
Would appreciate a look when you have time. 🙏 |
NousResearch#57954) When skill_manage(action='patch'/'edit') triggers a false-positive security alert, the whitelist in known-false-positives.json was not consulted, forcing agents to rewrite content to avoid blocked patterns. This change: - Adds _load_known_false_positives() to read the whitelist from /root/.hermes/known-false-positives.json - Adds _is_finding_whitelisted() to match findings against both by_skill (skill name + pattern_id) and by_pattern (pattern_id or match text substring) entries - Adds _filter_whitelisted_findings() to remove whitelisted findings from a ScanResult and re-evaluate the verdict - Modifies _security_scan_skill() to apply the whitelist before deciding whether to block Closes NousResearch#57954
3e48ffa to
789176c
Compare
teknium1
left a comment
There was a problem hiding this comment.
Thanks for the focused fix. The current-main premise is real: tools/skill_manager_tool.py:121-145 sends agent-created scan results directly to should_allow_install() without a whitelist path.
Problems
tools/skill_manager_tool.py:160assumesby_skillis iterable. Valid JSON such as{"by_skill": null}raises there; the broad exception handler at proposed lines 249-250 then returnsNone, allowing a write despite an enabled scan. Normalize malformed sections to an empty whitelist before filtering.- The PR changes only
tools/skill_manager_tool.py(+106/-0), whiletests/tools/test_skill_manager_tool.py:633-710has no coverage for whitelist filtering or malformed whitelist input.
Suggested changes
- Validate the JSON schema and preserve the original scan result for invalid entries.
- Add temp-
HERMES_HOMEbehavior tests for matching, partial verdict recomputation, all-filtered allow, and malformed input retaining the block.
Automated hermes-sweeper review.
| the finding's match/description text (match_contains). | ||
| """ | ||
| # by_skill: exact skill name match, pattern_id in skip list | ||
| for entry in whitelist.get("by_skill", []): |
There was a problem hiding this comment.
Please validate/normalize by_skill before iterating. Valid JSON such as {"by_skill": null} raises TypeError here; the broad exception handler in _security_scan_skill() then returns None, so an enabled scan silently allows the write. Treat malformed sections as an empty whitelist and add a regression test.
Summary
Closes #57954. When
skill_manage(action="patch"/"edit")triggers a false-positive security alert, the whitelist inknown-false-positives.jsonwas not consulted, forcing agents to rewrite content to avoid blocked patterns.Changes
_load_known_false_positives(): Reads/root/.hermes/known-false-positives.json(returns{}when missing — no special case needed)_is_finding_whitelisted(): Matches findings against:by_skillentries: exact skill name +pattern_idinfindings_to_skipby_patternentries: directpattern_idmatch ormatch_containssubstring (case-insensitive)_filter_whitelisted_findings(): Removes whitelisted findings from aScanResult, re-evaluates the verdict via_determine_verdict(), and returnsNonewhen all findings are whitelisted (allows silently)_security_scan_skill(): Applies the whitelist filter before callingshould_allow_install()Example whitelist file
{ "by_skill": [ {"skill": "my-skill", "findings_to_skip": ["persistence_cron", "sudo_usage"]} ], "by_pattern": [ {"pattern_id": "remote_fetch"}, {"match_contains": "base64"} ] }Whitelist format notes
by_skill: Theskillfield is compared against the skill directory name (not the full path), andfindings_to_skiplists findingpattern_idvalues to suppress for that skill. Only the exactpattern_idis matched (the substring issue in by_skill whitelist matching logic in skill-security-pipeline.sh has inverted substring check #57956 is a separate fix).by_pattern: Each entry can have either:pattern_id: suppresses all findings with that exactpattern_idmatch_contains: suppresses findings whosematchordescriptionfield contains the given substring (case-insensitive)Testing
All test scenarios pass:
by_skillentries match on skill name + pattern_idby_patternentries match on pattern_id or match text substringNone(silently allowed)