Skip to content

feat(skills): add hermes skills lint structure validator - #43555

Open
l3ad3r1 wants to merge 1 commit into
NousResearch:mainfrom
l3ad3r1:feature/skills-lint
Open

l3ad3r1 wants to merge 1 commit into
NousResearch:mainfrom
l3ad3r1:feature/skills-lint

Conversation

@l3ad3r1

@l3ad3r1 l3ad3r1 commented Jun 10, 2026

Copy link
Copy Markdown

What

Adds hermes skills lint, a structural validator for SKILL.md files. It complements the existing hermes skills audit (security scanning) and the hub update checker: audit asks "is this skill dangerous?", lint asks "is this skill well-formed?".

Why

Hermes is deliberately forgiving at runtime — when a skill's frontmatter is malformed, it degrades quietly rather than crashing. That is good for end users but bad for skill authors, who never learn their skill is subtly broken. Today:

  • parse_frontmatter() silently falls back to naive key:value parsing on broken YAML, dropping nested fields.
  • _get_required_environment_variables() silently drops malformed env-var entries.
  • Over-length name/description values are silently truncated.
  • skills publish only checked that a description exists and that the security scan passes.

lint surfaces exactly those hidden problems, with stable rule IDs and CI-friendly exit codes.

What changed

  • tools/skills_lint.py (new) — data-driven rule registry (18 rules, HSL001–HSL060), lint_skill(), format_lint_report(), and installed-skill enumeration helpers. Structural failures (e.g. broken YAML) suppress the field rules so authors do not get a misleading "name missing" cascade on top of the real error.
  • agent/skill_utils.py — adds parse_frontmatter_strict(), which surfaces YAML errors instead of papering over them. The existing lenient parse_frontmatter() is untouched.
  • hermes_cli/ — new subcommand hermes skills lint [targets ...] [--all] [--json] [--fail-on error|warning] with exit codes 0 (clean) / 1 (findings at/above threshold) / 2 (usage error). do_publish() now runs lint before the security scan; lint errors block publishing, warnings do not.
  • Docs — website/docs/user-guide/features/skills.md gains a full rule reference, severity/exit-code semantics, and CI guidance.

Severity model

  • error — the skill will silently misbehave or lose data at runtime/install. Fix these.
  • warning — style, migration, or heuristic issues.

Testing

  • 74 tests pass: per-rule unit tests in tests/tools/test_skills_lint.py (including a test that pins the gap — the same broken YAML HSL002 flags is swallowed without error by the lenient runtime parser), plus CLI tests for exit codes, --json, --fail-on, and publish-blocks-on-lint-error in tests/hermes_cli/.
  • Swept all 78 locally installed skills: 0 errors, 51 warnings — no false-positive errors, and warnings map to genuine issues (legacy prerequisites blocks, name/dir mismatches, missing referenced files).

Example

$ hermes skills lint --all --fail-on warning
Lint: apple-notes (.../skills/apple/apple-notes)  1 warning(s)
  HSL032  WARNING  prerequisites: legacy 'prerequisites' block; migrate env_vars to
                   'required_environment_variables' entries with prompt/help text

🤖 Generated with Claude Code

Adds a structural linter for SKILL.md files, complementing `skills audit`
(security) and the hub update checker. It surfaces problems the runtime
otherwise hides: lenient YAML fallback, silently dropped env-var entries,
and silent name/description truncation.

- tools/skills_lint.py: data-driven rule registry (HSL001-HSL060, 18 rules),
  lint_skill(), format_lint_report(), installed-skill enumeration helpers.
  Structural failures suppress field rules to avoid misleading cascades.
- agent/skill_utils.py: parse_frontmatter_strict() - surfaces YAML errors
  instead of falling back to naive key:value parsing.
- hermes_cli: `skills lint [targets ...] [--all] [--json] [--fail-on]`
  with CI-friendly exit codes (0/1/2); do_publish() now lints before the
  security scan, so lint errors block publishing.
- Docs: full rule reference, severity/exit-code semantics, CI guidance.
- Tests: 74 passing (per-rule unit tests + CLI exit codes, --json,
  --fail-on, publish-blocks-on-error).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/feature New feature or request comp/cli CLI entry point, hermes_cli/, setup wizard tool/skills Skills system (list, view, manage) P3 Low — cosmetic, nice to have labels Jun 10, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the structural-validator contribution. The current runtime still has the lenient YAML fallback at agent/skill_utils.py:145-155, and current main has no equivalent lint command, so the premise is valid.

Problems

  • tools/skills_lint.py:53 omits termux and android, although agent/skill_utils.py:183 explicitly accepts both under Termux. HSL020 would falsely emit a blocking error, including during the proposed publish gate at hermes_cli/skills_hub.py:1391-1396.
  • website/docs/user-guide/features/skills.md:714 describes --all as repository-wide CI linting, but hermes_cli/skills_hub.py:1051-1056 only enumerates installed profile/external skill paths.

Suggested changes

  • Align HSL020 with the runtime platform matcher and add Termux/Android regression tests.
  • Correct the CI guidance, or implement and test an explicit recursive repository/root lint mode.

Automated hermes-sweeper review.

Comment thread tools/skills_lint.py
}
)

VALID_PLATFORMS = frozenset({"linux", "macos", "windows"})

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agent/skill_utils.py:183 explicitly accepts termux and android platform tags in a Termux session. Excluding both here makes HSL020 a false blocking error and can cause the new publish gate to reject a runtime-supported skill; include them or share the runtime's accepted-tag definition.


```bash
# Fail the build on any structural error in any skill under this repo
hermes skills lint --all || exit 1

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--all uses find_installed_skill_paths(), which scans the active profile and configured external skill directories, not this repository. This command does not guarantee that repo skills are linted in CI; revise the example or add a repository-root recursive mode.

@chrisyoung2005

Copy link
Copy Markdown

Heads-up: #115176 wires the linter that merged in #81896 as hermes skills lint, following the targets/--all/--json shape from this PR, and picks up the review point still open here — --all scanning the profile rather than the repository — with --dir PATH for a repository-root recursive pass (CI). It adds no second validator module, which is the main way it differs from this branch. Credited you in the body; if you'd rather carry it here, happy to close mine or graft the pieces over.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/skills Skills system (list, view, manage) type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants