Skip to content

feat(capabilities): add capabilities_list tool for orchestrator profile discovery - #25247

Closed
cypres0099 wants to merge 4 commits into
NousResearch:mainfrom
cypres0099:feat/profile-capabilities-discovery
Closed

feat(capabilities): add capabilities_list tool for orchestrator profile discovery#25247
cypres0099 wants to merge 4 commits into
NousResearch:mainfrom
cypres0099:feat/profile-capabilities-discovery

Conversation

@cypres0099

@cypres0099 cypres0099 commented May 13, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a capabilities_list model tool that enumerates installed profiles' enabled skills, so an orchestrator profile can route work via kanban_create(assignee=..., skills=[...]) without maintaining a hand-edited specialist roster in its SOUL or skill.

The tool walks profiles.list_profiles(), reuses _find_skills_in_profile from web_server.py (added in #25116), and applies the skills.disabled filter via skills_config.get_disabled_skills. Registration is gated to orchestrator profiles via a check_fn modeled on _check_kanban_orchestrator_mode (tools/kanban_tools.py:76) — workers spawned via kanban_create see the tool dropped with a warning even if their platform_toolsets config includes it. Updates the bundled kanban-orchestrator skill to call capabilities_list at the start of every routing decision instead of consulting a static roster, and updates the KANBAN_CREATE_SCHEMA skills parameter docstring to point at the new discovery primitive.

Stacked on #25116 — depends on _find_skills_in_profile from that PR. The diff currently shows both commits because #25116 has not merged yet; once it merges the diff here will collapse to just the capabilities tool and its tests. Reviewers can also look at this branch's single feature commit (feat(capabilities): add capabilities_list tool...) in isolation.

Type of Change

  • ✨ New feature

Changes Made

  • tools/capabilities_tool.py (new): handler + schema + check_fn gate + symlink hardening + description sanitizer + INFO audit log
  • toolsets.py: add capabilities_list to _HERMES_CORE_TOOLS; add capabilities entry to TOOLSETS
  • tools/kanban_tools.py: docstring update on KANBAN_CREATE_SCHEMA.skills pointing at the new discovery primitive
  • skills/devops/kanban-orchestrator/SKILL.md: Step 0 reframed to call capabilities_list at the start of every routing decision (no caching across turns) with a worked example
  • tests/tools/test_capabilities_tool.py (new): 17 tests covering gating, happy path, filters, symlink isolation (dir-level, skills_dir-as-symlink, file-level), description sanitization (C0/DEL + Unicode Cf), defensive fallbacks for malformed config and skills: null, audit log
  • tests/tools/test_registry.py: add tools.capabilities_tool to the builtin-discovery expected set
  • scripts/release.py: add cypres0099@users.noreply.github.com to AUTHOR_MAP (resolves check-attribution failure)

Security model

Three concrete in-deployment threats are mitigated in this PR:

# Mitigation Threat
1 check_fn registration gate Prompt-injected worker profile with capabilities in its platform_toolsets config enumerates sibling skill rosters
2 Symlink hardening in _skill_path_is_symlink_free (rejects skills_dir-as-symlink, leaf-SKILL.md-as-symlink, AND resolved-real-path-escapes-skills_dir) Worker plants a symlink anywhere inside profiles/<worker>/skills/ (or replaces skills/ itself) pointing at a sibling's skills/ tree. _find_skills_in_profile invokes os.walk(..., followlinks=True) so each layer of the filter is required.
3 Description sanitization: 500-char cap + strip Unicode Cc (C0/C1/DEL) and Cf (bidi overrides, zero-width chars, BOM) categories via unicodedata.category Worker writes a SKILL.md description containing prompt-injection content (including bidi-override or zero-width payloads) that lands in the orchestrator's LLM prompt context

The handler also logs WARNING on profile config-load failure (rather than silently treating all skills as enabled, which would be a soft-fail security regression on operator intent) and is defensive against skills: null configs that would otherwise crash get_disabled_skills for the rest of the host.

Accepted tradeoffs

  • Session-token holder can enumerate all profiles. Same trust boundary as /api/profiles/{name}/skills from feat(dashboard): per-profile skills toggle UI #25116. No new network exposure (no HTTP surface in this PR).
  • Skill descriptions are confidential by frontmatter convention only. Operators with sensitive skill names should keep them out of the description field.
  • No stable skill ID. Renames in worker profiles silently break orchestrator routing because kanban_create(skills=[...]) matches by exact name string. Tracked as a follow-up.

Deferred mitigations (known limitations)

  • Per-skill or profile-level discoverable: false opt-out — no current multi-tenant use case.
  • HTTP-specific guards (non-loopback refusal, names-only default) — moot in v1 (no HTTP surface). Required when the HTTP endpoint ships in PR-2.

How to Test

  1. Create 2-3 test profiles each with a SKILL.md in their skills/ dir.
  2. From an orchestrator profile (no HERMES_KANBAN_TASK in env, toolsets: [..., capabilities] in config.yaml):
    hermes chat -q "List sibling capabilities"
    
    Confirm the tool fires and returns the expected JSON shape.
  3. From a worker spawn (with HERMES_KANBAN_TASK set): confirm the tool is not exposed in the model's schema.
  4. Run scripts/run_tests.sh tests/tools/test_capabilities_tool.py.

Platform tested: macOS 15 (Apple Silicon, Python 3.11).

Scope discipline — explicitly deferred to follow-ups

Per CONTRIBUTING.md: "Keep PRs focused: one logical change per PR. Don't mix a bug fix with a refactor with a new feature." This PR ships the model tool only. Three things explicitly deferred:

  • CLI command hermes capabilities — operator-debugging convenience, no current consumer. Defer to PR-2.
  • HTTP endpoint GET /api/profiles/capabilities — no current dashboard consumer. Defer to PR-2 with non-loopback refusal and names-only default.
  • Refactor lifting _find_skills_in_profile into a shared module — currently the new tool imports it directly from web_server.py (a known cross-module smell, called out inline). The lift is hygiene, not a feature dependency. Defer to PR-3.

CI Status

First CI run (commit 8bdbdb58c) — 8 of 11 checks passed. Three failures:

Check Status Resolution
check-attribution ❌ failed Fixed in follow-up commit 402fa80ef — added cypres0099@users.noreply.github.com to scripts/release.py AUTHOR_MAP. Awaiting maintainer to approve workflow re-run on the new commit (fork-PR contributor gate).
Scan PR for critical supply chain risks ❌ failed False positive: the scanner's `(^
test ❌ failed Pre-existing upstream failures — the same failing tests (test_bedrock_adapter, test_dingtalk, test_wecom_callback, test_weixin, test_feishu_bot_admission) fail on the latest main CI run from 1 hour before this PR was opened. Root causes: botocore module not installed in CI image, cffi library _openssl symbol mismatch on the runner, and a handful of AsyncMock interaction bugs in the dingtalk adapter tests. None touch any code modified by this PR. The 17 tests added in this PR all pass under the project's hermetic scripts/run_tests.sh runner.

The attribution fix is the only iteration-applicable change; the other two are not fixable from within this PR.

Known limitations

  • Session-token holder can enumerate all profiles (inherited trust boundary from feat(dashboard): per-profile skills toggle UI #25116).
  • No stable skill ID — sibling profile skill renames silently break orchestrator routing. Tracked as follow-up.
  • No profile-level opt-out. Deferred until a multi-tenant use case surfaces.

Residual Review Findings

The implementation was audited by a 9-reviewer parallel code review pass (security, adversarial, correctness, reliability, maintainability, testing, project-standards, kieran-python, agent-native). Seven safe-auto findings landed in the feature commit; six P2/P3 items are deferred:

  • P1 gated_auto — Schema descriptions hardcode cross-tool references (capabilities_listkanban_create). AGENTS.md guidance is to use get_tool_definitions() post-processing for cross-tool refs so a profile that enables only one of the two toolsets doesn't see the model hallucinate calls to the missing one. Considered acceptable for v1 since the natural pairing is documented (both expected to be enabled together on orchestrator profiles); will lift if a real misconfiguration surfaces.
  • P2 manual — No total-response cap. A host with hundreds of skills could exceed the global DEFAULT_RESULT_SIZE_CHARS budget and silently truncate. Suggested cap: 200KB / 500 entries with a truncated: true sentinel. Deferred to a follow-up once a real workload exists to size against.
  • P2 gated_autogateway_running not surfaced in the output dict. An orchestrator can route to a profile whose gateway is down; the kanban card sits in ready forever (same silent-failure surface the bundled skill already warns about for unknown assignees). The field is already populated by list_profiles() at no extra I/O cost; adding it is a small follow-up.
  • P3 gated_autoname and category fields not sanitized (only description is). Worker-writable directory names could theoretically carry control chars; lower threat surface than description text.
  • P2 manual (pre-existing in feat(dashboard): per-profile skills toggle UI #25116)os.walk(..., followlinks=True) in iter_skill_index_files has no cycle detection. A circular symlink inside any profile's skills/ tree would hang the gateway thread indefinitely. Out of scope for this PR; should be fixed alongside the helper module lift in PR-3.
  • P3 advisory (pre-existing in feat(dashboard): per-profile skills toggle UI #25116)_find_skills_in_profile raises TypeError on description: null frontmatter (len(None) path). Per-profile isolation in this PR's handler converts the crash into a WARNING + skip-profile, but the root cause is in the helper. Out of scope here.

CONTRIBUTING checklist

  • Conventional Commits format
  • One logical change per PR (model tool only; CLI/HTTP/refactor deferred)
  • Tests added — tests/tools/test_capabilities_tool.py (17 tests, all passing under scripts/run_tests.sh)
  • No Windows footguns (scripts/check-windows-footguns.py clean)
  • No new credentials, no new env-var requirements
  • AUTHOR_MAP entry added for the contributor email

cypres0099 and others added 2 commits May 13, 2026 11:53
Adds a profile selector to the dashboard's Skills page so each installed
profile's skills.disabled list can be managed from the same dashboard
daemon. Until now, /api/skills only knew the active profile (whichever
HERMES_HOME the dashboard process was launched under), so toggling skills
for a non-active profile required spinning up a second dashboard daemon
bound to that profile's HERMES_HOME — operationally awkward for users
running multiple profiles (e.g. a default + a worker/specialist profile).

Backend
-------
- Add GET  /api/profiles/{name}/skills        — list a profile's skills
- Add PUT  /api/profiles/{name}/skills/toggle — toggle for one profile
- Add is_active to ProfileInfo so the UI can identify the daemon's
  resident profile (the one served by the legacy /api/skills routes).
- Reads/writes go directly against the profile's config.yaml
  (skills.disabled). The load_config/save_config helpers are bound to
  the process-level HERMES_HOME via get_config_path(), so they can't
  be reused for cross-profile mutation without invasive global state
  changes.
- v1 omits skills.external_dirs scanning for non-active profiles. The
  dropdown targets profile-installed skills; external dirs are still
  respected by the gateway at runtime.

Frontend
--------
- SkillsPage gains a Select dropdown next to the enabled-of count
  (hidden when there's only one installed profile, so default-only
  installs are unchanged).
- Default selection is the dashboard's own profile (is_active, or
  is_default for older gateways that don't emit the field).
- Switching profile refetches the skills list from the profile-scoped
  endpoint; the active-profile selection still uses the legacy
  /api/skills route to stay in sync with the gateway's skill index.
- Toggles route through the appropriate endpoint based on selection.

The legacy /api/skills and /api/skills/toggle routes are untouched and
remain the canonical path for the active profile.
…le discovery

Adds a `capabilities_list` model tool that enumerates every Hermes
profile's enabled skills, so an orchestrator profile can route work via
`kanban_create(assignee=..., skills=[...])` without maintaining a
static specialist roster in its SOUL or skill.

The tool walks `profiles.list_profiles()`, reuses
`_find_skills_in_profile` from `web_server.py` (added in NousResearch#25116), and
applies the `skills.disabled` filter via
`skills_config.get_disabled_skills`. Registration is gated to
orchestrator profiles via a `check_fn` modeled on
`_check_kanban_orchestrator_mode` (`tools/kanban_tools.py:76`) —
workers spawned via `kanban_create` see the tool dropped with a
warning even if their `platform_toolsets` config includes it.

Updates the bundled `kanban-orchestrator` skill to call
`capabilities_list` at the start of every routing decision instead of
consulting a static roster, and updates the `KANBAN_CREATE_SCHEMA`
docstring to point at the new discovery primitive as a follow-up signal
for model callers.

Security mitigations land with the tool, each tied to a concrete
deployment threat:

1. check_fn registration gate — workers cannot enumerate sibling
   skill rosters from inside the dispatched task.
2. Symlink hardening — `_skill_path_is_symlink_free` rejects when
   `skills_dir` itself is a symlink, when the leaf SKILL.md file is a
   symlink, and when the resolved real path escapes the profile's
   `skills/` tree. `_find_skills_in_profile` invokes
   `os.walk(..., followlinks=True)` so the filter is required at this
   layer.
3. Description sanitization — strips C0/C1/DEL and Unicode `Cf`
   format chars (bidi overrides, zero-width chars, BOM) and caps to
   500 chars. Frontmatter is worker-writable and lands in the
   orchestrator's LLM prompt context.

The handler also logs WARNING on config-load failure (instead of
silently treating all skills as enabled) and is defensive against
`skills: null` profile configs that would otherwise crash
`get_disabled_skills` for the rest of the host.

Per CONTRIBUTING.md scope discipline, three follow-ups are explicitly
deferred:
- `hermes capabilities` CLI command (no current consumer)
- `GET /api/profiles/capabilities` HTTP endpoint (no current consumer)
- Lifting `_find_skills_in_profile` into a shared module (hygiene,
  not a feature dependency)

Tests (17 in tests/tools/test_capabilities_tool.py):
- 3 gating tests (worker hidden, orchestrator visible, toolset off)
- happy path 3 profiles × 2 skills with shape assertion
- profile filter, unknown profile
- global disabled + platform overlay
- 3 symlink isolation tests (dir, skills_dir itself, file-level)
- 2 description sanitization tests (C0/DEL + Unicode Cf)
- 2 defensive fallback tests (malformed config, skills: null)
- INFO audit log on every call

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/tools Tool registry, model_tools, toolsets comp/cli CLI entry point, hermes_cli/, setup wizard comp/tui Terminal UI (ui-tui/ + tui_gateway/) tool/skills Skills system (list, view, manage) labels May 13, 2026
cypres0099 and others added 2 commits May 13, 2026 17:06
Resolves contributor-attribution check failure on PR NousResearch#25247. The
``cypres0099@users.noreply.github.com`` git email was not yet mapped
to a GitHub username in ``scripts/release.py``, which the
``check-attribution`` workflow flags as a blocker.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The registry dispatches handlers as ``handler(args, **kwargs)`` (see
``tools/registry.py:387``) and the gateway injects context kwargs like
``task_id`` on every invocation. Without a ``**kw`` catch-all the
handler raised ``TypeError: unexpected keyword argument 'task_id'`` on
every model call. Mirrors the signature kanban handlers use.

Adds a regression test calling the handler with the dispatcher's exact
``(args, **kwargs)`` shape so the contract doesn't drift again.

Smoke-tested live on Artemis after the fix: the model invoked
``capabilities_list`` successfully and returned the full per-profile
skill roster as a grouped table (previously fell back to manual
filesystem walking through ``search_files``).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@cypres0099

Copy link
Copy Markdown
Contributor Author

Closing this stale broad branch rather than trying to revive it in-place. The capability-discovery idea may still be useful, but this PR spans CLI/tools/TUI/skills across 11 files and has drifted too far from current main. If we pursue it again, it should start as a fresh issue/design note or a much smaller PR from current main with the minimal surface area maintainers want.

@cypres0099 cypres0099 closed this Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/tools Tool registry, model_tools, toolsets comp/tui Terminal UI (ui-tui/ + tui_gateway/) P3 Low — cosmetic, nice to have tool/skills Skills system (list, view, manage) type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants