Skip to content

fix(toolsets): validate configured plugin toolsets before discovery finishes - #111322

Closed
budokai-msi wants to merge 1 commit into
NousResearch:mainfrom
budokai-msi:fix/plugin-toolset-validation-race
Closed

budokai-msi wants to merge 1 commit into
NousResearch:mainfrom
budokai-msi:fix/plugin-toolset-validation-race

Conversation

@budokai-msi

Copy link
Copy Markdown

Symptom

An explicitly enabled plugin toolset is reported as unknown on every CLI launch:

$ hermes chat -q "hi"
Warning: Unknown toolsets: dsh

dsh is a real, valid, enabled plugin toolset (hermes tools list shows it as
enabled, and its tool registers correctly). The name is right; only the timing
is wrong.

Root cause

validate_toolset() resolved plugin toolset names against the live tool
registry:

def _get_plugin_toolset_names() -> Set[str]:
    return {n for n in _registry_call("get_registered_toolset_names", ()) if n not in TOOLSETS}

Plugin discovery runs on a background thread to overlap the rest of CLI
startup (plugins.start_background_plugin_discovery()), and plugins register
their toolsets during that load. The CLI validates configured toolset names on
its own fast path (cli.HermesCLI._init_toolsets, via
hermes_cli/main.py:cmd_chat), so when it gets there first the registry is
still empty and every configured plugin toolset looks unknown:

# cli.py
invalid = [t for t in toolsets if not validate_toolset(t) and t not in mcp_names]
if invalid:
    self._console_print(f"[bold red]Warning: Unknown toolsets: {', '.join(invalid)}[/]")

The warning is unactionable: there is nothing wrong with the config, and the
plugin's tools do load once discovery lands.

Fix

Resolve declared plugin keys without joining discovery.
hermes_cli.plugins.get_plugin_toolset_keys_nowait() already exists for exactly
this case — it serves last launch's persisted key set while a background scan is
in flight, and blocks via discover_plugins() only when there is no scan to
wait on. Its persisted-cache staleness contract is explicitly safe for callers
that only validate names, which is all this does.

get_toolset() deliberately keeps reading the live registry alone, so the fix
cannot invent tooling: a declared-but-not-yet-loaded plugin toolset still
resolves to zero tools. A regression test pins that.

Test plan

scripts/run_tests.sh tests/tools/test_toolsets.py → 28 passed, 0 failed

New coverage:

  • a configured plugin toolset validates while discovery is in flight (previously
    False → warning);
  • the same declared key still resolves to no tools, so name validation is not
    mistaken for capability.

Verified on the affected host

hermes chat -q "..." no longer prints the warning, with the plugin enabled and
its tool working.

…inishes

`validate_toolset()` resolved plugin toolset names against the live tool
registry, which is empty until plugin discovery completes. Plugin discovery
is started on a background thread to overlap the rest of CLI startup, so any
fast path that validates configured toolset names first — `hermes chat` does,
via `_init_toolsets` — compared them against an empty set and printed

    Warning: Unknown toolsets: dsh

for a perfectly valid, explicitly enabled plugin toolset. The name is
correct; only the timing was wrong, so the warning was unactionable noise on
every launch (and the plugin's tools still loaded once discovery landed).

Resolve declared plugin keys without joining discovery:
`hermes_cli.plugins.get_plugin_toolset_keys_nowait()` already exists for
exactly this — it serves last launch's persisted key set while a background
scan is in flight and blocks via `discover_plugins()` only when there is no
scan to wait on, with a persisted-cache staleness contract that is harmless
for callers that only validate names.

`get_toolset()` deliberately keeps reading the live registry alone: name
validation must not invent tooling, so a declared-but-not-yet-loaded plugin
toolset still resolves to zero tools. A regression test pins that, alongside
the validation itself.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard comp/plugins Plugin system and bundled plugins comp/tools Tool registry, model_tools, toolsets labels Sep 15, 2026
@alt-glitch

Copy link
Copy Markdown

This was generated by AI during triage.

Related: #84499 and #89345 are open PRs for the same startup race (plugin toolsets flagged unknown before background discovery finishes; issue #86231). This PR resolves declared keys via get_plugin_toolset_keys_nowait() instead of waiting on discovery; the others await/serialize discovery. Flagging so reviewers compare the three rather than merge two.

@ailinter-dev

Copy link
Copy Markdown

Independent reproduction on a different install shape; the timing analysis here matches what I see.

git install on Debian 13, hermes-agent 0.21.x. A plugin registers its own toolset (omh: 18 tools, 8 hooks) and the profile lists it in platform_toolsets.cli. Every launch prints Warning: Unknown toolsets: omh, both in profile-scoped CLI sessions (hermes -p <profile> chat) and in the gateway daemon startup log. The tools themselves load and work, so the name is right and only the read is early, same as reported.

One data point on the shape of the fix: a local workaround that exempts names produced by the config resolver (_get_platform_tools) also silences the warning, but it suppresses it for every config-derived list rather than fixing the read. Resolving the declared keys (what this PR does) is the better shape, and the regression test pinning that a declared-but-unloaded toolset still resolves to zero tools is the part worth keeping.

@teknium1

Copy link
Copy Markdown
Collaborator

The startup Unknown-toolsets warning now exempts plugin toolset keys via get_plugin_toolset_keys_nowait() (#116425 salvaged, credit #89345); #71650 / #91757 / #95529 are closed.

Superseded by #118841 (merge 74f726c), which credits this PR. Closing.

@teknium1 teknium1 closed this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/plugins Plugin system and bundled plugins comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants