Skip to content

fix(cli): don't flag plugin toolsets as unknown at startup - #116425

Closed
byjaps wants to merge 1 commit into
NousResearch:mainfrom
byjaps:fix/plugin-toolset-startup-warning
Closed

byjaps wants to merge 1 commit into
NousResearch:mainfrom
byjaps:fix/plugin-toolset-startup-warning

Conversation

@byjaps

@byjaps byjaps commented Sep 19, 2026

Copy link
Copy Markdown

What

Every launch reports configured plugin toolsets as unknown, before plugin discovery has had a chance to register them:

$ hermes chat -q "hi" -Q
Warning: Unknown toolsets: voice_stack

HermesCLI._init_toolsets() validates the configured list while the instance is being constructed. Plugins register their toolsets later, during background discovery (discover_plugins()), so judging by the live registry alone (validate_toolset) flags every configured plugin toolset as a typo — even though the very same key set was just used to build enabled_toolsets (via get_plugin_toolset_keys_nowait) two lines earlier in the startup path.

Why it matters beyond the noise

The warning is written to the console, so it lands on stdout for machine-parsed runs. Reproduced case: an integration drives hermes chat -q -Q as a subprocess and reads stdout as the answer, so the warning line was consumed as the reply — in that deployment a TTS pipeline read it out loud instead of the user's answer. --format stream-json is affected the same way (its stdout is meant to be a single JSON object / JSONL stream).

Fix

Skip names the plugin registry reports, mirroring the MCP-server-name exemption already in this block:

try:
    from hermes_cli.plugins import get_plugin_toolset_keys_nowait

    plugin_ts_names = get_plugin_toolset_keys_nowait()
except Exception:
    plugin_ts_names = set()

invalid = [
    t
    for t in toolsets
    if not validate_toolset(t) and t not in mcp_names and t not in plugin_ts_names
]

get_plugin_toolset_keys_nowait() serves the key set persisted by the previous launch's discovery sweep, so the check stays non-blocking on startup (no join on the discovery thread). A genuinely unknown name still warns.

Tests

New TestPluginToolsetStartupValidation in tests/hermes_cli/test_cli_init.py:

  • a toolset that a plugin owns but that is not in the live registry yet → no warning, and enabled_toolsets is kept verbatim (the fix silences the warning, it does not drop names);
  • a real typo still warns, and the message names the typo only.

Proven RED on base (same tree, only the cli.py hunk reverted):

[100.0% | 45/~45 | ✓1 | ✗1] ✗ tests/hermes_cli/test_cli_init.py (1✓ 1✗, 1.8s)
  ║ >       assert printed == []
  ║ E       AssertionError: assert ['[bold red]W...ice_stack[/]'] == []
  ║ E         Left contains one more item: '[bold red]Warning: Unknown toolsets: voice_stack[/]'

GREEN with the fix:

$ scripts/run_tests.sh tests/hermes_cli/test_cli_init.py tests/tools/test_toolsets.py
=== Summary: 2 files, 70 tests passed, 0 failed, 1 skipped (100% complete) in 15.2s ===

Relation to other work

  • Same approach as fix(cli): don't warn on plugin toolsets at startup; accept them in subagent lifecycle #89345 (credit to @adamkrawczyk, added as co-author). That PR no longer applies to main: its agent/subagent_lifecycle.py hunk conflicts (that validation was rewritten upstream), and its startup half is what this PR re-lands against current main.
  • test(config): cover enabled plugin toolset discovery during migration #86233 covers the separate config-migration entry point, not this one.
  • Still open on main, not addressed here: agent/subagent_lifecycle.py rejects plugin toolsets in allowed_toolsets through the same static-membership test (set(request.allowed_toolsets) - set(TOOLSETS)). Worth a follow-up on the same class of bug rather than widening this PR.
  • Also not addressed here: for a real typo the warning still lands on stdout, which breaks the -Q / --format stream-json contract documented in _configure_quiet_agent() ("stdout carries only the final response"). Separate follow-up.

Refs #71650 · Refs #95529 (duplicate report — the one this was reproduced from)

Plugins register their toolsets during background discovery, but the CLI
validates the configured toolset list while the instance is being
constructed -- i.e. before that thread has landed. Judging by the live
registry alone therefore reports every configured plugin toolset as a
typo on every launch.

The warning is not merely cosmetic: it is written to the console, so
one-shot and quiet runs (-q / -Q, --format stream-json) hand it to
whatever parses their output; an integration reading the response can
receive the warning line instead of the answer.

Skip names the plugin registry knows about, mirroring how MCP server
names are already skipped here. Names persisted by the previous launch's
discovery sweep are served by get_plugin_toolset_keys_nowait, so this
stays non-blocking on startup. A genuinely unknown name still warns.

Refs NousResearch#71650 (NousResearch#95529 is the duplicate report of the same bug)

Co-authored-by: adamkrawczyk <adam-krawczyk@outlook.com>
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks @byjaps and @adamkrawczyk — cherry-picked (re-sited onto hermes_cli/cli_init_mixin.py); resolves #71650 / #91757 / #95529.

Salvaged into #118841 with your authorship preserved (merge 74f726c). Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants