Skip to content

fix: defer toolset validation to after plugin discovery (#71650) - #71742

Closed
JonthanaHanh wants to merge 1 commit into
NousResearch:mainfrom
JonthanaHanh:fix/defer-toolset-validation-post-plugin-discovery
Closed

JonthanaHanh wants to merge 1 commit into
NousResearch:mainfrom
JonthanaHanh:fix/defer-toolset-validation-post-plugin-discovery

Conversation

@JonthanaHanh

Copy link
Copy Markdown

Summary

validate_platform_toolsets() runs inside migrate_config() during early startup, before discover_plugins() populates the tool registry. Plugin-registered toolsets (e.g. beads from a third-party plugin) always trigger a false-positive "unknown toolset" warning on every startup.

Fixes #71650

Root Cause

The startup sequence is:

  1. migrate_config() → calls validate_platform_toolsets() → checks validate_toolset() → _get_plugin_toolset_names() returns empty set (plugins not loaded yet)
  2. discover_plugins() → plugins register their toolsets in the registry

Any plugin-registered toolset listed in platform_toolsets config triggers the warning at step 1 because the registry is empty at that point.

Fix

  • hermes_cli/config.py: When migrate_config() detects that the tool registry has no plugin toolsets yet, store warnings in a module-level deferred buffer (_deferred_toolset_warnings) instead of printing them. When plugins are already loaded (e.g. standalone hermes config migrate), emit inline as before.
  • hermes_cli/main.py: In _prepare_agent_startup(), after discover_plugins() completes, re-validate platform_toolsets with the now-populated plugin registry and emit only truly invalid toolset names.

Behavior

Scenario Before After
Plugin toolset in config, startup ⚠ false-positive warning No warning (deferred, then re-validated clean)
Invalid toolset in config, startup ⚠ warning (at migration time) ⚠ warning (after plugin discovery)
hermes config migrate (plugins loaded) ⚠ warning if invalid Same (inline path, plugins already loaded)

Test Plan

  • 13 existing test_toolset_validation.py tests pass
  • Syntax check passes on both modified files
  • Manual test: add a plugin toolset to platform_toolsets config, verify no false-positive warning on startup

…#71650)

validate_platform_toolsets() runs inside migrate_config() during early
startup, before discover_plugins() populates the tool registry. Plugin-
registered toolsets (e.g. "beads" from a third-party plugin) always
trigger a false-positive "unknown toolset" warning on every startup.

Fix: when the registry has no plugin toolsets yet (plugins not loaded),
store warnings in a deferred buffer instead of printing them. After
discover_plugins() in _prepare_agent_startup(), re-validate with the
now-populated registry and emit only truly invalid toolset names.

The "hermes config migrate" standalone path is unaffected: if plugins
are already loaded (registry populated), warnings emit inline as before.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard comp/plugins Plugin system and bundled plugins area/config Config system, migrations, profiles sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades needs-decision Awaiting maintainer decision before any implementation labels Jul 26, 2026
@alt-glitch

Copy link
Copy Markdown

This was generated by AI during triage.

Related: #71729 and #29774 address the same false warning by discovering plugins before validation, while this patch defers warnings and revalidates after discovery. These have different startup timing and failure trade-offs; maintainer decision needed rather than a duplicate closure.

@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for isolating the plugin-registry timing concern. This automated hermes-sweeper review found that current main already provides the requested startup ordering.

  • hermes_cli/main.py:12360 runs _prepare_agent_startup() before chat dispatch; hermes_cli/main.py:10633-10644 then discovers plugins for the default/chat agent paths.
  • cli.py:4446-4453 performs the explicit toolset validation only when HermesCLI is constructed later in that path.
  • toolsets.py:793-805 and toolsets.py:867-884 recognize plugin-registered toolsets from the live registry.
  • The related-discussion note about discover-before-validation matches the behavior already on main. The proposed deferred buffer would also defer real warnings for non-agent commands such as hermes config migrate, whose path does not enter _prepare_agent_startup().

Closing as already implemented on current main.

@teknium1 teknium1 closed this Jul 30, 2026
@teknium1 teknium1 added sweeper:implemented-on-main Sweeper: behavior already present on current main sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/cli CLI entry point, hermes_cli/, setup wizard comp/plugins Plugin system and bundled plugins needs-decision Awaiting maintainer decision before any implementation P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:implemented-on-main Sweeper: behavior already present on current main sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Toolset validation runs before plugin load -- plugin-registered toolsets always trigger false-positive warning

3 participants