Skip to content

fix(config): surface invalid platform_toolsets instead of silently dropping tools (#38798) - #52920

Merged
kshitijk4poor merged 1 commit into
mainfrom
salvage/38798-toolset-validation
Jun 26, 2026
Merged

fix(config): surface invalid platform_toolsets instead of silently dropping tools (#38798)#52920
kshitijk4poor merged 1 commit into
mainfrom
salvage/38798-toolset-validation

Conversation

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Summary

A corrupted or hand-edited platform_toolsets entry no longer silently disables all tools — invalid toolset names are now surfaced loudly.

Root cause (#38798)

When platform_toolsets contains an invalid toolset name (the reported case: a migration rewrote hermes-cli → the non-existent hermes), resolve_toolset() returns [] for that name. The platform ends up with zero tools, the system prompt is built without tool guidance, and the agent silently degrades to text-only replies — no error, no warning, no log entry. The cause took significant debugging to find.

Fix

Surface the failure at two points (hermes_cli/):

  • After migration (migrate_config) — validate platform_toolsets and record/print a warning per unknown name, with a hermes-<platform> suggestion when that would have been valid (the exact bug: config migration (v25->v26) corrupts platform_toolsets, silently kills all tools #38798 shape: cli: [hermes] → "did you mean 'hermes-cli'?").
  • At runtime (_get_platform_tools) — if a platform was explicitly configured but every toolset name is invalid, log a warning when tools are resolved for a session. This catches an already-corrupted config at startup, not only on the next hermes update.

The logic lives in a new pure, side-effect-free helper (hermes_cli/toolset_validation.py) with validate_toolset injected, so it's unit-testable without the tool registry or launching Hermes.

Scope note (verify-before-build)

The original v25→v26 migration that caused the corruption no longer exists — config format is now v30, and a full-history pickaxe shows no migration step that rewrites toolset names. So the prevent PR (#39581) is obsolete and the repair-migration PRs (#41006/#40208) target a corruption main no longer produces. This change is the durable defense against the silent-failure mode regardless of cause, which is exactly the issue's stated "Expected: log a warning."

Validation

Salvaged from #39207 by @lEWFkRAD (authorship preserved via cherry-pick; AUTHOR_MAP entry added).

Closes #38798. Supersedes #39581 (prevent-in-v25→v26 — that path is gone), #41006 / #40208 (repair-migration for already-corrupted configs).

@github-actions

github-actions Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

🔎 Lint report: salvage/38798-toolset-validation vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11484 on HEAD, 11485 on base (✅ -1)

🆕 New issues (2):

Rule Count
invalid-assignment 1
unresolved-import 1
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`
tests/hermes_cli/test_toolset_validation.py:7: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`

✅ Fixed issues (2):

Rule Count
unresolved-attribute 2
First entries
run_agent.py:3002: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`

Unchanged: 6032 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard comp/tools Tool registry, model_tools, toolsets area/config Config system, migrations, profiles P2 Medium — degraded but workaround exists labels Jun 26, 2026
…opping tools (#38798)

A config migration (or hand-edit) that leaves an invalid toolset name in
`platform_toolsets` — e.g. the #38798 corruption that rewrote `hermes-cli` to
the non-existent `hermes` — silently disabled all affected tools:
resolve_toolset() returns [] for an unknown name, so the agent quietly lost its
tools with no error, warning, or log entry and degraded to text-only replies.

Surface it loudly at two points:
- After migration (migrate_config): validate platform_toolsets and record/print
  a warning per unknown name, with a `hermes-<platform>` suggestion when that
  would have been valid (the exact #38798 shape).
- At runtime (_get_platform_tools): if a platform was explicitly configured but
  every toolset name is invalid, log a warning when tools are resolved for a
  session — so an ALREADY-corrupted config is caught at startup, not only on the
  next `hermes update`.

Logic lives in a new pure, side-effect-free helper (toolset_validation.py) with
validate_toolset injected, so it is unit-testable without the tool registry.

Note: the original v25→v26 migration that caused the corruption no longer
exists (config format is now v30; no migration step rewrites toolset names).
This change is the durable defense against the silent-failure mode regardless
of cause, matching the issue's "Expected: log a warning".

Salvaged from #39207 by @lEWFkRAD (authorship preserved via cherry-pick).
Tests: 9 helper cases (incl. the #38798 corruption shape, mixed valid/invalid,
zero-tools state, non-dict/scalar/non-string) + a runtime caplog test — both the
helper warning and the runtime guard mutation-verified to fail without the fix.

Closes #38798. Supersedes #39581 (prevent-in-v25→v26 — that path is gone),
#41006 / #40208 (repair-migration for already-corrupted configs).
@kshitijk4poor
kshitijk4poor force-pushed the salvage/38798-toolset-validation branch from ae36d36 to 41ede84 Compare June 26, 2026 08:37
@kshitijk4poor
kshitijk4poor enabled auto-merge June 26, 2026 08:42
@kshitijk4poor
kshitijk4poor merged commit 1aa458a into main Jun 26, 2026
27 checks passed
@kshitijk4poor
kshitijk4poor deleted the salvage/38798-toolset-validation branch June 26, 2026 08:44
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…-toolset-validation

fix(config): surface invalid platform_toolsets instead of silently dropping tools (NousResearch#38798)
HexLab98 added a commit to HexLab98/hermes-agent-fork that referenced this pull request Jul 5, 2026
…silently dropping tools

A YAML indentation slip nests toolset names under a mapping, e.g.

    discord:
      - hermes-discord:
          - browser
          - terminal

which parses to `[{'hermes-discord': ['browser', 'terminal', ...]}]`.
`_get_platform_tools` normalised with `[str(ts) for ts in toolset_names]`,
turning the mapping into the literal string `"{'hermes-discord': [...]}"` —
a name that matches no toolset. `has_explicit_config` stayed False for the
real toolsets and every nested toolset was silently dropped, so the platform
loaded almost no tools (the model could reach cronjob/tts but not
terminal/file/web/browser) while config.yaml looked correct, with no warning
anywhere.

Sibling fixes (NousResearch#38798/NousResearch#52920 invalid names -> zero tools, NousResearch#57063 missing
plugin bundles / list-stored-as-string) don't cover this shape: the result
is non-empty-but-wrong, so their zero-tools guards never fire. Add
`_flatten_toolset_names`, which recovers the intended names (mapping keys +
nested values) so the platform still works and logs a loud warning so the
malformed config is visible and fixable. Well-formed flat lists pass through
unchanged with no warning.

This resolves through the shared resolver, so it fixes CLI, messaging
gateway, and TUI alike.
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
…-toolset-validation

fix(config): surface invalid platform_toolsets instead of silently dropping tools (NousResearch#38798)
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…-toolset-validation

fix(config): surface invalid platform_toolsets instead of silently dropping tools (NousResearch#38798)
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…-toolset-validation

fix(config): surface invalid platform_toolsets instead of silently dropping tools (NousResearch#38798)
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…-toolset-validation

fix(config): surface invalid platform_toolsets instead of silently dropping tools (NousResearch#38798)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/cli CLI entry point, hermes_cli/, setup wizard comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: config migration (v25->v26) corrupts platform_toolsets, silently kills all tools

3 participants