Skip to content

test(tools_config): a platform-toolsets save for one home keeps that home's plugin keys - #120597

Closed
nca7777 wants to merge 1 commit into
NousResearch:mainfrom
nca7777:plugin-toolset-keys-target-home
Closed

nca7777 wants to merge 1 commit into
NousResearch:mainfrom
nca7777:plugin-toolset-keys-target-home

Conversation

@nca7777

@nca7777 nca7777 commented Sep 23, 2026

Copy link
Copy Markdown

What does this PR do?

Locks in the invariant that a platform_toolsets save for ONE home can only carry plugin toolset
keys THAT home can resolve — never the keys of whatever home the calling process happens to have its
plugins loaded from. Tests only; no behaviour change.

hermes_cli/tools_config.py::_save_platform_tools writes plugin toolset names into
platform_toolsets.<platform> and records them in known_plugin_toolsets.<platform>, and
hermes_cli/tools_config.py::_get_platform_tools auto-enables a plugin key that is not yet "known"
for that platform. hermes_cli/plugins_cmd.py::_toggle_plugin_toolset (the plugin enable path)
writes the same pair. In all three the keys come from _get_plugin_toolset_keys().

If those keys are resolved from the caller's process-wide plugin registry instead of the home the
config is being written for, one save stamps a foreign name into another home's list — after which
every run in that home logs Warning: Unknown toolsets: <x>, because a home with no <home>/plugins/<x>
can never resolve the name. That is the shape of a real fleet-wide drift here: one write put two
community plugins' toolset keys into 60 homes' cli lists (byte-identical, together with their
plugins.enabled entries), 50 of which never had those plugin directories, so those homes warned on
every CLI/gateway run for five days.

Per-home plugin managers (keyed by the resolved Hermes home, #4e1b2e43) are what keeps the two apart
today, so this PR is the regression fence rather than a fix: the same code that leaked before that
change must fail these tests, and does.

Related Issue

No issue filed; opening the tests directly so the invariant has an owner in CI. Related review queue
(not duplicates): #109844 (fix(config): stop flagging declared plugin toolsets as unknown) is the
READ side in toolset_validation, and #88003 / #55801 are warning-suppression discussions — none of
them assert anything about which home's plugin keys get WRITTEN.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • tests/hermes_cli/test_platform_toolsets_plugin_key_scope.py (new, 2 tests):
    • test_save_for_another_home_does_not_stamp_the_callers_plugin_toolset — home A is the process
      home and has a real (synthetic) directory plugin registering toolset probe_ts; home B has the
      same config and no plugin dir. Warm the site under A (assert the key is live there — the positive
      control), then run the resolve-then-save path the dashboard toggle and hermes tools use under
      B: neither B's platform_toolsets nor B's known_plugin_toolsets may name the key. Final
      assertion: the same save for A does keep the key, so the negative isn't vacuous.
    • test_enabling_a_plugin_does_not_stamp_it_into_a_home_without_it — the plugin-enable writer
      (plugins_cmd.dashboard_set_agent_plugin_enabled) under home B must refuse (not installed) and
      leave B's plugins.enabled and platform_toolsets untouched.
  • No mocks: two real temp homes, real config.yaml, real directory-plugin discovery, real
    set_hermes_home_override scope switch (the same seam a multiplexed dashboard uses).

How to Test

  1. scripts/run_tests.sh tests/hermes_cli/test_platform_toolsets_plugin_key_scope.py -q → 2 passed.

  2. scripts/run_tests.sh tests/hermes_cli/test_tools_config.py tests/hermes_cli/test_platform_toolsets_plugin_key_scope.py tests/plugins/ -q
    → 144 files, 1895 tests passed, 0 failed, 6 skipped.

  3. Red-on-defect proof: the first test run against the code immediately before the keyed per-home
    plugin manager (4e1b2e43^, i.e. 22af80bcfd) fails with

    assert _TOOLSET_KEY not in saved_in_b, (
        "a platform_toolsets save for home B carried the CALLER home's plugin toolset "
        f"{_TOOLSET_KEY!r}; B has no such plugin and would warn 'Unknown toolsets' on every run")
    E   AssertionError: a platform_toolsets save for home B carried the CALLER home's plugin toolset
    E   'probe_ts'; B has no such plugin and would warn 'Unknown toolsets' on every run
    

    (that tree: git worktree add --detach <dir> 4e1b2e43^, copy the test file in, run it — 1 failed,
    1 passed). On this branch: 2 passed.

Checklist

  • Tests pass locally via scripts/run_tests.sh
  • No behaviour change — test-only
  • Invariant is proven to fail on the code the change fences

…home's plugin keys

_save_platform_tools writes plugin toolset names into platform_toolsets.<platform>
and records them in known_plugin_toolsets.<platform>, and hermes_cli/plugins_cmd.py
writes the same pair when a plugin is enabled. A key has to belong to the home the
config is being written FOR: previously the caller's process-wide plugin registry
decided, so a save made for home B by a process whose own home had plugin X stamped
X's toolset into B, and every later run in B warned "Unknown toolsets: <x>".
Per-home plugin managers are what keep the two apart; these tests lock that in.

Red on the code before the keyed per-home manager (4e1b2e4^), green after.
@nca7777
nca7777 marked this pull request as ready for review September 23, 2026 19:47
@nca7777

nca7777 commented Sep 23, 2026

Copy link
Copy Markdown
Author

Evidence (all run from a worktree of origin/main, scripts/run_tests.sh):

  • branch: nca7777:plugin-toolset-keys-target-home @ 0ce274017f68 — 1 file, +147/-0, compare ahead_by=1 behind_by=0.
  • green on this branch: scripts/run_tests.sh tests/hermes_cli/test_platform_toolsets_plugin_key_scope.py -q → 2 passed; with the neighbouring suites (tests/hermes_cli/test_tools_config.py + tests/plugins/) → 144 files, 1895 passed, 0 failed, 6 skipped.
  • red on the code this fences: same test file in a detached worktree at 4e1b2e43^ (22af80bcfd, the commit before the keyed per-home plugin manager) → 1 failed, 1 passed, with the assertion naming the leaked key (probe_ts written into the non-owning home's config).
  • the failure is the exact fleet symptom this came from: two community plugins' toolset keys present in 60 homes' platform_toolsets.cli while 50 of those homes had no plugin directory, so they logged Warning: Unknown toolsets: ... on every run.

No production file is touched by this branch, so there is nothing to regression-review beyond the test file itself.

@alt-glitch alt-glitch added type/test Test coverage or test infrastructure P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard comp/plugins Plugin system and bundled plugins area/config Config system, migrations, profiles labels Sep 23, 2026
@nca7777 nca7777 closed this by deleting the head repository Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/cli CLI entry point, hermes_cli/, setup wizard comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have type/test Test coverage or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants