fix(plugins): bound background discovery handoff

[KoNit-K]

What does this PR do?

• bounds the handoff from synchronous callers to an in-flight background plugin discovery - returns with safely available partial plugin state when the 30-second join bound expires instead of re-entering the held discovery lock - preserves healthy background completion and idempotent discovery behavior Fixes [Bug]: discover_plugins() has no bounded timeout on MCP-backed plugins, blocks ACP session/new (and 30+ other call sites) indefinitely #108139 - RED: HERMES_PYTHON=/Users/blockkonit./Dev/hermes/hermes-agent/.venv/bin/python scripts/run_tests.sh tests/hermes_cli/test_plugins.py -k test_discover_plugins_returns_when_background_discovery_exceeds_bound -v - failed as expected because discover_plugins() re-entered the locked discovery path - GREEN: HERMES_PYTHON=/Users/blockkonit./Dev/hermes/hermes-agent/.venv/bin/python scripts/run_tests.sh tests/hermes_cli/test_plugins.py -v - 78 passed, 0 failed - git diff --check - clean - Developer self-review: P0=0 - Independent review not triggered: 2 files, 91 diff lines, no auth/crypto/state.db changes - Residual behavior is intentionally fail-open: after the timeout, a caller may temporarily observe partial plugin state while the daemon discovery thread completes. A force reload requested during that interval must be retried after completion.

Related Issue

Fixes #108139

Type of Change

• ✅ Bug fix (non-breaking change that fixes an issue)

Changes Made

• CONTRIBUTING.md — production or test change in this diff
• AGENTS.md — production or test change in this diff

How to Test

• scripts/run_tests.sh on the files in Changes Made — focused verification
• RED: HERMES_PYTHON=/Users/blockkonit./Dev/hermes/hermes-agent/.venv/bin/python scripts/run_tests.sh tests/hermes_cli/test_plugins.py -k test_discover_plugins_returns_when_background_discovery_exceeds_bound -v
• failed as expected because discover_plugins() re-entered the locked discovery path
• GREEN: HERMES_PYTHON=/Users/blockkonit./Dev/hermes/hermes-agent/.venv/bin/python scripts/run_tests.sh tests/hermes_cli/test_plugins.py -v
• 78 passed, 0 failed
• git diff --check
• clean
• scripts/run_tests.sh tests/hermes_cli/test_plugins.py -k test_discover_plugins_returns_when_background_discovery_exceeds_bound -v - failed as expected because discover_plugins() re-entered the locked discovery path - GREEN: HERMES_PYTHON=/Users/blockkonit./Dev/hermes/hermes-agent/.venv/bin/python scripts/run_tests.sh tests/hermes_cli/test_plugins.py -v - 78 passed, 0 failed - git diff --check - clean - Developer self-review: P0=0 - Independent review not triggered: 2 files, 91 diff lines, no auth/crypto/state.db changes - Residual behavior is intentionally fail-open: after the timeout, a caller may temporarily observe partial plugin state while the daemon discovery thread completes. A force reload requested during that interval must be retried after completion.

Evidence

• BEFORE: the focused probe was RED on origin/main before this change
• AFTER: - [x] scripts/run_tests.sh on the files in Changes Made — focused verification — 78 passed
• CONTROL: neighboring paths listed in How to Test still pass

Checklist

Code

• ✅ I've read the Contributing Guide
• ✅ My commit messages follow Conventional Commits
• ✅ I searched for existing PRs to make sure this isn't a duplicate
• ✅ My PR contains only changes related to this fix
• ✅ I've run relevant tests locally (see How to Test)
• ✅ I've added tests for my changes
• ✅ I've tested on my platform: macOS

Documentation & Housekeeping

• ✅ Documentation update: N/A unless noted in Changes Made
• ✅ cli-config.yaml.example: N/A
• ✅ CONTRIBUTING.md or AGENTS.md: N/A
• ✅ Cross-platform impact considered
• ✅ Tool descriptions/schemas: N/A

[kvnloo]

There is still an uncovered path from the issue: this only bounds the handoff when a background discovery thread already exists.

With this head:

def discover_plugins(force=False):
    if not _join_background_discovery():
        ...
        return
    get_plugin_manager().discover_and_load(force=force)

If no background thread was started, _join_background_discovery() returns True immediately and the caller still enters discover_and_load() synchronously, with no wall-clock bound on the discovery body.

That matters for the reported ACP path. Current model_tools.py still imports and calls discover_plugins() as a module-level side effect; its comment notes ACP session init runs the import under asyncio.to_thread, which protects the event loop but does not make session/new complete. I do not see ACP pre-arming start_background_plugin_discovery() the way hermes_cli.main::_prepare_agent_startup does for ordinary CLI launches.

So the regression added here proves:

synchronous caller does not re-enter the manager lock after an already-running background scan exceeds the join bound.

It does not yet prove the issue's stronger invariant:

a first discovery triggered by ACP / another direct caller cannot remain blocked indefinitely inside a plugin's discovery/register path.

Suggested additional regression before claiming #108139 fixed:

1. reset manager + _background_discovery_thread=None;
2. make the actual discovery body stall;
3. call discover_plugins() through the direct/first-call shape;
4. assert the public call returns under the intended bound and the stalled work remains isolated/backgrounded (or explicitly narrow the PR/issue claim to background-handoff only).

If the intended architecture is instead “every agent-capable surface must call start_background_plugin_discovery() before any possible discover_plugins()”, enforce and test that invariant for ACP/gateway/cron rather than relying on the CLI launcher's ordering.

[teknium1]

Thanks @KoNit-K — closing as superseded by #118915 (merge SHA 9863e315f1fe1dab65f279b77eeb70b8de1fc8c4), which credits this PR.

Main already carried the 30 s bound on the join onto an in-flight background discovery (_join_background_discovery(timeout=30.0)), so the handoff half of this change was in place; what remained — and what @kvnloo's review pointed at — was the first synchronous discovery, where the hang originates in the plugin's own import/register(). #118915 bounds it there: a per-plugin plugins.load_timeout_seconds deadline (default 10 s), the offender skipped with a named reason, late registrations from the abandoned loader refused, abandoned loaders capped.