fix(plugins): per-session hook single-flight + matcher gate before the flight window

[emirsaffar-collab]

Problem (upstream #105223)

pre_tool_call is a fail-closed policy hook: when its callback is skipped (still running for another fire, or in the 60s post-timeout suppression window), the tool call is blocked. Because single-flight is keyed per (hook_name, callback) process-wide, parallel sessions sharing one process (e.g. an API/webui server that imports AIAgent directly and runs many concurrent sessions) block each other's tool calls whenever their hook runs overlap — even though no callback ever actually hangs.

Two amplifiers make it routine rather than rare:

1. Shell-hook tool matchers are evaluated inside the callback — i.e. inside the single-flight window — so every configured pre_tool_call hook contends on every tool call of every session, matching or not.
2. Hooks dispatch sequentially per call, widening the overlap window.

Observed on a 10+ parallel-session host: 333 skip events in two days, all "skipped ... or while still running" — zero real timeouts. Sessions hit same_tool_failure_halt tool-guardrail halts from the resulting blocked-call streaks.

Fix (two minimal, independent changes)

1. Matcher gate before the flight window (hermes_cli/plugins_dispatch.py): if a callback exposes _hermes_matches_tool (shell-hook callbacks do, agent/shell_hooks.py), non-matching tools continue before acquiring the flight token. A non-matching hook can never contend (nor be skipped because of) another session's in-flight run.
2. Session-scoped flight keys (hermes_cli/plugins_dispatch.py): callback_key = (hook_name, id(cb), session_id) — same composition as the shell-hook payload (falls back to "" when identity kwargs are absent). Parallel sessions each get their own flight token + suppression window; same-session single-flight (hang protection, one stacked worker per session) is fully preserved.

Tests

New TestHookSingleFlightContentionUpstream105223 (5 polarity tests) + synthetic load test:

• non-matching hook across threads → never contends
• matching hook, same session → still single-flight (skip preserved)
• matching hook, different sessions → no skip, both run
• matching hook still dispatched; real shell-hook e2e matcher gate
• load: 8 parallel sessions × 6 sequential calls = 48 fires → 48 ran, 0 blocked (pre-patch: same scenario produces the fail-closed storm); same-session polarity: 1 ran / 5 blocked (hang protection intact)

tests/hermes_cli/test_plugins.py 83/83 green on this branch (incl. existing timeout/fail-closed/suppression regression tests untouched).

Notes

• pre_tool_call fail-closed semantics on actual timeout are unchanged — only cross-session false contention is removed.
• Python-plugin callbacks without _hermes_matches_tool behave exactly as before (the gate is opt-in via the attribute).

[gaoanze888]

Moving the shell matcher before admission is sound, but the new per-session key removes the original process-wide resource bound at exact head 32ccc9b18225e597a0c607af6c5d9611ffa60b86.

session_id is request/session controlled. A hung callback can now create one abandoned daemon worker for every distinct session, whereas (hook_name, id(cb)) limited a callback to one worker process-wide. I reproduced one hung callback with 100 unique session IDs producing 100 simultaneously running hook workers. This contradicts the existing invariant that a stuck policy hook cannot spawn a new abandoned thread on every fire. Keep session-level contention isolation, but add a bounded global per-callback worker/admission budget so distinct sessions cannot create unbounded abandoned threads; add a many-session hung-callback regression.

The timeout suppression map has the same cardinality problem: each timed-out (hook, callback, session) remains until that exact session fires again or the manager unloads. Continually new API/gateway session IDs can grow it indefinitely. Please prune expired entries and/or use a bounded cache, with a many-session timeout cleanup test.

Matcher placement itself is correct and retains the callback-side defensive match check; manager scoping also keeps identical session IDs in different profiles separate. Focused plugin/shell tests pass 130/130 with one platform skip; Ruff and diff checks are clean. The current tests cover same-session and cross-session polarity but not bounded global worker/cache behavior. The old (hook_name, id(cb)) wording in plugins.py should also be updated.

[kshitijk4poor]

Partial overlap note: the single-flight half of this (concurrent same-tool calls collapsing on the per-callback key) landed on main via #111177 (73f808e47f) — the gate is now keyed on tool_call_id/turn_id with abandoned-worker tracking. The matcher-gate-before-flight-window half for shell hooks is not covered by that change and still stands on its own; a rebase onto main would shrink this PR to that piece.

[teknium1]

Closing as superseded by #118844 (#118844), commit 75e9567ca7, which fixes the same defect (you are credited there as a co-author / in the PR body).

Thanks for the contribution — the underlying problem this PR addresses has been resolved on current main. If you believe this was closed in error, please comment and we'll reopen.