fix(runtime): detach concurrent tool batches on interrupt

[qWaitCrypto]

What does this PR do?

Fixes a runtime bug where interrupting a concurrent tool batch did not actually release the current turn promptly.

Before this change, _execute_tool_calls_concurrent() used with ThreadPoolExecutor(...). On interrupt, Hermes cancelled futures that had not started yet, but ThreadPoolExecutor.__exit__() still called shutdown(wait=True). In practice, /stop or a new user message could appear to succeed while the current turn was still blocked on already-running tools like web_search, file reads, or custom plugins that do not actively poll for interrupts.

This approach keeps the normal concurrent path intact, but makes interrupted concurrent batches detachable so the current turn can return promptly.

Related Issue

NA

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• updated run_agent.py to track the active concurrent tool batch on the agent
• replaced the implicit ThreadPoolExecutor lifetime with explicit attach/detach/shutdown handling
• detach the active concurrent batch during interrupt() so the current turn does not wait for long-running workers to drain
• added a short poll interval in the concurrent wait loop so interrupts are noticed promptly
• reject late callback delivery and discard late tool results after a batch has been detached
• handled the submit-time race where the executor can be shut down while a concurrent batch is being detached
• added a regression test in tests/run_agent/test_concurrent_interrupt.py covering prompt interrupt return and late-result suppression

How to Test

1. Run:

   PYTHONPATH=. /home/cyt/miniconda3/envs/meta_workflow_py312/bin/python -m pytest --override-ini=addopts='' -p no:cacheprovider tests/run_agent/test_concurrent_interrupt.py -q
   HOME=/tmp/hermes-agent-pytest HERMES_HOME=/tmp/hermes-agent-pytest/.hermes XDG_CACHE_HOME=/tmp/hermes-agent-pytest/.cache PYTHONPATH=. /home/cyt/miniconda3/envs/meta_workflow_py312/bin/python -m pytest --override-ini=addopts='' -p no:cacheprovider tests/run_agent/test_run_agent.py -q -k concurrent
   HOME=/tmp/hermes-agent-pytest HERMES_HOME=/tmp/hermes-agent-pytest/.hermes XDG_CACHE_HOME=/tmp/hermes-agent-pytest/.cache PYTHONPATH=. /home/cyt/miniconda3/envs/meta_workflow_py312/bin/python -m pytest --override-ini=addopts='' -p no:cacheprovider tests/run_agent/test_tool_executor_contextvar_propagation.py tests/run_agent/test_interrupt_propagation.py tests/run_agent/test_tool_call_guardrail_runtime.py -q

2. Confirm the results are:
   • tests/run_agent/test_concurrent_interrupt.py: 3 passed
   • tests/run_agent/test_run_agent.py -k concurrent: 22 passed, 303 deselected
   • tests/run_agent/test_tool_executor_contextvar_propagation.py tests/run_agent/test_interrupt_propagation.py tests/run_agent/test_tool_call_guardrail_runtime.py: 19 passed
3. Optionally reproduce the original issue by interrupting a concurrent tool batch with a slow non-cooperative tool and confirm the turn returns promptly instead of waiting for executor drain.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: WSL2 Ubuntu / local Linux pytest via /home/cyt/miniconda3/envs/meta_workflow_py312/bin/python

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Manual reproduction harness against upstream/main and this branch, forcing _MAX_TOOL_WORKERS = 1 so one tool is running while the batch is interrupted:

$ # upstream/main baseline
$ HOME=/tmp/hermes-agent-pytest HERMES_HOME=/tmp/hermes-agent-pytest/.hermes /home/cyt/miniconda3/envs/meta_workflow_py312/bin/python repro_concurrent_interrupt.py
Could not import tool module tools.browser_dialog_tool: No module named 'websockets'
{'phase': 'before_fix', 'started_wait': True, 'alive_after_1s': True, 'elapsed_after_1s': 1.0, 'total_elapsed': 1.0, 'message_count': 2, 'messages': ['{"ok": true, "late": true}', '{"ok": true, "late": true}']}

$ # fix/concurrent-tool-interrupt
$ HOME=/tmp/hermes-agent-pytest HERMES_HOME=/tmp/hermes-agent-pytest/.hermes /home/cyt/miniconda3/envs/meta_workflow_py312/bin/python repro_concurrent_interrupt.py
Could not import tool module tools.browser_dialog_tool: No module named 'websockets'
{'phase': 'after_fix', 'started_wait': True, 'alive_after_1s': False, 'elapsed_after_1s': 0.43, 'total_elapsed': 0.43, 'message_count': 2, 'messages': ['[Tool execution cancelled — tool_a was skipped due to user interrupt]', '[Tool execution cancelled — tool_b was skipped due to user interrupt]']}

The baseline reproduces the bug: one second after interrupt, the concurrent batch is still alive and the late tool result eventually lands. The fixed branch returns promptly and preserves interrupt placeholders instead of accepting late completions.

$ PYTHONPATH=. /home/cyt/miniconda3/envs/meta_workflow_py312/bin/python -m pytest --override-ini=addopts='' -p no:cacheprovider tests/run_agent/test_concurrent_interrupt.py -q
...                                                                      [100%]
3 passed in 6.52s

$ HOME=/tmp/hermes-agent-pytest HERMES_HOME=/tmp/hermes-agent-pytest/.hermes XDG_CACHE_HOME=/tmp/hermes-agent-pytest/.cache PYTHONPATH=. /home/cyt/miniconda3/envs/meta_workflow_py312/bin/python -m pytest --override-ini=addopts='' -p no:cacheprovider tests/run_agent/test_run_agent.py -q -k concurrent
......................                                                   [100%]
22 passed, 303 deselected in 19.50s

$ HOME=/tmp/hermes-agent-pytest HERMES_HOME=/tmp/hermes-agent-pytest/.hermes XDG_CACHE_HOME=/tmp/hermes-agent-pytest/.cache PYTHONPATH=. /home/cyt/miniconda3/envs/meta_workflow_py312/bin/python -m pytest --override-ini=addopts='' -p no:cacheprovider tests/run_agent/test_tool_executor_contextvar_propagation.py tests/run_agent/test_interrupt_propagation.py tests/run_agent/test_tool_call_guardrail_runtime.py -q
...................                                                      [100%]
19 passed in 12.21s

[teknium1]

Thanks for isolating a real concurrent-interrupt responsiveness issue. Current agent/tool_executor.py already abandons interrupted batches with shutdown(wait=False, cancel_futures=True) at lines 776-785, but it can first block in the five-second wait() at lines 695-710 and then takes a further three-second grace wait at line 761.

Problems

• The PR modifies the pre-extraction implementation in run_agent.py; current main forwards from run_agent.py:5760 into agent/tool_executor.py after commit 79559214a.
• Porting the PR's stdlib ThreadPoolExecutor construction would lose the daemon-worker invariant at agent/tool_executor.py:641-647. Commit 3f2a56d1a added that executor because abandoned stdlib workers can hold interpreter exit open.

Suggested changes

• Re-scope the detach/short-poll behavior to agent/tool_executor.py, preserving its daemon executor, timeout path, and current result/persistence handling.
• Port the regression test to exercise the extracted implementation with a non-cooperative worker and verify prompt return plus late-result suppression.

Automated hermes-sweeper review.