fix(gateway): keep silent in-flight tools alive

[Zeraphim]

What does this PR do?

Prevents the gateway turn-inactivity watchdog from abandoning turns while a tool is still executing silently.

The watchdog now treats an active tool as in-flight activity in both timeout paths. Concurrent tool execution keeps the activity marker set until the final worker exits, while genuinely idle turns retain the existing timeout behavior.

Related Issue

Fixes #84491

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• Updated gateway/run.py to skip inactivity timeout handling while current_tool is active.
• Updated agent/tool_executor.py to preserve activity state across concurrent tool workers.
• Added gateway watchdog regression coverage for silent in-flight tools.
• Added concurrent tool lifecycle coverage.

How to Test

1. Run scripts/run_tests.sh tests/gateway/test_abandoned_turn_process_cleanup.py tests/run_agent/test_run_agent.py -q.
2. Confirm an active silent tool is not interrupted or reaped.
3. Confirm a genuinely idle turn still triggers Execution timed out (inactivity).

Validation completed:

• python -m py_compile agent/tool_executor.py gateway/run.py tests/run_agent/test_run_agent.py tests/gateway/test_abandoned_turn_process_cleanup.py
• git diff --check
• Direct watchdog probes for active-tool protection and idle timeout behavior

The targeted pytest command was blocked because no available virtualenv contains pytest.

Checklist

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix
• I've run pytest tests/ -q and all tests pass
• I've added tests for this bug
• I've tested on macOS 26
• I've considered cross-platform impact
• Documentation updates are not applicable
• Configuration updates are not applicable
• Tool descriptions/schemas are not applicable

Screenshots / Logs

Not applicable for this backend behavior fix.

[spfcraze]

This was generated by AI during triage.

Summary:
A genuinely hung tool — one that never returns and never emits activity — now holds the gateway turn open with no hard bound: both inactivity paths skip any turn whose current_tool is set, and a hung single tool never clears it.

Problems:

• gateway/run.py:3055 and gateway/run.py:26414 short-circuit the inactivity check whenever get_activity_summary()['current_tool'] is truthy; _current_tool is set when a tool begins and cleared only after it returns, so a deadlocked tool keeps it set for the whole hang.
• The single-tool path runs the tool synchronously with no per-tool deadline (the 420s concurrent deadline only bounds the batch executor), so the inactivity watchdog was the only bound that hard-interrupted and reaped a genuinely hung tool's processes — and it now never fires for a turn with a tool in flight.
• Issue [Bug]: Gateway turn-inactivity watchdog abandons turns whose tool call runs silently >30 min #84491's suggested fix direction 2 pairs this exact exclusion with a separate (and configurable) hard ceiling for genuinely hung tools; this diff ships the exclusion without that ceiling.

Solution:
Add a separate, configurable hard ceiling on in-flight tool wall-time, independent of current_tool, so a genuinely hung tool is still abandoned and its processes reaped, as issue #84491's fix direction 2 calls for.

Evidence

no deterministic fact backs this claim — model belief, not executed or read evidence

---

Checked against 227686e — the PR head when this was written — and a871948, main at the same moment.

[Zeraphim]

This was generated by AI during triage.

Summary: A genuinely hung tool — one that never returns and never emits activity — now holds the gateway turn open with no hard bound: both inactivity paths skip any turn whose current_tool is set, and a hung single tool never clears it.

Problems:

• gateway/run.py:3055 and gateway/run.py:26414 short-circuit the inactivity check whenever get_activity_summary()['current_tool'] is truthy; _current_tool is set when a tool begins and cleared only after it returns, so a deadlocked tool keeps it set for the whole hang.
• The single-tool path runs the tool synchronously with no per-tool deadline (the 420s concurrent deadline only bounds the batch executor), so the inactivity watchdog was the only bound that hard-interrupted and reaped a genuinely hung tool's processes — and it now never fires for a turn with a tool in flight.
• Issue [Bug]: Gateway turn-inactivity watchdog abandons turns whose tool call runs silently >30 min #84491's suggested fix direction 2 pairs this exact exclusion with a separate (and configurable) hard ceiling for genuinely hung tools; this diff ships the exclusion without that ceiling.

Solution: Add a separate, configurable hard ceiling on in-flight tool wall-time, independent of current_tool, so a genuinely hung tool is still abandoned and its processes reaped, as issue #84491's fix direction 2 calls for.

Evidence

no deterministic fact backs this claim — model belief, not executed or read evidence

Checked against 227686e — the PR head when this was written — and a871948, main at the same moment.

Addressed in 227686e

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

fix(gateway): keep silent in-flight tools alive

1. agent/tool_executor.py — the concurrent fan-out path (_run_tool) sets _current_tool when it is unset, but the new _current_tool_started_at marker is only assigned in _begin_tool_execution. If the concurrent path does not go through _begin_tool_execution, get_activity_summary()["tool_started_at"] stays None, and _watch_gateway_turn_inactivity treats started_at is None as "continue" — meaning the new in-flight wall-time ceiling never applies to concurrent batches (they remain bounded only by the batch executor's own HERMES_CONCURRENT_TOOL_TIMEOUT_S). Please confirm the concurrent path populates tool_started_at, and if not, set it in _run_tool as well.

2. gateway/run.py _watch_gateway_turn_inactivity — once in_flight_tool_timeout expires for a genuinely hung tool, the watchdog falls through to _abandon_timed_out_gateway_turn. That is the intended behavior, but note the single-tool path runs the tool synchronously in the agent thread: an in-process agent thread blocked in the tool may not observe interrupt() until the tool returns, so the reap may not actually stop the hung call — worth verifying the interrupt/reap path is effective for the synchronous single-tool case this PR is targeting.

3. The default in_flight_tool_timeout = 420.0 now applies to every tool that emits no activity, including legitimate long-running silent operations (large downloads, long terminal commands without progress output). 7 minutes is reasonable, but a tool that is healthy and merely quiet will now be reaped where the old inactivity logic (which reset on _touch_activity at tool start) would have waited. Consider documenting the default in the env var reference, and confirm the existing _touch_activity("executing tool: ...") call gives genuinely silent tools at least the 420s grace.

4. Minor: _begin_tool_execution keeps the earliest _current_tool_started_at while _current_tool itself can be overwritten by a later start — the log line ("In-flight tool %s exceeded...") may pair a later tool name with an earlier timestamp. Harmless for the wall-clock decision, but slightly misleading in logs.

[Zeraphim]

Hey @Enough1122, thanks for catching these edge cases. Addressed on the latest head:

• Added a separate 420-second wall-time ceiling for active tools, using the existing HERMES_CONCURRENT_TOOL_TIMEOUT_S setting for single-tool execution and batches.
• Anchored concurrent worker timing even when tool preflight is bypassed, so the watchdog cannot miss a silent batch.
• Documented the timeout behavior and added regression coverage for the watchdog timestamp and cleanup path.

Validation:

• python3 -m compileall -q agent/tool_executor.py gateway/run.py run_agent.py tests/run_agent/test_tool_activity_heartbeat.py tests/gateway/test_abandoned_turn_process_cleanup.py — passed.
• git diff --check — passed.
• scripts/run_tests.sh tests/run_agent/test_tool_activity_heartbeat.py tests/gateway/test_abandoned_turn_process_cleanup.py tests/run_agent/test_sequential_tool_timeout.py -q — blocked; no checkout virtualenv with pytest is installed.q

[Enough1122]

Thanks @Zeraphim — a separate 420s wall-time ceiling reusing HERMES_CONCURRENT_TOOL_TIMEOUT_S is a clean fix for the silent in-flight tool hang; noted at 23f8bd69f.