fix: add activity heartbeats to prevent false gateway inactivity timeouts by teknium1 · Pull Request #10501 · NousResearch/hermes-agent

teknium1 · 2026-04-15T20:28:59Z

Summary

Multiple gaps in activity tracking could cause the gateway's inactivity timeout (30 min default) to fire while the agent is actively working.

Gap 1: Streaming wait loop (highest impact)

The outer thread in _api_call_with_interrupt only touched activity when the stale-stream detector fired (every 180-300s). For local providers (Ollama/oMLX), the stale timeout is infinity — meaning zero heartbeats during the entire stream wait. A reasoning model thinking for 30+ minutes before producing tokens would trigger the gateway timeout despite actively working.

Fix: Added 30-second periodic heartbeat to the streaming wait loop, covering both Anthropic and OpenAI paths.

Gap 2: Concurrent tool execution

_execute_tool_calls_concurrent ran tools in ThreadPoolExecutor workers but never called set_activity_callback(). Since the callback uses threading.local(), the main thread's callback was invisible to worker threads. Terminal commands running concurrently had no activity heartbeat.

Fix: Each worker thread now sets the callback. Also added _current_tool tracking and replaced concurrent.futures.wait() with a 30-second polling loop that reports which tools are still running.

Gap 3: Modal backend

BaseModalExecutionEnvironment.execute() overrides the base class with its own polling loop that had no activity callback.

Fix: Added activity callback matching _wait_for_process cadence (10s).

Changes

run_agent.py — Streaming heartbeat + concurrent tool tracking (+54 lines)
tools/environments/modal_utils.py — Modal activity heartbeat (+20 lines)
tests/tools/test_managed_modal_environment.py — Updated mocked time.monotonic() sequence

Testing

115 directly-related tests pass (streaming, gateway timeout, cron timeout, delegate, managed modal)
E2E verified: thread-local callback confirmed invisible to ThreadPoolExecutor workers (Gap 2 root cause)
E2E verified: sequential terminal path callback fires correctly every 10s

…outs Multiple gaps in activity tracking could cause the gateway's inactivity timeout to fire while the agent is actively working: 1. Streaming wait loop had no periodic heartbeat — the outer thread only touched activity when the stale-stream detector fired (180-300s), and for local providers (Ollama) the stale timeout was infinity, meaning zero heartbeats. Now touches activity every 30s. 2. Concurrent tool execution never set the activity callback on worker threads (threading.local invisible across threads) and never set _current_tool. Workers now set the callback, and the concurrent wait uses a polling loop with 30s heartbeats. 3. Modal backend's execute() override had its own polling loop without any activity callback. Now matches _wait_for_process cadence (10s).

…outs (NousResearch#10501) Multiple gaps in activity tracking could cause the gateway's inactivity timeout to fire while the agent is actively working: 1. Streaming wait loop had no periodic heartbeat — the outer thread only touched activity when the stale-stream detector fired (180-300s), and for local providers (Ollama) the stale timeout was infinity, meaning zero heartbeats. Now touches activity every 30s. 2. Concurrent tool execution never set the activity callback on worker threads (threading.local invisible across threads) and never set _current_tool. Workers now set the callback, and the concurrent wait uses a polling loop with 30s heartbeats. 3. Modal backend's execute() override had its own polling loop without any activity callback. Now matches _wait_for_process cadence (10s).

The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR #10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.

…outs (NousResearch#10501) Multiple gaps in activity tracking could cause the gateway's inactivity timeout to fire while the agent is actively working: 1. Streaming wait loop had no periodic heartbeat — the outer thread only touched activity when the stale-stream detector fired (180-300s), and for local providers (Ollama) the stale timeout was infinity, meaning zero heartbeats. Now touches activity every 30s. 2. Concurrent tool execution never set the activity callback on worker threads (threading.local invisible across threads) and never set _current_tool. Workers now set the callback, and the concurrent wait uses a polling loop with 30s heartbeats. 3. Modal backend's execute() override had its own polling loop without any activity callback. Now matches _wait_for_process cadence (10s).

…esearch#11245) The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR NousResearch#10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.

…outs (NousResearch#10501) Multiple gaps in activity tracking could cause the gateway's inactivity timeout to fire while the agent is actively working: 1. Streaming wait loop had no periodic heartbeat — the outer thread only touched activity when the stale-stream detector fired (180-300s), and for local providers (Ollama) the stale timeout was infinity, meaning zero heartbeats. Now touches activity every 30s. 2. Concurrent tool execution never set the activity callback on worker threads (threading.local invisible across threads) and never set _current_tool. Workers now set the callback, and the concurrent wait uses a polling loop with 30s heartbeats. 3. Modal backend's execute() override had its own polling loop without any activity callback. Now matches _wait_for_process cadence (10s).

…esearch#11245) The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR NousResearch#10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.

…outs (NousResearch#10501) Multiple gaps in activity tracking could cause the gateway's inactivity timeout to fire while the agent is actively working: 1. Streaming wait loop had no periodic heartbeat — the outer thread only touched activity when the stale-stream detector fired (180-300s), and for local providers (Ollama) the stale timeout was infinity, meaning zero heartbeats. Now touches activity every 30s. 2. Concurrent tool execution never set the activity callback on worker threads (threading.local invisible across threads) and never set _current_tool. Workers now set the callback, and the concurrent wait uses a polling loop with 30s heartbeats. 3. Modal backend's execute() override had its own polling loop without any activity callback. Now matches _wait_for_process cadence (10s).

…esearch#11245) The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR NousResearch#10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.

…outs (NousResearch#10501) Multiple gaps in activity tracking could cause the gateway's inactivity timeout to fire while the agent is actively working: 1. Streaming wait loop had no periodic heartbeat — the outer thread only touched activity when the stale-stream detector fired (180-300s), and for local providers (Ollama) the stale timeout was infinity, meaning zero heartbeats. Now touches activity every 30s. 2. Concurrent tool execution never set the activity callback on worker threads (threading.local invisible across threads) and never set _current_tool. Workers now set the callback, and the concurrent wait uses a polling loop with 30s heartbeats. 3. Modal backend's execute() override had its own polling loop without any activity callback. Now matches _wait_for_process cadence (10s).

…esearch#11245) The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR NousResearch#10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.

teknium1 merged commit a418ddb into main Apr 15, 2026
6 of 7 checks passed

teknium1 deleted the hermes/hermes-5b019ce9 branch April 15, 2026 20:29

teknium1 mentioned this pull request Apr 16, 2026

fix(approval): heartbeat activity during gateway approval wait #11245

Merged

github-actions Bot mentioned this pull request Apr 24, 2026

chore: bump NousResearch/hermes-agent version from v2026.4.16 to v2026.4.23 Docker-Hub-sirmark/docker-hermes-agent#3

Merged

subinium mentioned this pull request Apr 27, 2026

feat(tools): emit periodic activity heartbeat during long-running tools subinium/CrowClaw#95

Closed

Navigation Menu

Uh oh!

fix: add activity heartbeats to prevent false gateway inactivity timeouts - #10501

fix: add activity heartbeats to prevent false gateway inactivity timeouts#10501
teknium1 merged 1 commit into
mainfrom
hermes/hermes-5b019ce9

teknium1 commented Apr 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

teknium1 commented Apr 15, 2026

Summary

Gap 1: Streaming wait loop (highest impact)

Gap 2: Concurrent tool execution

Gap 3: Modal backend

Changes

Testing

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant