fix(approval): heartbeat activity during gateway approval wait - #11245
Merged
Conversation
The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR #10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.
Contributor
|
aj-nt
pushed a commit
to aj-nt/hermes-agent
that referenced
this pull request
May 1, 2026
…esearch#11245) The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR NousResearch#10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
…esearch#11245) The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR NousResearch#10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
…esearch#11245) The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR NousResearch#10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
…esearch#11245) The blocking gateway approval wait at tools/approval.py called `entry.event.wait(timeout=...)` which never touched the agent's activity tracker. When a user was slow to respond to a /approve prompt (or the gateway_timeout config was set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay. The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem. At the default 10s heartbeat cadence, a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. Observed in community user logs: 12 repeated 'Agent idle 1800s, last_activity=executing tool: terminal' events across April 12-14. Companion to PR NousResearch#10501 which covered streaming / concurrent-tool / Modal-backend gaps but did not touch approval.py. Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats fire during the wait, (2) user responses are still near-instant, and (3) the approval path stays functional when the heartbeat helper can't be imported.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the agent calls the terminal tool with a dangerous command that needs gateway approval, the agent thread blocks in
tools/approval.pyonentry.event.wait(timeout=...). Before this PR, no activity heartbeats fired during that wait. If the user was slow to respond (or hadapprovals.gateway_timeoutset higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay.The fix polls the event in 1s slices and calls
touch_activity_if_duebetween slices, mirroring the_wait_for_process()pattern intools/environments/base.pythat covers the subprocess-waiting side of the same problem.Evidence
Community user MRB reported
Stuck on terminal toolon Discord (April 2026). Their logs show 12 repeated idle-timeout events with this exact shape:Events #1, #5, #6 were preceded by Discord approval button clicks where the user took 9-10 minutes to respond. Events #2, #3, #4 were covered by the already-shipped PR #10501 (streaming / concurrent tools / Modal backend heartbeats) — this PR closes the remaining gap in
approval.py.How it works
At the default 10s heartbeat cadence (from
touch_activity_if_due), a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. The polling slice is 1s, so user approvals are still essentially instant.The lazy
from tools.environments.base import touch_activity_if_dueinside the function avoids any import-order coupling and degrades gracefully if the helper can't be imported.Test plan
tests/tools/test_approval_heartbeat.py(new, 3 tests):touch_activity_if_due, runs the fullcheck_all_command_guardspath in a thread, asserts the heartbeat callback fires before resolve.check_all_command_guardsruntime is under 3s (proves the polling slice doesn't add visible latency).tools.environments.base, asserts approval still completes.E2E validation with the real
set_activity_callback→ thread-local →touch_activity_if_duechain confirms a 12s simulated user wait produces a heartbeat at the 10s mark withlast_activity='waiting for user approval (10s elapsed)', anduser_approved=Trueis preserved.Re-ran 199 approval-adjacent tests across 7 test files — all pass:
tests/tools/test_approval_heartbeat.pytests/tools/test_approval.pytests/gateway/test_approve_deny_commands.pytests/gateway/test_gateway_inactivity_timeout.pytests/gateway/test_{telegram,feishu,slack}_approval_buttons.pyCompanion to PR #10501
PR #10501 added activity heartbeats for streaming waits, concurrent tool dispatch (thread-local callback propagation), and the Modal backend's custom
execute(). This PR closes the remaining known gap — the gateway approvalEvent.wait()that covers the human-in-the-loop case.