Skip to content

fix(approval): heartbeat activity during gateway approval wait - #11245

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-5fdde1c8
Apr 16, 2026
Merged

fix(approval): heartbeat activity during gateway approval wait#11245
teknium1 merged 1 commit into
mainfrom
hermes/hermes-5fdde1c8

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

When the agent calls the terminal tool with a dangerous command that needs gateway approval, the agent thread blocks in tools/approval.py on entry.event.wait(timeout=...). Before this PR, no activity heartbeats fired during that wait. If the user was slow to respond (or had approvals.gateway_timeout set higher than the default 300s), the agent thread sat silent long enough for the gateway's inactivity watchdog (agent.gateway_timeout, default 1800s) to kill it — even though the agent was doing exactly the right thing and the user was the one causing the delay.

The fix polls the event in 1s slices and calls touch_activity_if_due between slices, mirroring the _wait_for_process() pattern in tools/environments/base.py that covers the subprocess-waiting side of the same problem.

Evidence

Community user MRB reported Stuck on terminal tool on Discord (April 2026). Their logs show 12 repeated idle-timeout events with this exact shape:

ERROR gateway.run: Agent idle for 1800s (timeout 1800s)
  | last_activity=executing tool: terminal | iteration=X/70 | tool=terminal

Events #1, #5, #6 were preceded by Discord approval button clicks where the user took 9-10 minutes to respond. Events #2, #3, #4 were covered by the already-shipped PR #10501 (streaming / concurrent tools / Modal backend heartbeats) — this PR closes the remaining gap in approval.py.

How it works

At the default 10s heartbeat cadence (from touch_activity_if_due), a 300s approval wait now pings activity ~30 times, well under the 1800s idle threshold. The polling slice is 1s, so user approvals are still essentially instant.

while True:
    _remaining = _deadline - time.monotonic()
    if _remaining <= 0:
        break
    if entry.event.wait(timeout=min(1.0, _remaining)):
        resolved = True
        break
    if touch_activity_if_due is not None:
        touch_activity_if_due(_activity_state, "waiting for user approval")

The lazy from tools.environments.base import touch_activity_if_due inside the function avoids any import-order coupling and degrades gracefully if the helper can't be imported.

Test plan

  • tests/tools/test_approval_heartbeat.py (new, 3 tests):

    1. Heartbeats fire while waiting — patches touch_activity_if_due, runs the full check_all_command_guards path in a thread, asserts the heartbeat callback fires before resolve.
    2. User response returns near-instant — resolve at t=0.1s, asserts total check_all_command_guards runtime is under 3s (proves the polling slice doesn't add visible latency).
    3. Import failure doesn't break the wait — simulates ImportError on tools.environments.base, asserts approval still completes.
  • E2E validation with the real set_activity_callback → thread-local → touch_activity_if_due chain confirms a 12s simulated user wait produces a heartbeat at the 10s mark with last_activity='waiting for user approval (10s elapsed)', and user_approved=True is preserved.

  • Re-ran 199 approval-adjacent tests across 7 test files — all pass:

    • tests/tools/test_approval_heartbeat.py
    • tests/tools/test_approval.py
    • tests/gateway/test_approve_deny_commands.py
    • tests/gateway/test_gateway_inactivity_timeout.py
    • tests/gateway/test_{telegram,feishu,slack}_approval_buttons.py

Companion to PR #10501

PR #10501 added activity heartbeats for streaming waits, concurrent tool dispatch (thread-local callback propagation), and the Modal backend's custom execute(). This PR closes the remaining known gap — the gateway approval Event.wait() that covers the human-in-the-loop case.

The blocking gateway approval wait at tools/approval.py called
`entry.event.wait(timeout=...)` which never touched the agent's
activity tracker.  When a user was slow to respond to a /approve prompt
(or the gateway_timeout config was set higher than the default 300s),
the agent thread sat silent long enough for the gateway's inactivity
watchdog (agent.gateway_timeout, default 1800s) to kill it — even
though the agent was doing exactly the right thing and the user was
the one causing the delay.

The fix polls the event in 1s slices and calls touch_activity_if_due
between slices, mirroring the _wait_for_process() pattern in
tools/environments/base.py that covers the subprocess-waiting side of
the same problem.  At the default 10s heartbeat cadence, a 300s
approval wait now pings activity ~30 times, well under the 1800s
idle threshold.

Observed in community user logs: 12 repeated 'Agent idle 1800s,
last_activity=executing tool: terminal' events across April 12-14.
Companion to PR #10501 which covered streaming / concurrent-tool /
Modal-backend gaps but did not touch approval.py.

Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats
fire during the wait, (2) user responses are still near-instant, and
(3) the approval path stays functional when the heartbeat helper
can't be imported.
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Supply Chain Risk Detected

This PR contains patterns commonly associated with supply chain attacks. This does not mean the PR is malicious — but these patterns require careful human review before merging.

⚠️ WARNING: Install hook files modified

These files can execute code during package installation or interpreter startup.

Files:

hermes_cli/setup.py

Automated scan triggered by supply-chain-audit. If this is a false positive, a maintainer can approve after manual review.

@teknium1
teknium1 merged commit 387aa9a into main Apr 16, 2026
4 of 5 checks passed
@teknium1
teknium1 deleted the hermes/hermes-5fdde1c8 branch April 16, 2026 21:48
aj-nt pushed a commit to aj-nt/hermes-agent that referenced this pull request May 1, 2026
…esearch#11245)

The blocking gateway approval wait at tools/approval.py called
`entry.event.wait(timeout=...)` which never touched the agent's
activity tracker.  When a user was slow to respond to a /approve prompt
(or the gateway_timeout config was set higher than the default 300s),
the agent thread sat silent long enough for the gateway's inactivity
watchdog (agent.gateway_timeout, default 1800s) to kill it — even
though the agent was doing exactly the right thing and the user was
the one causing the delay.

The fix polls the event in 1s slices and calls touch_activity_if_due
between slices, mirroring the _wait_for_process() pattern in
tools/environments/base.py that covers the subprocess-waiting side of
the same problem.  At the default 10s heartbeat cadence, a 300s
approval wait now pings activity ~30 times, well under the 1800s
idle threshold.

Observed in community user logs: 12 repeated 'Agent idle 1800s,
last_activity=executing tool: terminal' events across April 12-14.
Companion to PR NousResearch#10501 which covered streaming / concurrent-tool /
Modal-backend gaps but did not touch approval.py.

Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats
fire during the wait, (2) user responses are still near-instant, and
(3) the approval path stays functional when the heartbeat helper
can't be imported.
02356abc pushed a commit to 02356abc/hermes-agent that referenced this pull request May 14, 2026
…esearch#11245)

The blocking gateway approval wait at tools/approval.py called
`entry.event.wait(timeout=...)` which never touched the agent's
activity tracker.  When a user was slow to respond to a /approve prompt
(or the gateway_timeout config was set higher than the default 300s),
the agent thread sat silent long enough for the gateway's inactivity
watchdog (agent.gateway_timeout, default 1800s) to kill it — even
though the agent was doing exactly the right thing and the user was
the one causing the delay.

The fix polls the event in 1s slices and calls touch_activity_if_due
between slices, mirroring the _wait_for_process() pattern in
tools/environments/base.py that covers the subprocess-waiting side of
the same problem.  At the default 10s heartbeat cadence, a 300s
approval wait now pings activity ~30 times, well under the 1800s
idle threshold.

Observed in community user logs: 12 repeated 'Agent idle 1800s,
last_activity=executing tool: terminal' events across April 12-14.
Companion to PR NousResearch#10501 which covered streaming / concurrent-tool /
Modal-backend gaps but did not touch approval.py.

Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats
fire during the wait, (2) user responses are still near-instant, and
(3) the approval path stays functional when the heartbeat helper
can't be imported.
gweeteve pushed a commit to gweeteve/hermes-agent that referenced this pull request Jun 2, 2026
…esearch#11245)

The blocking gateway approval wait at tools/approval.py called
`entry.event.wait(timeout=...)` which never touched the agent's
activity tracker.  When a user was slow to respond to a /approve prompt
(or the gateway_timeout config was set higher than the default 300s),
the agent thread sat silent long enough for the gateway's inactivity
watchdog (agent.gateway_timeout, default 1800s) to kill it — even
though the agent was doing exactly the right thing and the user was
the one causing the delay.

The fix polls the event in 1s slices and calls touch_activity_if_due
between slices, mirroring the _wait_for_process() pattern in
tools/environments/base.py that covers the subprocess-waiting side of
the same problem.  At the default 10s heartbeat cadence, a 300s
approval wait now pings activity ~30 times, well under the 1800s
idle threshold.

Observed in community user logs: 12 repeated 'Agent idle 1800s,
last_activity=executing tool: terminal' events across April 12-14.
Companion to PR NousResearch#10501 which covered streaming / concurrent-tool /
Modal-backend gaps but did not touch approval.py.

Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats
fire during the wait, (2) user responses are still near-instant, and
(3) the approval path stays functional when the heartbeat helper
can't be imported.
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…esearch#11245)

The blocking gateway approval wait at tools/approval.py called
`entry.event.wait(timeout=...)` which never touched the agent's
activity tracker.  When a user was slow to respond to a /approve prompt
(or the gateway_timeout config was set higher than the default 300s),
the agent thread sat silent long enough for the gateway's inactivity
watchdog (agent.gateway_timeout, default 1800s) to kill it — even
though the agent was doing exactly the right thing and the user was
the one causing the delay.

The fix polls the event in 1s slices and calls touch_activity_if_due
between slices, mirroring the _wait_for_process() pattern in
tools/environments/base.py that covers the subprocess-waiting side of
the same problem.  At the default 10s heartbeat cadence, a 300s
approval wait now pings activity ~30 times, well under the 1800s
idle threshold.

Observed in community user logs: 12 repeated 'Agent idle 1800s,
last_activity=executing tool: terminal' events across April 12-14.
Companion to PR NousResearch#10501 which covered streaming / concurrent-tool /
Modal-backend gaps but did not touch approval.py.

Test: tests/tools/test_approval_heartbeat.py — verifies (1) heartbeats
fire during the wait, (2) user responses are still near-instant, and
(3) the approval path stays functional when the heartbeat helper
can't be imported.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant