Skip to content

fix(gateway): unblock clarify wait on /stop or interrupt-mode messages - #84119

Open
yflmq001 wants to merge 1 commit into
NousResearch:mainfrom
yflmq001:fix/83889-clarify-wait-interrupt
Open

fix(gateway): unblock clarify wait on /stop or interrupt-mode messages#84119
yflmq001 wants to merge 1 commit into
NousResearch:mainfrom
yflmq001:fix/83889-clarify-wait-interrupt

Conversation

@yflmq001

Copy link
Copy Markdown
Contributor

Fixes #83889 (root cause 1: wait_for_response never observes interrupts).

Bug Description

When an agent in a gateway session calls clarify, the agent thread blocks in tools/clarify_gateway.wait_for_response on a threading.Event with a 1s-slice poll loop whose only side-channel is the inactivity heartbeat. The gateway's interrupt path (_interrupt_and_clear_sessionrequest_hard_interrupt → agent _set_interrupt(True, thread_id)) sets the per-thread interrupt flag in tools.interrupt, but the blocked wait never checks it. Because the agent thread is stuck inside the tool, the run's finally cleanup (clear_session in run_sync) cannot execute either, so the clarify stays blocked until the full timeout (600s default) — /stop does nothing, and any user reply typed in the meantime is dead. Confirmed on multiplexed Feishu groups (#83889); the same mechanism applies to every platform with text-fallback clarify.

Root Cause

wait_for_response (tools/clarify_gateway.py:135) polls only:

  1. entry.event.wait(timeout=slice_s) — set by resolve_gateway_clarify / clear_session
  2. touch_activity_if_due — inactivity heartbeat

No interrupt flag check. The per-thread interrupt mechanism (tools.interrupt.is_interrupted()) is exactly designed for this: the wait runs on the agent thread, and /stop marks that same thread. This is the same pattern tools/environments/base.py already uses for the heartbeat — the interrupt check is missing.

Fix

  • tools/clarify_gateway.pywait_for_response polls tools.interrupt.is_interrupted() each 1s slice (same frequency as the heartbeat, try/except-guarded like the heartbeat import). On interrupt it breaks, returns None, and the existing post-loop cleanup removes the entry — identical exit path to a timeout.
  • gateway/run.py_clarify_callback_sync now checks the interrupt flag after wait_for_response returns and surfaces "[interrupted by user]" instead of the misleading "[user did not respond within 10m]" timeout message (the agent is in the interrupted state and winds the turn down).

How to Verify

PYTHONPATH=. python -m pytest tests/tools/test_clarify_gateway.py -q -k unblocks_on_interrupt

The new test registers a pending clarify, signals the per-thread interrupt flag from a second thread 50ms later, and asserts wait_for_response returns None well before the 10s timeout. Verified as a genuine regression: reverting the fix makes the test block the full timeout and fail (11.3s, AssertionError); with the fix it returns in ~0.05s.

Test Plan

  • tests/tools/test_clarify_gateway.py — 24 passed (incl. new interrupt test)
  • tests/gateway/test_clarify_active_session_bypass.py — 25 passed with the above (no regression)
  • tests/gateway/test_clarify_progress_leak.py, test_telegram_clarify_buttons.py, test_clarify_thread_followup_not_swallowed.py — 9 passed

Risk Assessment

Low. The interrupt check only shortens a wait that would otherwise run to its full timeout; the return path and entry cleanup are byte-for-byte the same as an existing timeout exit. tools.interrupt is a stdlib-only module with no new dependencies, and the import is guarded (falls back to a no-op predicate, same as the heartbeat import).

`wait_for_response` (tools/clarify_gateway.py) blocked the agent thread
on a threading.Event with only an inactivity heartbeat poll — no check of
the per-thread interrupt flag. When the gateway's interrupt path
(`_interrupt_and_clear_session` → `request_hard_interrupt`) fires, it
sets the agent thread's interrupt bit via tools.interrupt, but the
blocked wait never observes it and the run's `finally` cleanup can't run
(the thread is stuck inside the tool), so the clarify stays blocked until
the full 600s timeout even though the user hit /stop. The user's typed
reply is also dead — see NousResearch#83889 (multiplexed Feishu groups; same
mechanism on every platform with text-fallback clarify).

Fix: poll `tools.interrupt.is_interrupted()` each 1s slice, exactly like
the existing inactivity heartbeat. On interrupt the wait returns None and
the entry is cleaned up through the same exit path as a timeout; the
gateway clarify callback now surfaces it as "[interrupted by user]"
instead of a misleading "did not respond within Nm".

Regression test signals the per-thread interrupt flag from a second
thread while a clarify is pending and asserts the wait unblocks well
before the timeout (verified: without the fix the test blocks the full
10s and fails).
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages duplicate This issue or pull request already exists labels Aug 11, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #25506, which already implements the same interrupt check in the clarify wait loop.

@yflmq001

Copy link
Copy Markdown
Contributor Author

You're right — this is a duplicate of #25506 (same interrupt check in the wait_for_response loop, opened 2026-05-14, which I missed when scanning for existing work; #84047's "no fix in flight" listing for family E is what misled me). Closing this PR.

One small addition from this PR that #25506 doesn't have, in case it helps there: when the interrupt unblocks the wait, the gateway's clarify callback (gateway/run.py) should return an explicit [interrupted by user] sentinel instead of the misleading [user did not respond within Nm] timeout message. I've posted that as a suggestion on #25506.

@yflmq001

Copy link
Copy Markdown
Contributor Author

Reopening: the duplicate rationale (vs #25506) no longer holds.

Keeping it open so maintainers can compare both lines; happy to close again if #84560 is preferred.

@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference, author can ignore or act on any point.

fix(gateway): unblock clarify wait on /stop or interrupt-mode messages

No blocking issues found. A few minor observations:

  1. The thread already notes this duplicates fix(clarify): check interrupt flag in gateway wait_for_response loop #25506 — worth closing/superseding one of them so two implementations of the same interrupt check in wait_for_response don't drift apart.
  2. tools/clarify_gateway.py::wait_for_response returns None both on interrupt and on timeout, and gateway/run.py::_clarify_callback_sync distinguishes the two by re-checking is_interrupted() after the call returns. If anything clears the per-thread interrupt flag between the wait loop's check and the caller's check, a cancelled run would be reported as a misleading "user did not respond within Xm" timeout. Returning a dedicated sentinel from wait_for_response (e.g. an Interrupted marker) would make the distinction race-free.
  3. Minor: the is_interrupted() import fallback to lambda: False means a broken/removed tools.interrupt silently re-introduces the original hang — the try/except swallows the failure. A one-time WARNING log on the fallback path would keep the failure visible.

@alt-glitch alt-glitch added needs-decision Awaiting maintainer decision before any implementation and removed duplicate This issue or pull request already exists labels Aug 16, 2026
ayushnangia added a commit to ayushnangia/hermes-agent that referenced this pull request Aug 17, 2026
Salvages NousResearch#25506 (liuhao1024): the wait loop now checks
tools.interrupt.is_interrupted() once per slice, so /stop and
interrupt-mode messages unblock a pending clarify instead of wedging the
agent thread for the full timeout (600s default) or forever in unlimited
mode. The end-of-run clear_session cleanup cannot fire while this wait
blocks, so the interrupt flag was set but never observed (NousResearch#83889 RC1).

Also folds yflmq001's callsite gap from the NousResearch#84119 review: an
interrupted wait now reports '[interrupted by user]' instead of the
misleading '[user did not respond within Nm]'.

Sweeper verdict on NousResearch#25506: keep_open, salvageability=high.
ayushnangia added a commit to ayushnangia/hermes-agent that referenced this pull request Aug 23, 2026
Salvages NousResearch#25506 (liuhao1024): the wait loop now checks
tools.interrupt.is_interrupted() once per slice, so /stop and
interrupt-mode messages unblock a pending clarify instead of wedging the
agent thread for the full timeout (600s default) or forever in unlimited
mode. The end-of-run clear_session cleanup cannot fire while this wait
blocks, so the interrupt flag was set but never observed (NousResearch#83889 RC1).

Also folds yflmq001's callsite gap from the NousResearch#84119 review: an
interrupted wait now reports '[interrupted by user]' instead of the
misleading '[user did not respond within Nm]'.

Sweeper verdict on NousResearch#25506: keep_open, salvageability=high.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] clarify deadlock + silent queue on multiplexed Feishu groups (follow-up to #25482, with code refs)

3 participants