fix(computer_use): reset _started in _lifecycle_coro finally block + expose dispatch param by heidis168 · Pull Request #55048 · NousResearch/hermes-agent

heidis168 · 2026-06-29T17:37:32Z

Summary

Two fixes for computer_use on Windows — one for the session hang that
made list_apps / capture return empty after the MCP session died, and
one for the click being a no-op on Qt apps like WeChat and Telegram.

Bug 1: session hang (the original report)

When the cua-driver MCP connection drops, _lifecycle_coro exits but
_started stays True. The next call to list_apps / capture then
hits _require_started() and hangs forever instead of reconnecting.
This made the wrapper look broken from the very first MCP blip.

Fix — _lifecycle_coro.finally now sets _started = False on exit,
so subsequent calls re-enter start() and rebuild the session.

Bug 2: click is a no-op on Qt apps (discovered while testing #1)

Default background dispatch walks UIA hit-test → UIA Invoke, which Qt
clients silently drop. Even when the daemon returned "Posted click to
pid 460592", the WeChat window never opened the chat. Foreground dispatch
sends real SendInput events — works on every Windows app — but the
wrapper was hiding the override.

Fix — wire dispatch through _dispatch(). Both element- and
coordinate-based clicks now honour it:

click(element=42, dispatch="foreground")          # SendInput via cua-driver
click(coordinate=[300, 500], dispatch="foreground") # raw pixel click

Without this fix, the wrapper could observe any Windows desktop but
couldn't actually drive most non-Microsoft apps.

Operator note

Foreground dispatch requires Hermes to be running as Administrator on
Windows. cua-driver needs UIAAccess integrity to bypass the Windows
foreground lock — without it, SendInput events are silently dropped by
the desktop. Background dispatch works without elevation but only delivers
to apps that implement UIA InvokePattern.

The computer-use-setup skill (heidis168/hermes-tools) documents the
full Windows setup, including the elevation requirement and the
mcp_discovery_timeout fix.

Verification

End-to-end on Windows 11 24H2, Hermes running as Administrator:

Target	Path	Result
`list_apps`	default	12 apps returned (was: empty after MCP blip)
`capture(app=Telegram)`	default	200+ elements, window state, screenshots
`click(element=196, dispatch=foreground)`	Telegram chat list	chat switched to "太子爺高端選爺"
`click(element=81, dispatch=foreground)`	WeChat chat list	chat switched to "好兄弟，挺住💪🏻(3)"
`type_text`, `hotkey`, `scroll`	default	all working with foreground-eligible paths

Repro of original bug — kill cua-driver daemon while a Hermes session is
active, then call list_apps(). Without this fix the call hangs; with
this fix it transparently reconnects (see PR diff for the exact line).

When the cua-driver MCP connection drops or the lifecycle coroutine exits abnormally, _started remains True even though _session is None. Subsequent calls hang because the wrapper believes the session is alive. This sets self._started = False in the finally block so callers either get an explicit 'session not started' error or trigger a reconnect via the existing _is_closed_session_error retry path, instead of waiting indefinitely for a session that no longer exists.

Companion to the previous _lifecycle_coro fix: when the lifecycle coroutine dies (sets _started = False in finally), the next list_apps call returned an empty apps list instead of triggering a reconnect, because list_apps treats the empty data path as 'no apps running'. Add _ensure_session_alive() that re-runs _session.start() when _started is False. start() is idempotent (self-loop guarded at the top), so this only fires when the session genuinely died, and callers get a fresh session + session id without needing to know the session ever dropped. E2E verified on Windows 11 + cua-driver 0.6.8: after cua-driver daemon restart, list_apps now returns 11 running apps instead of 0.

…ound SendInput Companion to the session-hang fix (NousResearch#55048). The default background dispatch walks UIA hit-test → Invoke, which Qt clients (WeChat, Telegram) silently drop. Foreground dispatch sends real SendInput events — works on every Windows app — but the wrapper was hiding the override. Wiring `dispatch` through _dispatch() lets the agent pick the path explicitly. Element- and coordinate-based clicks both honour it. This unlocks the whole Windows desktop to the agent — without it, click() on Qt apps was a no-op even though the daemon returned OK. Note for operators: foreground dispatch requires Hermes to be running as Administrator (cua-driver needs UIAccess integrity to bypass the Windows foreground lock). Background dispatch works without elevation but only delivers to apps that implement UIA InvokePattern.

teknium1

Thanks for isolating the lifecycle-state inconsistency; current main still clears _session without resetting _started in tools/computer_use/cua_backend.py:658-663.

Problems

The new recovery is only called by list_apps() (PR cua_backend.py:1406). capture() still calls the session directly (tools/computer_use/cua_backend.py:1261 on main), so after _started is reset it will fail instead of reconnecting.
dispatch is not declared in tools/computer_use/schema.py:29-214, so the model cannot discover the new argument.
The explicit-dispatch branch at PR tools/computer_use/tool.py:385-405 bypasses ComputerUseBackend.click() via private _action() and drops modifiers from its payload.

Suggested changes

Centralize recovery at the session boundary and add forced-lifecycle-exit tests for both list_apps() and capture().
Expose a constrained schema field and thread it through the public backend interface, preserving modifiers.

Automated hermes-sweeper review.

teknium1 · 2026-07-15T09:04:59Z

+            # If the lifecycle coroutine exited without stop() being called
+            # (e.g. MCP connection dropped), mark the session as not started
+            # so callers don't hang on a dead session.
+            self._started = False


Resetting _started fixes the inconsistent state, but only list_apps() invokes the new recovery helper. capture() still calls the session directly, so it will now raise “session not started” after this path instead of reconnecting. Put recovery at the session boundary or cover every direct caller.

teknium1 · 2026-07-15T09:04:59Z

        x, y = (coord[0], coord[1]) if coord and coord[0] is not None else (None, None)
+        dispatch = args.get("dispatch")
+        modifiers = args.get("modifiers")
+        if dispatch:


dispatch is not declared in tools/computer_use/schema.py, so this branch is not exposed to models. Thread a constrained field through the schema and public backend interface rather than relying on the private Cua-only _action() path.

teknium1 · 2026-07-15T09:04:59Z

+                        "element_index": element,
+                        "window_id": backend._active_window_id,
+                        "button": button or "left",
+                        "dispatch": dispatch,


This raw payload omits modifiers, although they are read on line 384 and passed by the normal backend.click() path. A foreground modifier-click will silently lose its modifiers.

teknium1 · 2026-07-18T21:51:37Z

Both bugs here are addressed:

Bug 1 (session hang — _started not reset on MCP drop): fixed in fix(computer_use): reconnect a dead cua-driver session instead of hanging (#55048) #67138. _lifecycle_coro's finally now resets _started=False on any exit and call_tool() rebuilds a dead session before use, so a connection drop reconnects instead of hanging _require_started() on a None session.
Bug 2 (click is a no-op on Qt apps — need foreground dispatch): shipped in feat(computer_use): follow cua-driver's verify → escalate ladder (#67052) #67123 as delivery_mode="foreground", wired through all input actions (click/drag/scroll/type/key), capability-gated with a foreground_unsupported refusal on old drivers. The operator note about foreground requiring elevated integrity on Windows is preserved in the skill guidance.

Will close once #67138 merges. Thanks for the two-part diagnosis and the draft.

…ging (#67138) Bug 1 of #55048: when the MCP connection dropped (driver crash / restart), _lifecycle_coro exited but left _started=True, so the next list_apps/capture passed _require_started() and then operated on a None session — hanging forever instead of reconnecting. - _lifecycle_coro's finally now resets _started=False on ANY exit, so a dead session is re-enterable (idempotent no-op on the normal stop() path; atomic bool write, safe from the bridge-loop thread without the lock stop() holds). - call_tool() re-enters start() when the session isn't active, rebuilding it before the call. The start_session/end_session handshake (driven by start()/ stop() themselves) is exempted so bootstrap doesn't recurse. Tests: two cases in test_computer_use_delivery_ladder.py — finally resets _started, and call_tool restarts a dead session exactly once. Full computer_use suite green (233). Refs #55048 (Bug 1). Bug 2 (expose foreground dispatch) is covered by the delivery_mode work in #67123.

teknium1 · 2026-07-18T22:07:51Z

Thanks for this — your two-part diagnosis was spot on, and both fixes have now landed on main:

Bug 1 (session hang — _started not reset in _lifecycle_coro's finally): fixed in fix(computer_use): reconnect a dead cua-driver session instead of hanging (#55048) #67138 (commit 7a43ab042). _lifecycle_coro now resets _started=False on any exit, and call_tool() rebuilds a dead session before use, so a dropped MCP connection reconnects instead of hanging _require_started() on a None session.
Bug 2 (click no-op on Qt apps — expose foreground dispatch): shipped in feat(computer_use): follow cua-driver's verify → escalate ladder (#67052) #67123 (commit 9d6d77283) as delivery_mode="foreground", wired through all input actions (click/drag/scroll/type/key), capability-gated with a structured foreground_unsupported refusal on drivers that predate it. Your operator note (foreground needs elevated integrity on Windows) is reflected in the skill guidance.

Both were part of the broader cua-driver verify→escalate ladder work (#67052). Closing this as superseded by those merges — but the credit for surfacing both bugs, with a clear repro path, is yours. Appreciated.

…esearch#57623) Add parameter (background|foreground|auto) to the computer_use schema and wire it through _dispatch() for all mutating actions: click, double_click, right_click, middle_click, drag, scroll, type, key, and set_value. Previously the dispatch override was only discussed in PR NousResearch#55048 (click variants) and was never actually declared in the tool schema, so the model could never produce it. This left drag, scroll, type, key, and set_value with no way to request foreground delivery on Windows apps (Explorer SysListView32, Qt clients like WeChat/Telegram) where background dispatch silently fails. The parameter flows through the backend ABC, CuaDriverBackend, and NoopBackend. When omitted or None, behavior is unchanged (background default).

…ging (NousResearch#67138) Bug 1 of NousResearch#55048: when the MCP connection dropped (driver crash / restart), _lifecycle_coro exited but left _started=True, so the next list_apps/capture passed _require_started() and then operated on a None session — hanging forever instead of reconnecting. - _lifecycle_coro's finally now resets _started=False on ANY exit, so a dead session is re-enterable (idempotent no-op on the normal stop() path; atomic bool write, safe from the bridge-loop thread without the lock stop() holds). - call_tool() re-enters start() when the session isn't active, rebuilding it before the call. The start_session/end_session handshake (driven by start()/ stop() themselves) is exempted so bootstrap doesn't recurse. Tests: two cases in test_computer_use_delivery_ladder.py — finally resets _started, and call_tool restarts a dead session exactly once. Full computer_use suite green (233). Refs NousResearch#55048 (Bug 1). Bug 2 (expose foreground dispatch) is covered by the delivery_mode work in NousResearch#67123.

heidis168 added 2 commits June 30, 2026 01:35

alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/tools Tool registry, model_tools, toolsets platform/windows Native Windows-specific behavior or breakage labels Jun 29, 2026

heidis168 changed the title ~~fix(computer_use): reset _started in _lifecycle_coro finally block~~ fix(computer_use): reset _started in _lifecycle_coro finally block + expose dispatch param Jun 29, 2026

fishlikeX mentioned this pull request Jul 3, 2026

[Bug]:computer_use background dispatch silently fails on certain Windows UI frameworks — need dispatch param for all actions + auto-fallback** #57623

Open

Sahil-SS9 mentioned this pull request Jul 3, 2026

fix(computer_use): expose dispatch param for all input actions #57686

Open

teknium1 reviewed Jul 15, 2026

View reviewed changes

f-trycua mentioned this pull request Jul 18, 2026

computer_use cannot follow cua-driver's verify/escalate ladder #67052

Closed

10 tasks

teknium1 mentioned this pull request Jul 18, 2026

fix(computer_use): reconnect a dead cua-driver session instead of hanging (#55048) #67138

Merged

teknium1 closed this Jul 18, 2026

Navigation Menu

Uh oh!

fix(computer_use): reset _started in _lifecycle_coro finally block + expose dispatch param - #55048

fix(computer_use): reset _started in _lifecycle_coro finally block + expose dispatch param#55048
heidis168 wants to merge 3 commits into
NousResearch:mainfrom
heidis168:fix/computer-use-session-hang

heidis168 commented Jun 29, 2026 •

edited

Loading

Uh oh!

teknium1 left a comment

Uh oh!

teknium1 Jul 15, 2026

Uh oh!

teknium1 Jul 15, 2026

Uh oh!

teknium1 Jul 15, 2026

Uh oh!

teknium1 commented Jul 18, 2026

Uh oh!

teknium1 commented Jul 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

heidis168 commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Bug 1: session hang (the original report)

Bug 2: click is a no-op on Qt apps (discovered while testing #1)

Operator note

Verification

Uh oh!

teknium1 left a comment

Choose a reason for hiding this comment

Problems

Suggested changes

Uh oh!

teknium1 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 commented Jul 18, 2026

Uh oh!

teknium1 commented Jul 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

heidis168 commented Jun 29, 2026 •

edited

Loading