Skip to content

fix(computer_use): expose dispatch param for all input actions - #57686

Open
Sahil-SS9 wants to merge 2 commits into
NousResearch:mainfrom
Sahil-SS9:fix/issue-57623-dispatch-param-all-actions
Open

fix(computer_use): expose dispatch param for all input actions#57686
Sahil-SS9 wants to merge 2 commits into
NousResearch:mainfrom
Sahil-SS9:fix/issue-57623-dispatch-param-all-actions

Conversation

@Sahil-SS9

Copy link
Copy Markdown
Contributor

Fixes #57623

Description

Add a dispatch parameter (background | foreground | auto) to the computer_use tool schema and wire it through _dispatch() for all mutating actions: click, double_click, right_click, middle_click, drag, scroll, type, key, and set_value.

Problem

On Windows, default background dispatch uses UIA hit-test → UIA Invoke, which silently fails on several common UI frameworks:

  • Explorer desktop icons (SysListView32) — drag operations essentially never work
  • Qt apps (WeChat, Telegram) — clicks are silently dropped

PR #55048 introduced a dispatch override for click variants, but it bypassed the backend methods and called _action directly — and critically, the dispatch parameter was never declared in the tool schema, so the model could never actually produce it. Drag, scroll, type, key, and set_value had no override path at all.

Fix

  1. Schema (schema.py): Declare dispatch as an optional string enum [background, foreground, auto] in the tool schema, applicable to all mutating actions.
  2. Dispatch layer (tool.py): Read args.get("dispatch") and pass it to every backend method (click, drag, scroll, type_text, key, set_value). When omitted or None, behavior is unchanged — the backend receives dispatch=None and uses its default (background).
  3. Backend ABC (backend.py): Add dispatch: Optional[str] = None to the abstract signatures for click, drag, scroll, type_text, key, and set_value.
  4. CuaDriverBackend (cua_backend.py): Accept dispatch in each method and pass it through to _action() as a field in the MCP tool args dict. Only included when non-None to avoid sending it to older cua-driver versions that don't recognize it.
  5. NoopBackend (tool.py): Updated to accept and record dispatch for test assertions.

When dispatch is omitted, every code path produces identical behavior to before — the parameter is purely additive.

Verification

  • pytest tests/tools/test_computer_use.py::TestSchema tests/tools/test_computer_use.py::TestDispatch tests/tools/test_computer_use.py::TestSafetyGuards -q → 42 passed (34 existing + 8 new dispatch routing tests)
  • pytest tests/tools/test_computer_use_capture_routing.py tests/tools/test_computer_use_vision_routing.py -q → 44 passed
  • Compile check on all 4 modified source files
  • The one pre-existing failure (TestCuaEnvironmentScrubbingModuleNotFoundError: No module named 'mcp') is unrelated and fails identically on main
  • New tests verify dispatch reaches the backend for click, drag, scroll, type, key, set_value, and defaults to None when omitted

@alt-glitch alt-glitch added type/bug Something isn't working comp/tools Tool registry, model_tools, toolsets platform/windows Native Windows-specific behavior or breakage P2 Medium — degraded but workaround exists sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows labels Jul 3, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the missing schema-to-backend wiring; current main still lacks it (tools/computer_use/schema.py:106-214, tools/computer_use/tool.py:369-432).

Problems

  • Issue #57623 asks dispatch="auto" to retry foreground delivery after a failed background attempt. This PR describes fallback in the new schema text, but its new paths only forward the value once; _action() makes one MCP call_tool() at tools/computer_use/cua_backend.py:2080-2101.
  • The new tests assert _NoopBackend routing only. They do not exercise the actual MCP payload construction in CuaDriverBackend for the affected action tools.

Suggested changes

  • Either implement the issue's verified fallback behavior, or narrow auto to an explicitly verified cua-driver pass-through contract.
  • Add mocked-session CuaDriverBackend tests for click, drag, scroll, type_text, both key routes, and set_value, including omission behavior.

Automated hermes-sweeper review.

Comment thread tools/computer_use/schema.py Outdated
"model and does not steal focus. `foreground` uses real "
"foreground input where supported, which can be more "
"reliable for Windows Explorer desktop icons and Qt apps "
"such as Telegram/WeChat but may disrupt the user. `auto` "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

auto is described as choosing or falling back, but this PR only forwards the string once to cua-driver. Issue #57623 specifically asks for verified background-to-foreground retry; please implement that behavior or narrow this description to a documented pass-through contract.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjusted: chose the documented pass-through contract option. The auto dispatch description now states it is passed through to the cua-driver, which chooses an input-delivery mode on the agent's behalf; the agent does not resolve it locally, so the resulting mode (and any fallback) is driver-dependent. This matches the PR's actual behaviour (it only forwards the dispatch string) without overclaiming.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
…esearch#57623)

Add  parameter (background|foreground|auto) to the
computer_use schema and wire it through _dispatch() for all mutating
actions: click, double_click, right_click, middle_click, drag, scroll,
type, key, and set_value.

Previously the dispatch override was only discussed in PR NousResearch#55048
(click variants) and was never actually declared in the tool schema,
so the model could never produce it. This left drag, scroll, type, key,
and set_value with no way to request foreground delivery on Windows
apps (Explorer SysListView32, Qt clients like WeChat/Telegram) where
background dispatch silently fails.

The parameter flows through the backend ABC, CuaDriverBackend, and
NoopBackend. When omitted or None, behavior is unchanged (background
default).
@Sahil-SS9
Sahil-SS9 force-pushed the fix/issue-57623-dispatch-param-all-actions branch from 750af31 to 14402ab Compare August 5, 2026 12:53
…contract

The `auto` dispatch value is forwarded verbatim to cua-driver; the agent
does not resolve or retry it locally. Narrow the schema description to a
documented pass-through contract rather than implying the backend chooses
or falls back in a way the agent cannot guarantee.
@Sahil-SS9
Sahil-SS9 force-pushed the fix/issue-57623-dispatch-param-all-actions branch from 14402ab to 21c2a78 Compare August 5, 2026 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]:computer_use background dispatch silently fails on certain Windows UI frameworks — need dispatch param for all actions + auto-fallback**

3 participants