fix(computer_use): expose dispatch param for all input actions - #57686
fix(computer_use): expose dispatch param for all input actions#57686Sahil-SS9 wants to merge 2 commits into
Conversation
teknium1
left a comment
There was a problem hiding this comment.
Thanks for tracing the missing schema-to-backend wiring; current main still lacks it (tools/computer_use/schema.py:106-214, tools/computer_use/tool.py:369-432).
Problems
- Issue #57623 asks
dispatch="auto"to retry foreground delivery after a failed background attempt. This PR describes fallback in the new schema text, but its new paths only forward the value once;_action()makes one MCPcall_tool()attools/computer_use/cua_backend.py:2080-2101. - The new tests assert
_NoopBackendrouting only. They do not exercise the actual MCP payload construction inCuaDriverBackendfor the affected action tools.
Suggested changes
- Either implement the issue's verified fallback behavior, or narrow
autoto an explicitly verified cua-driver pass-through contract. - Add mocked-session CuaDriverBackend tests for click, drag, scroll, type_text, both key routes, and set_value, including omission behavior.
Automated hermes-sweeper review.
| "model and does not steal focus. `foreground` uses real " | ||
| "foreground input where supported, which can be more " | ||
| "reliable for Windows Explorer desktop icons and Qt apps " | ||
| "such as Telegram/WeChat but may disrupt the user. `auto` " |
There was a problem hiding this comment.
auto is described as choosing or falling back, but this PR only forwards the string once to cua-driver. Issue #57623 specifically asks for verified background-to-foreground retry; please implement that behavior or narrow this description to a documented pass-through contract.
There was a problem hiding this comment.
Adjusted: chose the documented pass-through contract option. The auto dispatch description now states it is passed through to the cua-driver, which chooses an input-delivery mode on the agent's behalf; the agent does not resolve it locally, so the resulting mode (and any fallback) is driver-dependent. This matches the PR's actual behaviour (it only forwards the dispatch string) without overclaiming.
…esearch#57623) Add parameter (background|foreground|auto) to the computer_use schema and wire it through _dispatch() for all mutating actions: click, double_click, right_click, middle_click, drag, scroll, type, key, and set_value. Previously the dispatch override was only discussed in PR NousResearch#55048 (click variants) and was never actually declared in the tool schema, so the model could never produce it. This left drag, scroll, type, key, and set_value with no way to request foreground delivery on Windows apps (Explorer SysListView32, Qt clients like WeChat/Telegram) where background dispatch silently fails. The parameter flows through the backend ABC, CuaDriverBackend, and NoopBackend. When omitted or None, behavior is unchanged (background default).
750af31 to
14402ab
Compare
…contract The `auto` dispatch value is forwarded verbatim to cua-driver; the agent does not resolve or retry it locally. Narrow the schema description to a documented pass-through contract rather than implying the backend chooses or falls back in a way the agent cannot guarantee.
14402ab to
21c2a78
Compare
Fixes #57623
Description
Add a
dispatchparameter (background|foreground|auto) to thecomputer_usetool schema and wire it through_dispatch()for all mutating actions: click, double_click, right_click, middle_click, drag, scroll, type, key, and set_value.Problem
On Windows, default background dispatch uses UIA hit-test → UIA Invoke, which silently fails on several common UI frameworks:
PR #55048 introduced a
dispatchoverride for click variants, but it bypassed the backend methods and called_actiondirectly — and critically, thedispatchparameter was never declared in the tool schema, so the model could never actually produce it. Drag, scroll, type, key, and set_value had no override path at all.Fix
schema.py): Declaredispatchas an optional string enum[background, foreground, auto]in the tool schema, applicable to all mutating actions.tool.py): Readargs.get("dispatch")and pass it to every backend method (click, drag, scroll, type_text, key, set_value). When omitted orNone, behavior is unchanged — the backend receivesdispatch=Noneand uses its default (background).backend.py): Adddispatch: Optional[str] = Noneto the abstract signatures for click, drag, scroll, type_text, key, and set_value.cua_backend.py): Acceptdispatchin each method and pass it through to_action()as a field in the MCP tool args dict. Only included when non-None to avoid sending it to older cua-driver versions that don't recognize it.tool.py): Updated to accept and recorddispatchfor test assertions.When
dispatchis omitted, every code path produces identical behavior to before — the parameter is purely additive.Verification
pytest tests/tools/test_computer_use.py::TestSchema tests/tools/test_computer_use.py::TestDispatch tests/tools/test_computer_use.py::TestSafetyGuards -q→ 42 passed (34 existing + 8 new dispatch routing tests)pytest tests/tools/test_computer_use_capture_routing.py tests/tools/test_computer_use_vision_routing.py -q→ 44 passedTestCuaEnvironmentScrubbing—ModuleNotFoundError: No module named 'mcp') is unrelated and fails identically onmaindispatchreaches the backend for click, drag, scroll, type, key, set_value, and defaults toNonewhen omitted