fix(desktop): stop Computer Use from silently acting on the wrong machine over a remote gateway - #90374
Open
OutThisLife wants to merge 3 commits into
Open
fix(desktop): stop Computer Use from silently acting on the wrong machine over a remote gateway#90374OutThisLife wants to merge 3 commits into
OutThisLife wants to merge 3 commits into
Conversation
…er machine read_window_below round-trips to the desktop renderer, so it describes the user's screen. computer_use spawns cua-driver on whatever host the gateway runs on. Those are the same computer on a local backend and two different ones on SSH, URL, or cloud — and nothing said so, so on a remote gateway with a display the agent would identify a window on your Mac and then click the same-named app over there. Only the client can answer this: behind an SSH tunnel the backend sees a loopback peer either way. So the renderer stamps the bare flag onto the window.read answer and the backend names itself, resolved when the agent asks rather than stamped on the session. Local sessions send nothing, so the common case costs no tokens.
…ng machine The per-turn HUD note tells the model to carry work out in the app behind the strip, naming computer_use as the way. On a remote gateway that instruction is wrong, and it is our own prompt that issues it. The note is built before any tool call, so it cannot know. Point it at the answer that can — read_window_below's agent_host — rather than carrying a locality bit that would be stale by the time anyone read it back.
The card reads /api/tools/computer-use/status from the gateway host and frames it as "this machine", "your Mac", "drives your desktop". On a remote backend every one of those is about a computer the user isn't sitting at, so a green "ready" told them Computer Use worked on their own screen.
Contributor
૮ >ﻌ< ა ci reviewran on 96e411c — fix(desktop): say which machine the Computer Use card is des
|
alt-glitch
added
type/bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When Desktop drives a remote gateway, the HUD and
computer_useare on two different computers and nothing says so. This makes the gap visible — to the agent, and to the person reading the Computer Use card.The bug
read_window_belowround-trips to the desktop renderer (tools/read_window_tool.py), so it describes the window on the user's screen.computer_useresolves a process-global backend that spawns cua-driver on the gateway host (tools/computer_use/tool.py:336-342). On a local backend those are one machine; on SSH, URL, or cloud they are two.Our own per-turn HUD note then closes the loop in the wrong direction:
So on a remote backend that has a display, the agent identifies Figma on your Mac and clicks the same-named app on the other box. Where the remote host has no display the tool is stripped by its
check_fnand the user just gets silence — reported on #61507 as "computer_use never appears, enabling it and restarting doesn't help".The fix
Only the client can answer "is the agent on this machine" — behind an SSH tunnel the backend sees a loopback peer either way. So the renderer stamps a bare flag onto the existing
window.readanswer and the backend names itself. Resolved when the agent asks rather than stamped on the session, for the reason #82187 landed on: a transcript outlives whoever was watching when it was written.agent-locality.tsowns the predicate, the host label, and the stamp. Local sessions carry nothing, so the common case costs no tokens.read_window_belowturns the flag into anagent_hostthe model can act on, documented in the schema.This does not make Computer Use work across the gap — that is the bridge in #61507 / #61311, which is a much larger change. This is the part that should not wait for it.
Test plan
scripts/run_tests.sh tests/agent/test_prompt_builder.py tests/agent/test_platform_hint_desktop.py tests/tui_gateway/ tests/tools/test_read_window_tool.py— 608 passednpx vitest run src/lib/agent-locality.test.ts— 10 passed, covering SSH identity over forwarded loopback port, cloud/URL labels, and that no host detail crosses the wirenpm run typecheck, eslint, prettier clean on changed filesruff check,git diff --checkclean