Skip to content

fix(bot-screen): restore the configured screen size when a human hands control back - #125329

Open
Julientalbot wants to merge 1 commit into
NousResearch:mainfrom
Julientalbot:fix/bot-screen-restore-geometry
Open

Julientalbot wants to merge 1 commit into
NousResearch:mainfrom
Julientalbot:fix/bot-screen-restore-geometry

Conversation

@Julientalbot

Copy link
Copy Markdown

What does this PR do?

The lease holder may resize the Bot Screen. Xvnc runs -AcceptSetDesktopSize, and the RFB filter lets the holder send SetDesktopSize, which noVNC does with resizeSession (for example a phone viewer, so that sites render at the phone's size). Xvnc keeps that size after hand-back. The agent then goes on capturing and clicking a phone-sized desktop, and bot_desktop.geometry is no longer what the agent gets.

With this change, handing control back restores the configured size.

Related Issue

Related: #108914 (Bot Screen), #92524 (taking over from a hosted instance, where phone viewers are the norm).

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • tools/bot_desktop/runtime.py: new restore_geometry(profile_home=None).
    • If the screen is up and not at bot_desktop.geometry, it runs xrandr -s WxH.
    • Xvnc drops the configured mode from its list after a SetDesktopSize, so the function falls back to --newmode / --addmode / --output … --mode.
    • It is best effort: 5 s timeout per call, never raises, and does nothing when the screen is down or already at size.
    • profile_home selects the profile's screen through the HERMES_HOME override, because the RFB bridge serves several profiles from one process.
  • tools/bot_desktop/lease.py: release() calls it when control actually returns to the agent. It is not called for an ignored stale-viewer release, nor when the agent already holds. This is the single human→agent path, whichever process performs it (dashboard bridge, TUI gateway, CLI).
  • tests/tools/test_bot_desktop_geometry.py: checks that:
    • the restore runs exactly once per real hand-back;
    • the mode is re-added when Xvnc dropped it;
    • a listed mode is used directly;
    • nothing happens at size or with the screen down;
    • the requested profile is the one queried.

How to Test

Live evidence, on a real Bot Screen: TigerVNC Xvnc + Xfce from the -desktop image, with this patch applied to its runtime.py/lease.py. The viewer is a phone-sized noVNC (resizeSession = true), followed by its hand-back.

Moment Before After
Before the takeover 1440 × 900 1440 × 900
During the takeover (viewer 393 × 607) 393 × 607 393 × 607
After hand-back 393 × 607; xrandr -s 1440x900 fails with "Size 1440x900 not found in available modes" 1440 × 900

To reproduce the bug: take over with a noVNC client that has resizeSession on, from a small window, hand back, then run xrandr -q on the bot's display.

Note for #121169: restore_geometry only goes through published_env() and geometry(). With terminal placement it will have to run xrandr inside the sandbox, like the rest of runtime. I'm happy to adapt whichever lands second.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix
  • tests/tools/test_bot_desktop_geometry.py and tests/tools/test_bot_desktop_lease.py: 13 passed, 1 skipped (an existing Linux-only case, on macOS); live test on Debian as above
  • I've added tests for my changes
  • I've tested on my platform: macOS 26 (unit), Debian 13 VM with the -desktop image (live)

Documentation & Housekeeping

  • Documentation: docstrings; no user-facing change beyond the fix
  • cli-config.yaml.example: N/A (uses the existing bot_desktop.geometry)
  • CONTRIBUTING.md / AGENTS.md: N/A
  • Cross-platform impact: Linux-only path (Bot Screen); a no-op elsewhere

🤖 Generated with Claude Code

https://claude.ai/code/session_01NNnn19Se2Kq86QQrAuGEZr

…s control back

The lease holder may resize the screen (Xvnc runs -AcceptSetDesktopSize; noVNC does it with
resizeSession, e.g. a phone viewer) and Xvnc keeps that size after hand-back, so the agent went on
capturing and clicking a phone-sized desktop. Measured on a real Bot Screen: 1440x900 before a phone
takeover, 393x607 during, still 393x607 after hand-back; 1440x900 after with this change.

lease.release() now calls runtime.restore_geometry(profile) when control actually returns to the
agent: xrandr -s to the configured size, or, since Xvnc drops that mode from its list after a
SetDesktopSize, --newmode/--addmode/--output. Best effort, never raises; no-op when already at size
or when the screen is down.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNnn19Se2Kq86QQrAuGEZr

kvnloo commented Sep 27, 2026

Copy link
Copy Markdown

Independent exact-head check on d7ef7fc0621e43305324199af0e4fe90af292b84: I mirrored this patch in kvnloo#188 and ran tests/tools/test_bot_desktop_geometry.py.

  • patched: 5 passed
  • negative control: 5 failed
  • restored patch: 5 passed

That directly covers restoring configured Bot Screen geometry after human hand-back, including the mode re-add path.

@Enough1122

Copy link
Copy Markdown

AI code review — automated follow-up for reference; not a maintainer.

At head d7ef7fc0621e43305324199af0e4fe90af292b84 (base e54d55052113f24f3edf6ba942ea610e90147e4f).

1. P1 — the mode re-add retargets the first connected output, which is not the bot's screen.
tools/bot_desktop/runtime.py:495 picks the output via re.search(r"^(\S+) connected", query.stdout, re.M) and feeds it to --addmode/--output (:502-503) — the first connected line xrandr prints. But the screen this PR is about is Xvnc's VNC-0 (-AcceptSetDesktopSize; hand-back path hermes_cli/web_routers/display.py:214). With a monitor also attached, xrandr lists that one first, so the mode is added to and applied to the wrong output: the resize either fails (returncode != 0 → returns False, phone-sized desktop survives hand-back) or reconfigures the operator's real display while the bot screen stays wrong.

Probe on the real head source (subprocess stubbed; provenance asserted via __file__ + sha256):

case xrandr -q shows --output target
control VNC-0 connected only VNC-0
multi-output HDMI-1 connected, then VNC-0 connected HDMI-1

Same function, same geometry, diverging target. The new test only feeds single-output fixtures (tests/tools/test_bot_desktop_geometry.py:35-37), so this branch is uncovered. Prefer selecting the VNC output explicitly to the first match — the "first match wins" shape #125048 just corrected.

2. P2 — the restore runs up to four blocking subprocess.run calls on the asyncio event loop.
tools/bot_desktop/runtime.py:483 calls subprocess.run(..., timeout=5) up to four times (:485, :491, :499, :502, :503), and lease.release is called synchronously from websocket teardown (hermes_cli/web_routers/display.py:214, inside async def _bridge). A slow or wedged xrandr therefore stalls the gateway loop for up to ~20s — the loop carrying every other bot's display bridge. Base had no subprocess here; this diff adds it. Off-loop execution (or a tighter timeout) seems cheap for a cosmetic resize.

Unverified / please confirm: both findings come from the post-image plus a stubbed-subprocess probe on a Windows host; I could not run a real Xvnc, so the line ordering of xrandr -q on a real multi-head Linux box is the one input worth confirming — it is what finding 1 rests on. Target selection, call count and call site are read straight from the post-image.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/desktop Electron desktop app (apps/desktop/*) comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have tool/browser Browser automation (CDP, Playwright) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants