Skip to content

Run Harbor computer-1 on a desktop its image already runs - #9

Merged
celve merged 3 commits into
LIN-1985/investigate-existing-recipes-forfrom
LIN-2149/train-harbor-computer-1-through
Oct 22, 2026
Merged

celve merged 3 commits into
LIN-1985/investigate-existing-recipes-forfrom
LIN-2149/train-harbor-computer-1-through

Conversation

@celve

@celve celve commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Lets Harbor's computer-1 (as Qwen3VLComputer1) run on a desktop its sandbox image already runs, such as CUA-Lite's GNOME template, and lets a launch pick the agent and verifier classes. This is what a run needs to train computer-1 on CUA-Lite's Lite.OSWorld tasks through the session server.

What changes

  • examples/experimental/harbor_cua/desktop_computer.py: DesktopQwen3VLComputer1 uses the image's display on :1 as it is.
    • It starts no Xvfb, XFCE, VNC or Chromium, judges liveness from the X socket alone, and never relaunches a browser after a failed action.
    • It waits settle_sec after each screen-changing action and can shrink screenshots in the sandbox (screenshot_width/height).
    • It caps the screenshots a request carries (keep_screenshots), counting what each prompt actually holds, so a JSON-parse retry keeps the current screen and the turn-cap final answer stays within the cap.
    • It refuses zoom (crop-relative clicks are mis-scaled for Qwen3-VL's grid) and the browser actions, which also leave its prompt.
    • An image without ImageMagick on its PATH supplies its own capture through _capture_script and session_class.
  • harbor_agent_function.py:
    • HARBOR_AGENT_IMPORT_PATH and HARBOR_AGENT_KWARGS pick and configure the agent class; a kwarg the binding sets is refused.
    • HARBOR_VERIFIER_IMPORT_PATH and HARBOR_VERIFIER_KWARGS run a Harbor verifier class in place of the task's tests/.
    • launch_common.harbor_env_vars forwards all four.

Verification

At 63279ff273, with the merge base 4d17c365df unchanged on the target:

  • Unit tests (Python 3.12, Harbor ac63e51a8f, no sglang):
    • test_desktop_computer.py + test_qwen_computer.py: 28 passed.
    • tests/fast/examples/experimental/harbor/: 104 passed, 1 failed. The failure is test_agentic_wiring_is_the_shipped_wiring, which imports sglang and fails without this change too.
  • Live, on 2×H200 with SGLang 3a93821 plus Keep message images on pretokenized chat requests sglang#5, CUA-Lite 2af6926 and the template lin2139-liteosworld:
    • Untouched desktops: 30 of 30 trials set up and scored 0 at 1280×720.
    • Oracle replays: 10 of 10 scored 1.
    • Three GRPO rounds with CUA-Lite's recipe at small scale, 30 turns:
      • engine weight versions 0, 2, 4;
      • trainer–engine gap 0.0064–0.0144 nats on all 6 steps;
      • TITO mismatch 0 on every round;
      • three evaluations ran.
    • Screening of the base model on an earlier commit: 16 tasks × 8, 4 of 16 groups mixed. CUA-Lite's own harness had 3 of 16.
    • Record: https://github.com/softlash/LIN-2149/blob/main/plans/harbor-liteosworld-run.md

Review

sol, two rounds; the second passed with no open P1, P2 or P3.

  • Round 1 found two P2s, both fixed here with tests:
    • a JSON-parse retry dropped the current screen;
    • the turn-cap fallback bypassed the screenshot cap.
  • Its P1 (my stage-4 gate) was met by a narrowed gate that sol accepted. The 16×8 probe is screening only, and readiness rests on the 3-round run on this commit.

Still owed to the human

The full 20-round run needs an 8-GPU node, which is Linyu's call. This PR does not certify that run.

celve added 3 commits October 22, 2026 16:52
- examples/experimental/harbor_cua/desktop_computer.py: DesktopQwen3VLComputer1 uses the sandbox's
  own display (no Xvfb, XFCE, VNC or Chromium), waits settle_sec after each screen-changing action,
  can shrink screenshots in the sandbox and cap the screenshots a request carries, and refuses zoom
  and the browser actions.
- harbor_agent_function.py: HARBOR_AGENT_IMPORT_PATH and HARBOR_AGENT_KWARGS pick and configure the
  agent class; HARBOR_VERIFIER_IMPORT_PATH and HARBOR_VERIFIER_KWARGS run a Harbor verifier class in
  place of a task's tests. launch_common forwards all four.
An image without ImageMagick overrides DesktopSession._capture_script, and an agent picks its session through session_class.
…inal answer included

A JSON-parse retry sends a text-only prompt, so the history now keeps the current screen; the turn-cap fallback's history is trimmed to leave room for the screen it attaches.
@celve
celve marked this pull request as ready for review October 22, 2026 11:20
Copilot AI balanced review requested due to automatic review settings October 22, 2026 11:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@celve
celve merged commit e951088 into LIN-1985/investigate-existing-recipes-for Oct 22, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-22T11:21:27.103649Z 63279ff Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants