feat(sandbox)!: boot the same Windows containerDisk locally and in Fleet cloud - #3118
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
e9a11f4 to
3442f8c
Compare
📦 Publishable packages changedThis comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Ask the release owner to apply |
3442f8c to
6cb5316
Compare
📦 Publishable packages changedThis comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Ask the release owner to apply |
6cb5316 to
c305da5
Compare
📦 Publishable packages changedThis comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Ask the release owner to apply |
c305da5 to
bfa5a7f
Compare
📦 Publishable packages changedThis comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Ask the release owner to apply |
|
@copilot rebase this with latest main |
Done — the branch is now current with |
1885c94 to
de06b75
Compare
📦 Publishable packages changedThis comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Ask the release owner to apply |
…eet cloud
BREAKING CHANGE: two Windows defaults move from "11" to "2022". `Image.windows()`
now defaults to `version="2022"` (Windows Server 2022), and cua-cli's bare
`windows` image alias follows it. Callers of either bare default previously got
a Windows 11 evaluation-ISO install locally and NotImplementedError on Fleet
cloud; they now get the pinned Server 2022 containerDisk, which works on both
paths. `Image.windows("11")` and `windows:11` still mean client Windows 11 and
are unchanged.
The Linux row is repinned to public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04 to match
bit-identical, so this is a string change; Linux was not re-tested here. Three
user-facing docs that still pointed at private ECR were moved to the public
refs as well.
`Image.windows(...)` had no registry counterpart, so the Fleet cloud path
rejected it and the local QEMU path fell through to `_build_windows_base`,
which downloads a ~6 GB Windows ISO and runs an unattended install. Both paths
now resolve the same pinned KubeVirt containerDisk, so a local run and a cloud
run boot identical bytes.
Generalises `cloud_registry_image` into a `BUILTIN_REGISTRY_IMAGES` descriptor
table keyed on (os_type, distro, version, kind), replacing the single hardcoded
Linux branch. Linux keeps its existing pin; Windows Server 2022 is the new row.
The image is `public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3` — an
anonymously pullable mirror, so the built-in Windows image needs no registry
credentials. It is an OCI index whose children are the linux/amd64
containerDisk and a buildx provenance attestation.
Selecting an oras auth backend now asks the registry instead of guessing.
Private ECR challenges with Basic, where oras' token backend raises "Cannot
respond to request for authentication"; public.ecr.aws and ghcr.io challenge
with Bearer, where basic auth has no credential to send. Since the pinned Linux
image is on private ECR and the pinned Windows image is on public ECR, no fixed
backend works for both, so `_detect_auth_backend` reads the scheme off the
registry's /v2/ endpoint.
Also fixes UEFI firmware discovery, without which the local Windows path
cannot boot on current Ubuntu. The bare-metal runtime looked only for
`/usr/share/OVMF/OVMF_CODE.fd` (Ubuntu 24.04 ships `OVMF_CODE_4M.fd`), so it
found no firmware and added no pflash drives. The WSL-hosted runtime had a
narrower defect: it did try the 4M file first, but chose the code file and the
varstore in two independent loops, so a host with 2M code and 4M vars got a
mismatched pair, and with no vars present at all it fabricated a zero-filled
varstore valid at neither size. Both now take code and vars from the same
candidate entry.
Verifying that on a real Windows host surfaced a second WSL bug: the session
overlay is created by the Windows-side builder, so the backing path recorded in
the qcow2 is a Windows path, and QEMU inside WSL parsed the drive letter as a
URI scheme -- "Could not open backing file: Unknown protocol 'C'". It now
repoints the overlay with a metadata-only `qemu-img rebase -u`. This affected
any layered or base-image disk on WSL; it was previously unreachable because
Image.windows() had no disk to overlay.
Windows examples that mean "give me a Windows sandbox" move to the bare
`Image.windows()`: the two sandbox_sdk integration examples, the CLI's MCP
`create_sandbox` tool, the CLI's bare `windows` alias, and the images guide.
`tests/test_runtime.py` deliberately stays on `Image.windows("11")` — it is the
remaining coverage of the Windows 11 ISO-install path.
Verified end to end on a bare-metal host with KVM, from a clean HOME with no
docker credentials present:
[e2e] docker config = /home/ubuntu/e2e-home/.docker/config.json exists=False
[e2e] image = Image(windows/windows:2022, kind=vm, 0 layers)
[e2e] default windows() = public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3
[e2e] chosen auth = token
Cached containerDisk public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3
Created overlay: .../sessions/e2e-windows-public.qcow2
Bare-metal QEMU VM e2e-windows-public is ready
[e2e] pull + boot in 446s
$ ver -> Microsoft Windows [Version 10.0.20348.587]
$ hostname -> DOCKERW-K5E4442
screenshot: 170369 bytes, PNG
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
de06b75 to
6e18c03
Compare
📦 Publishable packages changedThis comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Ask the release owner to apply |
Changes default local behaviour for Image.linux(). Image.linux() is a VM, but auto-selection sent it to Docker-wrapped QEMU, and resolve_image() hands that the trycua/cua-xfce *container* image regardless of kind. So asking for a VM produced a container byte-identical to Image.linux(kind='container'), never touched the pinned containerDisk, and then could not become ready at all because that image binds computer-server to 127.0.0.1. A Linux VM now boots the same containerDisk Fleet cloud boots, under bare-metal QEMU — the parity #3091 and #3118 were built for, which until now was reachable only by passing runtime=QEMURuntime(mode="bare-metal") by hand. Verified end to end with the plain documented invocation (Sandbox.ephemeral(Image.linux(), local=True), no runtime=): boots in 45s on the pinned disk, with apt_install, run and env layers all landing. A host with no QEMU raises and names the dependency instead of falling back. The fallback would be the XFCE container, which cannot become ready, so it would trade a clear error for a 120s timeout. Image.linux(kind='container') is unchanged and still runs under Docker. Note the image pin is untouched: DEFAULT_LINUX_REGISTRY_IMAGE still points at main-38352d34, and it feeds both cloud and local, so moving it moves the Fleet default too.
…ot the pinned disk locally (#3128) * fix(sandbox): make the documented Image builder methods actually work Verifying every method on the sandbox images guide turned up four code bugs and several wrong claims on the page. Code: * `_make_transport()` never forwarded `image`, so every cloud sandbox created with an explicit API key died with "Cannot create a cloud VM without an image". The Fleet branch directly above it passed `image=image`; this one silently did not. * `.env()` and `.copy()` were dropped on the QEMU VM path — the local path for `Image.linux()` and `Image.windows()`. The builder gated on `_layers` alone, so an image carrying only env vars or files was never built, and `/etc/profile.d/cua-env.sh` was never written. Both are now build inputs and both participate in the user-image cache key (file contents included, so editing a copied file rebuilds). Layers-only images keep their historical key, so existing caches stay valid. * `LayerExecutor` was constructed without `os_type`, so `run` layers on a Windows image were wrapped in `sudo bash -c`. * Concurrent local sandboxes collided: the bare-metal runtime pinned VNC display 0 and QMP port 4444, and pointed every UEFI guest at one shared `sessions/efivars.fd`. VNC and QMP are now allocated like the API port already was, and each VM gets its own efivars file. * `cua-cli` dead-ended on headless hosts with "Configure an OS keyring", naming no way out. The error now names the `keyrings.alt` backend and `FLEETS_TOKEN`. We deliberately do not fall back to on-disk storage automatically — that would silently downgrade an OAuth refresh token to cleartext. Docs: * `to_dict()` showed `kind: container` for `Image.linux()`, which is a VM. * The page presented every constructor as equivalent; only `Image.linux()` and `Image.windows()` have a pinned disk and run in the cloud. * Nothing said that customization is local-only — Fleet rejects any image carrying layers, env, or files. * `Image.from_registry()` needs an explicit `runtime=` and always reports `os_type=linux`, including for the macOS example the page shows. * Documented that `.env()` lands in profile.d and so needs a login shell. * fix(cli,sandbox): stop reporting failures and no-ops as success Follow-up from verifying the sandbox images guide. * `cua sb ls` wrapped both listings in `except Exception: pass`, so a user hitting an outage was told "No sandboxes found." and got exit 0. Failures are now reported per source, partial results still print, and the exit code is non-zero. `--json` carries an `errors` array. * `Sandbox.delete(local=True)` dispatched on `runtime_type` from a state file and took no branch when there was none: deleting a name that never existed reported success, while a container from a launch that timed out before writing state could not be deleted at all. Unknown names now raise, and an orphaned container is removed. Docs, from things the verification pass established by running them: * `Image.linux()` is documented as a QEMU VM, but started locally with no explicit runtime it auto-selects Docker-wrapped QEMU, which resolves to the same `trycua/cua-xfce` container `kind='container'` uses — no QEMU, no KVM, no containerDisk. Added the bare-metal recipe that does boot the pinned disk. * `Image.from_registry()` needs a KubeVirt containerDisk carrying `/disk/disk.img`. The `ubuntu:22.04` example on the page cannot work: the puller rejects it, and it authenticates anonymously, which Docker Hub refuses. * Separated the two caches the page conflated — `image-cache/` holds disks downloaded from a URL, `images/container-disks/` holds registry pulls. * docs(sandbox): say plainly which documented paths do not work today The verification pass proved three documented paths are broken. Where the fix lives elsewhere, the page should still not assert something we know is false — a reader hitting a 120s timeout is not helped by the fix being in another repo. * `Image.linux(kind='container')` cannot start at all: the published trycua/cua-xfce runs computer-server on 127.0.0.1:8000, so the published port mapping forwards to nothing. Same callout on the first-local-sandbox tutorial, whose only example uses that path. * `Image.windows()` in the cloud is blocked by a UEFI/GPT disk meeting a BIOS-by-default template; a firmware fix is in flight. * `expose()` is honoured in the cloud and silently ignored on local VMs. * fix(cli): recommend a keyring backend that works when followed literally The previous message pointed headless users at keyrings.alt.file.EncryptedKeyring, which fails at construction with "ModuleNotFoundError: No module named 'Crypto'" — keyrings.alt does not depend on a crypto library. Replacing a dead-end error with advice that dead-ends one step later is no improvement. Now keyrings.cryptfile, which depends on pycryptodome and works from a single install, and FLEETS_TOKEN is listed first because an encrypted keyring prompts for a passphrase on every command and so cannot be used unattended at all — the CI case the original error stranded. Every command in the message was run end to end against a clean CLI install: the bare CLI reproducing the dead end, then each of the three options taking it through to "Not logged in". * feat(sandbox)!: boot the pinned containerDisk for local Linux VMs Changes default local behaviour for Image.linux(). Image.linux() is a VM, but auto-selection sent it to Docker-wrapped QEMU, and resolve_image() hands that the trycua/cua-xfce *container* image regardless of kind. So asking for a VM produced a container byte-identical to Image.linux(kind='container'), never touched the pinned containerDisk, and then could not become ready at all because that image binds computer-server to 127.0.0.1. A Linux VM now boots the same containerDisk Fleet cloud boots, under bare-metal QEMU — the parity #3091 and #3118 were built for, which until now was reachable only by passing runtime=QEMURuntime(mode="bare-metal") by hand. Verified end to end with the plain documented invocation (Sandbox.ephemeral(Image.linux(), local=True), no runtime=): boots in 45s on the pinned disk, with apt_install, run and env layers all landing. A host with no QEMU raises and names the dependency instead of falling back. The fallback would be the XFCE container, which cannot become ready, so it would trade a clear error for a 120s timeout. Image.linux(kind='container') is unchanged and still runs under Docker. Note the image pin is untouched: DEFAULT_LINUX_REGISTRY_IMAGE still points at main-38352d34, and it feeds both cloud and local, so moving it moves the Fleet default too. * style: satisfy isort and black on the touched files CI runs isort, black and ruff in one step with bash -e, so the isort failure masked a black failure in build.py behind it. --------- Co-authored-by: Robert Wendt <robert@trycua.com>
📦 Publishable packages changedThis comment is status-only. Editing it or adding task-list checkboxes cannot authorize a release.
Ask the release owner to apply |
Behavior change: two Windows defaults move from 11 to Server 2022
1. SDK —
Image.windows()'s defaultversionchanges from"11"to"2022". Callers of the bare default previously got a Windows 11 evaluation-ISO install locally andNotImplementedErroron Fleet cloud; they now get the pinned Windows Server 2022 containerDisk, which works on both paths.2. CLI —
cua-cli's barewindowsimage alias changes from"11"to"2022"(cua_cli/commands/sandbox.py), so the CLI and the SDK agree on what an unqualified "windows" means.windows:11still selects Windows 11 explicitly.Nothing is silently relabelled —
Image.windows("11")still means client Windows 11 and keeps its current behaviour (local ISO install, unsupported on cloud). Only the defaults move, to the one version that actually has a disk.What this does
Image.windows(...)had no registry counterpart. The Fleet cloud path rejected it outright:and the local QEMU path fell through to
_build_windows_base, which downloads a ~6 GB Windows ISO and runs an unattended install.Image.windows()now resolves to a pinned KubeVirt containerDisk on both paths, so a local run and a cloud run boot identical bytes — the Windows counterpart of #3091.The descriptor table
#3091 added
cloud_registry_imagewith a single hardcoded Linux branch. Rather than bolting a second branch on for Windows, this generalises it into a lookup keyed on the built-in descriptor tuple:Resolved at runtime:
An explicit
Image.from_registry(...)still wins; descriptors with no pinned disk still returnNoneand keep their existing behaviour.The Linux row is repinned here to the public mirror to match #3091, so both built-ins are credential-free. That is a string change only — the public and private refs are bit-identical, and Linux was not re-tested for this PR; the Linux evidence lives in #3091.
Which image
public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3HOMEwith no~/.docker/config.jsonpresent.verin the booted guest and by the desktop watermark.main-bac7daa3is git-sha derived and matches themain-<sha>convention the Linux row already uses. Pinned rather thanlatest, which will move. Index digestsha256:6d341afc26a37c4072d22ba403a89ecdad9a29aebab79570b5a38da6b8e16370, recorded in a code comment.linux/amd64(the containerDisk) andunknown/unknown(a buildx provenance attestation), so index resolution and attestation-skipping are both required to reach the disk. The platform islinux/amd64, notwindows/amd64— the OCI platform describes the container wrapping the disk, not the guest inside it.The
"2022"naming stays coherent on the fallback path too:windows_unattend.pyalready maps"2022"to the Windows Server 2022 evaluation ISO, so if the containerDisk pull ever fails, the local install builds the same OS.Auth backend: ask the registry, don't guess
Registries disagree about the authentication challenge, and oras fixes its backend at construction time:
*.dkr.ecr.*.amazonaws.com)Basicbasic—tokenraisesValueError: Cannot respond to request for authentication.public.ecr.aws,ghcr.ioBearertoken—basicraisesAttributeError: 'BasicAuth' object has no attribute '_basic_auth'A fixed backend cannot serve both, and the pinned Linux image is on private ECR while the pinned Windows image is on public ECR.
_detect_auth_backendprobes the registry's/v2/endpoint and reads the scheme out of the challenge, so no host list has to be maintained and credentialed bearer registries still work.pull_container_disktakes an optionalauth_backendto skip the probe. The extra round-trip happens only on a cold pull; a cached disk returns before it.UEFI firmware fix
Local Windows could not boot on current Ubuntu regardless of the disk.
QEMUBaremetalRuntimelooked for/usr/share/OVMF/OVMF_CODE.fd, but Ubuntu 24.04 shipsOVMF_CODE_4M.fd; with no firmware found, no pflash drives were added and the GPT/EFI disk had nothing to boot from. When firmware was found, it was paired with a 256 KB zero-filled varstore rather than the matchingOVMF_VARS*.fd.Firmware is now discovered as matched
(code, vars)pairs.The WSL-hosted path (
_build_wsl_cmd) had a different, narrower defect, now fixed too. It already triedOVMF_CODE_4M.fdfirst, so it never hit the missing-firmware failure above. What it got wrong was the pairing: it picked the code file in one loop and the varstore in a second, independent loop, so a host carrying 2 MBOVMF_CODE.fdalongside 4 MBOVMF_VARS_4M.fdwould be handed a mismatched pair. When none of its three vars candidates existed it fell back tob"\x00" * (256 * 1024)— a zero-filled varstore valid at neither size. It now selects both halves from the same entry and raises a clear error naming the missing package instead of fabricating a varstore.Scoped honestly: on a stock WSL Ubuntu the pairing defect is latent, not active. The
ovmfpackage installs only the 4M generation, so old and new code both select the same correct pair (measured inventory below). This part is a robustness fix, not a repair of a failure users are hitting today.WSL2 path: verified on a real Windows host
Previously unverifiable; now measured on a bare-metal Windows Server 2022 host running WSL2 Ubuntu 24.04 (
ovmf 2024.02-2ubuntu0.9,qemu-system-x86 1:8.2.2+ds-0ubuntu1.18).Firmware inventory — of the six paths the code probes, only the 4M pair exists:
That inventory also explains the asymmetry with bare metal: bare metal was genuinely broken because its candidate list lacked the
_4Mname entirely and Ubuntu 24.04 ships only 4M, so it found no firmware at all.A second WSL bug this surfaced: the overlay's backing file
Booting a containerDisk through WSL2 failed outright:
The session overlay is created by the Windows-side builder, so the backing path recorded inside the qcow2 is a Windows path. The runtime translated the drive path to
/mnt/c/...but not the backing path, and QEMU inside WSL parsed the drive letter as a URI scheme. This affects any layered or base-image disk on WSL, not just containerDisks — it was simply unreachable before, becauseImage.windows()had no disk to overlay._rebase_backing_filenow repoints the overlay with a metadata-onlyqemu-img rebase -u.Real boot, accelerated
WSL2 on a bare-metal host exposes
vmxand a working/dev/kvm, so this ran under real KVM, not TCG. The launched command line:with the result:
The generated
efivars.fdis 540672 bytes — byte-for-byte the size ofOVMF_VARS_4M.fd, confirming it came from the paired template and not the 256 KB zero fallback.Files touched outside
libs/python/cua-sandboxThe default change reaches beyond the sandbox package. Examples and defaults that just mean "give me a Windows sandbox" now use the bare
Image.windows():tests/integration/sandbox_sdk/test_windows_cloud_vm.py,test_windows_local_vm.py— including the docstrings, which now state which version is which.libs/python/cua-cli/cua_cli/commands/mcp.py— the MCPcreate_sandboxtool builtImage.windows("11")and would have hitNotImplementedErroron cloud.libs/python/cua-cli/cua_cli/commands/sandbox.py— the CLI's barewindowsalias defaulted to"11"; it now defaults to"2022"to match the SDK.windows:11still selects Windows 11 explicitly.docs/content/docs/how-to-guides/sandbox/images.mdx— theImage.windows()line was annotated "Windows 11".docs/content/docs/how-to-guides/sandbox/configure-pool-with-terraform.mdx— twocontainer_disk_imageexamples pointed at private ECR (desktop-workspace-duo,cua-server-windows:latest); both now use the public refs, and the Windows one is pinned rather thanlatest.docs/content/docs/how-to-guides/sandbox/create-pool-with-python.mdx— same, for the Linux example.libs/python/cua-sandbox/tests/live/test_fleet_ephemeral.py— repinned to the public Linux digest, matching feat(sandbox): boot the same Linux containerDisk locally and in Fleet cloud #3091.tests/test_runtime.pydeliberately keepsImage.windows("11"). Those are Windows-host-only local-runtime tests and they are the remaining coverage of the Windows 11 ISO-install path; moving them to the default would have converted an ISO test into a containerDisk test and silently dropped that coverage.tests/test_windows_cloud.pyandtest_windows_timing.pyare left on their explicit"server-2025".Verification
Real end-to-end run on a bare-metal host with
/dev/kvm, viaSandbox.ephemeral(Image.windows(), local=True), from a cleanHOMEwith no registry credentials:No ISO install, and no KubeVirt-specific setup was needed — the containerDisk boots under plain QEMU + KVM with OVMF and a virtio disk. The
shell.runcalls exercise the cua computer-server on port 8000 inside the guest, so they double as proof it is installed and running.The Fleet cloud path is covered by unit tests against the typed template request, not by a live cloud call.
ruff checkclean in bothcua-sandboxandcua-cli.main. The 21 failures and 19 errors are identical before and after — they are pre-existing and need a local display.Relationship to #3091
This branch is cut from
mainand carries #3091's code so it stands alone.builder/build.py'sresolve_backing_diskandregistry/container_disk.pyare byte-identical to #3091's current head, so once it merges they rebase away to nothing and the remaining diff is the descriptor table, the Windows pin, the default change, the UEFI fix, the updated call sites, and tests.