feat(sandbox): boot the same Linux containerDisk locally and in Fleet cloud - #3091
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Collaborator
Author
|
Hillclimb complete.
|
Contributor
There was a problem hiding this comment.
Pull request overview
Defaults Linux sandboxes to a shared pinned containerDisk for Fleet and local QEMU execution.
Changes:
- Maps the default Ubuntu 24.04 VM descriptor to the containerDisk.
- Adds ORAS-based streaming extraction and local qcow2 caching.
- Updates Fleet routing and adds focused tests.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
cua_sandbox/image.py |
Defines default registry-image resolution. |
cua_sandbox/registry/container_disk.py |
Pulls and caches containerDisks. |
cua_sandbox/builder/build.py |
Uses containerDisks as QEMU backing images. |
cua_sandbox/transport/fleet_cloud.py |
Supports the default image in Fleet templates. |
cua_sandbox/sandbox.py |
Routes supported defaults through Fleet. |
cua_sandbox/pool.py |
Includes resolved images in pool identities. |
tests/test_container_disk.py |
Tests pulling, caching, and QEMU integration. |
tests/test_image.py |
Tests default-image resolution. |
tests/test_fleet_cloud_transport.py |
Tests Fleet template behavior. |
tests/test_cloud.py |
Tests default Linux Fleet routing. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+30
to
+31
| destination.parent.mkdir(parents=True, exist_ok=True) | ||
| temporary = destination.with_suffix(".tmp") |
Comment on lines
+31
to
+33
| DEFAULT_LINUX_REGISTRY_IMAGE = ( | ||
| "296062593712.dkr.ecr.us-west-2.amazonaws.com/" "desktop-workspace-duo:main-38352d34" | ||
| ) |
Co-authored-by: r33drichards <57335981+r33drichards@users.noreply.github.com>
pull_container_disk() was never reachable and did not work against the real registry, so Image.linux() still fell through to a locally built base image instead of the disk Fleet cloud boots. - Use oras' basic auth backend. ECR answers WWW-Authenticate: Basic, so the token backend failed with "This endpoint requires a token. Please use basic auth with a username or password." after ~2 minutes of retry backoff. - Follow OCI image indexes to the platform child manifest. The real image is a multi-arch index, which carries "manifests" and no "layers", so the extraction loop never ran and the pull always raised FileNotFoundError. Buildx attestation entries are skipped. - Skip chunked VM-disk layers (lume/tart/qemu) instead of streaming GBs of a non-containerDisk image before failing. - Wire the pull into create_session_disk() via resolve_backing_disk(), off the event loop with asyncio.to_thread. The session overlay is now backed by the pulled containerDisk; images with no registry counterpart still fall back to ensure_base_image(). Verified end to end on bare metal with KVM: Image.linux() pulls the pinned ECR containerDisk, overlays it, boots it under qemu-system-x86_64 -enable-kvm, and serves shell.run() and screenshot() from the guest computer-server. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Heads-up: #3114 migrates all tracked Ubuntu/Windows containerDisk consumers to public ECR and adds a repository-wide guard against the old private repositories. This PR currently introduces |
Repinning the Linux default to public.ecr.aws invalidated the hardcoded basic
auth backend. oras fixes its backend at construction and cannot negotiate one,
but the two registries want opposite schemes:
public.ecr.aws WWW-Authenticate: Bearer token works; basic raises
AttributeError: 'BasicAuth' object has no attribute '_basic_auth'
private ECR WWW-Authenticate: Basic basic works; token raises
ValueError: Cannot respond to request for authentication
So the backend is now read off the registry's /v2/ challenge instead of being
hardcoded, which keeps both the public default and explicit private
Image.from_registry(...) refs working. Callers can force one with the new
auth_backend= keyword; the probe falls back to the OCI-standard bearer flow when
the registry is unreachable or sends no challenge.
Verified against both real registries: detection picks token for public.ecr.aws
and basic for private ECR, and a cold-cache pull succeeds on each. The public
image is also an OCI index, so the index-descent path is still exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…2d34 Adopts the Windows branch's _auth_backend_for() verbatim so registry/container_disk.py stays byte-identical between the two branches. oras fixes its auth backend at construction and cannot negotiate, but the registries want opposite schemes, measured live: basic public.ecr.aws FAIL AttributeError: 'BasicAuth' object has no attribute '_basic_auth' token public.ecr.aws OK basic private ECR OK token private ECR FAIL ValueError: Cannot respond to request for authentication So the scheme is read off the registry's 401 challenge instead of guessed. pull_container_disk() takes an optional auth_backend; production omits it and probes, tests pass it explicitly so no unit test reaches the network. Repins the Linux default to public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04:main-38352d34 rather than :main-e5d853a9. The public main-38352d34 is bit-identical to the private image the PR was written against — the cold pull fetches the same layer sha256:7bd992eca8bf... (1438806575 bytes) the private registry served — so only the registry host changes and existing boot evidence still describes these bytes. Index descent now tests against the real two-child shape: the linux/amd64 disk plus the buildx provenance manifest that reports platform unknown/unknown, with a case proving the attestation is skipped even when listed first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dpoint
Converges registry/container_disk.py on the Windows branch's shape so the file is
byte-identical across both branches and rebases away to nothing when one merges.
_detect_auth_backend() pings https://{host}/v2/ rather than fetching a manifest:
/v2/ is the documented OCI discovery endpoint for reading a WWW-Authenticate
challenge, and it is cheaper than pulling a manifest just to be refused. The
_registry_host() helper returns None for refs with an implicit registry
("ubuntu:24.04"), which short-circuit to the token backend instead of having a
manifest URL built for a host that was never named.
Behavior is unchanged: public.ecr.aws still resolves to token, private ECR to
basic, and a cold-cache pull plus KVM boot of the pinned public image still
succeeds. Also collapses DEFAULT_LINUX_REGISTRY_IMAGE into a single literal now
that it no longer needs the private registry's longer host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
r33drichards
changed the title
r33drichards
pushed a commit
that referenced
this pull request
Aug 12, 2026
…eet cloud
BREAKING CHANGE: two Windows defaults move from "11" to "2022". `Image.windows()`
now defaults to `version="2022"` (Windows Server 2022), and cua-cli's bare
`windows` image alias follows it. Callers of either bare default previously got
a Windows 11 evaluation-ISO install locally and NotImplementedError on Fleet
cloud; they now get the pinned Server 2022 containerDisk, which works on both
paths. `Image.windows("11")` and `windows:11` still mean client Windows 11 and
are unchanged.
The Linux row is repinned to public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04 to match
#3091, so both built-ins are credential-free. The public and private refs are
bit-identical, so this is a string change; Linux was not re-tested here. Three
user-facing docs that still pointed at private ECR were moved to the public
refs as well.
`Image.windows(...)` had no registry counterpart, so the Fleet cloud path
rejected it and the local QEMU path fell through to `_build_windows_base`,
which downloads a ~6 GB Windows ISO and runs an unattended install. Both paths
now resolve the same pinned KubeVirt containerDisk, so a local run and a cloud
run boot identical bytes.
Generalises `cloud_registry_image` into a `BUILTIN_REGISTRY_IMAGES` descriptor
table keyed on (os_type, distro, version, kind), replacing the single hardcoded
Linux branch. Linux keeps its existing pin; Windows Server 2022 is the new row.
The image is `public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3` — an
anonymously pullable mirror, so the built-in Windows image needs no registry
credentials. It is an OCI index whose children are the linux/amd64
containerDisk and a buildx provenance attestation.
Selecting an oras auth backend now asks the registry instead of guessing.
Private ECR challenges with Basic, where oras' token backend raises "Cannot
respond to request for authentication"; public.ecr.aws and ghcr.io challenge
with Bearer, where basic auth has no credential to send. Since the pinned Linux
image is on private ECR and the pinned Windows image is on public ECR, no fixed
backend works for both, so `_detect_auth_backend` reads the scheme off the
registry's /v2/ endpoint.
Also fixes UEFI firmware discovery, without which the local Windows path
cannot boot on current Ubuntu. The bare-metal runtime looked only for
`/usr/share/OVMF/OVMF_CODE.fd` (Ubuntu 24.04 ships `OVMF_CODE_4M.fd`), so it
found no firmware and added no pflash drives. The WSL-hosted runtime had a
narrower defect: it did try the 4M file first, but chose the code file and the
varstore in two independent loops, so a host with 2M code and 4M vars got a
mismatched pair, and with no vars present at all it fabricated a zero-filled
varstore valid at neither size. Both now take code and vars from the same
candidate entry.
Verifying that on a real Windows host surfaced a second WSL bug: the session
overlay is created by the Windows-side builder, so the backing path recorded in
the qcow2 is a Windows path, and QEMU inside WSL parsed the drive letter as a
URI scheme -- "Could not open backing file: Unknown protocol 'C'". It now
repoints the overlay with a metadata-only `qemu-img rebase -u`. This affected
any layered or base-image disk on WSL; it was previously unreachable because
Image.windows() had no disk to overlay.
Windows examples that mean "give me a Windows sandbox" move to the bare
`Image.windows()`: the two sandbox_sdk integration examples, the CLI's MCP
`create_sandbox` tool, the CLI's bare `windows` alias, and the images guide.
`tests/test_runtime.py` deliberately stays on `Image.windows("11")` — it is the
remaining coverage of the Windows 11 ISO-install path.
Verified end to end on a bare-metal host with KVM, from a clean HOME with no
docker credentials present:
[e2e] docker config = /home/ubuntu/e2e-home/.docker/config.json exists=False
[e2e] image = Image(windows/windows:2022, kind=vm, 0 layers)
[e2e] default windows() = public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3
[e2e] chosen auth = token
Cached containerDisk public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3
Created overlay: .../sessions/e2e-windows-public.qcow2
Bare-metal QEMU VM e2e-windows-public is ready
[e2e] pull + boot in 446s
$ ver -> Microsoft Windows [Version 10.0.20348.587]
$ hostname -> DOCKERW-K5E4442
screenshot: 170369 bytes, PNG
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 12, 2026
r33drichards
pushed a commit
that referenced
this pull request
Aug 13, 2026
Changes default local behaviour for Image.linux(). Image.linux() is a VM, but auto-selection sent it to Docker-wrapped QEMU, and resolve_image() hands that the trycua/cua-xfce *container* image regardless of kind. So asking for a VM produced a container byte-identical to Image.linux(kind='container'), never touched the pinned containerDisk, and then could not become ready at all because that image binds computer-server to 127.0.0.1. A Linux VM now boots the same containerDisk Fleet cloud boots, under bare-metal QEMU — the parity #3091 and #3118 were built for, which until now was reachable only by passing runtime=QEMURuntime(mode="bare-metal") by hand. Verified end to end with the plain documented invocation (Sandbox.ephemeral(Image.linux(), local=True), no runtime=): boots in 45s on the pinned disk, with apt_install, run and env layers all landing. A host with no QEMU raises and names the dependency instead of falling back. The fallback would be the XFCE container, which cannot become ready, so it would trade a clear error for a 120s timeout. Image.linux(kind='container') is unchanged and still runs under Docker. Note the image pin is untouched: DEFAULT_LINUX_REGISTRY_IMAGE still points at main-38352d34, and it feeds both cloud and local, so moving it moves the Fleet default too.
r33drichards
added a commit
that referenced
this pull request
Aug 13, 2026
…ot the pinned disk locally (#3128) * fix(sandbox): make the documented Image builder methods actually work Verifying every method on the sandbox images guide turned up four code bugs and several wrong claims on the page. Code: * `_make_transport()` never forwarded `image`, so every cloud sandbox created with an explicit API key died with "Cannot create a cloud VM without an image". The Fleet branch directly above it passed `image=image`; this one silently did not. * `.env()` and `.copy()` were dropped on the QEMU VM path — the local path for `Image.linux()` and `Image.windows()`. The builder gated on `_layers` alone, so an image carrying only env vars or files was never built, and `/etc/profile.d/cua-env.sh` was never written. Both are now build inputs and both participate in the user-image cache key (file contents included, so editing a copied file rebuilds). Layers-only images keep their historical key, so existing caches stay valid. * `LayerExecutor` was constructed without `os_type`, so `run` layers on a Windows image were wrapped in `sudo bash -c`. * Concurrent local sandboxes collided: the bare-metal runtime pinned VNC display 0 and QMP port 4444, and pointed every UEFI guest at one shared `sessions/efivars.fd`. VNC and QMP are now allocated like the API port already was, and each VM gets its own efivars file. * `cua-cli` dead-ended on headless hosts with "Configure an OS keyring", naming no way out. The error now names the `keyrings.alt` backend and `FLEETS_TOKEN`. We deliberately do not fall back to on-disk storage automatically — that would silently downgrade an OAuth refresh token to cleartext. Docs: * `to_dict()` showed `kind: container` for `Image.linux()`, which is a VM. * The page presented every constructor as equivalent; only `Image.linux()` and `Image.windows()` have a pinned disk and run in the cloud. * Nothing said that customization is local-only — Fleet rejects any image carrying layers, env, or files. * `Image.from_registry()` needs an explicit `runtime=` and always reports `os_type=linux`, including for the macOS example the page shows. * Documented that `.env()` lands in profile.d and so needs a login shell. * fix(cli,sandbox): stop reporting failures and no-ops as success Follow-up from verifying the sandbox images guide. * `cua sb ls` wrapped both listings in `except Exception: pass`, so a user hitting an outage was told "No sandboxes found." and got exit 0. Failures are now reported per source, partial results still print, and the exit code is non-zero. `--json` carries an `errors` array. * `Sandbox.delete(local=True)` dispatched on `runtime_type` from a state file and took no branch when there was none: deleting a name that never existed reported success, while a container from a launch that timed out before writing state could not be deleted at all. Unknown names now raise, and an orphaned container is removed. Docs, from things the verification pass established by running them: * `Image.linux()` is documented as a QEMU VM, but started locally with no explicit runtime it auto-selects Docker-wrapped QEMU, which resolves to the same `trycua/cua-xfce` container `kind='container'` uses — no QEMU, no KVM, no containerDisk. Added the bare-metal recipe that does boot the pinned disk. * `Image.from_registry()` needs a KubeVirt containerDisk carrying `/disk/disk.img`. The `ubuntu:22.04` example on the page cannot work: the puller rejects it, and it authenticates anonymously, which Docker Hub refuses. * Separated the two caches the page conflated — `image-cache/` holds disks downloaded from a URL, `images/container-disks/` holds registry pulls. * docs(sandbox): say plainly which documented paths do not work today The verification pass proved three documented paths are broken. Where the fix lives elsewhere, the page should still not assert something we know is false — a reader hitting a 120s timeout is not helped by the fix being in another repo. * `Image.linux(kind='container')` cannot start at all: the published trycua/cua-xfce runs computer-server on 127.0.0.1:8000, so the published port mapping forwards to nothing. Same callout on the first-local-sandbox tutorial, whose only example uses that path. * `Image.windows()` in the cloud is blocked by a UEFI/GPT disk meeting a BIOS-by-default template; a firmware fix is in flight. * `expose()` is honoured in the cloud and silently ignored on local VMs. * fix(cli): recommend a keyring backend that works when followed literally The previous message pointed headless users at keyrings.alt.file.EncryptedKeyring, which fails at construction with "ModuleNotFoundError: No module named 'Crypto'" — keyrings.alt does not depend on a crypto library. Replacing a dead-end error with advice that dead-ends one step later is no improvement. Now keyrings.cryptfile, which depends on pycryptodome and works from a single install, and FLEETS_TOKEN is listed first because an encrypted keyring prompts for a passphrase on every command and so cannot be used unattended at all — the CI case the original error stranded. Every command in the message was run end to end against a clean CLI install: the bare CLI reproducing the dead end, then each of the three options taking it through to "Not logged in". * feat(sandbox)!: boot the pinned containerDisk for local Linux VMs Changes default local behaviour for Image.linux(). Image.linux() is a VM, but auto-selection sent it to Docker-wrapped QEMU, and resolve_image() hands that the trycua/cua-xfce *container* image regardless of kind. So asking for a VM produced a container byte-identical to Image.linux(kind='container'), never touched the pinned containerDisk, and then could not become ready at all because that image binds computer-server to 127.0.0.1. A Linux VM now boots the same containerDisk Fleet cloud boots, under bare-metal QEMU — the parity #3091 and #3118 were built for, which until now was reachable only by passing runtime=QEMURuntime(mode="bare-metal") by hand. Verified end to end with the plain documented invocation (Sandbox.ephemeral(Image.linux(), local=True), no runtime=): boots in 45s on the pinned disk, with apt_install, run and env layers all landing. A host with no QEMU raises and names the dependency instead of falling back. The fallback would be the XFCE container, which cannot become ready, so it would trade a clear error for a 120s timeout. Image.linux(kind='container') is unchanged and still runs under Docker. Note the image pin is untouched: DEFAULT_LINUX_REGISTRY_IMAGE still points at main-38352d34, and it feeds both cloud and local, so moving it moves the Fleet default too. * style: satisfy isort and black on the touched files CI runs isort, black and ruff in one step with bash -e, so the isort failure masked a black failure in build.py behind it. --------- Co-authored-by: Robert Wendt <robert@trycua.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Image.linux()now boots the same disk locally that it boots in Fleet cloud, and that disk is served from a public, credential-free registry.Image.linux()descriptor (ubuntu,24.04,vm) to the pinnedpublic.ecr.aws/k5j5w0x5/cua-ubuntu-24.04:main-38352d34containerDisk, viacloud_registry_image(). This is bit-identical to the privatedesktop-workspace-duo:main-38352d34the PR was originally written against — only the registry host changescreate_session_disk()resolves its backing disk through the newresolve_backing_disk(): it pulls the containerDisk withoras-py(loading Docker config / credential helpers — no Docker or Skopeo executable needed), stream-extracts/disk/disk.img, caches it as qcow2 under~/.cua/cua-sandbox/images/container-disks/<sha256(ref)>/disk.qcow2, and creates the normal per-session overlay on top of itasyncio.to_threadensure_base_image()Image.from_registry(...)behavior and reject unsupported built-in descriptors in FleetThree bugs fixed in
pull_container_disk(all only reachable against a real registry)1. The auth backend cannot be hardcoded. oras fixes its backend at construction and cannot negotiate, but the registries want opposite schemes. All four combinations, measured live:
public.ecr.aws is anonymously readable but still requires the Docker Bearer token flow — an unauthenticated manifest GET is a 401. The pull now reads the scheme off the
WWW-Authenticatechallenge served by the registry's/v2/discovery endpoint (the documented OCI place to read it, and cheaper than fetching a manifest just to be refused) and picks the matching backend, so the public default and explicit privateImage.from_registry(...)refs both work.pull_container_diskgained an optionalauth_backend=; production omits it and probes, tests pass it explicitly. Costs one extra unauthenticated round-trip per cold pull; the cached-disk fast path returns before it.2. The image is an OCI image index. A multi-arch index carries
manifests(per-platform children), notlayers, somanifest.get("layers", [])was always empty, the extraction loop never ran, and every pull ended inFileNotFoundError: ... does not contain /disk/disk.img. The pull now descends to thelinux/<host arch>child. The index has exactly two children — thelinux/amd64disk and a buildx provenance manifest reporting platformunknown/unknown— so selection matches explicitly onplatform.os/platform.architectureand skips attestations; picking "the first" or "any" child would grab the provenance manifest and fail confusingly.3. Chunked VM-disk layers were streamed before failing. They are now skipped, so a non-containerDisk registry image fails fast instead of downloading GBs first.
Validation
Run on a c5n.metal box with real
/dev/kvm, against the real registries.Unit/regression suite
ruff checkpasses on the touched files.tests/test_container_disk.pycovers challenge→backend selection (Basic → basic, Bearer → token, absent header → token, unreachable registry → token), the real two-child index shape including a case where the attestation is listed first, the explicit-auth_backendoverride, and thatImage.linux()overlays the pulled containerDisk rather than a locally built base — off the event loop thread. No unit test reaches the network.End-to-end boot (cold cache)
qemu-img info --backing-chainon the booted session disk:From inside the guest:
sb.screenshot()returned a 25 KB PNG of the XFCE desktop. The layer digest, byte count, kernel and/opt/computer-servermtime pulled from public ECR all match what the private registry served for the same tag, independently confirming the two images are the same bytes.Note:
QEMURuntime()defaults tomode="docker", which wraps QEMU in a container and never consultscreate_session_disk— the disk-parity path ismode="bare-metal".Known gaps
e3ef139), so they are unrelated to this change.linux/amd64was exercised against a real registry; the pinned image publishes noarm64child, so an arm64 host gets a clearno linux/arm64 manifest (available: ...)error rather than a silent wrong-arch boot.FleetCloudTransportstill setsimage_pull_secret("ecr-credentials"). Harmless for a public image and still needed for privatefrom_registryrefs, so the cloud path is left untouched here.