fix(cua-sandbox): boot Windows Fleet sandboxes with UEFI firmware

[r33drichards]

Why

Image.windows() sandboxes on Fleet cloud never become ready. The VM starts, the guest never boots, port 8000 never opens, the readiness probe never passes, and the warm pool sits at readyReplicas: 0 until claims fail with BindDeadlineExceeded.

The Windows containerDisk is GPT/UEFI-only. I read the guest's own partition table straight out of the published layer (public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3, range-GET → gunzip → qemu-io on the qcow2):

• LBA 0 — MBR partition entry type byte 0xEE: a GPT protective MBR. Its bootstrap is the stock "Invalid partition table / Missing operating system" stub.
• LBA 1 — EFI PART signature, valid GPT header.
• Partition entries — c12a7328-f81f-11d2-ba4b-00a0c93ec93b "EFI system partition", then "Microsoft reserved partition", then "Basic data partition".

No active MBR partition and no BIOS boot partition. There is nothing for SeaBIOS to chainload.

And SeaBIOS is exactly what it got. From the virt-launcher log of a live failing VM, the generated domain XML has an empty <os> block with no <loader>/<nvram>, and the qemu command line has no -drive if=pflash — no OVMF anywhere. The VMI reports:

phase=Running
Ready=False  ContainersNotReady  "containers with unready status: [compute]"

Running, but nothing listening. _template_request() never called .firmware(...), so the template inherited the Fleet schema default:

// libs/fleet/sdk-schema/src/common.rs:70-72
pub(crate) fn default_firmware() -> Option<Firmware> {
    Some(Firmware::Bios)
}

Verified end to end, not just asserted

Given that the whole bug is "we asserted the artifact instead of booting it", asserting the artifact again would be a poor fix. So I booted it — a throwaway pool on run.cua.ai built from this patch's own output, same image, same defaults (4 cpu / 4Gi), efi the only difference from the failing pool:

patched transport emits: firmware=Firmware.EFI  image=public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3

  42s  fw-verify-efi-2-3a90ed73  phase=Running ready=False ContainersNotReady
  62s  fw-verify-efi-2-3a90ed73  phase=Running ready=False ContainersNotReady
  82s  fw-verify-efi-2-3a90ed73  phase=Running ready=False ContainersNotReady
 103s  fw-verify-efi-2-3a90ed73  phase=Running ready=False ContainersNotReady
 123s  fw-verify-efi-2-3a90ed73  phase=Running ready=True

RESULT: READY after 123s

Ready in 123 seconds. The bios pool with the same image had been Running-but-never-Ready for 22 minutes at that point. Test namespaces were deleted and confirmed gone.

Unit tests: 37 passed, including two new ones — test_windows_image_boots_uefi (asserts the Windows case specifically) and test_linux_image_leaves_firmware_at_the_schema_default (pins the no-change-for-Linux behaviour).

This is not a regression from #3091 / #3114 / #3118

Please don't go hunting today's commits. This code path has never been able to boot a Windows image on Fleet. git log -S"firmware" on fleet_cloud.py returns exactly two commits, and neither ever set EFI:

• refactor(cua-sandbox): use generated Cyclops SDK for Fleet cloud transport #2398 created the SDK-based transport and passed firmware=None explicitly.
• fix(cua-sandbox): provision Fleet sandboxes with SDK builders #2979 moved to builders and deleted that line — same result, unset → bios.

It only became reachable now. Before #3118, Image.windows() defaulted to version "11", which has no entry in BUILTIN_REGISTRY_IMAGES, so cloud_registry_image() returned None and the transport rejected it early — Windows-on-Fleet never got as far as the firmware question. #3118 flipped the default to "2022" and pinned a real, pullable Windows containerDisk, which walked the path straight into a latent BIOS default that had been sitting there since July.

The lesson worth keeping

The existing Fleet unit tests passed the entire time while shipping a template that cannot boot. They assert the template we build — image reference, service ports — and a template asserting only "we set the fields we set" will happily describe an unbootable VM. #3118 added tests/test_uefi_firmware.py (75 lines) for the local path in the very same commit that made the cloud path reachable with no firmware handling at all. The local runtime has had this guard for ages:

# runtime/qemu.py:300
_locate_uefi_firmware(qemu_dir) if image.os_type == "windows" else (None, None)

and registry/qemu_builder.py:354 outright refuses to build a Windows VM without OVMF. The local path had the rule; the cloud path had nothing.

The tests that would have caught it exist — tests/integration/sandbox_sdk/test_windows_cloud_vm.py and its sandbox_cli twin, both touched by #3118 — but they are gated skipif(not os.environ.get("CUA_API_KEY")), so they skip in CI. That is the concrete mechanism behind that PR's "covered by unit tests against the typed template request, not by a live cloud call". Same local/cloud parity gap that motivated #3091 and #3118 — this time it bit the code those PRs shipped.

The fix

if self._image.os_type == "windows":
    vm_template_builder = vm_template_builder.firmware(Firmware.EFI)

Derived from the image rather than hardcoded by OS name, using the identical predicate the local runtime uses. Linux keeps the field omitted and the server-side BIOS default, matching the Linux pools that already run fine.

Known gaps deliberately left open

Image.from_registry(...) still gets BIOS. image.py:179 hardcodes os_type="linux", so a private Windows containerDisk reached that way — e.g. the cua-server-windows references used by the existing hermes-cua-pool and windows-fleet templates — still gets BIOS and still won't boot. Closing this properly needs a firmware= passthrough on FleetCloudTransport plumbed through Sandbox.create, which widens the public API surface. Left out of this PR on purpose; flagging it so someone can pick it up deliberately.

cpu/memory defaults left alone. The broken pool ran 4 cpu / 4Gi (the Fleet schema defaults from default_cpu_cores() / default_memory()) where the older hand-authored Windows pools use 10 cpu / 20Gi, which looks like a suspect. It isn't: the hand-authored windows-fleet-template also uses 4 cpu / 4Gi — with efi — and the verification boot above reached Ready on 4/4Gi. Firmware is the only differentiator between templates that boot and templates that don't, so no defaults are changed here.

.image_pull_secret("ecr-credentials") is now dead weight, not load-bearing — the public ECR pull succeeded with it attached on every VM in this investigation, including the one that booted. It was my leading suspect for this outage and it was wrong. Not removed here, because it turns out to have a real side effect worth deciding on separately: libs/fleet/backend/auth/pool_admission.rego only enforces its image allowlist when imagePullSecret == "ecr-credentials", and a template with no pull secret is allowed any image. Setting it unconditionally is therefore what makes non-allowlisted registries — e.g. ghcr.io/trycua/mini-swe:latest, which libs/python/cua-cli/README.md documents in a GitHub Actions example — fail with 403 k8s request is not allowed. That deserves its own change, not a ride-along on a firmware fix.

Not in scope, filed separately

Two other things surfaced while diagnosing this and are not addressed here:

1. OSGymSandboxWarmPool status goes stale — a fully healthy pool can report readyReplicas: 0 indefinitely. This misdirected the entire investigation into a false cluster-wide outage theory. Written up on its own.
2. The Fleet SDK holds one short-lived OIDC access token with no refresh, so any operation outliving the token (a long time_to_start, a 300s bind deadline) dies mid-flight with 401 auth token is invalid. The stored credential already carries a refresh_token; nothing consumes it.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!