diff --git a/CLAUDE.md b/CLAUDE.md index 3e008b1c6..cf07a74dd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -47,7 +47,7 @@ The system is orchestrated by `src/cli.ts` and managed by `src/docker-manager.ts - **[docs/releasing.md](docs/releasing.md)** - Release process and versioning instructions - **[docs/INTEGRATION-TESTS.md](docs/INTEGRATION-TESTS.md)** - Integration test coverage guide with gap analysis - **[docs/enclaves-architecture.md](docs/enclaves-architecture.md)** - Unified enclave architecture, MCP gateway handoff, migration, and coverage notes -- **[docs/cloud-hypervisor-foundation.md](docs/cloud-hypervisor-foundation.md)** - Cloud Hypervisor microVM foundation (config/artifacts/preflight only; not yet a runnable backend) +- **[docs/cloud-hypervisor-foundation.md](docs/cloud-hypervisor-foundation.md)** - Cloud Hypervisor v53.0 microVM backend (preview): REST API client, secure launcher (network-namespace join + privilege drop + Landlock/seccomp in place of a jailer), manager/backend, GitHub-hosted Ubuntu x86_64 KVM runners only ## Development Workflow diff --git a/README.md b/README.md index 4fb7e31e0..27699ae76 100644 --- a/README.md +++ b/README.md @@ -95,7 +95,7 @@ See [GitHub Actions](docs/github_actions.md) for advanced setup and `awf logs su - [Auth Doctor Updater workflow](.github/workflows/auth-doctor-updater.md) — daily/manual audit that opens bounded PRs with evidence-backed authentication and API-proxy documentation corrections - [Image verification](docs/image-verification.md) — cosign signature verification - [Firecracker integration (preview)](docs/firecracker-integration.md) — Firecracker v1.16.1 microVM backend: explicit opt-in, Linux/KVM only, macOS/Windows unsupported, operator-managed artifacts with mandatory SHA-256 digests, fail-closed egress, mandatory API proxy credential isolation -- [Cloud Hypervisor foundation](docs/cloud-hypervisor-foundation.md) — config/artifact plumbing, preflight validation, and guest artifact pipeline for a future Cloud Hypervisor microVM backend; **not yet a runnable runtime** +- [Cloud Hypervisor integration (preview)](docs/cloud-hypervisor-foundation.md) — Cloud Hypervisor v53.0 microVM backend: explicit opt-in, GitHub-hosted Ubuntu x86_64 KVM runners only, operator-managed artifacts with mandatory SHA-256 digests, Landlock/seccomp-confined launcher in place of a jailer, fail-closed egress, mandatory API proxy credential isolation ## Development diff --git a/docs/awf-config-spec.md b/docs/awf-config-spec.md index f789b6fe8..2e62ee7a7 100644 --- a/docs/awf-config-spec.md +++ b/docs/awf-config-spec.md @@ -76,7 +76,7 @@ following top-level properties. All are OPTIONAL: | `security` | object | Security and isolation settings | | `container` | object | Container and Docker settings | | `firecracker` | object | Firecracker v1.16.1 control-plane preview settings | -| `cloudHypervisor` | object | Cloud Hypervisor v53.0 foundation settings (artifacts/digests only; no lifecycle backend yet — see §4.1) | +| `cloudHypervisor` | object | Cloud Hypervisor v53.0 microVM preview settings (see §4.1) | | `chroot` | object | Chroot execution overrides for split-filesystem ARC/DinD runners | | `dind` | object | Bootstrap helpers for ARC/DinD split runner/daemon filesystems | | `runner` | object | Runner topology declaration (standard vs. ARC/DinD) | @@ -89,20 +89,27 @@ following top-level properties. All are OPTIONAL: Property-level constraints, types, and descriptions are defined normatively by `docs/awf-config.schema.json`. -### 4.1 Cloud Hypervisor foundation (not yet a runnable backend) +### 4.1 Cloud Hypervisor microVM preview The `cloudHypervisor` surface pins Cloud Hypervisor v53.0 artifacts and digests (binary, PCI-capable guest kernel, rootfs, and the shared AWF guest -supervisor) so a config document can be prepared and round-tripped ahead of -time. **It has no lifecycle backend in this release**: `cloud-hypervisor` is -not a valid `container.containerRuntime` value, and supplying -`cloudHypervisor` options does not execute any workload. Supported host -target is GitHub-hosted Ubuntu `x86_64` runners with KVM only; self-hosted -and non-Ubuntu/non-x86_64 hosts are out of scope. See +supervisor) and, like Firecracker, requires explicit +`--cloud-hypervisor-preview` opt-in plus `container.containerRuntime: +"cloud-hypervisor"` to execute a workload. Supported host target is +GitHub-hosted Ubuntu `x86_64` runners with KVM only — self-hosted and +non-Ubuntu/non-x86_64 hosts are rejected explicitly by +[`src/cloud-hypervisor/host-eligibility.ts`](../src/cloud-hypervisor/host-eligibility.ts), +unlike Firecracker's preview which permits self-hosted hosts. See [`src/cloud-hypervisor/preflight.ts`](../src/cloud-hypervisor/preflight.ts) -for the artifact/host trust-check module and -[`guest/cloud-hypervisor/`](../guest/cloud-hypervisor/) for the guest -artifact build/verification pipeline. +for the artifact/host trust-check module, +[`src/cloud-hypervisor/launcher.ts`](../src/cloud-hypervisor/launcher.ts) +for the secure host launcher (network-namespace join, privilege drop, and +Landlock-based filesystem confinement in place of Firecracker's jailer), +[`src/cloud-hypervisor/manager.ts`](../src/cloud-hypervisor/manager.ts) for +the VM lifecycle, and [`guest/cloud-hypervisor/`](../guest/cloud-hypervisor/) +for the guest artifact build/verification pipeline. See +[docs/cloud-hypervisor-foundation.md](./cloud-hypervisor-foundation.md) for +the full architecture and security-boundary writeup. ## 5. CLI Mapping @@ -220,7 +227,7 @@ AWF settings MAY be supplied via config files, including stdin (`--config -`). - `firecracker.sha256.kernel` → `--firecracker-kernel-sha256` - `firecracker.sha256.rootfs` → `--firecracker-rootfs-sha256` - `firecracker.sha256.supervisor` → `--firecracker-supervisor-sha256` -- `cloudHypervisor.previewEnabled` → `--cloud-hypervisor-preview` *(foundation only; does not enable workload execution)* +- `cloudHypervisor.previewEnabled` → `--cloud-hypervisor-preview` *(requires `container.containerRuntime: "cloud-hypervisor"` and a GitHub-hosted Ubuntu x86_64 KVM runner to execute a workload)* - `cloudHypervisor.cloudHypervisorBinary` → `--cloud-hypervisor-binary` - `cloudHypervisor.kernelPath` → `--cloud-hypervisor-kernel` - `cloudHypervisor.rootfsPath` → `--cloud-hypervisor-rootfs` diff --git a/docs/awf-config.schema.json b/docs/awf-config.schema.json index ff1f1f99d..8a6b3527c 100644 --- a/docs/awf-config.schema.json +++ b/docs/awf-config.schema.json @@ -730,13 +730,13 @@ }, "cloudHypervisor": { "type": "object", - "description": "Cloud Hypervisor v53.0 foundation configuration (artifacts and digests only). There is no lifecycle backend yet: this cannot be selected via container.containerRuntime and cannot execute workloads.", + "description": "Cloud Hypervisor v53.0 microVM preview configuration. Requires container.containerRuntime: \"cloud-hypervisor\" and previewEnabled to execute workloads; supported only on GitHub-hosted Ubuntu x86_64 KVM runners.", "additionalProperties": false, "properties": { "previewEnabled": { "type": "boolean", "default": false, - "description": "Reserve Cloud Hypervisor configuration for a future preview. Has no effect on workload execution in this release." + "description": "Enable the Cloud Hypervisor v53.0 workload-execution preview. Requires container.containerRuntime: \"cloud-hypervisor\" and a GitHub-hosted Ubuntu x86_64 KVM runner." }, "cloudHypervisorBinary": { "type": "string", diff --git a/docs/cloud-hypervisor-foundation.md b/docs/cloud-hypervisor-foundation.md index 3b7d67655..af83b7828 100644 --- a/docs/cloud-hypervisor-foundation.md +++ b/docs/cloud-hypervisor-foundation.md @@ -1,100 +1,349 @@ --- -title: Cloud Hypervisor foundation (not yet a runnable backend) -description: Pinned versions/digests, configuration surface, preflight/artifact validation module, and guest artifact pipeline for the Cloud Hypervisor microVM foundation. There is no lifecycle backend yet. +title: Cloud Hypervisor integration (preview) +description: Cloud Hypervisor v53.0 microVM backend — REST API client, secure launcher, manager/backend, GitHub-hosted Ubuntu x86_64 KVM runners only, Landlock/seccomp confinement in place of a jailer. --- -:::caution[Foundation only — no lifecycle backend] -This document describes **preparatory** work for a future Cloud Hypervisor -microVM backend: configuration/artifact plumbing, fail-closed preflight -validation, and a guest artifact build pipeline. **`cloud-hypervisor` is not a -valid `--container-runtime` value in this release** and no workload can be -executed with it. Firecracker (see -[Firecracker microVM integration (preview)](./firecracker-integration.md)) -continues to work unchanged and is unaffected by this foundation. +:::caution[Preview — GitHub-hosted Ubuntu x86_64 KVM runners only] +Cloud Hypervisor is an explicit-opt-in preview, exactly like +[Firecracker](./firecracker-integration.md). It requires +`--cloud-hypervisor-preview` **and** `--container-runtime cloud-hypervisor`, +runs only on GitHub-hosted Ubuntu x86_64 KVM runners (self-hosted runners +are rejected, unlike Firecracker's preview), and is otherwise fail-closed. +Firecracker continues to work unchanged and is unaffected by this backend. ::: -## What this adds - -- A `cloudHypervisor` config-file block and matching `--cloud-hypervisor-*` - CLI flags (see [`docs/awf-config-spec.md`](./awf-config-spec.md) §4.1 for - the normative property list and CLI mapping) so artifact paths and SHA-256 - digests can be pinned and round-tripped through config today. -- [`src/cloud-hypervisor/preflight.ts`](../src/cloud-hypervisor/preflight.ts): - fail-closed host and artifact validation mirroring - [`src/firecracker/preflight.ts`](../src/firecracker/preflight.ts) — pinned - version parsing, trusted-owner/non-writable regular-file checks for the - binary/kernel/rootfs/supervisor, digest verification, `/dev/kvm` access, - and required trusted host tools (`ip`, `nft`, `sysctl`, `mke2fs`, - `debugfs`, `e2fsck`, `rsync`). -- [`src/cloud-hypervisor/host-eligibility.ts`](../src/cloud-hypervisor/host-eligibility.ts): - a narrow, independently-tested helper distinguishing GitHub-hosted Ubuntu - x86_64 KVM runners from self-hosted or non-Ubuntu hosts. This is a - necessary-but-not-sufficient check — it does not open `/dev/kvm` or verify - artifacts; `runCloudHypervisorPreflight` does that. -- [`guest/cloud-hypervisor/build-test-artifacts.sh`](../guest/cloud-hypervisor/build-test-artifacts.sh) - and - [`verify-test-artifacts.sh`](../guest/cloud-hypervisor/verify-test-artifacts.sh): - a reproducible guest artifact pipeline that produces a pinned Cloud - Hypervisor binary, a PCI-capable guest Linux kernel, a deterministic raw - ext4 rootfs (BusyBox + CA bundle + the shared AWF guest supervisor), - `SHA256SUMS`, `manifest.json`, and an SPDX SBOM. - -## Pinned versions and digests +This is **stack layer 3** of a 4-layer PR stack: it builds a complete, +runnable Cloud Hypervisor microVM backend on top of the layer 1 VMM-neutral +`src/microvm/` primitives and the layer 2 configuration/artifact/preflight +foundation. Layer 4 adds the live-KVM GitHub Actions CI workflow. + +## Part 1 — What Cloud Hypervisor adds and why it is a preview + +### What Cloud Hypervisor is + +[Cloud Hypervisor](https://www.cloudhypervisor.org/) is a Rust VMM built on +`rust-vmm` crates, offering a REST API (`/api/v1`) over a Unix domain socket +for VM lifecycle management. AWF uses it as a second, alternative microVM +backend to Firecracker — same threat model (hypervisor-isolated agent +execution, mandatory network egress control, mandatory API proxy credential +isolation), different VMM implementation and host launch strategy. + +### Why it is a preview + +- It is supported **only** on GitHub-hosted Ubuntu x86_64 KVM runners (see + Part 4). Self-hosted runners, other architectures, and other operating + systems are all explicitly rejected. +- Cloud Hypervisor has **no jailer-equivalent process**. AWF replaces + jailer's chroot+pivot_root+capability-drop with a different (not weaker) + boundary: network-namespace join, non-root privilege drop, and + kernel-enforced Landlock filesystem confinement (see Part 3). +- The live-KVM GitHub Actions workflow that exercises this backend end to + end on real hardware is a later layer; this layer's validation is unit + tests plus static analysis only (see Part 15). + +### Comparison with Firecracker + +| Aspect | Firecracker | Cloud Hypervisor | +|---|---|---| +| Control plane | REST over UDS, Firecracker-specific endpoints | REST over UDS, `/api/v1`, upstream OpenAPI-documented | +| Privileged launcher | `jailer` binary (chroot, cgroup, netns join, uid/gid drop) | None — AWF's own launcher (netns join via `ip netns exec`, privilege drop via `setpriv`, Landlock via VM config) | +| Bus for block/net/vsock | MMIO (`pci=off`) | virtio-**pci** (PCI required; no MMIO transport) | +| Host support | Linux/KVM, GitHub-hosted or self-hosted | GitHub-hosted Ubuntu x86_64 KVM only | +| Guest kernel, rootfs, supervisor | Own pinned artifacts | **Shared** pinned kernel config and guest supervisor binary with Firecracker | +| Resource limits | jailer's own cgroup (no explicit quotas set by AWF) | AWF creates and assigns an explicit memory/CPU/PID cgroup | + +## Part 2 — Architecture + +### Host-side components + +1. **`src/cloud-hypervisor/api-client.ts`** — `CloudHypervisorApiClient`, a + typed REST client over the Unix domain socket, implementing exactly the + endpoints AWF needs: `vmm.ping`, `vm.create`, `vm.boot`, `vm.info`, + `vm.counters`, `vm.shutdown`, `vmm.shutdown`. Every request has a bounded + wall-clock timeout and a 1 MiB response cap; error bodies (Cloud + Hypervisor's chained-error-message JSON arrays) are parsed into a single + readable message. +2. **`src/cloud-hypervisor/launcher.ts`** — pure functions and small classes + for the secure host launch: + - `buildCloudHypervisorLaunchCommand()` builds the exact argv AWF spawns: + `ip netns exec setpriv --reuid= --regid= + --groups= --no-new-privs --inh-caps=-all --bounding-set=-all + -- cloud-hypervisor --api-socket path= --log-file -v + --seccomp true`. `--groups=` replaces the operator's full + supplementary group list with only the group that owns `/dev/kvm` + (resolved by preflight) — a blanket `--clear-groups` would also drop + kvm-group access and make every real launch fail with EACCES. No + shell is ever invoked — this argv is passed directly to `execa`, + never interpolated into a shell string. + - `computeCloudHypervisorLandlockRules()` computes the minimal + `landlock_rules` list sent in the `vm.create` payload. + - `CloudHypervisorCgroup` manages a cgroup v2 hierarchy: it enables + `cpu`/`memory`/`pids` delegation (`cgroup.subtree_control`) at the + cgroup root and the shared parent directory before creating the + per-run leaf cgroup (cgroup v2 only materializes a controller's + interface files in a child once the parent delegates it), writes + explicit `memory.max`/`cpu.max`/`pids.max`, and assigns the launched + process's PID to it. Cleanup uses a plain `rmdir` on the leaf — + cgroupfs's controller files are virtual and a recursive `rm` fails. + `runCloudHypervisorPreflight` rejects cgroup v1-only hosts explicitly + (see Part 4) rather than falling back to a v1 hierarchy this class + does not manage. +3. **`src/cloud-hypervisor/manager.ts`** — `CloudHypervisorManager` owns one + run end to end: preflight → network namespace setup (reusing + `src/microvm/network.ts` unchanged) → workspace image preparation + (reusing `src/microvm/workspace.ts` unchanged) → private run-directory + staging → cgroup setup → launch → API-socket readiness → `vm.create` → + (later) `vm.boot` → VSOCK guest-supervisor connect (reusing + `src/microvm/vsock-client.ts` and `guest-protocol.ts` unchanged) → + execution → graceful `vm.shutdown`/`vmm.shutdown` → process termination + → workspace extraction → network/cgroup/run-directory cleanup, with + aggregated cleanup-error reporting matching Firecracker's manager. +4. **`src/cloud-hypervisor-runtime-backend.ts`** — `CloudHypervisorRuntimeBackend` + implements `ExternalAgentRuntimeBackend`: infrastructure discovery + (`src/microvm/infrastructure.ts`, unchanged), credential-safe guest + environment construction, guest connectivity probing (Squid + API + proxy), cancellation/timeout/exit-code handling, and diagnostics + collection — structurally identical to `FirecrackerRuntimeBackend`. + +### Guest contents + +Unchanged from layer 2 and shared with Firecracker: the PCI-capable guest +kernel (built from Firecracker's pinned `microvm-kernel-ci-x86_64-6.1.config`), +a deterministic BusyBox + CA-bundle ext4 rootfs, and the VMM-neutral +`awf-supervisor` guest binary (`guest/firecracker-supervisor/`, unmodified). + +### Control flow + +``` +awf --container-runtime cloud-hypervisor --cloud-hypervisor-preview ... + ↓ +assertCloudHypervisorRuntimeCompatibility() — security mode, topology, GitHub-hosted host eligibility, artifact/digest completeness + ↓ +runCloudHypervisorPreflight() — Linux/KVM/x86_64, /dev/kvm + owning gid, cgroup v2 (v1-only hosts rejected), trusted host tools incl. setpriv, trusted+pinned+digest-verified artifacts + ↓ +CloudHypervisorManager.start() + ↓ +MicrovmNetworkManager.setup() (netns, veth, TAP, nftables — shared with Firecracker) + ↓ +MicrovmWorkspaceImage.prepare() (workspace + rootfs staging — shared with Firecracker) + ↓ +private run directory under /run/awf-cloud-hypervisor (0711 ancestors, 0700 leaf owned by the non-root identity) + per-run cgroup v2 (subtree_control delegated root→parent→leaf) + ↓ +buildCloudHypervisorLaunchCommand() → ip netns exec → setpriv --groups= → cloud-hypervisor --api-socket ... --seccomp true (minimal PATH-only environment) + ↓ +wait for API socket → vmm.ping → vm.create (landlock_enable: true, minimal landlock_rules) + ↓ +CloudHypervisorRuntimeBackend.start(): vm.boot → VSOCK connect (CID 3, CONNECT \n) → Squid/API-proxy connectivity probe + ↓ +Agent command executes inside the guest via the VSOCK guest-protocol transport (unchanged) + ↓ +graceful guest shutdown → vm.shutdown → vmm.shutdown → SIGTERM/SIGKILL fallback → workspace extraction → network/cgroup/run-directory cleanup +``` + +## Part 3 — Security boundary: the launcher in place of a jailer + +Cloud Hypervisor ships as a single static binary with **no jailer +equivalent** — nothing that atomically joins a network namespace, chroots, +drops capabilities, and execs the VMM. Reimplementing jailer's chroot + +`pivot_root` for a foreign binary was judged impractical and risky within +this layer, so AWF instead documents and tests an explicit replacement +boundary: + +1. **Network namespace join.** `ip netns exec ...` execs + directly into the namespace `src/microvm/network.ts` already prepared + (the same TAP/veth/nftables topology Firecracker uses), without an + intermediate fork — the resulting process keeps the PID the host + observes for cgroup assignment. +2. **Privilege drop.** `setpriv --reuid= --regid= + --groups= --no-new-privs --inh-caps=-all --bounding-set=-all` + execs Cloud Hypervisor as the same non-root operator identity + Firecracker's jailer targets (`SUDO_UID`/`SUDO_GID`), with an empty + capability bounding set and `no_new_privs` set, before any guest code + runs. `--groups=` replaces the operator's supplementary group + list with only the group that owns `/dev/kvm` (resolved by preflight); + a blanket `--clear-groups` would also drop that membership and make + every real launch fail opening `/dev/kvm` even though root-run + preflight passed. +3. **Filesystem confinement.** In place of jailer's userspace chroot, AWF + combines: + - a **private run directory** under `/run/awf-cloud-hypervisor///` + — deliberately **outside** `workDir` (which is root-owned `0700` + because it holds `docker-compose.yml`'s plaintext secrets). Since + there is no `chroot()` to make host-side ancestor permissions + irrelevant, the non-root launched process must be able to really + traverse down to the run directory: the two ancestor levels are + `0711` (traversable/executable by any uid, but not listable), and + only the per-run leaf directory is chowned to the target identity + with `0700` (so only that identity, or root, can read its contents); + - **Landlock**, a Linux LSM, enabled via `landlock_enable: true` in the + `vm.create` payload with a minimal `landlock_rules` list (kernel image + read-only; rootfs, workspace, and the run directory read-write; + `/dev/kvm` and `/dev/net/tun` read-write for KVM ioctls and TAP + attachment). Any path not listed becomes inaccessible to the Cloud + Hypervisor process the instant Landlock is enabled — enforced by the + kernel, not a userspace boundary a compromised process could bypass. + - Cloud Hypervisor's own **default seccomp filter** (`--seccomp true`, + its default kill-on-violation mode). + + This is a **different** boundary than jailer's chroot — kernel-LSM-based + rather than mount-namespace-based — not a weaker one. It is exercised by + `src/cloud-hypervisor/launcher.test.ts` (argv construction, Landlock rule + computation, cgroup lifecycle) rather than silently degraded to "no + filesystem confinement". +4. **Resource limits.** A dedicated cgroup **v2** hierarchy is created + before launch: `cpu`/`memory`/`pids` delegation is enabled at the + cgroup root and the shared parent directory (`cgroup.subtree_control`) + before the per-run leaf cgroup is created, then explicit + `memory.max`/`cpu.max`/`pids.max` are written and the launched + process's PID is assigned to it immediately after spawn. Cgroup v1-only + hosts are rejected explicitly at preflight (see Part 4) rather than + silently constructing a broken multi-controller v1 hierarchy. +5. **The management API socket is never guest-accessible.** It lives only + in the host-side private run directory; it is never passed to the guest + as a drive, vsock, or virtio-fs device, and Landlock additionally blocks + any *new* open() of it by the Cloud Hypervisor process after `vm.create` + (the already-open listening socket is unaffected, matching how a + jailer-chrooted Firecracker keeps its already-open resources). +6. **Minimal launcher environment.** The launched process receives an + explicit minimal environment (just `PATH`), never `process.env` — + Cloud Hypervisor directly parses untrusted guest/device input, so a VMM + compromise reading its own inherited environment could otherwise read + provider/GitHub credentials and bypass the API-proxy credential + isolation boundary entirely. + +## Part 4 — Prerequisites and supported hosts + +### Supported host configurations + +| Requirement | Value | +|-------------|-------| +| Operating system | **Linux only**, and additionally **GitHub-hosted only** (`GITHUB_ACTIONS=true`, `RUNNER_ENVIRONMENT=github-hosted`) | +| Distribution | Ubuntu (`ImageOS` must start with `ubuntu`) | +| Architecture | x86_64 only | +| KVM device | `/dev/kvm` must exist and be readable + writable | +| Cgroup hierarchy | **cgroup v2 unified only** (`/sys/fs/cgroup/cgroup.controllers` must exist) — cgroup v1-only hosts are rejected explicitly; see `CloudHypervisorCgroup` in Part 3 | +| Self-hosted runners | **Explicitly rejected** — see `src/cloud-hypervisor/host-eligibility.ts` | + +Host eligibility is checked in two layers: `evaluateGithubHostedRunnerEligibility()` +(host identity only — cheap, environment-variable based) and the full +`runCloudHypervisorPreflight()` (live capability checks: `/dev/kvm`, cgroup +version, trusted host tools, artifact trust/digests). + +### Required host tools + +Same as Firecracker (`ip`, `nft`, `sysctl`, `mke2fs`, `debugfs`, `e2fsck`, +`rsync`), plus: + +| Tool | Purpose | +|------|---------| +| `setpriv` | Drops to the non-root operator uid/gid with an empty capability set before Cloud Hypervisor execs (util-linux; standard on Ubuntu) | + +### Operator account + +Same as Firecracker: AWF must be invoked through `sudo` from a **non-root** +account; `SUDO_UID`/`SUDO_GID` determine the target identity for both the +launcher's `setpriv` step and the guest execution identity. The target +account must have `/dev/kvm` access (typically via `kvm` group membership). + +## Part 5 — Artifact policy + +Identical trust model to Firecracker (see +[Firecracker's Part 5](./firecracker-integration.md#part-5--artifact-policy)): +root/operator-owned non-writable regular files, trusted ancestor +directories, pinned version, mandatory SHA-256 digests. Cloud Hypervisor has +no jailer-equivalent binary, so there is one fewer artifact to pin than +Firecracker (no jailer digest). | Artifact | Version | SHA-256 | |---|---|---| | `cloud-hypervisor` (x86_64 static) | v53.0 | `448af3d4e59b22c2987f7df94c213ad40fb53a10d437e42b5ee6c4fce7c29ecc` | | Linux kernel source | 6.1.141 | `bc3c45faf6f5f0450666c75fa9dad9bc7c0cf7c7cba0dbd94e5cfdc58229c116` | -| Kernel config | firecracker v1.16.1 `microvm-kernel-ci-x86_64-6.1.config` | `adbc70ab5e89213ba00594b12d25e09bdf8bb1ed3c252d7449326bb14c22963b` | +| Kernel config (from Firecracker v1.16.1) | `microvm-kernel-ci-x86_64-6.1.config` | `adbc70ab5e89213ba00594b12d25e09bdf8bb1ed3c252d7449326bb14c22963b` | | BusyBox source | 1.36.1 | `b8cc24c9574d809e7279c3be349795c5d5ceb6fdf19ca709f80cde50e47de314` | | CA bundle | 2025-02-25 | `50a6277ec69113f00c5fd45f09e8b97a4b3e32daa35d3a95ab30137a55386cef` | -Cloud Hypervisor v53.0 was the current upstream release as of this writing -([cloud-hypervisor/cloud-hypervisor releases](https://github.com/cloud-hypervisor/cloud-hypervisor/releases)). -Both the binary digest and the source-tarball digest were independently -verified against the GitHub release assets before pinning. - -### Why the guest kernel reuses the Firecracker config - -`guest/cloud-hypervisor/build-test-artifacts.sh` intentionally builds from the -**same Linux kernel source and the same pinned Firecracker -`microvm-kernel-ci-x86_64-6.1.config`** used by -[`guest/firecracker/build-test-artifacts.sh`](../guest/firecracker/build-test-artifacts.sh). -That config already enables everything Cloud Hypervisor's direct-kernel-boot, -virtio-pci transport needs — `CONFIG_PCI`, `CONFIG_VIRTIO_PCI`, -`CONFIG_PCI_MMCONFIG` (ACPI MCFG/PCIe ECAM), `CONFIG_VIRTIO_BLK`, -`CONFIG_VIRTIO_NET`, `CONFIG_VIRTIO_CONSOLE`, `CONFIG_VSOCKETS` / -`CONFIG_VIRTIO_VSOCKETS`, `CONFIG_EXT4_FS`, and `CONFIG_PVH` for -firmware-less direct boot — while leaving virtio-fs, VFIO, vhost-user, vDPA, -snapshot/restore, hotplug, and confidential-computing options off. Reusing -one reviewed, pinned kernel config keeps both VMM backends' guest kernels -identical instead of maintaining a second hand-curated config. - -### Why the guest supervisor is shared, unmodified - -[`guest/firecracker-supervisor/`](../guest/firecracker-supervisor/) documents -itself as VMM-neutral: its length-prefixed JSON framing protocol -(`protocol.go`) mirrors `src/microvm/guest-protocol.ts` on the host side and -does not depend on any Firecracker-specific transport. The Cloud Hypervisor -guest pipeline invokes -`guest/firecracker-supervisor/build.sh` as-is to produce the same -`awf-supervisor` binary used in both guest rootfs images. - -## Explicit scope limits (this layer) +## Part 6 — Devices, boot, and networking + +- **Boot**: direct kernel boot (no UEFI/firmware layer), root device + `/dev/vda`, workspace device `/dev/vdb`, `rootfstype=ext4`, `rw`, + `net.ifnames=0 biosdevname=0` for deterministic `eth0` naming. Unlike + Firecracker, `pci=off` is **not** set — Cloud Hypervisor requires PCI. +- **Devices**: virtio-**pci** block (rootfs, workspace), net (single TAP, + pre-created and owned exactly like Firecracker's), vsock (CID 3, same + `CONNECT \n` transport and guest-protocol framing as Firecracker), + serial console redirected to a bounded host log file, virtio-console + disabled (`mode: "Off"`). No virtio-fs, snapshots, migration, hotplug, + VFIO, vhost-user, vDPA, TDX/SEV, or TPM. +- **Networking**: the exact same TAP/netns/nftables design as Firecracker + (`src/microvm/network.ts`, unmodified) — mandatory network isolation, + mandatory API proxy credential isolation, identical egress ACL. + +## Part 7 — Bounded diagnostics + +`CloudHypervisorManager.collectDiagnostics()` writes, all under a +0700-mode directory with 0600-mode files bounded to 1 MiB each: launcher +stdout/stderr capture, the Cloud Hypervisor log file, the guest serial +console log, `vm.counters()` output (best-effort — failures don't block +diagnostics), the resolved network plan, and a `runtime.json` summary. This +mirrors Firecracker's `collectDiagnostics()` shape exactly. + +## Part 8 — CLI reference + +```bash +sudo awf \ + --container-runtime cloud-hypervisor \ + --cloud-hypervisor-preview \ + --cloud-hypervisor-binary /usr/local/bin/cloud-hypervisor \ + --cloud-hypervisor-kernel /opt/awf/vmlinux \ + --cloud-hypervisor-rootfs /opt/awf/rootfs.ext4 \ + --cloud-hypervisor-supervisor /opt/awf/awf-supervisor \ + --cloud-hypervisor-binary-sha256 \ + --cloud-hypervisor-kernel-sha256 \ + --cloud-hypervisor-rootfs-sha256 \ + --cloud-hypervisor-supervisor-sha256 \ + --enable-api-proxy \ + --allow-domains github.com \ + -- npx @github/copilot --prompt "list files" +``` + +See [`docs/awf-config-spec.md`](./awf-config-spec.md) §4.1 for the full +config-file/CLI mapping. + +## Part 9 — Explicit scope limits (this layer) - **Direct kernel boot only.** No UEFI/firmware layer. -- **Raw ext4 disks only.** No virtio-fs, snapshot/restore, hotplug, VFIO, - vhost-user, vDPA, or confidential computing. +- **Raw ext4 disks only**, `backing_files: false`. No virtio-fs, + snapshot/restore, hotplug, VFIO, vhost-user, vDPA, or confidential + computing. - **virtio-pci transport only** for block, net, and vsock devices. - **GitHub-hosted Ubuntu x86_64 KVM runners only.** Self-hosted runners and - non-Ubuntu/non-x86_64 hosts are explicitly out of scope; see - `evaluateGithubHostedRunnerEligibility()` in - [`src/cloud-hypervisor/host-eligibility.ts`](../src/cloud-hypervisor/host-eligibility.ts). -- **No REST control-plane client, launcher, or manager.** No - `--container-runtime cloud-hypervisor` registration. No live KVM - integration test. These arrive in a later layer once the lifecycle backend - exists. - -Passing `--container-runtime cloud-hypervisor` fails immediately with an -explicit error (`assertCloudHypervisorNotYetAvailable`) instead of silently -falling through to the generic unknown-runtime passthrough. + non-Ubuntu/non-x86_64 hosts are explicitly rejected. +- **No TTY, DinD, host access, extra volume mounts, enclaves, or topology + peers** — same restrictions as Firecracker's preview. +- **No live-KVM GitHub Actions workflow yet.** This layer's validation is + unit tests, `tsc --noEmit`, and the existing Firecracker live-KVM + workflow (unaffected). A dedicated Cloud Hypervisor live-KVM smoke test + is a later layer's responsibility — see Part 15. + +## Part 15 — Validation performed in this layer + +- `tsc --noEmit -p tsconfig.check.json`: clean. +- Full Jest suite: all suites passing, including new coverage for the API + client (typed requests, chained-error parsing, timeout/size bounds), + launcher (argv construction, kvm-gid retention, Landlock rule + computation, cgroup v2 subtree_control delegation ordering and rmdir-only + cleanup), manager (launch, partial-start rollback, workspace/vsock + lifecycle, keep-mode preservation, termination-confirmation retry, + natural-exit wait, `vm.create` failure rollback, signal-exit fast-fail, + bounded diagnostics with `vm.counters`), backend (host-eligibility gating, + stdin serialization, TTY rejection, credential-safe environment, + cancellation), and runtime registration/preview-gate wiring. +- No new shell scripts were introduced — the launcher builds argv arrays + passed directly to `execa`; there is nothing to `shellcheck`/`bash -n`. +- `guest/firecracker-supervisor` Go tests (`go vet`, `go test`): unaffected, + confirming the shared guest supervisor still works for both backends. +- **Not performed in this layer**: an actual boot on real KVM hardware. + This environment has no `/dev/kvm` and no built Cloud Hypervisor guest + artifacts, so a live smoke test would be faked, not validated. This is + explicitly deferred to the layer that adds the dedicated live-KVM GitHub + Actions workflow. diff --git a/src/awf-config-schema.json b/src/awf-config-schema.json index ff1f1f99d..8a6b3527c 100644 --- a/src/awf-config-schema.json +++ b/src/awf-config-schema.json @@ -730,13 +730,13 @@ }, "cloudHypervisor": { "type": "object", - "description": "Cloud Hypervisor v53.0 foundation configuration (artifacts and digests only). There is no lifecycle backend yet: this cannot be selected via container.containerRuntime and cannot execute workloads.", + "description": "Cloud Hypervisor v53.0 microVM preview configuration. Requires container.containerRuntime: \"cloud-hypervisor\" and previewEnabled to execute workloads; supported only on GitHub-hosted Ubuntu x86_64 KVM runners.", "additionalProperties": false, "properties": { "previewEnabled": { "type": "boolean", "default": false, - "description": "Reserve Cloud Hypervisor configuration for a future preview. Has no effect on workload execution in this release." + "description": "Enable the Cloud Hypervisor v53.0 workload-execution preview. Requires container.containerRuntime: \"cloud-hypervisor\" and a GitHub-hosted Ubuntu x86_64 KVM runner." }, "cloudHypervisorBinary": { "type": "string", diff --git a/src/cli-options.ts b/src/cli-options.ts index c2394aeb0..680917a7f 100644 --- a/src/cli-options.ts +++ b/src/cli-options.ts @@ -12,7 +12,7 @@ const optionGroupHeaders: Record = { 'allow-domains': 'Domain Filtering:', 'build-local': 'Image Management:', 'firecracker-preview': 'Firecracker Preview:', - 'cloud-hypervisor-preview': 'Cloud Hypervisor Preview (foundation only, not yet runnable):', + 'cloud-hypervisor-preview': 'Cloud Hypervisor Preview (GitHub-hosted Ubuntu x86_64 KVM only):', 'env': 'Container Configuration:', 'dns-servers': 'Network & Security:', 'upstream-proxy': 'Network & Security:', @@ -178,6 +178,8 @@ program ' "gvisor" — OCI runtime via Docker Compose (translates to runsc).\n' + ' "sbx" — Docker sbx microVM with hypervisor isolation.\n' + ' "firecracker" — explicit Linux/KVM Firecracker v1.16.1 preview.\n' + + ' "cloud-hypervisor" — explicit GitHub-hosted Ubuntu x86_64 KVM\n' + + ' Cloud Hypervisor v53.0 preview.\n' + ' Unknown values are passed through as raw Docker runtime names.' ) .option( @@ -200,15 +202,16 @@ program .option('--firecracker-rootfs-sha256 ', 'Expected SHA-256 digest of the guest rootfs.') .option('--firecracker-supervisor-sha256 ', 'Expected SHA-256 digest of the AWF guest supervisor.') - // -- Cloud Hypervisor Preview (foundation only) -- + // -- Cloud Hypervisor Preview -- // - // NOTE: these flags only accept and validate configuration/artifact inputs. - // There is no lifecycle backend yet, so "cloud-hypervisor" is NOT a valid - // --container-runtime value and no workload can be executed with it. + // NOTE: like Firecracker, this requires explicit --cloud-hypervisor-preview + // opt-in plus --container-runtime cloud-hypervisor and is supported only + // on GitHub-hosted Ubuntu x86_64 KVM runners (self-hosted is unsupported). .option( '--cloud-hypervisor-preview', - `Reserve Cloud Hypervisor v${CLOUD_HYPERVISOR_RELEASE_VERSION} configuration for a future preview.\n` + - ' Foundation only: no runtime is registered yet and this cannot execute workloads.', + `Enable the Cloud Hypervisor v${CLOUD_HYPERVISOR_RELEASE_VERSION} workload-execution preview.\n` + + ' GitHub-hosted Ubuntu x86_64 KVM runners only. Requires local Docker\n' + + ' and pinned guest artifacts.', false ) .option('--cloud-hypervisor-binary ', `Path to the Cloud Hypervisor v${CLOUD_HYPERVISOR_RELEASE_VERSION} binary.`) diff --git a/src/cloud-hypervisor-runtime-backend.test.ts b/src/cloud-hypervisor-runtime-backend.test.ts new file mode 100644 index 000000000..3de1dc895 --- /dev/null +++ b/src/cloud-hypervisor-runtime-backend.test.ts @@ -0,0 +1,453 @@ +import { PassThrough } from 'stream'; +import type { WrapperConfig } from './types'; +import * as hostEligibility from './cloud-hypervisor/host-eligibility'; +import { + CloudHypervisorRuntimeBackend, + assertCloudHypervisorPreSecurityCompatibility, + buildCloudHypervisorGuestEnvironment, + cloudHypervisorRuntimeTestHelpers, + createCloudHypervisorRuntimeBackend, + type CloudHypervisorRuntimeBackendDependencies, +} from './cloud-hypervisor-runtime-backend'; +import { assertCloudHypervisorSelection } from './cloud-hypervisor/runtime-validation'; +import type { MicrovmInfrastructureSnapshot } from './microvm/infrastructure'; + +const digest = 'a'.repeat(64); + +function config(overrides: Partial = {}): WrapperConfig { + return { + containerRuntime: 'cloud-hypervisor', + cloudHypervisor: { + previewEnabled: true, + cloudHypervisorBinary: '/opt/cloud-hypervisor', + kernelPath: '/opt/kernel', + rootfsPath: '/opt/rootfs', + supervisorPath: '/opt/supervisor', + vcpuCount: 2, + memoryMib: 512, + apiTimeoutMs: 5000, + sha256: { + cloudHypervisor: digest, + kernel: digest, + rootfs: digest, + supervisor: digest, + }, + }, + agentCommand: 'printf hello', + allowedDomains: ['github.com'], + workDir: '/tmp/awf', + keepContainers: false, + networkIsolation: true, + legacySecurity: false, + enableApiProxy: true, + enableDind: false, + enableHostAccess: false, + tty: false, + logLevel: 'info', + buildLocal: false, + skipPull: true, + imageRegistry: 'registry', + imageTag: 'tag', + envAll: false, + sslBump: false, + enableDlp: false, + ...overrides, + } as WrapperConfig; +} + +function infrastructure(): MicrovmInfrastructureSnapshot { + return { + networkId: 'a'.repeat(64), + bridgeName: 'br-aaaaaaaaaaaa', + subnet: '172.30.0.0/24', + gateway: '172.30.0.1', + squidIp: '172.30.0.10', + apiProxyIp: '172.30.0.30', + revalidate: jest.fn().mockResolvedValue(undefined), + }; +} + +const preflightResult = { + version: '53.0', + cloudHypervisorBinary: '/opt/cloud-hypervisor', + kernelPath: '/opt/kernel', + rootfsPath: '/opt/rootfs', + supervisorPath: '/opt/supervisor', + cgroupVersion: 2 as const, + kvmGid: 978, + tools: { + ip: '/usr/bin/ip', + nft: '/usr/sbin/nft', + sysctl: '/usr/sbin/sysctl', + mke2fs: '/usr/sbin/mke2fs', + debugfs: '/usr/sbin/debugfs', + e2fsck: '/usr/sbin/e2fsck', + rsync: '/usr/bin/rsync', + setpriv: '/usr/bin/setpriv', + }, +}; + +function harness(overrides: Partial = {}) { + const order: string[] = []; + const stdin = new PassThrough(); + const manager = { + paths: { runDirectory: '/tmp/awf/cloud-hypervisor-run/cloud-hypervisor/test' }, + guestIp: '100.64.0.2', + networkNamespace: 'awffc-test', + start: jest.fn(async () => { order.push('vm-config'); }), + startInstance: jest.fn(async () => { order.push('vm-start'); }), + execute: jest.fn() + .mockImplementationOnce(async () => { + order.push('probe'); + return { requestId: 'probe', exitCode: 0, signal: null, timedOut: false }; + }) + .mockImplementationOnce(async () => ({ + requestId: 'agent', + exitCode: 23, + signal: null, + timedOut: false, + })), + cancel: jest.fn().mockResolvedValue(undefined), + writeStdin: jest.fn().mockResolvedValue(undefined), + endStdin: jest.fn().mockResolvedValue(undefined), + collectDiagnostics: jest.fn().mockResolvedValue(undefined), + stop: jest.fn(async () => { order.push('vm-stop'); }), + }; + const infra = infrastructure(); + (infra.revalidate as jest.Mock).mockImplementation(async () => { + order.push('revalidate'); + }); + const deps: CloudHypervisorRuntimeBackendDependencies = { + startInfrastructure: jest.fn(async () => { order.push('compose'); }), + preflight: jest.fn(async () => { order.push('preflight'); return preflightResult; }), + resolveInfrastructure: jest.fn(async () => infra), + createManager: jest.fn(() => manager), + workspacePath: () => '/workspace-host', + homePath: () => '/home/runner', + identity: () => ({ uid: 1000, gid: 1000 }), + stdin, + stdout: new PassThrough(), + stderr: new PassThrough(), + logger: { + debug: jest.fn(), + info: jest.fn(), + warn: jest.fn(), + }, + ...overrides, + }; + return { order, manager, infra, deps, stdin }; +} + +describe('Cloud Hypervisor runtime backend', () => { + let eligibilitySpy: jest.SpyInstance; + + beforeEach(() => { + eligibilitySpy = jest.spyOn(hostEligibility, 'assertGithubHostedRunnerEligibility') + .mockImplementation(() => undefined); + }); + + afterEach(() => { + eligibilitySpy.mockRestore(); + }); + + it('constructs default backend dependencies and manager policy', () => { + const startInfrastructure = jest.fn(); + const defaults = cloudHypervisorRuntimeTestHelpers.defaultDependencies(startInfrastructure); + const previousWorkspace = process.env.GITHUB_WORKSPACE; + process.env.GITHUB_WORKSPACE = '/github/workspace'; + try { + expect(defaults.workspacePath()).toBe('/github/workspace'); + delete process.env.GITHUB_WORKSPACE; + expect(defaults.workspacePath()).toBe(process.cwd()); + expect(defaults.homePath()).toBeTruthy(); + expect(defaults.identity()).toEqual({ + uid: expect.any(Number), + gid: expect.any(Number), + }); + expect(defaults.createManager( + config().cloudHypervisor!, + '/tmp/awf', + infrastructure(), + '/workspace', + '/home/runner', + { uid: 1000, gid: 1000 }, + )).toBeDefined(); + expect(createCloudHypervisorRuntimeBackend(config(), startInfrastructure)) + .toBeInstanceOf(CloudHypervisorRuntimeBackend); + } finally { + if (previousWorkspace === undefined) delete process.env.GITHUB_WORKSPACE; + else process.env.GITHUB_WORKSPACE = previousWorkspace; + } + }); + + it('starts infrastructure, revalidates it, boots and probes before execution', async () => { + const { order, manager, deps, stdin } = harness(); + const backend = new CloudHypervisorRuntimeBackend(config(), deps); + + await backend.start('/tmp/awf', ['github.com']); + const execution = backend.exec('/tmp/awf', ['github.com'], undefined, 1); + stdin.end('input'); + await expect(execution).resolves.toEqual({ exitCode: 23 }); + await backend.stop(); + + expect(order).toEqual([ + 'preflight', + 'compose', + 'revalidate', + 'vm-config', + 'vm-start', + 'probe', + 'vm-stop', + ]); + expect(eligibilitySpy).toHaveBeenCalled(); + expect(manager.execute).toHaveBeenNthCalledWith(2, expect.objectContaining({ + argv: ['/bin/sh', '-lc', 'printf hello'], + cwd: '/workspace', + uid: 1000, + gid: 1000, + timeoutMs: 60_000, + })); + expect(manager.writeStdin).toHaveBeenCalledWith( + Buffer.from('input'), + expect.stringMatching(/^agent-/), + ); + }); + + it('rejects an ineligible host before infrastructure startup', async () => { + eligibilitySpy.mockImplementation(() => { + throw new Error('Cloud Hypervisor is supported only inside GitHub Actions runs'); + }); + const { deps } = harness(); + const backend = new CloudHypervisorRuntimeBackend(config(), deps); + + await expect(backend.start('/tmp/awf', ['github.com'])) + .rejects.toThrow(/supported only inside GitHub Actions runs/); + expect(deps.startInfrastructure).not.toHaveBeenCalled(); + }); + + it('rejects timeouts beyond the guest supervisor limit before infrastructure startup', async () => { + const { deps } = harness(); + const backend = new CloudHypervisorRuntimeBackend(config({ agentTimeout: 1441 }), deps); + + await expect(backend.start('/tmp/awf', ['github.com'])) + .rejects.toThrow(/up to 1440 minutes/); + expect(deps.startInfrastructure).not.toHaveBeenCalled(); + }); + + it('serializes stdin chunks before sending EOF', async () => { + const { manager, deps, stdin } = harness(); + let releaseFirstWrite!: () => void; + let resolveExecution!: (value: { + requestId: string; + exitCode: number; + signal: null; + timedOut: boolean; + }) => void; + manager.execute + .mockReset() + .mockResolvedValueOnce({ + requestId: 'probe', + exitCode: 0, + signal: null, + timedOut: false, + }) + .mockReturnValueOnce(new Promise((resolve) => { + resolveExecution = resolve; + })); + manager.writeStdin.mockImplementationOnce(() => new Promise((resolve) => { + releaseFirstWrite = resolve; + })); + const backend = new CloudHypervisorRuntimeBackend(config(), deps); + await backend.start('/tmp/awf', ['github.com']); + const execution = backend.exec('/tmp/awf', ['github.com']); + stdin.write(Buffer.alloc(70_000, 1)); + stdin.end('second'); + await new Promise((resolve) => setImmediate(resolve)); + + expect(manager.endStdin).not.toHaveBeenCalled(); + releaseFirstWrite(); + await new Promise((resolve) => setImmediate(resolve)); + resolveExecution({ + requestId: 'agent', + exitCode: 0, + signal: null, + timedOut: false, + }); + await execution; + expect(manager.writeStdin.mock.invocationCallOrder[0]) + .toBeLessThan(manager.writeStdin.mock.invocationCallOrder[1]); + expect(manager.writeStdin.mock.invocationCallOrder[1]) + .toBeLessThan(manager.endStdin.mock.invocationCallOrder[0]); + }); + + it('stops the partial VM when readiness probing fails', async () => { + const { manager, deps } = harness(); + manager.execute.mockReset().mockResolvedValue({ + requestId: 'probe', + exitCode: 41, + signal: null, + timedOut: false, + }); + const backend = new CloudHypervisorRuntimeBackend(config(), deps); + + await expect(backend.start('/tmp/awf', ['github.com'])) + .rejects.toThrow(/connectivity probe failed/); + expect(manager.stop).toHaveBeenCalledTimes(1); + }); + + it('fails closed when manager readiness or startup cleanup is unavailable', async () => { + const missingIp = harness(); + Reflect.set(missingIp.manager, 'guestIp', undefined); + const backend = new CloudHypervisorRuntimeBackend(config(), missingIp.deps); + await expect(backend.start('/tmp/awf', ['github.com'])) + .rejects.toThrow(/did not expose the configured guest IP/); + expect(missingIp.manager.stop).toHaveBeenCalledTimes(1); + + const dualFailure = harness(); + (dualFailure.infra.revalidate as jest.Mock).mockRejectedValue('topology moved'); + dualFailure.manager.stop.mockRejectedValue('cleanup failed'); + const failing = new CloudHypervisorRuntimeBackend(config(), dualFailure.deps); + await expect(failing.start('/tmp/awf', ['github.com'])).rejects.toMatchObject({ + message: expect.stringContaining('topology moved'), + cause: 'topology moved', + cleanupCause: 'cleanup failed', + }); + }); + + it('rejects execution before readiness and unsupported TTY execution', async () => { + const cold = harness(); + await expect(new CloudHypervisorRuntimeBackend(config(), cold.deps).exec( + '/tmp/awf', + ['github.com'], + )).rejects.toThrow(/microVM is not ready/); + + const ttyHarness = harness(); + const ttyConfig = config(); + const ttyBackend = new CloudHypervisorRuntimeBackend(ttyConfig, ttyHarness.deps); + await ttyBackend.start('/tmp/awf', ['github.com']); + ttyConfig.tty = true; + await expect(ttyBackend.exec('/tmp/awf', ['github.com'])) + .rejects.toThrow(/does not support TTY execution/); + await ttyBackend.stop(); + }); + + it('preserves a stopped VM once and logs retained artifacts', async () => { + const { manager, deps } = harness(); + const backend = new CloudHypervisorRuntimeBackend(config(), deps); + await backend.start('/tmp/awf', ['github.com']); + + await backend.preserve(); + await backend.preserve(); + await backend.collectDiagnostics(); + + expect(manager.stop).toHaveBeenCalledWith({ preserve: true }); + expect(manager.stop).toHaveBeenCalledTimes(1); + expect(deps.logger.info).toHaveBeenCalledWith( + '[cloud-hypervisor] Preserved network namespace: awffc-test', + ); + }); + + it('cancels an active guest command before stopping', async () => { + const { manager, deps } = harness(); + let resolveExecution!: (value: { + requestId: string; + exitCode: number; + signal: null; + timedOut: boolean; + }) => void; + manager.execute + .mockReset() + .mockResolvedValueOnce({ + requestId: 'probe', + exitCode: 0, + signal: null, + timedOut: false, + }) + .mockReturnValueOnce(new Promise((resolve) => { + resolveExecution = resolve; + })); + manager.cancel.mockImplementationOnce(async () => { + resolveExecution({ + requestId: 'agent', + exitCode: 130, + signal: null, + timedOut: false, + }); + }); + const backend = new CloudHypervisorRuntimeBackend(config(), deps); + await backend.start('/tmp/awf', ['github.com']); + const execution = backend.exec('/tmp/awf', ['github.com']); + + await backend.stop(); + await expect(execution).resolves.toEqual({ exitCode: 130 }); + await backend.stop(); + expect(manager.cancel).toHaveBeenCalledWith( + 'AWF cleanup', + expect.stringMatching(/^agent-/), + ); + expect(manager.stop).toHaveBeenCalledWith({ preserve: false }); + }); + + it('cancels after stdin forwarding failure without changing command output', async () => { + const { manager, deps, stdin } = harness(); + manager.writeStdin.mockRejectedValueOnce(new Error('closed stdin')); + const backend = new CloudHypervisorRuntimeBackend(config(), deps); + await backend.start('/tmp/awf', ['github.com']); + const execution = backend.exec('/tmp/awf', ['github.com']); + stdin.write('input'); + await new Promise((resolve) => setImmediate(resolve)); + + await expect(execution).resolves.toEqual({ exitCode: 23 }); + expect(deps.logger.warn).toHaveBeenCalledWith( + expect.stringContaining('stdin forwarding failed'), + ); + expect(manager.cancel).toHaveBeenCalledWith( + 'stdin forwarding failure', + expect.stringMatching(/^agent-/), + ); + await backend.stop(); + }); + + it('preserves sanitized env values without leaking real provider secrets', () => { + const secret = 'sk-real-provider-secret'; + const environment = buildCloudHypervisorGuestEnvironment( + config({ + openaiApiKey: secret, + additionalEnv: { + SAFE_SETTING: 'enabled', + OPENAI_API_KEY: secret, + }, + }), + infrastructure(), + ); + + expect(environment.SAFE_SETTING).toBe('enabled'); + expect(environment.OPENAI_API_KEY).not.toBe(secret); + expect(Object.values(environment)).not.toContain(secret); + expect(environment.HTTP_PROXY).toBe('http://172.30.0.10:3128'); + expect(environment.HOME).toBe('/workspace/.awf-home'); + + expect(() => buildCloudHypervisorGuestEnvironment( + config({ + openaiApiKey: 'enabled', + additionalEnv: { SAFE_SETTING: 'enabled' }, + }), + infrastructure(), + )).toThrow(/Refusing to pass a real provider credential/); + }); + + it('rejects unsupported strict-security and topology combinations', () => { + expect(() => assertCloudHypervisorPreSecurityCompatibility( + config({ enableDind: true }), + )).toThrow(/Docker-in-Docker/); + expect(() => assertCloudHypervisorPreSecurityCompatibility( + config({ enableHostAccess: true }), + )).toThrow(/host access/); + expect(() => assertCloudHypervisorPreSecurityCompatibility( + config({ enclaves: { enabled: true } } as Partial), + )).toThrow(/MCP gateway path/); + expect(() => assertCloudHypervisorSelection( + config({ containerRuntime: 'gvisor' }), + )).toThrow(/require --container-runtime cloud-hypervisor/); + }); +}); diff --git a/src/cloud-hypervisor-runtime-backend.ts b/src/cloud-hypervisor-runtime-backend.ts new file mode 100644 index 000000000..2e17f1ffa --- /dev/null +++ b/src/cloud-hypervisor-runtime-backend.ts @@ -0,0 +1,464 @@ +import type { Readable, Writable } from 'stream'; +import type { WorkflowDependencies } from './cli-workflow'; +import type { ExternalAgentRuntimeBackend } from './external-runtime-backend'; +import { + API_PROXY_IP, + NETWORK_SUBNET, + SQUID_IP, +} from './config/network-policy'; +import { + resolveMicrovmInfrastructure, + type MicrovmInfrastructureSnapshot, +} from './microvm/infrastructure'; +import type { + GuestExecutionRequest, + GuestExecutionResult, +} from './microvm/vsock-client'; +import type { CloudHypervisorPreflightResult } from './cloud-hypervisor/preflight'; +import { CloudHypervisorManager } from './cloud-hypervisor/manager'; +import { runCloudHypervisorPreflight } from './cloud-hypervisor/preflight'; +import { getRealUserHome, getSafeHostGid, getSafeHostUid } from './host-identity'; +import { logger } from './logger'; +import { buildAgentEnvironment } from './services/agent-service'; +import { buildAgentCredentialEnv } from './services/api-proxy-credential-env'; +import type { CloudHypervisorOptions, WrapperConfig } from './types'; +import { + assertCloudHypervisorRuntimeCompatibility, + requireCloudHypervisorConfig, +} from './cloud-hypervisor/runtime-validation'; +export { + assertCloudHypervisorPreSecurityCompatibility, + assertCloudHypervisorRuntimeCompatibility, +} from './cloud-hypervisor/runtime-validation'; + +const CLOUD_HYPERVISOR_GUEST_WORKSPACE = '/workspace'; +const CLOUD_HYPERVISOR_GUEST_HOME = `${CLOUD_HYPERVISOR_GUEST_WORKSPACE}/.awf-home`; +const CLOUD_HYPERVISOR_PROBE_TIMEOUT_MS = 15_000; +const CLOUD_HYPERVISOR_CANCEL_GRACE_MS = 3_000; +const CLOUD_HYPERVISOR_MAX_TIMEOUT_MS = 86_400_000; + +interface CloudHypervisorBackendLogger { + debug(message: string, ...args: unknown[]): void; + info(message: string, ...args: unknown[]): void; + warn(message: string, ...args: unknown[]): void; +} + +interface CloudHypervisorManagerAdapter { + readonly paths: Pick; + readonly guestIp?: string; + readonly networkNamespace?: string; + start(): Promise; + startInstance(): Promise; + execute(request: GuestExecutionRequest): Promise; + cancel(reason?: string, requestId?: string): Promise; + writeStdin(data: Buffer, requestId?: string): Promise; + endStdin(requestId?: string): Promise; + stop(options?: { preserve?: boolean }): Promise; + collectDiagnostics(directory: string): Promise; +} + +export interface CloudHypervisorRuntimeBackendDependencies { + startInfrastructure: WorkflowDependencies['startContainers']; + preflight(config: CloudHypervisorOptions): Promise; + resolveInfrastructure(enableApiProxy: boolean, ipPath?: string): Promise; + createManager( + config: CloudHypervisorOptions, + workDir: string, + infrastructure: MicrovmInfrastructureSnapshot, + workspacePath: string, + homePath: string, + identity: { uid: number; gid: number }, + ): CloudHypervisorManagerAdapter; + workspacePath(): string; + homePath(): string; + identity(): { uid: number; gid: number }; + stdin: Readable & { isTTY?: boolean }; + stdout: Writable; + stderr: Writable; + logger: CloudHypervisorBackendLogger; +} + +function defaultDependencies( + startInfrastructure: WorkflowDependencies['startContainers'], +): CloudHypervisorRuntimeBackendDependencies { + return { + startInfrastructure, + preflight: runCloudHypervisorPreflight, + resolveInfrastructure: (enableApiProxy, ipPath) => + resolveMicrovmInfrastructure(enableApiProxy, undefined, ipPath), + createManager: (config, workDir, infrastructure, workspacePath, homePath, identity) => + new CloudHypervisorManager( + config, + workDir, + undefined, + undefined, + { + infrastructureBridge: infrastructure.bridgeName, + enableApiProxy: Boolean(infrastructure.apiProxyIp), + }, + { + workspacePath, + homePath, + supervisorBinaryPath: config.supervisorPath!, + supervisorSha256: config.sha256!.supervisor!, + identity, + }, + ), + workspacePath: () => process.env.GITHUB_WORKSPACE || process.cwd(), + homePath: getRealUserHome, + identity: () => ({ + uid: Number(getSafeHostUid()), + gid: Number(getSafeHostGid()), + }), + stdin: process.stdin, + stdout: process.stdout, + stderr: process.stderr, + logger, + }; +} + +/** @internal Exposed only for focused default-policy tests. */ +export const cloudHypervisorRuntimeTestHelpers = { defaultDependencies }; + +/** Stateful adapter for an explicitly enabled, fail-closed Cloud Hypervisor microVM. */ +export class CloudHypervisorRuntimeBackend implements ExternalAgentRuntimeBackend { + readonly runtime = 'cloud-hypervisor'; + + private manager: CloudHypervisorManagerAdapter | undefined; + private environment: Record | undefined; + private activeExecution: + | { requestId: string; promise: Promise } + | undefined; + private stopped = false; + private stopping: Promise | undefined; + private identity: { uid: number; gid: number } | undefined; + private preflightResult: CloudHypervisorPreflightResult | undefined; + + constructor( + private readonly config: WrapperConfig, + private readonly dependencies: CloudHypervisorRuntimeBackendDependencies, + ) {} + + async preflight(): Promise { + const cloudHypervisor = requireCloudHypervisorConfig(this.config); + if ( + this.config.agentTimeout !== undefined && + this.config.agentTimeout * 60_000 > CLOUD_HYPERVISOR_MAX_TIMEOUT_MS + ) { + throw new Error( + `Cloud Hypervisor preview supports --agent-timeout values up to ${ + CLOUD_HYPERVISOR_MAX_TIMEOUT_MS / 60_000 + } minutes`, + ); + } + assertCloudHypervisorRuntimeCompatibility(this.config, cloudHypervisor); + this.preflightResult = await this.dependencies.preflight(cloudHypervisor); + } + + readonly start: WorkflowDependencies['startContainers'] = async ( + workDir, + allowedDomains, + proxyLogsDir, + skipPull, + onNetworkReady, + onInfrastructureReady, + ) => { + let stage = 'preflight'; + this.dependencies.logger.info( + '[cloud-hypervisor] runtime=cloud-hypervisor maturity=preview fallback=disabled', + ); + try { + await this.preflight(); + stage = 'compose-infrastructure'; + await this.dependencies.startInfrastructure( + workDir, + allowedDomains, + proxyLogsDir, + skipPull, + onNetworkReady, + onInfrastructureReady, + ); + + stage = 'infrastructure-discovery'; + const cloudHypervisor = requireCloudHypervisorConfig(this.config); + const infrastructure = await this.dependencies.resolveInfrastructure( + Boolean(this.config.enableApiProxy), + this.preflightResult?.tools.ip, + ); + this.identity = this.dependencies.identity(); + this.manager = this.dependencies.createManager( + cloudHypervisor, + workDir, + infrastructure, + this.dependencies.workspacePath(), + this.dependencies.homePath(), + this.identity, + ); + + stage = 'topology-revalidation'; + await infrastructure.revalidate(); + stage = 'vmm-configuration'; + await this.manager.start(); + if (!this.manager.guestIp) { + throw new Error('Cloud Hypervisor manager did not expose the configured guest IP'); + } + this.environment = buildCloudHypervisorGuestEnvironment( + this.config, + infrastructure, + this.manager.guestIp, + ); + stage = 'guest-boot'; + await this.manager.startInstance(); + stage = 'guest-connectivity'; + await this.probeGuestConnectivity(); + this.dependencies.logger.info('[cloud-hypervisor] stage=ready'); + } catch (error) { + this.dependencies.logger.warn( + `[cloud-hypervisor] stage=${stage} status=failed: ${formatError(error)}`, + ); + try { + await this.manager?.stop(); + } catch (cleanupError) { + const combined = new Error( + `Cloud Hypervisor startup failed: ${formatError(error)}; ` + + `microVM cleanup also failed: ${formatError(cleanupError)}`, + ); + Object.defineProperty(combined, 'cause', { value: error }); + Object.assign(combined, { cleanupCause: cleanupError }); + throw combined; + } + throw error; + } + }; + + readonly exec: WorkflowDependencies['runAgentCommand'] = async ( + _workDir, + _allowedDomains, + _proxyLogsDir, + agentTimeoutMinutes, + ) => { + const manager = this.manager; + const environment = this.environment; + const identity = this.identity; + if (!manager || !environment || !identity) { + throw new Error('Cloud Hypervisor microVM is not ready'); + } + if (this.config.tty) { + throw new Error( + 'Cloud Hypervisor preview guest supervisor does not support TTY execution', + ); + } + + const requestId = `agent-${process.pid}-${Date.now()}`; + const timeoutMs = agentTimeoutMinutes === undefined + ? undefined + : agentTimeoutMinutes * 60_000; + const execution = manager.execute({ + requestId, + argv: ['/bin/sh', '-lc', this.config.agentCommand], + env: environment, + cwd: CLOUD_HYPERVISOR_GUEST_WORKSPACE, + ...identity, + tty: false, + ...(timeoutMs === undefined ? {} : { timeoutMs }), + stdout: this.dependencies.stdout, + stderr: this.dependencies.stderr, + }); + this.activeExecution = { requestId, promise: execution }; + + let forwarding = Promise.resolve(); + let stdinEnded = false; + const forward = (operation: () => Promise): void => { + forwarding = forwarding.then(operation).catch((error) => { + this.dependencies.logger.warn( + `Cloud Hypervisor guest stdin forwarding failed: ${formatError(error)}`, + ); + return manager.cancel('stdin forwarding failure', requestId).catch(() => undefined); + }); + }; + const onData = (chunk: Buffer | string): void => { + const data = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); + forward(() => manager.writeStdin(data, requestId)); + }; + const onEnd = (): void => { + if (stdinEnded) return; + stdinEnded = true; + forward(() => manager.endStdin(requestId)); + }; + this.dependencies.stdin.on('data', onData); + this.dependencies.stdin.once('end', onEnd); + if (this.dependencies.stdin.readableEnded) onEnd(); + + try { + const result = await execution; + this.dependencies.logger.info( + `[cloud-hypervisor] Agent command exited with code ${result.exitCode}` + + (result.signal ? ` (${result.signal})` : ''), + ); + return { exitCode: result.exitCode }; + } finally { + this.dependencies.stdin.off('data', onData); + this.dependencies.stdin.off('end', onEnd); + await forwarding; + this.activeExecution = undefined; + } + }; + + async collectDiagnostics(): Promise { + if (!this.manager) return; + const directory = this.config.auditDir + ? `${this.config.auditDir}/cloud-hypervisor` + : `${this.config.workDir}/diagnostics/cloud-hypervisor`; + await this.manager.collectDiagnostics(directory); + } + + async stop(): Promise { + if (this.stopped) return; + if (this.stopping) return this.stopping; + this.stopping = this.stopManager(false); + try { + await this.stopping; + this.stopped = true; + } finally { + this.stopping = undefined; + } + } + + async preserve(): Promise { + if (this.stopped) return; + if (this.stopping) return this.stopping; + this.stopping = this.stopManager(true); + try { + await this.stopping; + this.stopped = true; + if (this.manager) { + this.dependencies.logger.info( + `[cloud-hypervisor] Preserved run directory: ${this.manager.paths.runDirectory}`, + ); + this.dependencies.logger.info( + `[cloud-hypervisor] Preserved images: ${this.config.workDir}/firecracker-images`, + ); + if (this.manager.networkNamespace) { + this.dependencies.logger.info( + `[cloud-hypervisor] Preserved network namespace: ${this.manager.networkNamespace}`, + ); + } + } + } finally { + this.stopping = undefined; + } + } + + private async stopManager(preserve: boolean): Promise { + const active = this.activeExecution; + if (active && this.manager) { + try { + await this.manager.cancel('AWF cleanup', active.requestId); + } catch { + // Process termination below remains authoritative. + } + await Promise.race([ + active.promise.catch(() => undefined), + new Promise((resolve) => setTimeout(resolve, CLOUD_HYPERVISOR_CANCEL_GRACE_MS)), + ]); + } + await this.manager?.stop({ preserve }); + } + + private async probeGuestConnectivity(): Promise { + const manager = this.manager!; + const environment = this.environment!; + const identity = this.identity; + if (!identity) { + throw new Error('Cloud Hypervisor guest identity is not ready'); + } + const squidProbe = + `curl --silent --show-error --max-time 5 --output /dev/null ` + + `http://${SQUID_IP}:3128/`; + const apiProxyProbe = this.config.enableApiProxy + ? ` && curl --fail --silent --show-error --max-time 5 --noproxy '*' ` + + `--output /dev/null http://${API_PROXY_IP}:10000/reflect` + : ''; + const result = await manager.execute({ + requestId: `probe-${process.pid}-${Date.now()}`, + argv: ['/bin/sh', '-c', `set -eu; ${squidProbe}${apiProxyProbe}`], + env: environment, + cwd: CLOUD_HYPERVISOR_GUEST_WORKSPACE, + ...identity, + timeoutMs: CLOUD_HYPERVISOR_PROBE_TIMEOUT_MS, + }); + if (result.exitCode !== 0) { + throw new Error( + `Cloud Hypervisor guest connectivity probe failed with exit code ${result.exitCode}`, + ); + } + this.dependencies.logger.info( + '[cloud-hypervisor] Guest supervisor, Squid, and API proxy connectivity verified', + ); + } +} + +export function buildCloudHypervisorGuestEnvironment( + config: WrapperConfig, + infrastructure: Pick, + guestIp = '100.64.0.2', +): Record { + const networkConfig = { + subnet: NETWORK_SUBNET, + squidIp: infrastructure.squidIp, + agentIp: guestIp, + proxyIp: infrastructure.apiProxyIp, + }; + const environment = buildAgentEnvironment({ + config, + networkConfig, + dnsServers: [], + }); + if (config.enableApiProxy) { + Object.assign(environment, buildAgentCredentialEnv({ config, networkConfig })); + } + Object.assign(environment, { + HOME: CLOUD_HYPERVISOR_GUEST_HOME, + PWD: CLOUD_HYPERVISOR_GUEST_WORKSPACE, + AWF_WORKDIR: CLOUD_HYPERVISOR_GUEST_WORKSPACE, + SQUID_PROXY_HOST: infrastructure.squidIp, + HOSTNAME: 'awf-cloud-hypervisor', + AWF_RUNTIME: 'cloud-hypervisor', + }); + assertNoProviderSecrets(config, environment); + return environment; +} + +function assertNoProviderSecrets( + config: WrapperConfig, + environment: Readonly>, +): void { + const secrets = [ + config.openaiApiKey, + config.anthropicApiKey, + config.copilotGithubToken, + config.copilotProviderApiKey, + config.geminiApiKey, + config.googleApiKey, + config.githubToken, + ] + .filter((value): value is string => typeof value === 'string' && value.length > 0); + for (const [name, value] of Object.entries(environment)) { + if (secrets.some((secret) => value === secret || value.includes(secret))) { + throw new Error( + `Refusing to pass a real provider credential through Cloud Hypervisor guest variable ${name}`, + ); + } + } +} + +function formatError(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +export function createCloudHypervisorRuntimeBackend( + config: WrapperConfig, + startInfrastructure: WorkflowDependencies['startContainers'], +): CloudHypervisorRuntimeBackend { + return new CloudHypervisorRuntimeBackend(config, defaultDependencies(startInfrastructure)); +} diff --git a/src/cloud-hypervisor/api-client.test.ts b/src/cloud-hypervisor/api-client.test.ts new file mode 100644 index 000000000..3a309109b --- /dev/null +++ b/src/cloud-hypervisor/api-client.test.ts @@ -0,0 +1,181 @@ +import * as http from 'http'; +import { promises as fs } from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { + CloudHypervisorApiClient, + CloudHypervisorApiError, +} from './api-client'; + +describe('CloudHypervisorApiClient', () => { + let directory: string; + let socketPath: string; + let server: http.Server; + + beforeEach(async () => { + directory = await fs.mkdtemp(path.join(os.tmpdir(), 'awf-ch-api-')); + socketPath = path.join(directory, 'api.socket'); + }); + + afterEach(async () => { + if (server?.listening) { + await new Promise((resolve, reject) => { + server.close((error) => error ? reject(error) : resolve()); + }); + } + await fs.rm(directory, { recursive: true, force: true }); + }); + + async function listen( + handler: http.RequestListener, + ): Promise { + server = http.createServer(handler); + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(socketPath, resolve); + }); + } + + it('sends typed JSON requests over the Unix socket under /api/v1', async () => { + const received: Array<{ method?: string; url?: string; body: string }> = []; + await listen((request, response) => { + const chunks: Buffer[] = []; + request.on('data', (chunk: Buffer) => chunks.push(chunk)); + request.on('end', () => { + received.push({ + method: request.method, + url: request.url, + body: Buffer.concat(chunks).toString('utf8'), + }); + if (request.url === '/api/v1/vmm.ping') { + response.writeHead(200, { 'Content-Type': 'application/json' }); + response.end(JSON.stringify({ version: '53.0' })); + return; + } + if (request.url === '/api/v1/vm.info') { + response.writeHead(200, { 'Content-Type': 'application/json' }); + response.end(JSON.stringify({ + config: { cpus: { boot_vcpus: 2, max_vcpus: 2 }, memory: { size: 1 }, payload: { kernel: '/kernel' } }, + state: 'Running', + })); + return; + } + response.writeHead(204).end(); + }); + }); + + const client = new CloudHypervisorApiClient({ socketPath }); + expect(await client.ping()).toEqual({ version: '53.0' }); + await client.vmCreate({ + cpus: { boot_vcpus: 2, max_vcpus: 2 }, + memory: { size: 512 * 1024 * 1024 }, + payload: { kernel: '/kernel', cmdline: 'console=ttyS0' }, + disks: [{ id: 'rootfs', path: '/rootfs', readonly: false }], + net: [{ id: 'net0', tap: 'chtap0', mac: '02:00:00:00:00:01' }], + vsock: { cid: 3, socket: '/run/vsock.socket' }, + landlock_enable: true, + landlock_rules: [{ path: '/kernel', access: 'r' }], + }); + await client.vmBoot(); + const info = await client.vmInfo(); + expect(info.state).toBe('Running'); + await client.vmShutdown(); + await client.vmmShutdown(); + + expect(received[0]).toMatchObject({ method: 'GET', url: '/api/v1/vmm.ping' }); + expect(received[1]).toMatchObject({ method: 'PUT', url: '/api/v1/vm.create' }); + expect(JSON.parse(received[1].body)).toMatchObject({ + cpus: { boot_vcpus: 2, max_vcpus: 2 }, + landlock_enable: true, + }); + expect(received[2]).toMatchObject({ method: 'PUT', url: '/api/v1/vm.boot', body: '' }); + expect(received[3]).toMatchObject({ method: 'GET', url: '/api/v1/vm.info' }); + expect(received[4]).toMatchObject({ method: 'PUT', url: '/api/v1/vm.shutdown', body: '' }); + expect(received[5]).toMatchObject({ method: 'PUT', url: '/api/v1/vmm.shutdown', body: '' }); + }); + + it('parses Cloud Hypervisor chained-error-message arrays', async () => { + await listen((_request, response) => { + response.writeHead(500, { 'Content-Type': 'application/json' }); + response.end(JSON.stringify(['failed to create VM', 'invalid disk path'])); + }); + + const client = new CloudHypervisorApiClient({ socketPath }); + const error = await client.vmCreate({ + cpus: { boot_vcpus: 1, max_vcpus: 1 }, + memory: { size: 1 }, + payload: { kernel: '/kernel' }, + }).catch((caught) => caught); + + expect(error).toBeInstanceOf(CloudHypervisorApiError); + expect(error).toMatchObject({ + method: 'PUT', + requestPath: '/api/v1/vm.create', + statusCode: 500, + }); + expect(error.message).toContain('failed to create VM: invalid disk path'); + }); + + it('falls back to the raw body when the error is not a string array', async () => { + await listen((_request, response) => { + response.writeHead(400, { 'Content-Type': 'text/plain' }); + response.end('not json'); + }); + + const client = new CloudHypervisorApiClient({ socketPath }); + const error = await client.vmBoot().catch((caught) => caught); + expect(error.message).toContain('not json'); + }); + + it('resolves undefined for empty 204 responses', async () => { + await listen((_request, response) => { + response.writeHead(204).end(); + }); + const client = new CloudHypervisorApiClient({ socketPath }); + await expect(client.vmBoot()).resolves.toBeUndefined(); + }); + + it('enforces a wall-clock timeout even when the peer keeps sending data', async () => { + await listen((_request, response) => { + response.writeHead(200, { 'Content-Type': 'application/json' }); + const interval = setInterval(() => { + response.write(' '); + }, 5); + response.on('close', () => clearInterval(interval)); + }); + + const client = new CloudHypervisorApiClient({ socketPath, timeoutMs: 30 }); + await expect(client.vmInfo()).rejects.toThrow(/timed out after 30ms/); + }); + + it('rejects when the response stream errors before completion', async () => { + await listen((_request, response) => { + response.writeHead(200, { 'Content-Type': 'application/json' }); + response.write('{"state":"Running"'); + response.destroy(new Error('socket closed')); + }); + + const client = new CloudHypervisorApiClient({ socketPath }); + await expect(client.vmInfo()).rejects.toThrow(); + }); + + it('rejects when the response exceeds the bounded size limit', async () => { + await listen((_request, response) => { + response.writeHead(200, { 'Content-Type': 'application/json' }); + response.write(Buffer.alloc(2 * 1024 * 1024, 'a')); + }); + + const client = new CloudHypervisorApiClient({ socketPath }); + await expect(client.vmInfo()).rejects.toThrow(/exceeded 1 MiB/); + }); + + it('rejects with invalid JSON error message on malformed success body', async () => { + await listen((_request, response) => { + response.writeHead(200, { 'Content-Type': 'application/json' }); + response.end('{not-json'); + }); + + const client = new CloudHypervisorApiClient({ socketPath }); + await expect(client.vmInfo()).rejects.toThrow(/returned invalid JSON/); + }); +}); diff --git a/src/cloud-hypervisor/api-client.ts b/src/cloud-hypervisor/api-client.ts new file mode 100644 index 000000000..f49f1de85 --- /dev/null +++ b/src/cloud-hypervisor/api-client.ts @@ -0,0 +1,295 @@ +import * as http from 'http'; + +/** + * Typed request/response shapes for the subset of the Cloud Hypervisor + * `/api/v1` REST surface AWF drives. Field names intentionally match the + * upstream OpenAPI document + * (`vmm/src/api/openapi/cloud-hypervisor.yaml`, v53.0) verbatim so this + * client stays a thin, auditable mapping instead of an abstraction layer. + */ + +export interface CloudHypervisorPayloadConfig { + kernel: string; + cmdline?: string; + initramfs?: string; +} + +export interface CloudHypervisorCpuTopology { + threads_per_core?: number; + cores_per_die?: number; + dies_per_package?: number; + packages?: number; +} + +export interface CloudHypervisorCpusConfig { + boot_vcpus: number; + max_vcpus: number; + topology?: CloudHypervisorCpuTopology; + kvm_hyperv?: boolean; + nested?: boolean; +} + +export interface CloudHypervisorMemoryConfig { + size: number; + mergeable?: boolean; + shared?: boolean; + hugepages?: boolean; + thp?: boolean; +} + +export interface CloudHypervisorDiskConfig { + id: string; + path: string; + readonly?: boolean; + direct?: boolean; + /** Must stay `false`: raw images/backing_files off (no qcow2 layering). */ + backing_files?: false; +} + +export interface CloudHypervisorNetConfig { + id: string; + /** Name of an already-created, already-up host TAP device. */ + tap: string; + mac: string; + num_queues?: number; + queue_size?: number; +} + +export interface CloudHypervisorRngConfig { + src: string; +} + +export type CloudHypervisorConsoleMode = 'Off' | 'Pty' | 'Tty' | 'File' | 'Socket' | 'Null'; + +export interface CloudHypervisorConsoleConfig { + mode: CloudHypervisorConsoleMode; + file?: string; +} + +export interface CloudHypervisorSerialConfig { + mode: CloudHypervisorConsoleMode; + file?: string; +} + +export interface CloudHypervisorVsockConfig { + cid: number; + socket: string; +} + +/** `access` is `"r"`, `"w"`, or `"rw"` per Cloud Hypervisor's Landlock rule parser. */ +export interface CloudHypervisorLandlockRule { + path: string; + access: 'r' | 'w' | 'rw'; +} + +export interface CloudHypervisorVmConfig { + cpus: CloudHypervisorCpusConfig; + memory: CloudHypervisorMemoryConfig; + payload: CloudHypervisorPayloadConfig; + disks?: CloudHypervisorDiskConfig[]; + net?: CloudHypervisorNetConfig[]; + rng?: CloudHypervisorRngConfig; + serial?: CloudHypervisorSerialConfig; + console?: CloudHypervisorConsoleConfig; + vsock?: CloudHypervisorVsockConfig; + watchdog?: boolean; + landlock_enable?: boolean; + landlock_rules?: CloudHypervisorLandlockRule[]; +} + +export interface CloudHypervisorVmmPingResponse { + build_version?: string; + version: string; + pid?: number; + features?: string[]; +} + +export type CloudHypervisorVmState = 'Created' | 'Running' | 'Shutdown' | 'Paused'; + +export interface CloudHypervisorVmInfo { + config: CloudHypervisorVmConfig; + state: CloudHypervisorVmState; + memory_actual_size?: number; +} + +/** Nested counter map: `{ "": { "": value } }`. */ +export type CloudHypervisorVmCounters = Record>; + +export class CloudHypervisorApiError extends Error { + constructor( + readonly method: string, + readonly requestPath: string, + readonly statusCode: number, + readonly responseBody: string, + message: string, + ) { + super(message); + this.name = 'CloudHypervisorApiError'; + } +} + +export interface CloudHypervisorApiClientOptions { + socketPath: string; + timeoutMs?: number; +} + +const API_ROOT = '/api/v1'; +const MAX_RESPONSE_BYTES = 1024 * 1024; + +/** + * Typed client for Cloud Hypervisor's `/api/v1` REST API over its Unix + * domain socket. Every call is a bounded-timeout, single JSON round trip; + * there is no retry or connection reuse logic beyond what Node's `http` + * module does per request. + */ +export class CloudHypervisorApiClient { + private readonly timeoutMs: number; + + constructor(private readonly options: CloudHypervisorApiClientOptions) { + this.timeoutMs = options.timeoutMs ?? 5_000; + } + + ping(): Promise { + return this.request('GET', '/vmm.ping'); + } + + vmmShutdown(): Promise { + return this.request('PUT', '/vmm.shutdown'); + } + + vmCreate(config: CloudHypervisorVmConfig): Promise { + return this.request('PUT', '/vm.create', config); + } + + vmBoot(): Promise { + return this.request('PUT', '/vm.boot'); + } + + vmInfo(): Promise { + return this.request('GET', '/vm.info'); + } + + vmCounters(): Promise { + return this.request('GET', '/vm.counters'); + } + + vmShutdown(): Promise { + return this.request('PUT', '/vm.shutdown'); + } + + vmDelete(): Promise { + return this.request('PUT', '/vm.delete'); + } + + private request( + method: string, + endpoint: string, + payload?: object, + ): Promise { + const requestPath = `${API_ROOT}${endpoint}`; + const body = payload === undefined ? undefined : JSON.stringify(payload); + + return new Promise((resolve, reject) => { + let settled = false; + const timer = setTimeout(() => { + const error = new Error( + `Cloud Hypervisor API ${method} ${requestPath} timed out after ${this.timeoutMs}ms`, + ); + rejectOnce(error); + request.destroy(error); + }, this.timeoutMs); + const clearTimer = () => clearTimeout(timer); + const resolveOnce = (value: TResponse) => { + if (settled) return; + settled = true; + clearTimer(); + resolve(value); + }; + const rejectOnce = (error: unknown) => { + if (settled) return; + settled = true; + clearTimer(); + reject(error); + }; + + const request = http.request({ + socketPath: this.options.socketPath, + path: requestPath, + method, + headers: body === undefined + ? undefined + : { + 'Content-Type': 'application/json', + 'Content-Length': Buffer.byteLength(body), + }, + }, (response) => { + const chunks: Buffer[] = []; + let totalBytes = 0; + response.on('error', rejectOnce); + response.on('aborted', () => { + rejectOnce(new Error(`Cloud Hypervisor API ${method} ${requestPath} response was aborted`)); + }); + response.on('data', (chunk: Buffer) => { + totalBytes += chunk.length; + if (totalBytes > MAX_RESPONSE_BYTES) { + const error = new Error('Cloud Hypervisor API response exceeded 1 MiB'); + rejectOnce(error); + request.destroy(error); + return; + } + chunks.push(chunk); + }); + response.on('end', () => { + const responseBody = Buffer.concat(chunks).toString('utf8'); + const statusCode = response.statusCode ?? 0; + if (statusCode < 200 || statusCode >= 300) { + rejectOnce(new CloudHypervisorApiError( + method, + requestPath, + statusCode, + responseBody, + `Cloud Hypervisor API ${method} ${requestPath} failed with HTTP ${statusCode}: ` + + `${parseErrorDetail(responseBody)}`, + )); + return; + } + + if (statusCode === 204 || !responseBody) { + resolveOnce(undefined as TResponse); + return; + } + try { + resolveOnce(JSON.parse(responseBody) as TResponse); + } catch (error) { + rejectOnce(new Error( + `Cloud Hypervisor API ${method} ${requestPath} returned invalid JSON: ` + + `${error instanceof Error ? error.message : String(error)}`, + )); + } + }); + }); + + request.on('error', rejectOnce); + if (body !== undefined) request.write(body); + request.end(); + }); + } +} + +/** + * Cloud Hypervisor error bodies are a JSON array of chained error messages + * (outermost first), e.g. `["failed to create VM", "invalid disk path"]`. + * Falls back to the raw body when the shape is unexpected. + */ +function parseErrorDetail(responseBody: string): string { + if (!responseBody) return 'empty response'; + try { + const parsed: unknown = JSON.parse(responseBody); + if (Array.isArray(parsed) && parsed.every((entry) => typeof entry === 'string')) { + return parsed.length > 0 ? parsed.join(': ') : responseBody; + } + } catch { + // Fall through to the raw body below. + } + return responseBody; +} diff --git a/src/cloud-hypervisor/launcher.test.ts b/src/cloud-hypervisor/launcher.test.ts new file mode 100644 index 000000000..2a8cd2174 --- /dev/null +++ b/src/cloud-hypervisor/launcher.test.ts @@ -0,0 +1,198 @@ +import { + CloudHypervisorCgroup, + buildCloudHypervisorLaunchCommand, + computeCloudHypervisorLandlockRules, + type CloudHypervisorCgroupDependencies, +} from './launcher'; + +describe('buildCloudHypervisorLaunchCommand', () => { + const baseOptions = { + tools: { ip: '/usr/sbin/ip', setpriv: '/usr/bin/setpriv' }, + namespaceName: 'awfch-abc123', + identity: { uid: 1000, gid: 1000 }, + kvmGid: 978, + cloudHypervisorBinary: '/opt/cloud-hypervisor', + apiSocketPath: '/run/awf/api.socket', + logFilePath: '/run/awf/cloud-hypervisor.log', + }; + + it('joins the namespace, drops privileges but retains the kvm group, then execs Cloud Hypervisor with no shell', () => { + const result = buildCloudHypervisorLaunchCommand(baseOptions); + expect(result.command).toBe('/usr/sbin/ip'); + expect(result.args).toEqual([ + 'netns', 'exec', 'awfch-abc123', + '/usr/bin/setpriv', + '--reuid=1000', + '--regid=1000', + '--groups=978', + '--no-new-privs', + '--inh-caps=-all', + '--bounding-set=-all', + '--', + '/opt/cloud-hypervisor', + '--api-socket', 'path=/run/awf/api.socket', + '--log-file', '/run/awf/cloud-hypervisor.log', + '-v', + '--seccomp', 'true', + ]); + expect(result.args).not.toContain('--clear-groups'); + // No argument contains shell metacharacters that would matter if ever + // interpolated; more importantly, args are a plain array (never joined + // into a shell string) so metacharacters have no special meaning here. + expect(result.args.every((arg) => typeof arg === 'string')).toBe(true); + }); + + it.each([ + ['unsafe namespace name', { namespaceName: '../etc' }, /Unsafe Cloud Hypervisor network namespace name/], + ['zero uid', { identity: { uid: 0, gid: 1000 } }, /uid must be a positive integer/], + ['negative gid', { identity: { uid: 1000, gid: -1 } }, /gid must be a positive integer/], + ['negative kvm gid', { kvmGid: -1 }, /\/dev\/kvm group id must be a non-negative integer/], + ['relative binary path', { cloudHypervisorBinary: 'cloud-hypervisor' }, /binary path must be absolute/], + ['relative socket path', { apiSocketPath: 'api.socket' }, /API socket path must be absolute/], + ])('rejects %s', (_label, overrides, error) => { + expect(() => buildCloudHypervisorLaunchCommand({ ...baseOptions, ...overrides })) + .toThrow(error); + }); + + it('accepts a kvm gid of 0 (root-owned /dev/kvm on unusual hosts)', () => { + expect(() => buildCloudHypervisorLaunchCommand({ ...baseOptions, kvmGid: 0 })).not.toThrow(); + }); +}); + +describe('computeCloudHypervisorLandlockRules', () => { + it('restricts the VMM to exactly the staged paths plus required device nodes', () => { + const rules = computeCloudHypervisorLandlockRules({ + kernelPath: '/run/awf/kernel', + rootfsPath: '/run/awf/rootfs.ext4', + workspacePath: '/run/awf/workspace.ext4', + runDirectory: '/run/awf/run', + apiSocketPath: '/run/awf/run/api.socket', + vsockSocketPath: '/run/awf/run/vsock.socket', + }); + + expect(rules).toEqual([ + { path: '/run/awf/kernel', access: 'r' }, + { path: '/run/awf/rootfs.ext4', access: 'rw' }, + { path: '/run/awf/run', access: 'rw' }, + { path: '/dev/kvm', access: 'rw' }, + { path: '/dev/net/tun', access: 'rw' }, + { path: '/run/awf/workspace.ext4', access: 'rw' }, + ]); + }); + + it('omits the workspace rule when no workspace disk is configured', () => { + const rules = computeCloudHypervisorLandlockRules({ + kernelPath: '/run/awf/kernel', + rootfsPath: '/run/awf/rootfs.ext4', + runDirectory: '/run/awf/run', + apiSocketPath: '/run/awf/run/api.socket', + vsockSocketPath: '/run/awf/run/vsock.socket', + }); + + expect(rules.some((rule) => rule.path.includes('workspace'))).toBe(false); + }); +}); + +describe('CloudHypervisorCgroup', () => { + function dependencies(): CloudHypervisorCgroupDependencies & { + mkdir: jest.Mock; + writeFile: jest.Mock; + rmdir: jest.Mock; + } { + return { + mkdir: jest.fn().mockResolvedValue(undefined), + writeFile: jest.fn().mockResolvedValue(undefined), + rmdir: jest.fn().mockResolvedValue(undefined), + }; + } + + it('enables cpu/memory/pids delegation at the cgroup root and parent before creating the leaf', async () => { + const deps = dependencies(); + const cgroup = new CloudHypervisorCgroup( + '/sys/fs/cgroup/awf-cloud-hypervisor/run-1', + { memoryMib: 512, vcpuCount: 2 }, + deps, + ); + await cgroup.setup(); + + expect(deps.writeFile).toHaveBeenCalledWith( + '/sys/fs/cgroup/cgroup.subtree_control', + '+cpu +memory +pids', + ); + expect(deps.writeFile).toHaveBeenCalledWith( + '/sys/fs/cgroup/awf-cloud-hypervisor/cgroup.subtree_control', + '+cpu +memory +pids', + ); + expect(deps.mkdir).toHaveBeenCalledWith('/sys/fs/cgroup/awf-cloud-hypervisor'); + expect(deps.mkdir).toHaveBeenCalledWith('/sys/fs/cgroup/awf-cloud-hypervisor/run-1'); + + // Cross-mock chronological order (jest's shared invocation counter, + // not per-mock array indices) must be: enable root -> mkdir parent -> + // enable parent -> mkdir leaf. A cgroup v2 child only gets a + // controller's interface files once that controller is enabled in + // the *parent's* subtree_control, so the parent directory must exist + // before its own subtree_control can be written, and the leaf must + // not be created until the parent has delegated the controllers down. + const writeFileCallIndex = (target: string): number => { + const index = deps.writeFile.mock.calls.findIndex(([path]) => path === target); + return deps.writeFile.mock.invocationCallOrder[index]; + }; + const mkdirCallIndex = (target: string): number => { + const index = deps.mkdir.mock.calls.findIndex(([path]) => path === target); + return deps.mkdir.mock.invocationCallOrder[index]; + }; + const rootEnableOrder = writeFileCallIndex('/sys/fs/cgroup/cgroup.subtree_control'); + const parentMkdirOrder = mkdirCallIndex('/sys/fs/cgroup/awf-cloud-hypervisor'); + const parentEnableOrder = writeFileCallIndex('/sys/fs/cgroup/awf-cloud-hypervisor/cgroup.subtree_control'); + const leafMkdirOrder = mkdirCallIndex('/sys/fs/cgroup/awf-cloud-hypervisor/run-1'); + + expect(parentMkdirOrder).toBeGreaterThan(rootEnableOrder); + expect(parentEnableOrder).toBeGreaterThan(parentMkdirOrder); + expect(leafMkdirOrder).toBeGreaterThan(parentEnableOrder); + }); + + it('writes cgroup v2 memory/cpu/pids limits derived from the guest configuration', async () => { + const deps = dependencies(); + const cgroup = new CloudHypervisorCgroup( + '/sys/fs/cgroup/awf-cloud-hypervisor/run-1', + { memoryMib: 512, vcpuCount: 2 }, + deps, + ); + await cgroup.setup(); + + expect(deps.writeFile).toHaveBeenCalledWith( + '/sys/fs/cgroup/awf-cloud-hypervisor/run-1/memory.max', + String((512 + 256) * 1024 * 1024), + ); + expect(deps.writeFile).toHaveBeenCalledWith( + '/sys/fs/cgroup/awf-cloud-hypervisor/run-1/cpu.max', + '200000 100000', + ); + expect(deps.writeFile).toHaveBeenCalledWith( + '/sys/fs/cgroup/awf-cloud-hypervisor/run-1/pids.max', + '256', + ); + }); + + it('assigns a PID into cgroup.procs and rejects invalid PIDs', async () => { + const deps = dependencies(); + const cgroup = new CloudHypervisorCgroup('/sys/fs/cgroup/awf-cloud-hypervisor/run-1', { memoryMib: 512, vcpuCount: 2 }, deps); + await cgroup.assign(4321); + expect(deps.writeFile).toHaveBeenCalledWith('/sys/fs/cgroup/awf-cloud-hypervisor/run-1/cgroup.procs', '4321'); + + await expect(cgroup.assign(0)).rejects.toThrow(/invalid PID/); + await expect(cgroup.assign(-5)).rejects.toThrow(/invalid PID/); + }); + + it('only rmdirs the leaf cgroup directory (never a recursive removal) if setup succeeded', async () => { + const deps = dependencies(); + const cgroup = new CloudHypervisorCgroup('/sys/fs/cgroup/awf-cloud-hypervisor/run-1', { memoryMib: 512, vcpuCount: 2 }, deps); + await cgroup.cleanup(); + expect(deps.rmdir).not.toHaveBeenCalled(); + + await cgroup.setup(); + await cgroup.cleanup(); + expect(deps.rmdir).toHaveBeenCalledWith('/sys/fs/cgroup/awf-cloud-hypervisor/run-1'); + expect(deps.rmdir).toHaveBeenCalledTimes(1); + }); +}); diff --git a/src/cloud-hypervisor/launcher.ts b/src/cloud-hypervisor/launcher.ts new file mode 100644 index 000000000..b48bd1597 --- /dev/null +++ b/src/cloud-hypervisor/launcher.ts @@ -0,0 +1,271 @@ +import { promises as fs } from 'fs'; +import * as path from 'path'; +import type { CloudHypervisorLandlockRule } from './api-client'; + +/** + * Secure host launch-argv construction and resource-confinement helpers for + * Cloud Hypervisor. + * + * Cloud Hypervisor has no jailer-equivalent process (unlike Firecracker), so + * there is nothing that natively joins a prepared network namespace, + * chroots, drops capabilities, and execs the VMM as one atomic operation. + * This module documents and implements the exact replacement boundary AWF + * uses instead: + * + * 1. **Network namespace join** — `ip netns exec ...` (the + * already-trusted, already-required `ip` tool) execs directly into the + * per-run namespace {@link https://man7.org/linux/man-pages/man8/ip-netns.8.html} + * without an intermediate fork, so the resulting process keeps the PID + * the host process observes. + * 2. **Privilege drop** — `setpriv --reuid --regid --clear-groups + * --no-new-privs --inh-caps=-all --bounding-set=-all` execs the Cloud + * Hypervisor binary as the non-root operator uid/gid with an empty + * capability bounding set and `no_new_privs` set, before any guest code + * runs. This is the same non-root identity Firecracker's jailer targets + * (see `resolveJailerIdentity` in `src/firecracker/manager.ts`) and + * requires the same operator preconditions (kvm-group membership, + * `/dev/kvm` access). + * 3. **Filesystem confinement** — Cloud Hypervisor has no chroot of its + * own, and jailer's userspace chroot+pivot_root cannot be replicated + * for a foreign static binary without reimplementing jailer itself. + * Instead AWF combines: + * - a **private run directory** (mode `0700`, owned by the target + * uid/gid) holding only the staged kernel/rootfs/workspace images + * and the API/vsock sockets — see `CloudHypervisorManager`; + * - **Landlock** (`landlock_enable`/`landlock_rules` in the + * `vm.create` payload — see {@link computeCloudHypervisorLandlockRules}) + * restricting the VMM process's own filesystem access to exactly + * those paths, enforced by the kernel LSM rather than a userspace + * boundary; + * - Cloud Hypervisor's own **default seccomp** filter + * (`--seccomp true`, its default "kill on violation" mode). + * This is a different (kernel-LSM-based) boundary than jailer's chroot, + * not a weaker one — it is intentionally documented and covered by + * tests instead of silently degrading to "no filesystem confinement". + * 4. **Resource limits** — a dedicated cgroup (v1 or v2, matching + * preflight's detected version) bounding memory and CPU time, created + * before launch and assigned by PID immediately after spawn (see + * {@link CloudHypervisorCgroup}). + * + * No shell is ever invoked: every argv below is passed as a plain array to + * `execa`, never interpolated into a shell string. + */ + +export const CLOUD_HYPERVISOR_GUEST_CID = 3; + +export interface CloudHypervisorLaunchPaths { + readonly kernelPath: string; + readonly rootfsPath: string; + readonly workspacePath?: string; + readonly runDirectory: string; + readonly apiSocketPath: string; + readonly vsockSocketPath: string; +} + +export interface CloudHypervisorLaunchIdentity { + readonly uid: number; + readonly gid: number; +} + +export interface CloudHypervisorLaunchCommand { + readonly command: string; + readonly args: readonly string[]; +} + +export interface CloudHypervisorLaunchToolPaths { + readonly ip: string; + readonly setpriv: string; +} + +/** + * Builds the argv AWF spawns to launch Cloud Hypervisor: join the prepared + * network namespace, drop to the non-root operator identity with an empty + * capability bounding set, then exec the pinned Cloud Hypervisor binary + * with only its API socket configured (the VM itself is created and booted + * afterwards over that socket, mirroring Firecracker's `--api-sock`-only + * jailer invocation). + * + * The launched process retains exactly one supplementary group: the group + * that owns `/dev/kvm` (resolved by preflight). A blanket `--clear-groups` + * would also drop that membership, and since the documented supported + * setup relies on kvm-group access for the non-root operator identity + * (see docs/cloud-hypervisor-foundation.md), that would make every real + * launch fail with EACCES opening `/dev/kvm` even though preflight (which + * runs as root) passed. + */ +export function buildCloudHypervisorLaunchCommand(options: { + readonly tools: CloudHypervisorLaunchToolPaths; + readonly namespaceName: string; + readonly identity: CloudHypervisorLaunchIdentity; + readonly kvmGid: number; + readonly cloudHypervisorBinary: string; + readonly apiSocketPath: string; + readonly logFilePath: string; +}): CloudHypervisorLaunchCommand { + assertSafeNamespaceName(options.namespaceName); + assertPositiveIdentity(options.identity.uid, 'uid'); + assertPositiveIdentity(options.identity.gid, 'gid'); + if (!Number.isSafeInteger(options.kvmGid) || options.kvmGid < 0) { + throw new Error(`Cloud Hypervisor launch /dev/kvm group id must be a non-negative integer: ${options.kvmGid}`); + } + if (!path.isAbsolute(options.cloudHypervisorBinary)) { + throw new Error(`Cloud Hypervisor binary path must be absolute: ${options.cloudHypervisorBinary}`); + } + if (!path.isAbsolute(options.apiSocketPath)) { + throw new Error(`Cloud Hypervisor API socket path must be absolute: ${options.apiSocketPath}`); + } + + return { + command: options.tools.ip, + args: [ + 'netns', 'exec', options.namespaceName, + options.tools.setpriv, + `--reuid=${options.identity.uid}`, + `--regid=${options.identity.gid}`, + // Replaces the operator's full supplementary group list with only + // the /dev/kvm-owning group, instead of --clear-groups (which would + // also drop kvm access). + `--groups=${options.kvmGid}`, + '--no-new-privs', + '--inh-caps=-all', + '--bounding-set=-all', + '--', + options.cloudHypervisorBinary, + '--api-socket', `path=${options.apiSocketPath}`, + '--log-file', options.logFilePath, + '-v', + '--seccomp', 'true', + ], + }; +} + +/** + * Computes the minimal set of Landlock filesystem rules Cloud Hypervisor's + * own process needs after `vm.create`: read access to the kernel image, + * read-write access to the rootfs and (if present) workspace disk images, + * read-write access to the private run directory (for the API and vsock + * UNIX domain sockets it creates there), and read-write access to the + * device nodes it must reopen for virtio-net TAP attachment and KVM + * ioctls. Any path not listed here becomes inaccessible to the Cloud + * Hypervisor process the instant Landlock is enabled, even to a + * hypothetical guest-escape. + */ +export function computeCloudHypervisorLandlockRules( + paths: CloudHypervisorLaunchPaths, +): CloudHypervisorLandlockRule[] { + const rules: CloudHypervisorLandlockRule[] = [ + { path: paths.kernelPath, access: 'r' }, + { path: paths.rootfsPath, access: 'rw' }, + { path: paths.runDirectory, access: 'rw' }, + { path: '/dev/kvm', access: 'rw' }, + { path: '/dev/net/tun', access: 'rw' }, + ]; + if (paths.workspacePath) { + rules.push({ path: paths.workspacePath, access: 'rw' }); + } + return rules; +} + +export interface CloudHypervisorResourceLimits { + readonly memoryMib: number; + readonly vcpuCount: number; +} + +/** Fixed VMM/guest-overhead headroom added on top of configured guest memory. */ +const CGROUP_MEMORY_HEADROOM_MIB = 256; +/** Bounds the number of Cloud Hypervisor host threads/tasks (defense in depth; it is a single process). */ +const CGROUP_MAX_PIDS = 256; +const CGROUP_V2_PERIOD_US = 100_000; +const CGROUP_V2_CONTROLLERS = '+cpu +memory +pids'; + +export interface CloudHypervisorCgroupDependencies { + mkdir(directory: string): Promise; + writeFile(filePath: string, contents: string): Promise; + /** Removes exactly the (now-empty) leaf cgroup directory. cgroupfs's + * controller/interface files are virtual and cannot be `unlink()`ed, so + * this must be a plain `rmdir`, not a recursive tree removal. */ + rmdir(directory: string): Promise; +} + +const defaultCgroupDependencies: CloudHypervisorCgroupDependencies = { + mkdir: (directory) => fs.mkdir(directory, { recursive: true, mode: 0o700 }), + writeFile: (filePath, contents) => fs.writeFile(filePath, contents), + rmdir: (directory) => fs.rmdir(directory), +}; + +/** + * Places one Cloud Hypervisor run under an explicit memory/CPU/PID cgroup, + * created before launch and assigned by PID immediately after spawn (moving + * a PID into `cgroup.procs` requires only host-root write access to that + * file, not any privilege from the moved process itself). + * + * Cgroup v2 only: `runCloudHypervisorPreflight` rejects cgroup v1-only + * hosts explicitly rather than falling back to a v1 hierarchy this class + * does not manage (v1's memory/cpu/pids controllers live under separate + * per-controller mount points, not a single directory). + */ +export class CloudHypervisorCgroup { + private created = false; + + constructor( + readonly cgroupPath: string, + private readonly limits: CloudHypervisorResourceLimits, + private readonly dependencies: CloudHypervisorCgroupDependencies = defaultCgroupDependencies, + ) {} + + async setup(): Promise { + // cgroup v2 only materializes a controller's interface files + // (memory.max, cpu.max, pids.max, ...) in a directory once that + // controller is enabled in the *parent's* `cgroup.subtree_control`. + // That delegation has to happen at every level from the cgroup root + // down to (but excluding) the leaf we actually place limits on. + const parentDir = path.dirname(this.cgroupPath); + const rootDir = path.dirname(parentDir); + await this.enableControllers(rootDir); + await this.dependencies.mkdir(parentDir); + await this.enableControllers(parentDir); + await this.dependencies.mkdir(this.cgroupPath); + this.created = true; + + const memoryMaxBytes = (this.limits.memoryMib + CGROUP_MEMORY_HEADROOM_MIB) * 1024 * 1024; + const cpuQuotaUs = this.limits.vcpuCount * CGROUP_V2_PERIOD_US; + await this.dependencies.writeFile(path.join(this.cgroupPath, 'memory.max'), String(memoryMaxBytes)); + await this.dependencies.writeFile( + path.join(this.cgroupPath, 'cpu.max'), + `${cpuQuotaUs} ${CGROUP_V2_PERIOD_US}`, + ); + await this.dependencies.writeFile(path.join(this.cgroupPath, 'pids.max'), String(CGROUP_MAX_PIDS)); + } + + async assign(pid: number): Promise { + if (!Number.isInteger(pid) || pid <= 0) { + throw new Error(`Cannot assign an invalid PID to the Cloud Hypervisor cgroup: ${pid}`); + } + await this.dependencies.writeFile(path.join(this.cgroupPath, 'cgroup.procs'), String(pid)); + } + + async cleanup(): Promise { + if (!this.created) return; + await this.dependencies.rmdir(this.cgroupPath); + this.created = false; + } + + private async enableControllers(directory: string): Promise { + await this.dependencies.writeFile( + path.join(directory, 'cgroup.subtree_control'), + CGROUP_V2_CONTROLLERS, + ); + } +} + +function assertSafeNamespaceName(value: string): void { + if (!/^[A-Za-z0-9_.-]+$/.test(value)) { + throw new Error(`Unsafe Cloud Hypervisor network namespace name: ${value}`); + } +} + +function assertPositiveIdentity(value: number, label: string): void { + if (!Number.isSafeInteger(value) || value <= 0) { + throw new Error(`Cloud Hypervisor launch ${label} must be a positive integer`); + } +} diff --git a/src/cloud-hypervisor/manager.test.ts b/src/cloud-hypervisor/manager.test.ts new file mode 100644 index 000000000..da3e959c1 --- /dev/null +++ b/src/cloud-hypervisor/manager.test.ts @@ -0,0 +1,790 @@ +import type { ExecaChildProcess } from 'execa'; +import { PassThrough } from 'stream'; +import type { + MicrovmNetworkLifecycle, + MicrovmNetworkPlan, +} from '../microvm/network'; +import type { MicrovmVsockClient } from '../microvm/vsock-client'; +import type { MicrovmWorkspaceImage } from '../microvm/workspace'; +import type { CloudHypervisorOptions } from '../types/runtime-options'; +import type { CloudHypervisorApiClient } from './api-client'; +import type { CloudHypervisorCgroup } from './launcher'; +import { + CloudHypervisorManager, + buildSupervisorBootArgs, + cloudHypervisorManagerTestHelpers, + createCloudHypervisorRunPaths, + type CloudHypervisorManagerDependencies, + type CloudHypervisorManagerNetworkConfig, +} from './manager'; +import type { CloudHypervisorHostToolPaths } from './preflight'; + +const hostTools: CloudHypervisorHostToolPaths = { + ip: '/usr/bin/ip', + nft: '/usr/sbin/nft', + sysctl: '/usr/sbin/sysctl', + mke2fs: '/usr/sbin/mke2fs', + debugfs: '/usr/sbin/debugfs', + e2fsck: '/usr/sbin/e2fsck', + rsync: '/usr/bin/rsync', + setpriv: '/usr/bin/setpriv', +}; + +function config(overrides: Partial = {}): CloudHypervisorOptions { + return { + previewEnabled: true, + cloudHypervisorBinary: '/opt/cloud-hypervisor', + kernelPath: '/opt/vmlinux', + rootfsPath: '/opt/rootfs.ext4', + supervisorPath: '/opt/awf-supervisor', + vcpuCount: 2, + memoryMib: 512, + apiTimeoutMs: 1, + ...overrides, + }; +} + +function processMock(): ExecaChildProcess { + const child = Promise.resolve({ exitCode: 0 }) as unknown as ExecaChildProcess; + Object.assign(child, { + exitCode: null, + signalCode: null, + killed: false, + pid: 4242, + kill: jest.fn(() => { + Object.assign(child, { exitCode: 0, killed: true }); + return true; + }), + }); + return child; +} + +function networkConfig( + overrides: Partial = {}, +): CloudHypervisorManagerNetworkConfig { + return { + infrastructureBridge: 'awfbr0', + enableApiProxy: true, + ...overrides, + }; +} + +function networkLifecycle(plan: MicrovmNetworkPlan): MicrovmNetworkLifecycle { + return { + plan, + setup: jest.fn().mockResolvedValue(plan), + cleanup: jest.fn().mockResolvedValue(undefined), + }; +} + +function cgroupMock(): CloudHypervisorCgroup { + return { + cgroupPath: '/sys/fs/cgroup/awf-cloud-hypervisor/run', + setup: jest.fn().mockResolvedValue(undefined), + assign: jest.fn().mockResolvedValue(undefined), + cleanup: jest.fn().mockResolvedValue(undefined), + } as unknown as CloudHypervisorCgroup; +} + +function dependencies( + overrides: Partial = {}, +): CloudHypervisorManagerDependencies { + const client = { + ping: jest.fn().mockResolvedValue({ version: '53.0' }), + vmCreate: jest.fn().mockResolvedValue(undefined), + vmBoot: jest.fn().mockResolvedValue(undefined), + vmInfo: jest.fn().mockResolvedValue({ state: 'Running' }), + vmCounters: jest.fn().mockResolvedValue({ net0: { rx_bytes: 0 } }), + vmShutdown: jest.fn().mockResolvedValue(undefined), + vmmShutdown: jest.fn().mockResolvedValue(undefined), + } as unknown as CloudHypervisorApiClient; + return { + preflight: jest.fn().mockResolvedValue({ + version: '53.0', + cloudHypervisorBinary: '/opt/cloud-hypervisor', + kernelPath: '/opt/vmlinux', + rootfsPath: '/opt/rootfs.ext4', + tools: hostTools, + supervisorPath: '/opt/awf-supervisor', + cgroupVersion: 2, + kvmGid: 978, + }), + launch: jest.fn().mockReturnValue(processMock()), + mkdir: jest.fn().mockResolvedValue(undefined), + copyFile: jest.fn().mockResolvedValue(undefined), + chmod: jest.fn().mockResolvedValue(undefined), + chown: jest.fn().mockResolvedValue(undefined), + writeFile: jest.fn().mockResolvedValue(undefined), + readFileTail: jest.fn().mockResolvedValue(Buffer.alloc(0)), + access: jest.fn().mockResolvedValue(undefined), + rm: jest.fn().mockResolvedValue(undefined), + sleep: jest.fn().mockResolvedValue(undefined), + createClient: jest.fn().mockReturnValue(client), + createNetwork: jest.fn((plan) => networkLifecycle(plan)), + createWorkspaceImage: jest.fn(), + createVsockClient: jest.fn(), + createCgroup: jest.fn(() => cgroupMock()), + resolveIdentity: jest.fn().mockReturnValue({ uid: 1000, gid: 1000 }), + ...overrides, + }; +} + +describe('CloudHypervisorManager', () => { + it('constructs the default host adapters and non-root identity', async () => { + const defaults = cloudHypervisorManagerTestHelpers.defaultDependencies; + const child = defaults.launch(process.execPath, ['-e', ''], { + reject: false, + stdio: ['ignore', 'pipe', 'pipe'], + env: { PATH: '/usr/bin' }, + extendEnv: false, + }); + await expect(child).resolves.toMatchObject({ exitCode: 0 }); + await expect(defaults.sleep(0)).resolves.toBeUndefined(); + expect(defaults.createClient('/tmp/api.socket', 100)).toBeDefined(); + expect(defaults.createNetwork({} as MicrovmNetworkPlan, hostTools)).toBeDefined(); + expect(defaults.createWorkspaceImage({ + runId: 'adapter-test', + workDir: '/tmp/awf', + workspacePath: '/workspace', + homePath: '/home/runner', + baseRootfsPath: '/opt/rootfs', + supervisorBinaryPath: '/opt/supervisor', + supervisorSha256: 'a'.repeat(64), + uid: 1000, + gid: 1000, + }, hostTools)).toBeDefined(); + expect(defaults.createVsockClient('/tmp/vsock.socket', 52, 100)).toBeDefined(); + expect(defaults.createCgroup('/sys/fs/cgroup/awf/run', { memoryMib: 512, vcpuCount: 2 })).toBeDefined(); + + const originalSudoUid = process.env.SUDO_UID; + const originalSudoGid = process.env.SUDO_GID; + const uidSpy = jest.spyOn(process, 'getuid').mockReturnValue(0); + const gidSpy = jest.spyOn(process, 'getgid').mockReturnValue(0); + try { + process.env.SUDO_UID = '2001'; + process.env.SUDO_GID = '2002'; + expect(cloudHypervisorManagerTestHelpers.resolveCloudHypervisorIdentity()).toEqual({ + uid: 2001, + gid: 2002, + }); + + delete process.env.SUDO_UID; + delete process.env.SUDO_GID; + expect(cloudHypervisorManagerTestHelpers.resolveCloudHypervisorIdentity) + .toThrow(/non-root target uid\/gid/); + } finally { + uidSpy.mockRestore(); + gidSpy.mockRestore(); + if (originalSudoUid === undefined) delete process.env.SUDO_UID; + else process.env.SUDO_UID = originalSudoUid; + if (originalSudoGid === undefined) delete process.env.SUDO_GID; + else process.env.SUDO_GID = originalSudoGid; + } + }); + + it('constructs unique, contained run paths outside workDir', () => { + const first = createCloudHypervisorRunPaths('/opt/cloud-hypervisor'); + const second = createCloudHypervisorRunPaths('/opt/cloud-hypervisor'); + expect(first.runId).not.toBe(second.runId); + expect(first.runDirectory).toContain('/run/awf-cloud-hypervisor/cloud-hypervisor/'); + expect(first.cgroupPath).toContain('/sys/fs/cgroup/awf-cloud-hypervisor/'); + expect(() => createCloudHypervisorRunPaths( + '/opt/cloud-hypervisor', + '../escape', + )).toThrow(/Unsafe microVM run id/); + }); + + it('launches via the secure launcher and creates/boots the VM over the API', async () => { + const deps = dependencies(); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'run-1', + networkConfig(), + ); + const client = await manager.start(); + + expect(deps.launch).toHaveBeenCalledWith( + '/usr/bin/ip', + expect.arrayContaining([ + 'netns', 'exec', expect.stringMatching(/^awffc-/), + '/usr/bin/setpriv', + '--reuid=1000', + '--regid=1000', + '--groups=978', + ]), + expect.objectContaining({ + reject: false, + extendEnv: false, + env: { PATH: expect.stringContaining('/bin') }, + }), + ); + const launchEnv = (deps.launch as jest.Mock).mock.calls[0][2].env as NodeJS.ProcessEnv; + expect(Object.keys(launchEnv)).toEqual(['PATH']); + expect(client.vmCreate).toHaveBeenCalledWith(expect.objectContaining({ + cpus: { boot_vcpus: 2, max_vcpus: 2 }, + memory: { size: 512 * 1024 * 1024 }, + payload: expect.objectContaining({ kernel: expect.stringContaining('/kernel') }), + landlock_enable: true, + })); + expect(client.vmCreate).not.toHaveBeenCalledWith(expect.objectContaining({ vsock: expect.anything() })); + expect(client.ping).toHaveBeenCalledTimes(1); + expect(deps.createCgroup).toHaveBeenCalledWith( + expect.stringContaining('awf-cloud-hypervisor/run-1'), + { memoryMib: 512, vcpuCount: 2 }, + ); + const cgroup = (deps.createCgroup as jest.Mock).mock.results[0].value as CloudHypervisorCgroup; + expect(cgroup.setup).toHaveBeenCalledTimes(1); + expect(cgroup.assign).toHaveBeenCalledWith(4242); + // Private run directory: ancestor levels stay traversable-only (0711, + // root-owned); only the leaf is chowned to the non-root identity. + expect(deps.mkdir).toHaveBeenCalledWith('/run/awf-cloud-hypervisor', { recursive: true, mode: 0o711 }); + expect(deps.chmod).toHaveBeenCalledWith('/run/awf-cloud-hypervisor', 0o711); + expect(deps.mkdir).toHaveBeenCalledWith('/run/awf-cloud-hypervisor/cloud-hypervisor', { recursive: true, mode: 0o711 }); + expect(deps.chmod).toHaveBeenCalledWith('/run/awf-cloud-hypervisor/cloud-hypervisor', 0o711); + expect(deps.mkdir).toHaveBeenCalledWith( + '/run/awf-cloud-hypervisor/cloud-hypervisor/run-1', + { recursive: true, mode: 0o700 }, + ); + expect(deps.chown).toHaveBeenCalledWith( + '/run/awf-cloud-hypervisor/cloud-hypervisor/run-1', + 1000, + 1000, + ); + expect(deps.createNetwork).toHaveBeenCalledWith( + expect.objectContaining({ + infrastructureBridge: 'awfbr0', + tapOwnerUid: 1000, + tapOwnerGid: 1000, + }), + hostTools, + ); + const lifecycle = (deps.createNetwork as jest.Mock).mock.results[0] + .value as MicrovmNetworkLifecycle; + expect(lifecycle.setup).toHaveBeenCalledTimes(1); + }); + + it('terminates the partial process and removes its run directory on readiness failure', async () => { + const child = processMock(); + const missing = Object.assign(new Error('missing'), { code: 'ENOENT' }); + const deps = dependencies({ + launch: jest.fn().mockReturnValue(child), + access: jest.fn().mockRejectedValue(missing), + sleep: jest.fn(async () => new Promise((resolve) => setTimeout(resolve, 2))), + }); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'partial', + networkConfig(), + ); + + await expect(manager.start()).rejects.toThrow(/API socket was not ready/); + expect(child.kill).toHaveBeenCalledWith( + 'SIGTERM', + { forceKillAfterTimeout: 2_000 }, + ); + expect(deps.rm).toHaveBeenCalledWith( + '/run/awf-cloud-hypervisor/cloud-hypervisor/partial', + { recursive: true, force: true }, + ); + const lifecycle = (deps.createNetwork as jest.Mock).mock.results[0] + .value as MicrovmNetworkLifecycle; + expect(lifecycle.cleanup).toHaveBeenCalledTimes(1); + const cgroup = (deps.createCgroup as jest.Mock).mock.results[0].value as CloudHypervisorCgroup; + expect(cgroup.cleanup).toHaveBeenCalledTimes(1); + }); + + it('refuses to launch without host-side network enforcement', async () => { + const deps = dependencies(); + const manager = new CloudHypervisorManager(config(), '/tmp/awf', deps, 'unsafe'); + + await expect(manager.start()).rejects.toThrow(/unfiltered microVM/); + expect(deps.preflight).not.toHaveBeenCalled(); + expect(deps.launch).not.toHaveBeenCalled(); + }); + + it('cleans up the network and cgroup before removing the run directory', async () => { + const order: string[] = []; + const deps = dependencies({ + createNetwork: jest.fn((plan) => ({ + plan, + setup: jest.fn().mockResolvedValue(plan), + cleanup: jest.fn(async () => { + order.push('network'); + }), + })), + createCgroup: jest.fn(() => ({ + cgroupPath: '/sys/fs/cgroup/awf-cloud-hypervisor/cleanup', + setup: jest.fn().mockResolvedValue(undefined), + assign: jest.fn().mockResolvedValue(undefined), + cleanup: jest.fn(async () => { + order.push('cgroup'); + }), + } as unknown as CloudHypervisorCgroup)), + rm: jest.fn(async () => { + order.push('run-directory'); + }), + }); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'cleanup', + networkConfig(), + ); + + await manager.start(); + await manager.stop(); + + expect(order).toEqual(['network', 'cgroup', 'run-directory']); + }); + + it('configures the workspace disk and vsock, then extracts only after VM termination', async () => { + const order: string[] = []; + const child = processMock(); + const workspace = { + prepare: jest.fn().mockResolvedValue({ + workspaceImagePath: '/tmp/prepared-workspace.ext4', + rootfsImagePath: '/tmp/prepared-rootfs.ext4', + imageBytes: 1024, + originalManifest: new Map(), + }), + extractAfterStop: jest.fn(async () => { + order.push('extract'); + expect(child.exitCode).toBe(0); + }), + cleanup: jest.fn().mockResolvedValue(undefined), + } as unknown as MicrovmWorkspaceImage; + const guestClient = { + connect: jest.fn().mockResolvedValue({ + version: 1, + type: 'ready', + requestId: 'control', + capabilities: { stdin: true, tty: false, resize: false }, + }), + execute: jest.fn().mockResolvedValue({ + requestId: 'command', + exitCode: 0, + signal: null, + timedOut: false, + }), + shutdown: jest.fn().mockResolvedValue(undefined), + destroy: jest.fn(), + } as unknown as MicrovmVsockClient; + const deps = dependencies({ + launch: jest.fn().mockReturnValue(child), + createWorkspaceImage: jest.fn().mockReturnValue(workspace), + createVsockClient: jest.fn().mockReturnValue(guestClient), + }); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'guest', + networkConfig(), + { + workspacePath: '/workspace', + homePath: '/home/runner', + supervisorBinaryPath: '/opt/awf-supervisor', + supervisorSha256: 'a'.repeat(64), + }, + ); + + const client = await manager.start(); + expect(client.vmCreate).toHaveBeenCalledWith(expect.objectContaining({ + payload: expect.objectContaining({ cmdline: expect.stringContaining('init=/sbin/awf-supervisor') }), + disks: expect.arrayContaining([ + expect.objectContaining({ id: 'rootfs' }), + expect.objectContaining({ id: 'workspace' }), + ]), + vsock: expect.objectContaining({ cid: 3 }), + landlock_rules: expect.arrayContaining([ + expect.objectContaining({ path: expect.stringContaining('workspace.ext4') }), + ]), + })); + await manager.startInstance(); + expect(client.vmBoot).toHaveBeenCalledTimes(1); + expect(deps.createVsockClient).toHaveBeenCalledWith( + expect.stringContaining('/run/awf-cloud-hypervisor/cloud-hypervisor/guest/awf-vsock.socket'), + 52, + 1, + ); + await expect(manager.execute({ + requestId: 'command', + argv: ['true'], + env: {}, + cwd: '/workspace', + uid: 1000, + gid: 1000, + })).resolves.toEqual(expect.objectContaining({ exitCode: 0 })); + await manager.stop(); + + expect(guestClient.shutdown).toHaveBeenCalledTimes(1); + expect(client.vmShutdown).toHaveBeenCalledTimes(1); + expect(client.vmmShutdown).toHaveBeenCalledTimes(1); + expect(workspace.extractAfterStop).toHaveBeenCalledWith( + expect.stringContaining('/run/awf-cloud-hypervisor/cloud-hypervisor/guest/workspace.ext4'), + ); + expect(order).toEqual(['extract']); + }); + + it('delegates guest cancellation, stdin, and resize only after readiness', async () => { + const cold = new CloudHypervisorManager( + config(), + '/tmp/awf', + dependencies(), + 'cold-guest', + networkConfig(), + ); + await expect(cold.cancel()).rejects.toThrow(/supervisor is not ready/); + await expect(cold.writeStdin(Buffer.from('input'))).rejects.toThrow(/supervisor is not ready/); + await expect(cold.endStdin()).rejects.toThrow(/supervisor is not ready/); + await expect(cold.resize(80, 24)).rejects.toThrow(/supervisor is not ready/); + + const guestClient = { + connect: jest.fn().mockResolvedValue(undefined), + execute: jest.fn(), + cancel: jest.fn().mockResolvedValue(undefined), + writeStdin: jest.fn().mockResolvedValue(undefined), + endStdin: jest.fn().mockResolvedValue(undefined), + resize: jest.fn().mockResolvedValue(undefined), + shutdown: jest.fn().mockResolvedValue(undefined), + destroy: jest.fn(), + } as unknown as MicrovmVsockClient; + const workspace = { + prepare: jest.fn().mockResolvedValue({ + workspaceImagePath: '/tmp/workspace.ext4', + rootfsImagePath: '/tmp/rootfs.ext4', + imageBytes: 1024, + originalManifest: new Map(), + }), + extractAfterStop: jest.fn().mockResolvedValue(undefined), + cleanup: jest.fn().mockResolvedValue(undefined), + } as unknown as MicrovmWorkspaceImage; + const deps = dependencies({ + createVsockClient: jest.fn().mockReturnValue(guestClient), + createWorkspaceImage: jest.fn().mockReturnValue(workspace), + }); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'ready-guest', + networkConfig(), + { + workspacePath: '/workspace', + homePath: '/home/runner', + supervisorBinaryPath: '/opt/supervisor', + supervisorSha256: 'a'.repeat(64), + }, + ); + await manager.start(); + await manager.startInstance(); + await manager.cancel('test', 'request'); + await manager.writeStdin(Buffer.from('input'), 'request'); + await manager.endStdin('request'); + await manager.resize(80, 24, 'request'); + + expect(guestClient.cancel).toHaveBeenCalledWith('test', 'request'); + expect(guestClient.writeStdin).toHaveBeenCalledWith(Buffer.from('input'), 'request'); + expect(guestClient.endStdin).toHaveBeenCalledWith('request'); + expect(guestClient.resize).toHaveBeenCalledWith(80, 24, 'request'); + await manager.stop(); + }); + + it('quiesces and copies back while preserving run directory, images, and network in keep mode', async () => { + const child = processMock(); + const workspace = { + prepare: jest.fn().mockResolvedValue({ + workspaceImagePath: '/tmp/prepared-workspace.ext4', + rootfsImagePath: '/tmp/prepared-rootfs.ext4', + imageBytes: 1024, + originalManifest: new Map(), + }), + extractAfterStop: jest.fn().mockResolvedValue(undefined), + cleanup: jest.fn().mockResolvedValue(undefined), + } as unknown as MicrovmWorkspaceImage; + const guestClient = { + connect: jest.fn().mockResolvedValue(undefined), + shutdown: jest.fn().mockResolvedValue(undefined), + destroy: jest.fn(), + } as unknown as MicrovmVsockClient; + const deps = dependencies({ + launch: jest.fn().mockReturnValue(child), + createWorkspaceImage: jest.fn().mockReturnValue(workspace), + createVsockClient: jest.fn().mockReturnValue(guestClient), + }); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'keep', + networkConfig(), + { + workspacePath: '/workspace', + homePath: '/home/runner', + supervisorBinaryPath: '/opt/awf-supervisor', + supervisorSha256: 'a'.repeat(64), + }, + ); + await manager.start(); + await manager.startInstance(); + + await manager.stop({ preserve: true }); + + const lifecycle = (deps.createNetwork as jest.Mock).mock.results[0] + .value as MicrovmNetworkLifecycle; + expect(workspace.extractAfterStop).toHaveBeenCalledTimes(1); + expect(lifecycle.cleanup).not.toHaveBeenCalled(); + expect(workspace.cleanup).not.toHaveBeenCalled(); + expect(deps.rm).not.toHaveBeenCalled(); + const cgroup = (deps.createCgroup as jest.Mock).mock.results[0].value as CloudHypervisorCgroup; + expect(cgroup.cleanup).toHaveBeenCalledTimes(1); + }); + + it('builds explicit supervisor boot cmdline with PCI-required root/interface naming', () => { + const args = buildSupervisorBootArgs({ + runId: 'run', + namespaceName: 'ns', + netnsPath: '/var/run/netns/ns', + nftTableName: 'table', + infrastructureBridge: 'awfbr0', + hostVethName: 'host', + namespaceVethName: 'namespace', + tapName: 'tap', + infrastructureIp: '172.30.0.20', + infrastructureCidr: '172.30.0.0/24', + hostGatewayIp: '172.30.0.1', + guestSubnet: '100.64.0.0/30', + guestIp: '100.64.0.2', + guestGatewayIp: '100.64.0.1', + guestPrefixLength: 30, + guestMac: '02:00:00:00:00:01', + tapOwnerUid: 1000, + tapOwnerGid: 1000, + allowedEndpoints: [], + networkInterface: { iface_id: 'eth0', host_dev_name: 'tap' }, + }, { + workspacePath: '/workspace', + homePath: '/home/runner', + supervisorBinaryPath: '/opt/supervisor', + supervisorSha256: 'a'.repeat(64), + }); + expect(args).toContain('root=/dev/vda'); + expect(args).toContain('awf.guest-ip=100.64.0.2'); + expect(args).toContain('awf.guest-gateway=100.64.0.1'); + expect(args).toContain('awf.workspace-device=/dev/vdb'); + expect(args).toContain('net.ifnames=0'); + expect(args).not.toContain('pci=off'); + expect(args).not.toContain('8.8.8.8'); + }); + + it('retains the workspace and network until process termination is confirmed', async () => { + const child = Promise.resolve({ exitCode: null }) as unknown as ExecaChildProcess; + Object.assign(child, { + exitCode: null, + signalCode: null, + killed: false, + pid: 9, + kill: jest.fn(() => { + Object.assign(child, { killed: true }); + return true; + }), + }); + const workspace = { + prepare: jest.fn().mockResolvedValue({ + workspaceImagePath: '/tmp/prepared-workspace.ext4', + rootfsImagePath: '/tmp/prepared-rootfs.ext4', + imageBytes: 1024, + originalManifest: new Map(), + }), + extractAfterStop: jest.fn().mockResolvedValue(undefined), + cleanup: jest.fn().mockResolvedValue(undefined), + } as unknown as MicrovmWorkspaceImage; + const guestClient = { + connect: jest.fn().mockResolvedValue(undefined), + shutdown: jest.fn().mockResolvedValue(undefined), + destroy: jest.fn(), + } as unknown as MicrovmVsockClient; + const deps = dependencies({ + launch: jest.fn().mockReturnValue(child), + createWorkspaceImage: jest.fn().mockReturnValue(workspace), + createVsockClient: jest.fn().mockReturnValue(guestClient), + }); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'termination', + networkConfig(), + { + workspacePath: '/workspace', + homePath: '/home/runner', + supervisorBinaryPath: '/opt/awf-supervisor', + supervisorSha256: 'a'.repeat(64), + }, + ); + await manager.start(); + await manager.startInstance(); + + await expect(manager.stop()).rejects.toThrow(/stopped before workspace\/network removal/); + const lifecycle = (deps.createNetwork as jest.Mock).mock.results[0] + .value as MicrovmNetworkLifecycle; + expect(lifecycle.cleanup).not.toHaveBeenCalled(); + expect(workspace.extractAfterStop).not.toHaveBeenCalled(); + expect(deps.rm).not.toHaveBeenCalled(); + + Object.assign(child, { exitCode: 0 }); + await expect(manager.stop()).resolves.toBeUndefined(); + expect(workspace.extractAfterStop).toHaveBeenCalledTimes(1); + expect(lifecycle.cleanup).toHaveBeenCalledTimes(1); + }); + + it('waits briefly for natural VM exit after guest shutdown before sending SIGTERM', async () => { + const child = processMock(); + const workspace = { + prepare: jest.fn().mockResolvedValue({ + workspaceImagePath: '/tmp/prepared-workspace.ext4', + rootfsImagePath: '/tmp/prepared-rootfs.ext4', + imageBytes: 1024, + originalManifest: new Map(), + }), + extractAfterStop: jest.fn().mockResolvedValue(undefined), + cleanup: jest.fn().mockResolvedValue(undefined), + } as unknown as MicrovmWorkspaceImage; + const guestClient = { + connect: jest.fn().mockResolvedValue(undefined), + shutdown: jest.fn().mockResolvedValue(undefined), + destroy: jest.fn(), + } as unknown as MicrovmVsockClient; + let sleepCalls = 0; + const deps = dependencies({ + launch: jest.fn().mockReturnValue(child), + createWorkspaceImage: jest.fn().mockReturnValue(workspace), + createVsockClient: jest.fn().mockReturnValue(guestClient), + sleep: jest.fn(async () => { + sleepCalls += 1; + if (sleepCalls === 3) Object.assign(child, { exitCode: 0 }); + }), + }); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'natural-exit', + networkConfig(), + { + workspacePath: '/workspace', + homePath: '/home/runner', + supervisorBinaryPath: '/opt/awf-supervisor', + supervisorSha256: 'a'.repeat(64), + }, + ); + await manager.start(); + await manager.startInstance(); + await manager.stop(); + expect(child.kill).not.toHaveBeenCalled(); + expect(sleepCalls).toBeGreaterThan(0); + }); + + it('rolls back the network and cgroup when vm.create fails', async () => { + const client = { + ping: jest.fn().mockResolvedValue({ version: '53.0' }), + vmCreate: jest.fn().mockRejectedValue(new Error('invalid disk path')), + vmBoot: jest.fn().mockResolvedValue(undefined), + vmInfo: jest.fn().mockResolvedValue({ state: 'Created' }), + vmCounters: jest.fn().mockResolvedValue({}), + vmShutdown: jest.fn().mockResolvedValue(undefined), + vmmShutdown: jest.fn().mockResolvedValue(undefined), + } as unknown as CloudHypervisorApiClient; + const deps = dependencies({ + createClient: jest.fn().mockReturnValue(client), + }); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'create-failure', + networkConfig(), + ); + + await expect(manager.start()).rejects.toThrow('invalid disk path'); + + const lifecycle = (deps.createNetwork as jest.Mock).mock.results[0] + .value as MicrovmNetworkLifecycle; + expect(lifecycle.cleanup).toHaveBeenCalledTimes(1); + expect(deps.rm).toHaveBeenCalled(); + const cgroup = (deps.createCgroup as jest.Mock).mock.results[0].value as CloudHypervisorCgroup; + expect(cgroup.cleanup).toHaveBeenCalledTimes(1); + }); + + it('fails fast when Cloud Hypervisor exits by signal before API readiness', async () => { + const child = processMock(); + Object.assign(child, { signalCode: 'SIGKILL', kill: jest.fn() }); + const missing = Object.assign(new Error('missing'), { code: 'ENOENT' }); + const deps = dependencies({ + launch: jest.fn().mockReturnValue(child), + access: jest.fn().mockRejectedValue(missing), + sleep: jest.fn().mockResolvedValue(undefined), + }); + const manager = new CloudHypervisorManager( + config({ apiTimeoutMs: 2000 }), + '/tmp/awf', + deps, + 'signal', + networkConfig(), + ); + + await expect(manager.start()).rejects.toThrow( + /exited before API readiness with code null and signal SIGKILL/, + ); + expect(deps.sleep).not.toHaveBeenCalled(); + }); + + it('collects bounded diagnostics including VM counters', async () => { + const oversized = Buffer.alloc(1024 * 1024 + 128, 0x61); + const child = processMock(); + const stdout = new PassThrough(); + const stderr = new PassThrough(); + Object.assign(child, { stdout, stderr }); + const deps = dependencies({ + launch: jest.fn().mockReturnValue(child), + readFileTail: jest.fn().mockImplementation((_source: string, maxBytes: number) => + Promise.resolve(oversized.subarray(oversized.length - maxBytes)), + ), + }); + const manager = new CloudHypervisorManager( + config(), + '/tmp/awf', + deps, + 'diagnostics', + networkConfig(), + ); + + const client = await manager.start(); + stdout.write(oversized); + stderr.write('launcher error'); + await manager.startInstance(); + await manager.collectDiagnostics('/tmp/diagnostics'); + + expect(client.vmCounters).toHaveBeenCalledTimes(1); + expect(deps.writeFile).toHaveBeenCalledWith( + '/tmp/diagnostics/launcher-stdout.log', + expect.objectContaining({ length: 1024 * 1024 }), + { mode: 0o600 }, + ); + expect(deps.writeFile).toHaveBeenCalledWith( + '/tmp/diagnostics/launcher-stderr.log', + Buffer.from('launcher error'), + { mode: 0o600 }, + ); + expect(deps.writeFile).toHaveBeenCalledWith( + '/tmp/diagnostics/counters.json', + expect.stringContaining('rx_bytes'), + { mode: 0o600 }, + ); + }); +}); diff --git a/src/cloud-hypervisor/manager.ts b/src/cloud-hypervisor/manager.ts new file mode 100644 index 000000000..f6285b8be --- /dev/null +++ b/src/cloud-hypervisor/manager.ts @@ -0,0 +1,831 @@ +import { randomBytes } from 'crypto'; +import { constants, promises as fs } from 'fs'; +import * as path from 'path'; +import execa, { type ExecaChildProcess } from 'execa'; +import { + CLOUD_HYPERVISOR_RELEASE_VERSION, + type CloudHypervisorOptions, +} from '../types/runtime-options'; +import { getSafeHostGid, getSafeHostUid } from '../host-identity'; +import { + LinuxNetworkCommands, + MicrovmNetworkManager, + assertSafeMicrovmRunId, + createMicrovmNetworkPlan, + type MicrovmControlPeer, + type MicrovmNetworkLifecycle, + type MicrovmNetworkPlan, +} from '../microvm/network'; +import { + MicrovmVsockClient, + type GuestExecutionRequest, + type GuestExecutionResult, +} from '../microvm/vsock-client'; +import { + MicrovmWorkspaceImage, + type MicrovmWorkspaceImageConfig, +} from '../microvm/workspace'; +import { CloudHypervisorApiClient } from './api-client'; +import { + CLOUD_HYPERVISOR_GUEST_CID, + CloudHypervisorCgroup, + buildCloudHypervisorLaunchCommand, + computeCloudHypervisorLandlockRules, + type CloudHypervisorResourceLimits, +} from './launcher'; +import { runCloudHypervisorPreflight } from './preflight'; +import type { CloudHypervisorHostToolPaths } from './preflight'; + +const API_SOCKET_NAME = 'api.socket'; +const VSOCK_SOCKET_NAME = 'awf-vsock.socket'; +const WORKSPACE_IMAGE_NAME = 'workspace.ext4'; +const KERNEL_RUN_NAME = 'kernel'; +const ROOTFS_RUN_NAME = 'rootfs.ext4'; +const CLOUD_HYPERVISOR_LOG_NAME = 'cloud-hypervisor.log'; +const CLOUD_HYPERVISOR_SERIAL_LOG_NAME = 'serial.log'; +const CLOUD_HYPERVISOR_CAPTURE_LIMIT_BYTES = 1024 * 1024; +export const CLOUD_HYPERVISOR_GUEST_VSOCK_PORT = 52; +const CLOUD_HYPERVISOR_GUEST_SHUTDOWN_GRACE_MS = 5_000; +/** + * Private run-directory root, deliberately **outside** `workDir`. + * + * `workDir` is created root-owned mode 0700 (it holds `docker-compose.yml` + * with plaintext secrets — see `validateAndPrepareWorkDir` in + * `src/config-writer.ts`), so a non-root process can never traverse into + * it no matter how a leaf directory underneath it is chowned. Since this + * backend has no jailer to `chroot()` the launched process (which would + * make host-side ancestor permissions irrelevant), Cloud Hypervisor must + * be able to really `stat()`/`open()` its way down to the run directory + * post-`setpriv`. `/run` is always present, root-owned tmpfs; the two + * ancestor levels created under it are `0711` (traversable/executable by + * any uid, but not listable/readable — `ls` still fails), and only the + * per-run leaf directory is chowned to the non-root target identity with + * `0700` (so only that identity, or root, can actually read its contents). + */ +const CLOUD_HYPERVISOR_RUN_ROOT = '/run/awf-cloud-hypervisor'; +const CGROUP_ROOT = '/sys/fs/cgroup'; + +export interface CloudHypervisorRunPaths { + runId: string; + runBaseDir: string; + runDirectory: string; + apiSocketPath: string; + kernelPath: string; + rootfsPath: string; + workspacePath: string; + vsockSocketPath: string; + logPath: string; + serialLogPath: string; + cgroupPath: string; +} + +export interface CloudHypervisorManagerDependencies { + preflight: typeof runCloudHypervisorPreflight; + launch( + command: string, + args: string[], + options: { + reject: false; + stdio: ['ignore', 'pipe', 'pipe']; + env: NodeJS.ProcessEnv; + extendEnv: false; + }, + ): ExecaChildProcess; + mkdir(directory: string, options: { recursive: true; mode: number }): Promise; + copyFile(source: string, destination: string, flags: number): Promise; + chmod(filePath: string, mode: number): Promise; + chown(filePath: string, uid: number, gid: number): Promise; + writeFile: typeof fs.writeFile; + readFileTail(filePath: string, maxBytes: number): Promise; + access(filePath: string): Promise; + rm(directory: string, options: { recursive: true; force: true }): Promise; + sleep(milliseconds: number): Promise; + createClient(socketPath: string, timeoutMs: number): CloudHypervisorApiClient; + createNetwork(plan: MicrovmNetworkPlan, tools: CloudHypervisorHostToolPaths): MicrovmNetworkLifecycle; + createWorkspaceImage(config: MicrovmWorkspaceImageConfig, tools: CloudHypervisorHostToolPaths): MicrovmWorkspaceImage; + createVsockClient(socketPath: string, guestPort: number, timeoutMs: number): MicrovmVsockClient; + createCgroup(cgroupPath: string, limits: CloudHypervisorResourceLimits): CloudHypervisorCgroup; + resolveIdentity(): { uid: number; gid: number }; +} + +export interface CloudHypervisorManagerNetworkConfig { + infrastructureBridge: string; + enableApiProxy: boolean; + controlPeer?: MicrovmControlPeer; +} + +export interface CloudHypervisorManagerGuestConfig { + readonly workspacePath: string; + readonly homePath: string; + readonly supervisorBinaryPath: string; + readonly supervisorSha256: string; + readonly maxWorkspaceImageBytes?: number; + readonly vsockPort?: number; + readonly identity?: { uid: number; gid: number }; +} + +async function readBoundedTail(filePath: string, maxBytes: number): Promise { + const handle = await fs.open(filePath, 'r'); + try { + const { size } = await handle.stat(); + const length = Math.min(size, maxBytes); + const buffer = Buffer.alloc(length); + if (length > 0) { + await handle.read(buffer, 0, length, size - length); + } + return buffer; + } finally { + await handle.close(); + } +} + +const defaultDependencies: CloudHypervisorManagerDependencies = { + preflight: runCloudHypervisorPreflight, + launch: (command, args, options) => execa(command, args, options), + mkdir: fs.mkdir, + copyFile: fs.copyFile, + chmod: fs.chmod, + chown: fs.chown, + writeFile: fs.writeFile, + readFileTail: (filePath, maxBytes) => readBoundedTail(filePath, maxBytes), + access: fs.access, + rm: fs.rm, + sleep: (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)), + createClient: (socketPath, timeoutMs) => new CloudHypervisorApiClient({ socketPath, timeoutMs }), + createNetwork: (plan, tools) => new MicrovmNetworkManager( + plan, + new LinuxNetworkCommands(undefined, tools), + ), + createWorkspaceImage: (config, tools) => new MicrovmWorkspaceImage(config, undefined, tools), + createVsockClient: (socketPath, guestPort, timeoutMs) => new MicrovmVsockClient({ + socketPath, + guestPort, + connectTimeoutMs: timeoutMs, + readTimeoutMs: Math.max(timeoutMs, 30_000), + writeTimeoutMs: timeoutMs, + }), + createCgroup: (cgroupPath, limits) => new CloudHypervisorCgroup(cgroupPath, limits), + resolveIdentity: resolveCloudHypervisorIdentity, +}; + +/** @internal Exposed only for focused host-adapter tests. */ +export const cloudHypervisorManagerTestHelpers = { + defaultDependencies, + resolveCloudHypervisorIdentity, +}; + +function resolveCloudHypervisorIdentity(): { uid: number; gid: number } { + const operatorUid = parsePositiveIdentity(process.env.SUDO_UID) ?? process.getuid?.(); + const operatorGid = parsePositiveIdentity(process.env.SUDO_GID) ?? process.getgid?.(); + if ( + operatorUid === undefined || + operatorGid === undefined || + operatorUid === 0 || + operatorGid === 0 + ) { + throw new Error( + 'Cloud Hypervisor requires a non-root target uid/gid; run through sudo from a non-root account', + ); + } + const uid = Number(getSafeHostUid()); + const gid = Number(getSafeHostGid()); + if (!Number.isSafeInteger(uid) || !Number.isSafeInteger(gid) || uid < 1 || gid < 1) { + throw new Error( + 'Cloud Hypervisor requires a non-root target uid/gid; run through sudo from a non-root account', + ); + } + return { uid, gid }; +} + +function parsePositiveIdentity(value: string | undefined): number | undefined { + if (!value || !/^[1-9]\d*$/.test(value)) return undefined; + return Number(value); +} + +export function createCloudHypervisorRunPaths( + cloudHypervisorBinary: string, + runId = `awf-${process.pid}-${randomBytes(6).toString('hex')}`, +): CloudHypervisorRunPaths { + assertSafeMicrovmRunId(runId); + const runBaseDir = CLOUD_HYPERVISOR_RUN_ROOT; + const runDirectory = path.join( + runBaseDir, + path.basename(cloudHypervisorBinary), + runId, + ); + return { + runId, + runBaseDir, + runDirectory, + apiSocketPath: path.join(runDirectory, API_SOCKET_NAME), + kernelPath: path.join(runDirectory, KERNEL_RUN_NAME), + rootfsPath: path.join(runDirectory, ROOTFS_RUN_NAME), + workspacePath: path.join(runDirectory, WORKSPACE_IMAGE_NAME), + vsockSocketPath: path.join(runDirectory, VSOCK_SOCKET_NAME), + logPath: path.join(runDirectory, CLOUD_HYPERVISOR_LOG_NAME), + serialLogPath: path.join(runDirectory, CLOUD_HYPERVISOR_SERIAL_LOG_NAME), + cgroupPath: path.join(CGROUP_ROOT, 'awf-cloud-hypervisor', runId), + }; +} + +/** + * Owns one Cloud Hypervisor process launched via the secure host launcher in + * `./launcher.ts` (network-namespace join + privilege drop + Landlock, in + * place of Firecracker's jailer) and its partial-start cleanup. + */ +export class CloudHypervisorManager { + paths: CloudHypervisorRunPaths; + private process: ExecaChildProcess | undefined; + private client: CloudHypervisorApiClient | undefined; + private network: MicrovmNetworkLifecycle | undefined; + private workspace: MicrovmWorkspaceImage | undefined; + private guestClient: MicrovmVsockClient | undefined; + private cgroup: CloudHypervisorCgroup | undefined; + private networkPlan: MicrovmNetworkPlan | undefined; + private instanceStarted = false; + private readonly stdoutCapture = new BoundedOutputCapture(CLOUD_HYPERVISOR_CAPTURE_LIMIT_BYTES); + private readonly stderrCapture = new BoundedOutputCapture(CLOUD_HYPERVISOR_CAPTURE_LIMIT_BYTES); + + get guestIp(): string | undefined { + return this.networkPlan?.guestIp; + } + + get networkNamespace(): string | undefined { + return this.networkPlan?.namespaceName; + } + + constructor( + private readonly config: CloudHypervisorOptions, + private readonly workDir: string, + private readonly dependencies: CloudHypervisorManagerDependencies = defaultDependencies, + runId?: string, + private readonly networkConfig?: CloudHypervisorManagerNetworkConfig, + private readonly guestConfig?: CloudHypervisorManagerGuestConfig, + ) { + this.paths = createCloudHypervisorRunPaths(config.cloudHypervisorBinary, runId); + } + + async start(): Promise { + if (!this.networkConfig) { + throw new Error( + 'Cloud Hypervisor network configuration is required; refusing to launch an unfiltered microVM', + ); + } + + let startupError: unknown; + try { + const artifacts = await this.dependencies.preflight(this.config); + const identity = this.guestConfig?.identity ?? this.dependencies.resolveIdentity(); + const networkPlan = createMicrovmNetworkPlan(this.paths.runId, { + ...this.networkConfig, + tapOwnerUid: identity.uid, + tapOwnerGid: identity.gid, + }); + this.networkPlan = networkPlan; + this.network = this.dependencies.createNetwork(networkPlan, artifacts.tools); + await this.network.setup(); + let rootfsSource = artifacts.rootfsPath; + let workspaceSource: string | undefined; + if (this.guestConfig) { + this.workspace = this.dependencies.createWorkspaceImage({ + runId: this.paths.runId, + workDir: this.workDir, + workspacePath: this.guestConfig.workspacePath, + homePath: this.guestConfig.homePath, + baseRootfsPath: artifacts.rootfsPath, + supervisorBinaryPath: this.guestConfig.supervisorBinaryPath, + supervisorSha256: this.guestConfig.supervisorSha256, + ...(this.guestConfig.maxWorkspaceImageBytes === undefined + ? {} + : { maxImageBytes: this.guestConfig.maxWorkspaceImageBytes }), + uid: identity.uid, + gid: identity.gid, + }, artifacts.tools); + const preparation = await this.workspace.prepare(); + rootfsSource = preparation.rootfsImagePath; + workspaceSource = preparation.workspaceImagePath; + } + + await this.prepareRunDirectory(identity); + + this.cgroup = this.dependencies.createCgroup( + this.paths.cgroupPath, + { memoryMib: this.config.memoryMib, vcpuCount: this.config.vcpuCount }, + ); + await this.cgroup.setup(); + + await this.stageArtifact(artifacts.kernelPath, this.paths.kernelPath, 0o400, identity); + await this.stageArtifact(rootfsSource, this.paths.rootfsPath, 0o600, identity); + if (workspaceSource) { + await this.stageArtifact(workspaceSource, this.paths.workspacePath, 0o600, identity); + } + await this.stageDiagnosticFile(this.paths.logPath, identity); + await this.stageDiagnosticFile(this.paths.serialLogPath, identity); + + const launchCommand = buildCloudHypervisorLaunchCommand({ + tools: { ip: artifacts.tools.ip, setpriv: artifacts.tools.setpriv }, + namespaceName: networkPlan.namespaceName, + identity, + kvmGid: artifacts.kvmGid, + cloudHypervisorBinary: this.config.cloudHypervisorBinary, + apiSocketPath: this.paths.apiSocketPath, + logFilePath: this.paths.logPath, + }); + this.process = this.dependencies.launch( + launchCommand.command, + [...launchCommand.args], + { + reject: false, + stdio: ['ignore', 'pipe', 'pipe'], + // Explicit minimal environment: the launched process must never + // inherit AWF's host environment (provider/GitHub credentials + // the guest environment deliberately excludes). Cloud Hypervisor + // directly processes untrusted guest/device input, so a VMM + // compromise reading `process.env` would bypass the API-proxy + // credential isolation boundary entirely. `extendEnv: false` + // stops execa from merging this back with `process.env`. + extendEnv: false, + env: buildLauncherEnvironment(), + }, + ); + this.process.stdout?.on('data', (chunk: Buffer | string) => { + this.stdoutCapture.append(chunk); + }); + this.process.stderr?.on('data', (chunk: Buffer | string) => { + this.stderrCapture.append(chunk); + }); + if (this.process.pid !== undefined) { + await this.cgroup.assign(this.process.pid); + } + + await this.waitForApiSocket(); + this.client = this.dependencies.createClient( + this.paths.apiSocketPath, + this.config.apiTimeoutMs, + ); + await this.client.ping(); + await this.client.vmCreate(this.buildVmConfig(networkPlan)); + return this.client; + } catch (error) { + startupError = error; + } + + try { + await this.stop(); + } catch (cleanupError) { + throw new Error( + `Cloud Hypervisor startup failed: ${formatError(startupError)}; ` + + `partial-start cleanup also failed: ${formatError(cleanupError)}`, + ); + } + throw startupError; + } + + private buildVmConfig(networkPlan: MicrovmNetworkPlan) { + const landlockRules = computeCloudHypervisorLandlockRules({ + kernelPath: this.paths.kernelPath, + rootfsPath: this.paths.rootfsPath, + workspacePath: this.guestConfig ? this.paths.workspacePath : undefined, + runDirectory: this.paths.runDirectory, + apiSocketPath: this.paths.apiSocketPath, + vsockSocketPath: this.paths.vsockSocketPath, + }); + return { + cpus: { + boot_vcpus: this.config.vcpuCount, + max_vcpus: this.config.vcpuCount, + }, + memory: { + size: this.config.memoryMib * 1024 * 1024, + }, + payload: { + kernel: this.paths.kernelPath, + ...(this.guestConfig + ? { cmdline: buildSupervisorBootArgs(networkPlan, this.guestConfig) } + : {}), + }, + disks: [ + { id: 'rootfs', path: this.paths.rootfsPath, readonly: false }, + ...(this.guestConfig + ? [{ id: 'workspace', path: this.paths.workspacePath, readonly: false }] + : []), + ], + net: [{ + id: 'net0', + tap: networkPlan.networkInterface.host_dev_name, + mac: networkPlan.networkInterface.guest_mac ?? '', + }], + rng: { src: '/dev/urandom' }, + serial: { mode: 'File' as const, file: this.paths.serialLogPath }, + console: { mode: 'Off' as const }, + ...(this.guestConfig + ? { vsock: { cid: CLOUD_HYPERVISOR_GUEST_CID, socket: this.paths.vsockSocketPath } } + : {}), + watchdog: false, + landlock_enable: true, + landlock_rules: landlockRules, + }; + } + + async startInstance(): Promise { + if (!this.client) throw new Error('Cloud Hypervisor API is not configured'); + await this.client.vmBoot(); + this.instanceStarted = true; + if (this.guestConfig) { + this.guestClient = this.dependencies.createVsockClient( + this.paths.vsockSocketPath, + this.guestConfig.vsockPort ?? CLOUD_HYPERVISOR_GUEST_VSOCK_PORT, + this.config.apiTimeoutMs, + ); + await this.guestClient.connect(); + } + } + + async execute( + request: GuestExecutionRequest, + ): Promise { + if (!this.guestClient) { + throw new Error('Cloud Hypervisor guest supervisor is not ready'); + } + return this.guestClient.execute(request); + } + + cancel(reason = 'host cancellation', requestId?: string): Promise { + if (!this.guestClient) { + return Promise.reject(new Error('Cloud Hypervisor guest supervisor is not ready')); + } + return this.guestClient.cancel(reason, requestId); + } + + writeStdin(data: Buffer, requestId?: string): Promise { + if (!this.guestClient) { + return Promise.reject(new Error('Cloud Hypervisor guest supervisor is not ready')); + } + return this.guestClient.writeStdin(data, requestId); + } + + endStdin(requestId?: string): Promise { + if (!this.guestClient) { + return Promise.reject(new Error('Cloud Hypervisor guest supervisor is not ready')); + } + return this.guestClient.endStdin(requestId); + } + + resize(columns: number, rows: number, requestId?: string): Promise { + if (!this.guestClient) { + return Promise.reject(new Error('Cloud Hypervisor guest supervisor is not ready')); + } + return this.guestClient.resize(columns, rows, requestId); + } + + async stop(options: { preserve?: boolean } = {}): Promise { + const errors: unknown[] = []; + const instanceWasStarted = this.instanceStarted; + let guestShutdownAcknowledged = false; + if (this.guestClient) { + try { + await this.guestClient.shutdown(); + guestShutdownAcknowledged = true; + } catch (error) { + if ( + !(error instanceof Error) || + error.message !== 'Cannot shut down guest while a request is running' + ) { + errors.push(error); + } + this.guestClient.destroy(); + } + } + this.guestClient = undefined; + + if (this.client && instanceWasStarted && guestShutdownAcknowledged) { + try { + await this.client.vmShutdown(); + } catch { + // The process-level termination below remains authoritative; a + // failed graceful vm.shutdown just means we fall through to SIGTERM. + } + } + if (this.client) { + try { + await this.client.vmmShutdown(); + } catch { + // Same as above: SIGTERM/SIGKILL below is authoritative. + } + } + + let terminationConfirmed = !this.process || + this.process.exitCode !== null || + this.process.signalCode !== null; + if ( + this.process && + this.process.exitCode === null && + this.process.signalCode === null + ) { + const child = this.process; + try { + terminationConfirmed = await this.waitForProcessExit( + child, + CLOUD_HYPERVISOR_GUEST_SHUTDOWN_GRACE_MS, + ); + if (!child.killed) { + if (child.exitCode === null && child.signalCode === null) { + child.kill('SIGTERM', { forceKillAfterTimeout: 2_000 }); + } + } + if (!terminationConfirmed) { + await child; + if (child.exitCode === null && child.signalCode === null) { + throw new Error('Cloud Hypervisor process termination was not confirmed'); + } + } + terminationConfirmed = true; + } catch (error) { + terminationConfirmed = child.exitCode !== null || child.signalCode !== null; + errors.push(error); + } + } + if (!terminationConfirmed && this.process) { + if (errors.length === 0) { + errors.push(new Error('Cloud Hypervisor process termination was not confirmed')); + } + throw new Error( + `Cloud Hypervisor cleanup stopped before workspace/network removal: ` + + `${errors.map(formatError).join('; ')}`, + ); + } + this.process = undefined; + this.client = undefined; + + if (this.workspace && instanceWasStarted) { + try { + await this.workspace.extractAfterStop(this.paths.workspacePath); + } catch (error) { + errors.push(error); + } + } + this.instanceStarted = false; + + if (options.preserve) { + try { + await this.cgroup?.cleanup(); + } catch (error) { + errors.push(error); + } + this.cgroup = undefined; + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new Error( + `Cloud Hypervisor preservation failed: ${errors.map(formatError).join('; ')}`, + ); + } + return; + } + + try { + await this.network?.cleanup(); + this.network = undefined; + this.networkPlan = undefined; + } catch (error) { + errors.push(error); + } + + try { + await this.cgroup?.cleanup(); + } catch (error) { + errors.push(error); + } + this.cgroup = undefined; + + if (!instanceWasStarted || terminationConfirmed) { + try { + await this.dependencies.rm( + path.join( + this.paths.runBaseDir, + path.basename(this.config.cloudHypervisorBinary), + this.paths.runId, + ), + { recursive: true, force: true }, + ); + } catch (error) { + errors.push(error); + } + } + + try { + await this.workspace?.cleanup(!instanceWasStarted); + } catch (error) { + errors.push(error); + } + this.workspace = undefined; + + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new Error( + `Cloud Hypervisor cleanup failed: ${errors.map(formatError).join('; ')}`, + ); + } + } + + private async waitForProcessExit( + child: ExecaChildProcess, + timeoutMs: number, + ): Promise { + const pollIntervalMs = 25; + const attempts = Math.max(1, Math.ceil(timeoutMs / pollIntervalMs)); + for (let attempt = 0; attempt < attempts; attempt += 1) { + if (child.exitCode !== null || child.signalCode !== null) return true; + await this.dependencies.sleep(pollIntervalMs); + } + return child.exitCode !== null || child.signalCode !== null; + } + + async collectDiagnostics(directory: string): Promise { + await this.dependencies.mkdir(directory, { recursive: true, mode: 0o700 }); + let counters: unknown = null; + if (this.client && this.instanceStarted) { + try { + counters = await this.client.vmCounters(); + } catch { + counters = null; + } + } + const writeBounded = async (fileName: string, contents: Buffer): Promise => { + const destination = path.join(directory, fileName); + await this.dependencies.writeFile(destination, contents, { mode: 0o600 }); + }; + await writeBounded('launcher-stdout.log', this.stdoutCapture.contents()); + await writeBounded('launcher-stderr.log', this.stderrCapture.contents()); + await this.copyBoundedDiagnostic( + this.paths.logPath, + path.join(directory, CLOUD_HYPERVISOR_LOG_NAME), + ); + await this.copyBoundedDiagnostic( + this.paths.serialLogPath, + path.join(directory, CLOUD_HYPERVISOR_SERIAL_LOG_NAME), + ); + await this.dependencies.writeFile( + path.join(directory, 'network-plan.json'), + `${JSON.stringify(this.networkPlan ?? null, null, 2)}\n`, + { mode: 0o600 }, + ); + await this.dependencies.writeFile( + path.join(directory, 'counters.json'), + `${JSON.stringify(counters, null, 2)}\n`, + { mode: 0o600 }, + ); + await this.dependencies.writeFile( + path.join(directory, 'runtime.json'), + `${JSON.stringify({ + runtime: 'cloud-hypervisor', + version: CLOUD_HYPERVISOR_RELEASE_VERSION, + runId: this.paths.runId, + vcpuCount: this.config.vcpuCount, + memoryMib: this.config.memoryMib, + instanceStarted: this.instanceStarted, + }, null, 2)}\n`, + { mode: 0o600 }, + ); + } + + private async waitForApiSocket(): Promise { + const deadline = Date.now() + this.config.apiTimeoutMs; + while (Date.now() < deadline) { + if (this.process && (this.process.exitCode != null || this.process.signalCode != null)) { + throw new Error( + `Cloud Hypervisor exited before API readiness with code ${this.process.exitCode ?? 'null'} ` + + `and signal ${this.process.signalCode ?? 'null'}`, + ); + } + try { + await this.dependencies.access(this.paths.apiSocketPath); + return; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code !== 'ENOENT') throw error; + } + await this.dependencies.sleep(25); + } + throw new Error( + `Cloud Hypervisor API socket was not ready after ${this.config.apiTimeoutMs}ms: ` + + this.paths.apiSocketPath, + ); + } + + /** + * Creates the private run-directory chain with real traversal + * permissions for the non-root target identity: the two ancestor + * levels (`CLOUD_HYPERVISOR_RUN_ROOT` and the per-binary directory + * beneath it) are `0711` root-owned (executable/traversable by any uid, + * but not listable), and only the per-run leaf directory is chowned to + * the target identity with `0700` (so only that identity, or root, can + * actually read its contents). See the `CLOUD_HYPERVISOR_RUN_ROOT` + * comment above for why this can't simply live under `workDir`. + */ + private async prepareRunDirectory(identity: { uid: number; gid: number }): Promise { + const binaryDir = path.dirname(this.paths.runDirectory); + await this.dependencies.mkdir(this.paths.runBaseDir, { recursive: true, mode: 0o711 }); + await this.dependencies.chmod(this.paths.runBaseDir, 0o711); + await this.dependencies.mkdir(binaryDir, { recursive: true, mode: 0o711 }); + await this.dependencies.chmod(binaryDir, 0o711); + await this.dependencies.mkdir(this.paths.runDirectory, { recursive: true, mode: 0o700 }); + await this.dependencies.chown(this.paths.runDirectory, identity.uid, identity.gid); + } + + private async stageArtifact( + source: string, + destination: string, + mode: number, + identity: { uid: number; gid: number }, + ): Promise { + await this.dependencies.copyFile(source, destination, constants.COPYFILE_EXCL); + await this.dependencies.chown(destination, identity.uid, identity.gid); + await this.dependencies.chmod(destination, mode); + } + + private async stageDiagnosticFile( + destination: string, + identity: { uid: number; gid: number }, + ): Promise { + await this.dependencies.writeFile(destination, '', { flag: 'wx', mode: 0o600 }); + await this.dependencies.chown(destination, identity.uid, identity.gid); + } + + private async copyBoundedDiagnostic(source: string, destination: string): Promise { + try { + const bounded = await this.dependencies.readFileTail(source, CLOUD_HYPERVISOR_CAPTURE_LIMIT_BYTES); + await this.dependencies.writeFile(destination, bounded, { mode: 0o600 }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + } + } +} + +export function buildSupervisorBootArgs( + networkPlan: MicrovmNetworkPlan, + guestConfig: CloudHypervisorManagerGuestConfig, +): string { + const port = guestConfig.vsockPort ?? CLOUD_HYPERVISOR_GUEST_VSOCK_PORT; + if (!Number.isInteger(port) || port < 1 || port > 65_535) { + throw new Error(`Cloud Hypervisor guest vsock port must be in 1-65535: ${port}`); + } + return [ + 'console=ttyS0', + 'reboot=k', + 'panic=1', + 'root=/dev/vda', + 'rootfstype=ext4', + 'rootflags=data=ordered', + 'rw', + // Cloud Hypervisor requires PCI (no `pci=off` MMIO-only mode like + // Firecracker); pin legacy `ethN` interface naming so the guest's + // single virtio-pci NIC has a deterministic name across boots. + 'net.ifnames=0', + 'biosdevname=0', + 'init=/sbin/awf-supervisor', + 'awf.workspace-device=/dev/vdb', + 'awf.workspace-mount=/workspace', + `awf.vsock-port=${port}`, + `awf.guest-ip=${networkPlan.guestIp}`, + `awf.guest-prefix=${networkPlan.guestPrefixLength}`, + `awf.guest-gateway=${networkPlan.guestGatewayIp}`, + 'awf.guest-interface=eth0', + ].join(' '); +} + +function formatError(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +/** + * Explicit, minimal environment for the launched `ip netns exec ... setpriv + * ... cloud-hypervisor` process. Deliberately does **not** include + * `process.env` — Cloud Hypervisor directly parses untrusted guest/device + * input, so a VMM compromise reading its own inherited environment could + * read provider/GitHub credentials and bypass the API-proxy credential + * isolation boundary. Callers must also pass `extendEnv: false` to execa; + * otherwise execa merges this object back into `process.env`. + */ +function buildLauncherEnvironment(): NodeJS.ProcessEnv { + return { + PATH: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', + }; +} + +class BoundedOutputCapture { + private buffer = Buffer.alloc(0); + + constructor(private readonly maximumBytes: number) {} + + append(chunk: Buffer | string): void { + const next = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); + this.buffer = Buffer.concat([this.buffer, next]); + if (this.buffer.length > this.maximumBytes) { + this.buffer = this.buffer.subarray(this.buffer.length - this.maximumBytes); + } + } + + contents(): Buffer { + return this.buffer; + } +} diff --git a/src/cloud-hypervisor/preflight.test.ts b/src/cloud-hypervisor/preflight.test.ts index 5714afc59..28e171a53 100644 --- a/src/cloud-hypervisor/preflight.test.ts +++ b/src/cloud-hypervisor/preflight.test.ts @@ -51,6 +51,7 @@ function dependencies( assertToolAvailable: jest.fn(async (tool: string) => `/usr/bin/${tool}`), assertHostPolicy: jest.fn().mockResolvedValue(2), assertDockerInfrastructure: jest.fn().mockResolvedValue(undefined), + resolveKvmGid: jest.fn().mockResolvedValue(978), ...overrides, }; } @@ -129,6 +130,10 @@ describe('Cloud Hypervisor preflight (foundation only)', () => { ['compose', 'version'], expect.objectContaining({ timeout: 10_000, reject: false }), ); + + const statSpy = jest.spyOn(fs, 'stat').mockResolvedValue({ gid: 978 } as never); + await expect(defaults.resolveKvmGid()).resolves.toBe(978); + expect(statSpy).toHaveBeenCalledWith('/dev/kvm'); }); it('reports host policy and Docker probe failures', async () => { @@ -145,6 +150,19 @@ describe('Cloud Hypervisor preflight (foundation only)', () => { .rejects.toThrow(/\/usr\/bin\/docker info failed.*daemon unavailable/); }); + it('rejects cgroup v1-only hosts explicitly instead of falling back', async () => { + const defaults = cloudHypervisorPreflightTestHelpers.defaultDependencies; + jest.spyOn(process, 'getuid').mockReturnValue(0); + jest.spyOn(fs, 'access').mockImplementation(async (target) => { + if (target === '/sys/fs/cgroup/cgroup.controllers') { + throw new Error('no cgroup v2'); + } + return undefined; + }); + await expect(defaults.assertHostPolicy()) + .rejects.toThrow(/requires the cgroup v2 unified hierarchy.*no cgroup v2/); + }); + it('parses Cloud Hypervisor release output', () => { expect(parseCloudHypervisorVersion('cloud-hypervisor v53.0')).toBe('53.0'); expect(parseCloudHypervisorVersion('53.0')).toBe('53.0'); @@ -164,12 +182,14 @@ describe('Cloud Hypervisor preflight (foundation only)', () => { expect(result.version).toBe('53.0'); expect(result.cgroupVersion).toBe(2); + expect(result.kvmGid).toBe(978); + expect(deps.resolveKvmGid).toHaveBeenCalledTimes(1); expect(deps.access).toHaveBeenCalledWith( '/dev/kvm', constants.R_OK | constants.W_OK, ); expect(deps.sha256).toHaveBeenCalledTimes(4); - expect(deps.assertToolAvailable).toHaveBeenCalledTimes(8); + expect(deps.assertToolAvailable).toHaveBeenCalledTimes(9); expect(deps.assertDockerInfrastructure).toHaveBeenCalledWith('/usr/bin/docker'); expect(result.tools).toEqual({ ip: '/usr/bin/ip', @@ -179,6 +199,7 @@ describe('Cloud Hypervisor preflight (foundation only)', () => { debugfs: '/usr/bin/debugfs', e2fsck: '/usr/bin/e2fsck', rsync: '/usr/bin/rsync', + setpriv: '/usr/bin/setpriv', }); }); diff --git a/src/cloud-hypervisor/preflight.ts b/src/cloud-hypervisor/preflight.ts index 5bfea4daf..0da043193 100644 --- a/src/cloud-hypervisor/preflight.ts +++ b/src/cloud-hypervisor/preflight.ts @@ -9,16 +9,18 @@ import { /** * Fail-closed host and artifact validation for the Cloud Hypervisor v53.0 - * foundation. This module intentionally mirrors + * runtime. This module intentionally mirrors * `src/firecracker/preflight.ts`'s trust-check patterns (absolute paths, * root/operator-owned non-writable regular files, trusted ancestor * directories, digest pinning, PATH-resolved but ownership-verified host * tools) so both VMM backends share the same fail-closed posture. * - * There is no lifecycle backend yet — this is exercised only by unit tests - * and by a future layer that wires up Cloud Hypervisor workload execution. * Cloud Hypervisor has no jailer-equivalent process, so there is no paired * binary version cross-check the way Firecracker cross-checks jailer. + * Instead `src/cloud-hypervisor/launcher.ts` builds an equivalent + * network-namespace-join + privilege-drop + Landlock/seccomp launch using + * the `setpriv` tool resolved here, and `src/cloud-hypervisor/manager.ts` + * stages artifacts into a private, non-world-readable run directory. */ export interface CloudHypervisorPreflightDependencies { @@ -35,8 +37,11 @@ export interface CloudHypervisorPreflightDependencies { runVersion(binaryPath: string): Promise; sha256(filePath: string): Promise; assertToolAvailable(tool: string): Promise; - assertHostPolicy(): Promise<1 | 2>; + assertHostPolicy(): Promise<2>; assertDockerInfrastructure(dockerBinaryPath: string): Promise; + /** Resolves the group ID that owns `/dev/kvm`, so the launcher can retain + * exactly that supplementary group instead of the full operator group set. */ + resolveKvmGid(): Promise; } export type CloudHypervisorHostToolPaths = Readonly<{ @@ -47,9 +52,16 @@ export type CloudHypervisorHostToolPaths = Readonly<{ debugfs: string; e2fsck: string; rsync: string; + /** + * util-linux `setpriv`, used by the launcher to drop to the non-root + * operator uid/gid and clear capabilities/groups after joining the + * per-run network namespace (there is no jailer-equivalent process to do + * this for Cloud Hypervisor). See `src/cloud-hypervisor/launcher.ts`. + */ + setpriv: string; }>; const CLOUD_HYPERVISOR_HOST_TOOLS: (keyof CloudHypervisorHostToolPaths)[] = [ - 'ip', 'nft', 'sysctl', 'mke2fs', 'debugfs', 'e2fsck', 'rsync', + 'ip', 'nft', 'sysctl', 'mke2fs', 'debugfs', 'e2fsck', 'rsync', 'setpriv', ]; const defaultDependencies: CloudHypervisorPreflightDependencies = { @@ -105,16 +117,21 @@ const defaultDependencies: CloudHypervisorPreflightDependencies = { try { await fs.access('/sys/fs/cgroup/cgroup.controllers', constants.R_OK); return 2; - } catch { - try { - await fs.access('/sys/fs/cgroup', constants.R_OK | constants.W_OK); - return 1; - } catch (error) { - throw new Error( - 'Cloud Hypervisor requires a writable cgroup v1 hierarchy or cgroup v2 controllers: ' + - `${error instanceof Error ? error.message : String(error)}`, - ); - } + } catch (error) { + // Cloud Hypervisor's launcher manages an explicit memory/CPU/PID + // cgroup for the launched process (see `src/cloud-hypervisor/launcher.ts` + // `CloudHypervisorCgroup`), which requires the cgroup v2 unified + // hierarchy's `cgroup.subtree_control` delegation model. A cgroup v1 + // fallback would need separate per-controller mount points + // (`memory`, `cpu,cpuacct`, `pids`) that this launcher does not + // manage, so it is rejected explicitly rather than silently + // constructing a broken cgroup. GitHub-hosted Ubuntu runners (the + // only supported host) always run cgroup v2. + throw new Error( + 'Cloud Hypervisor requires the cgroup v2 unified hierarchy ' + + '(/sys/fs/cgroup/cgroup.controllers); cgroup v1-only hosts are not supported: ' + + `${error instanceof Error ? error.message : String(error)}`, + ); } }, assertDockerInfrastructure: async (dockerBinaryPath) => { @@ -131,6 +148,10 @@ const defaultDependencies: CloudHypervisorPreflightDependencies = { } } }, + resolveKvmGid: async () => { + const stat = await fs.stat('/dev/kvm'); + return stat.gid; + }, }; /** @internal Exposed only for focused host-probe tests. */ @@ -143,7 +164,9 @@ export interface CloudHypervisorPreflightResult { rootfsPath: string; supervisorPath: string; tools: CloudHypervisorHostToolPaths; - cgroupVersion: 1 | 2; + cgroupVersion: 2; + /** Group ID that owns `/dev/kvm`, retained as the launcher's sole supplementary group. */ + kvmGid: number; } async function assertTrustedHostTool(label: string, filePath: string): Promise { @@ -321,6 +344,7 @@ export async function runCloudHypervisorPreflight( `${error instanceof Error ? error.message : String(error)}`, ); } + const kvmGid = await dependencies.resolveKvmGid(); const cgroupVersion = await dependencies.assertHostPolicy(); let dockerBinaryPath: string; try { @@ -411,5 +435,6 @@ export async function runCloudHypervisorPreflight( supervisorPath: config.supervisorPath, tools, cgroupVersion, + kvmGid, }; } diff --git a/src/cloud-hypervisor/runtime-validation.test.ts b/src/cloud-hypervisor/runtime-validation.test.ts index 668cf4fb1..f67a2813a 100644 --- a/src/cloud-hypervisor/runtime-validation.test.ts +++ b/src/cloud-hypervisor/runtime-validation.test.ts @@ -1,85 +1,139 @@ -import { assertCloudHypervisorNotYetAvailable, assertCloudHypervisorSelection, requireCloudHypervisorConfig } from './runtime-validation'; import type { WrapperConfig } from '../types'; +import * as hostEligibility from './host-eligibility'; +import { + assertCloudHypervisorPreSecurityCompatibility, + assertCloudHypervisorRuntimeCompatibility, + assertCloudHypervisorSelection, + requireCloudHypervisorConfig, +} from './runtime-validation'; -function baseConfig(overrides: Partial = {}): WrapperConfig { +const digest = 'a'.repeat(64); + +function config(overrides: Partial = {}): WrapperConfig { return { - allowedDomains: [], - agentCommand: 'echo test', - logLevel: 'info', - keepContainers: false, - workDir: '/tmp/awf-test', - buildLocal: false, - skipPull: false, - imageRegistry: 'registry', - imageTag: 'latest', - envAll: false, - dnsServers: [], - enableHostAccess: false, - sslBump: false, + containerRuntime: 'cloud-hypervisor', + networkIsolation: true, + legacySecurity: false, + enableApiProxy: true, enableDind: false, - enableDlp: false, - legacySecurity: undefined, + enableHostAccess: false, + tty: false, + cloudHypervisor: { + previewEnabled: true, + cloudHypervisorBinary: '/opt/cloud-hypervisor', + kernelPath: '/opt/kernel', + rootfsPath: '/opt/rootfs', + supervisorPath: '/opt/supervisor', + vcpuCount: 2, + memoryMib: 512, + apiTimeoutMs: 5000, + sha256: { + cloudHypervisor: digest, + kernel: digest, + rootfs: digest, + supervisor: digest, + }, + }, ...overrides, } as WrapperConfig; } -describe('Cloud Hypervisor runtime-selection guard (foundation only)', () => { - it('rejects --container-runtime cloud-hypervisor with an explicit not-yet-available error', () => { - expect(() => assertCloudHypervisorNotYetAvailable( - baseConfig({ containerRuntime: 'cloud-hypervisor' }), - )).toThrow(/not yet an available --container-runtime/); +describe('Cloud Hypervisor runtime validation', () => { + let eligibilitySpy: jest.SpyInstance; + + beforeEach(() => { + eligibilitySpy = jest.spyOn(hostEligibility, 'assertGithubHostedRunnerEligibility') + .mockImplementation(() => undefined); }); - it('allows any other runtime, including undefined', () => { - expect(() => assertCloudHypervisorNotYetAvailable(baseConfig())).not.toThrow(); - expect(() => assertCloudHypervisorNotYetAvailable( - baseConfig({ containerRuntime: 'gvisor' }), - )).not.toThrow(); + afterEach(() => { + eligibilitySpy.mockRestore(); }); - it('requires --container-runtime cloud-hypervisor when cloudHypervisor options are set', () => { - const config = baseConfig({ + it('accepts only a complete explicitly selected preview on an eligible host', () => { + const valid = config(); + expect(() => assertCloudHypervisorSelection(valid)).not.toThrow(); + expect(() => assertCloudHypervisorRuntimeCompatibility(valid)).not.toThrow(); + expect(eligibilitySpy).toHaveBeenCalled(); + expect(requireCloudHypervisorConfig(valid)).toBe(valid.cloudHypervisor); + + expect(() => assertCloudHypervisorSelection(config({ containerRuntime: 'gvisor', - cloudHypervisor: { - previewEnabled: true, - cloudHypervisorBinary: '/opt/cloud-hypervisor', - vcpuCount: 2, - memoryMib: 512, - apiTimeoutMs: 5000, - }, + }))).toThrow(/require --container-runtime cloud-hypervisor/); + expect(() => requireCloudHypervisorConfig(config({ + containerRuntime: 'gvisor', + }))).toThrow(/resolved without Cloud Hypervisor runtime configuration/); + }); + + it('rejects an ineligible host even with otherwise-complete configuration', () => { + eligibilitySpy.mockImplementation(() => { + throw new Error('Cloud Hypervisor is supported only inside GitHub Actions runs'); }); - expect(() => assertCloudHypervisorSelection(config)).toThrow( - /Cloud Hypervisor options require --container-runtime cloud-hypervisor/, - ); + expect(() => assertCloudHypervisorRuntimeCompatibility(config())) + .toThrow(/supported only inside GitHub Actions runs/); }); - it('accepts cloudHypervisor options when no other runtime is specified', () => { - const config = baseConfig({ + it.each([ + [{ cloudHypervisor: { ...config().cloudHypervisor!, previewEnabled: false } }, /explicit --cloud-hypervisor-preview/], + [{ networkIsolation: false }, /strict --network-isolation/], + [{ legacySecurity: true }, /strict --network-isolation/], + [{ enableApiProxy: false }, /API proxy credential isolation/], + [{ cloudHypervisor: { - previewEnabled: true, - cloudHypervisorBinary: '/opt/cloud-hypervisor', - vcpuCount: 2, - memoryMib: 512, - apiTimeoutMs: 5000, + ...config().cloudHypervisor!, + supervisorPath: undefined, }, - }); - expect(() => assertCloudHypervisorSelection(config)).not.toThrow(); + }, /explicit kernel, rootfs, and guest supervisor/], + [{ + cloudHypervisor: { + ...config().cloudHypervisor!, + sha256: { ...config().cloudHypervisor!.sha256, supervisor: undefined }, + }, + }, /requires SHA-256 digests/], + ] as const)('rejects incomplete runtime configuration %#', (overrides, error) => { + expect(() => assertCloudHypervisorRuntimeCompatibility( + config(overrides as Partial), + )).toThrow(error); }); - it('returns the Cloud Hypervisor config when present', () => { - const cloudHypervisor = { - previewEnabled: true, - cloudHypervisorBinary: '/opt/cloud-hypervisor', - vcpuCount: 2, - memoryMib: 512, - apiTimeoutMs: 5000, - }; - expect(requireCloudHypervisorConfig(baseConfig({ cloudHypervisor }))).toBe(cloudHypervisor); + it.each([ + [{ networkIsolation: false }, /cannot disable --network-isolation/], + [{ enableDind: true }, /Docker-in-Docker/], + [{ dockerHostPathPrefix: '/host' }, /split filesystems/], + [{ runnerTopology: 'arc-dind' }, /split filesystems/], + [{ enableHostAccess: true }, /host access/], + [{ allowHostPorts: ['8080'] }, /host access/], + [{ allowHostServicePorts: ['5432'] }, /host access/], + [{ volumeMounts: ['/tmp:/tmp'] }, /additional host volume mounts/], + [{ topologyAttach: ['gateway'] }, /MCP gateway path/], + [{ difcProxyHost: 'proxy:443' }, /MCP gateway path/], + [{ enclaves: { enabled: true } }, /MCP gateway path/], + [{ dnsOverHttps: 'https://dns.example/dns-query' }, /DNS-over-HTTPS/], + [{ tty: true }, /does not support --tty/], + [{ awfDockerHost: 'tcp://localhost:2375' }, /local Unix-socket Docker daemon/], + ] as const)('rejects unsupported preview policy %#', (overrides, error) => { + expect(() => assertCloudHypervisorPreSecurityCompatibility( + config(overrides as Partial), + )).toThrow(error); + }); + + it('accepts a local Unix Docker socket', () => { + expect(() => assertCloudHypervisorPreSecurityCompatibility(config({ + awfDockerHost: 'unix:///var/run/docker.sock', + }))).not.toThrow(); + }); + + it('rejects Cloud Hypervisor options paired with another --container-runtime', () => { + const invalid = config({ containerRuntime: 'gvisor' }); + expect(() => assertCloudHypervisorSelection(invalid)).toThrow( + /Cloud Hypervisor options require --container-runtime cloud-hypervisor/, + ); }); - it('throws when Cloud Hypervisor config is missing', () => { - expect(() => requireCloudHypervisorConfig(baseConfig())).toThrow( - /resolved without Cloud Hypervisor runtime configuration/, + it('rejects cloudHypervisor options with no --container-runtime selected at all', () => { + const invalid = config({ containerRuntime: undefined }); + expect(() => assertCloudHypervisorSelection(invalid)).toThrow( + /Cloud Hypervisor options require --container-runtime cloud-hypervisor/, ); }); }); diff --git a/src/cloud-hypervisor/runtime-validation.ts b/src/cloud-hypervisor/runtime-validation.ts index da93f4e7a..db3938bf1 100644 --- a/src/cloud-hypervisor/runtime-validation.ts +++ b/src/cloud-hypervisor/runtime-validation.ts @@ -1,41 +1,104 @@ +import { getLocalDockerEnv } from '../docker-host'; import type { CloudHypervisorOptions, WrapperConfig } from '../types'; +import { assertGithubHostedRunnerEligibility } from './host-eligibility'; /** - * Cloud Hypervisor is a foundation-only runtime in this release: its config - * surface, preflight/artifact validation, and guest artifact pipeline exist - * so a later layer can add a lifecycle backend, but no backend is registered - * yet (see `src/container-runtime.ts` and - * `src/external-runtime-backend-resolver.ts`). - * - * This guard produces a clear, actionable error instead of letting - * `--container-runtime cloud-hypervisor` fall through to the generic - * unknown-runtime passthrough (which would surface as an opaque Docker - * runtime error). + * Explicit, fail-closed compatibility guards for the Cloud Hypervisor + * v53.0 microVM runtime. This module mirrors + * `src/firecracker/runtime-validation.ts` closely — same security-mode, + * topology, and Docker-host requirements — with two Cloud + * Hypervisor-specific additions: no jailer digest is required (Cloud + * Hypervisor has no jailer-equivalent process) and host eligibility is + * additionally restricted to GitHub-hosted Ubuntu x86_64 KVM runners via + * {@link assertGithubHostedRunnerEligibility} (self-hosted runners are not + * supported, unlike Firecracker's preview). */ -export function assertCloudHypervisorNotYetAvailable(config: WrapperConfig): void { - if (config.containerRuntime === 'cloud-hypervisor') { + +export function assertCloudHypervisorSelection(config: WrapperConfig): void { + if (config.cloudHypervisor && config.containerRuntime !== 'cloud-hypervisor') { throw new Error( - 'Cloud Hypervisor is not yet an available --container-runtime. ' + - 'Its configuration surface is foundation-only in this release: no lifecycle backend is registered.', + 'Cloud Hypervisor options require --container-runtime cloud-hypervisor', ); } } -/** - * Cloud Hypervisor options may only be supplied alongside their own - * (currently non-selectable) runtime name, mirroring - * `assertFirecrackerSelection`'s intent for config-shape consistency. - */ -export function assertCloudHypervisorSelection(config: WrapperConfig): void { - if (config.cloudHypervisor && config.containerRuntime !== undefined && config.containerRuntime !== 'cloud-hypervisor') { +export function assertCloudHypervisorRuntimeCompatibility( + config: WrapperConfig, + cloudHypervisor = requireCloudHypervisorConfig(config), +): void { + if (!cloudHypervisor.previewEnabled) { + throw new Error( + 'Cloud Hypervisor workload execution requires explicit --cloud-hypervisor-preview opt-in', + ); + } + if (!config.networkIsolation || config.legacySecurity) { + throw new Error('Cloud Hypervisor preview requires strict --network-isolation security'); + } + if (!config.enableApiProxy) { + throw new Error('Cloud Hypervisor preview requires API proxy credential isolation'); + } + assertCloudHypervisorPreSecurityCompatibility(config); + assertGithubHostedRunnerEligibility(); + if (!cloudHypervisor.kernelPath || !cloudHypervisor.rootfsPath || !cloudHypervisor.supervisorPath) { + throw new Error( + 'Cloud Hypervisor preview requires explicit kernel, rootfs, and guest supervisor artifacts', + ); + } + const digests = cloudHypervisor.sha256; + if ( + !digests?.cloudHypervisor || + !digests.kernel || + !digests.rootfs || + !digests.supervisor + ) { + throw new Error( + 'Cloud Hypervisor preview requires SHA-256 digests for cloud-hypervisor, kernel, rootfs, and supervisor', + ); + } +} + +export function assertCloudHypervisorPreSecurityCompatibility(config: WrapperConfig): void { + if (config.networkIsolation === false) { + throw new Error('Cloud Hypervisor preview cannot disable --network-isolation'); + } + if ( + config.enableDind || + config.dockerHostPathPrefix || + config.runnerTopology === 'arc-dind' + ) { + throw new Error('Cloud Hypervisor preview does not support Docker-in-Docker or split filesystems'); + } + if (config.enableHostAccess || config.allowHostPorts || config.allowHostServicePorts) { + throw new Error('Cloud Hypervisor preview does not support host access'); + } + if (config.volumeMounts?.length) { + throw new Error('Cloud Hypervisor preview does not support additional host volume mounts'); + } + if ( + config.topologyAttach?.length || + config.difcProxyHost || + config.enclaves?.enabled + ) { + throw new Error( + 'Cloud Hypervisor preview does not yet prove the MCP gateway path; topology peers and enclaves are disabled', + ); + } + if (config.dnsOverHttps) { + throw new Error('Cloud Hypervisor preview does not support DNS-over-HTTPS'); + } + if (config.tty) { + throw new Error('Cloud Hypervisor preview guest supervisor does not support --tty'); + } + const dockerHost = config.awfDockerHost ?? getLocalDockerEnv().DOCKER_HOST; + if (dockerHost && !dockerHost.startsWith('unix://')) { throw new Error( - 'Cloud Hypervisor options require --container-runtime cloud-hypervisor (not yet an available runtime; foundation-only configuration).', + 'Cloud Hypervisor preview requires a local Unix-socket Docker daemon so its bridge is host-visible', ); } } export function requireCloudHypervisorConfig(config: WrapperConfig): CloudHypervisorOptions { - if (!config.cloudHypervisor) { + if (config.containerRuntime !== 'cloud-hypervisor' || !config.cloudHypervisor) { throw new Error('Cloud Hypervisor backend resolved without Cloud Hypervisor runtime configuration'); } return config.cloudHypervisor; diff --git a/src/commands/build-config.ts b/src/commands/build-config.ts index f9b453c30..69f4922a9 100644 --- a/src/commands/build-config.ts +++ b/src/commands/build-config.ts @@ -314,13 +314,13 @@ function parsePositiveIntegerOption( } /** - * Builds the Cloud Hypervisor foundation config (artifacts/digests only). - * - * There is no lifecycle backend for this runtime yet, so `selected` only - * mirrors the Firecracker pattern for config round-trip/test symmetry — - * `--container-runtime cloud-hypervisor` is rejected explicitly elsewhere - * (see `assertCloudHypervisorNotYetAvailable`) before this config could ever - * drive workload execution. + * Builds the Cloud Hypervisor microVM runtime config (artifacts/digests + * plus vcpu/memory/timeout settings). `selected` mirrors the Firecracker + * pattern: `--container-runtime cloud-hypervisor` requires explicit + * `--cloud-hypervisor-preview` opt-in and full artifact/digest + * configuration, enforced by + * `assertCloudHypervisorRuntimeCompatibility` in + * `src/cloud-hypervisor/runtime-validation.ts`. */ function buildCloudHypervisorConfig( options: Record, diff --git a/src/commands/validators/config-assembly.ts b/src/commands/validators/config-assembly.ts index a4faf5245..24a8962fe 100644 --- a/src/commands/validators/config-assembly.ts +++ b/src/commands/validators/config-assembly.ts @@ -15,7 +15,8 @@ import { assertFirecrackerSelection, } from '../../firecracker/runtime-validation'; import { - assertCloudHypervisorNotYetAvailable, + assertCloudHypervisorPreSecurityCompatibility, + assertCloudHypervisorRuntimeCompatibility, assertCloudHypervisorSelection, } from '../../cloud-hypervisor/runtime-validation'; @@ -81,7 +82,6 @@ export function assembleAndValidateConfig( validateInfrastructureOptions(config); try { assertFirecrackerSelection(config); - assertCloudHypervisorNotYetAvailable(config); assertCloudHypervisorSelection(config); } catch (error) { logger.error(`❌ ${error instanceof Error ? error.message : String(error)}`); @@ -95,6 +95,14 @@ export function assembleAndValidateConfig( process.exit(1); } } + if (config.containerRuntime === 'cloud-hypervisor') { + try { + assertCloudHypervisorPreSecurityCompatibility(config); + } catch (error) { + logger.error(`❌ ${error instanceof Error ? error.message : String(error)}`); + process.exit(1); + } + } applySecurityMode(config); if (config.containerRuntime === 'firecracker') { try { @@ -104,6 +112,14 @@ export function assembleAndValidateConfig( process.exit(1); } } + if (config.containerRuntime === 'cloud-hypervisor') { + try { + assertCloudHypervisorRuntimeCompatibility(config); + } catch (error) { + logger.error(`❌ ${error instanceof Error ? error.message : String(error)}`); + process.exit(1); + } + } applyAgentTimeout(options.agentTimeout as string | undefined, config, logger); applyRateLimitConfig(config, options); validateFeatureFlagCompatibility(config); diff --git a/src/container-runtime.test.ts b/src/container-runtime.test.ts index 4961c38f4..8c36326a8 100644 --- a/src/container-runtime.test.ts +++ b/src/container-runtime.test.ts @@ -17,6 +17,10 @@ describe('container-runtime', () => { expect(resolveDockerRuntime('firecracker')).toBeUndefined(); }); + it('returns undefined for Cloud Hypervisor (no OCI runtime)', () => { + expect(resolveDockerRuntime('cloud-hypervisor')).toBeUndefined(); + }); + it('passes through unknown runtime names unchanged', () => { expect(resolveDockerRuntime('kata')).toBe('kata'); expect(resolveDockerRuntime('custom-runtime')).toBe('custom-runtime'); @@ -40,6 +44,10 @@ describe('container-runtime', () => { expect(runtimeNeedsStaticDns('firecracker')).toBe(false); }); + it('returns false for Cloud Hypervisor', () => { + expect(runtimeNeedsStaticDns('cloud-hypervisor')).toBe(false); + }); + it('returns false for unknown runtimes', () => { expect(runtimeNeedsStaticDns('kata')).toBe(false); }); @@ -67,6 +75,10 @@ describe('container-runtime', () => { expect(runtimeUsesIptables('firecracker')).toBe(false); }); + it('returns false for Cloud Hypervisor (no host-agent iptables)', () => { + expect(runtimeUsesIptables('cloud-hypervisor')).toBe(false); + }); + it('returns true for unknown runtimes (share host netns)', () => { expect(runtimeUsesIptables('kata')).toBe(true); }); @@ -97,6 +109,10 @@ describe('container-runtime', () => { expect(runtimeUsesComposeAgent('firecracker')).toBe(false); }); + it('returns false for the Cloud Hypervisor microVM model', () => { + expect(runtimeUsesComposeAgent('cloud-hypervisor')).toBe(false); + }); + it('returns true for unknown runtimes (assumed compose)', () => { expect(runtimeUsesComposeAgent('kata')).toBe(true); expect(runtimeUsesComposeAgent('runsc')).toBe(true); diff --git a/src/container-runtime.ts b/src/container-runtime.ts index 10b74f490..3c19ca619 100644 --- a/src/container-runtime.ts +++ b/src/container-runtime.ts @@ -116,6 +116,12 @@ const RUNTIME_REGISTRY: Readonly> = { needsStaticDns: false, usesIptables: false, }, + 'cloud-hypervisor': { + executionModel: 'microvm', + dockerRuntime: undefined, + needsStaticDns: false, + usesIptables: false, + }, }; /** diff --git a/src/external-runtime-backend-resolver.ts b/src/external-runtime-backend-resolver.ts index 341e984ba..96b9b6f79 100644 --- a/src/external-runtime-backend-resolver.ts +++ b/src/external-runtime-backend-resolver.ts @@ -3,6 +3,7 @@ import { runtimeUsesComposeAgent } from './container-runtime'; import type { ExternalAgentRuntimeBackend } from './external-runtime-backend'; import { createSbxRuntimeBackend } from './sbx-runtime-backend'; import { createFirecrackerRuntimeBackend } from './firecracker-runtime-backend'; +import { createCloudHypervisorRuntimeBackend } from './cloud-hypervisor-runtime-backend'; import type { WrapperConfig } from './types'; interface ExternalRuntimeBackendFactoryContext { @@ -23,6 +24,8 @@ const EXTERNAL_RUNTIME_BACKENDS: ExternalRuntimeBackendRegistry = { createSbxRuntimeBackend(config, startInfrastructure), firecracker: ({ config, startInfrastructure }) => createFirecrackerRuntimeBackend(config, startInfrastructure), + 'cloud-hypervisor': ({ config, startInfrastructure }) => + createCloudHypervisorRuntimeBackend(config, startInfrastructure), }; /** @@ -46,6 +49,11 @@ export function resolveExternalRuntimeBackend( 'Firecracker workload execution requires explicit --firecracker-preview opt-in', ); } + if (runtime === 'cloud-hypervisor' && !config.cloudHypervisor?.previewEnabled) { + throw new Error( + 'Cloud Hypervisor workload execution requires explicit --cloud-hypervisor-preview opt-in', + ); + } const factory = runtime ? registry[runtime] : undefined; if (!factory) { throw new Error(`No external agent runtime backend is registered for "${runtime}"`); diff --git a/src/external-runtime-backend.test.ts b/src/external-runtime-backend.test.ts index bd940000e..750a44e69 100644 --- a/src/external-runtime-backend.test.ts +++ b/src/external-runtime-backend.test.ts @@ -77,6 +77,25 @@ describe('external runtime backend', () => { expect(backend?.runtime).toBe('firecracker'); }); + + it('requires explicit Cloud Hypervisor preview opt-in during resolution', () => { + const config = { + containerRuntime: 'cloud-hypervisor', + cloudHypervisor: { previewEnabled: false }, + } as WrapperConfig; + expect(() => resolveExternalRuntimeBackend(config, startInfrastructure)) + .toThrow(/explicit --cloud-hypervisor-preview/); + expect(startInfrastructure).not.toHaveBeenCalled(); + }); + + it('uses the registered Cloud Hypervisor factory after preview opt-in', () => { + const backend = resolveExternalRuntimeBackend({ + containerRuntime: 'cloud-hypervisor', + cloudHypervisor: { previewEnabled: true }, + } as WrapperConfig, startInfrastructure); + + expect(backend?.runtime).toBe('cloud-hypervisor'); + }); it('adapts start and exec without changing arguments or exit codes', async () => { const backend = createBackend(); const adapted = adaptExternalRuntimeBackend(backend);