feat: isolate Cloud Hypervisor with per-run VMM identities

docs/cloud-hypervisor-foundation.md

@@ -72,8 +72,10 @@ AWF performs these steps for each run:
    policy.
 5. Copy the rootfs, inject the guest supervisor, and stage files in a private
    run directory.
-6. Create a bounded cgroup v2 leaf and launch Cloud Hypervisor as the invoking
-   non-root identity.
+6. Allocate a random-named, dedicated per-run system account, grant its
+   host-assigned uid/gid temporary access only to KVM/TUN, the pre-created TAP,
+   staged files, and required sockets, then launch Cloud Hypervisor under that
+   identity in a bounded cgroup v2 leaf.
 7. After the API responds, verify the launched VMM's trusted host `/proc` and
    cgroup state. AWF fails closed before `vm.create` if the PID identity,
    executable, credentials, capabilities, `no_new_privs`, seccomp worker,
@@ -89,7 +91,8 @@ AWF performs these steps for each run:
 10. Execute the agent command and propagate its exit code. Timeouts return
    `124`.
 11. Sync and unmount guest filesystems, stop the VM and VMM, reap `virtiofsd`,
-    and remove network, cgroup, and run-directory resources.
+    remove network, cgroup, and run-directory resources, revoke device ACLs,
+    and delete the exact per-run account.
 
 Cleanup is idempotent and aggregates errors so one cleanup failure does not
 skip later cleanup steps.
@@ -163,8 +166,11 @@ release or production use.
 AWF launches Cloud Hypervisor through `ip netns exec` and `setpriv` without a
 shell. The process:
 
-- runs as the non-root identity recorded by `SUDO_UID` and `SUDO_GID`;
-- keeps only the KVM supplementary group;
+- runs as a random `awfvmm-<token>` system account allocated for that run,
+  independently of `SUDO_UID` and `SUDO_GID`;
+- has no home, login shell, or supplementary groups;
+- receives temporary uid-specific ACLs for `/dev/kvm` and `/dev/net/tun`;
+- owns only its run directory, staged VMM files and sockets, and TAP;
 - sets `no_new_privs`;
 - has empty inheritable, permitted, effective, bounding, and ambient capability
   sets;
@@ -191,7 +197,15 @@ not copy unbounded `/proc` content.
 
 The private run directory is under
 `/run/awf-cloud-hypervisor/<binary>/<runId>/`. Its per-run leaf is accessible
-only to the selected non-root identity and root.
+only to the dedicated VMM identity and root. Account allocation and deletion
+are serialized by an owner-token lock that validates both PID and process start
+time before reclaiming stale state, preventing PID reuse from stealing a live
+lock. Cleanup validates the exact account uid/gid before deletion and revokes
+only that run's ACL entries, so concurrent sibling runs remain untouched.
+
+The guest command still uses the invoking workspace uid/gid. That guest
+identity is carried separately through the supervisor protocol and is never
+reused as the host VMM identity.
 
 ### virtiofsd confinement
 
@@ -206,8 +220,8 @@ configuration, AWF verifies the live parent and worker through `/proc`:
 - parent and worker UIDs/GIDs match the reviewed root namespace identity;
 - every parent capability set is empty, while the worker effective and
   permitted masks equal the pinned minimal virtiofsd set, its inheritable and
-  ambient sets are empty, and its bounding set contains the capabilities
-  needed during sandbox setup (rendered non-acquirable after `NoNewPrivs`);
+  ambient sets are empty, and its host bounding set remains empty after
+  entering the user namespace;
 - the worker has `NoNewPrivs: 1` and seccomp filter mode `2`;
 - the worker mount, PID, and network namespaces differ from the host;
 - the worker root inode is the inode of the declared export, proving the

scripts/ci/cloud-hypervisor-live-smoke.sh

@@ -665,6 +665,24 @@ done
 proc_uid=$(sudo stat -c %u "/proc/$vmm_pid" 2>/dev/null || echo "")
 [ -n "$proc_uid" ] || fail_security "could not stat /proc/$vmm_pid"
 [ "$proc_uid" != "0" ] || fail_security "Cloud Hypervisor process is running as root"
+operator_uid=$(id -u)
+[ "$proc_uid" != "$operator_uid" ] \
+  || fail_security "Cloud Hypervisor process reused the invoking operator uid $operator_uid"
+vmm_passwd=$(getent passwd "$proc_uid" || true)
+[ -n "$vmm_passwd" ] || fail_security "no passwd entry for Cloud Hypervisor uid $proc_uid"
+vmm_name=$(printf '%s' "$vmm_passwd" | cut -d: -f1)
+vmm_gid=$(printf '%s' "$vmm_passwd" | cut -d: -f4)
+[ "$(printf '%s' "$vmm_passwd" | cut -d: -f6)" = "/nonexistent" ] \
+  || fail_security "Cloud Hypervisor account has an unexpected home directory"
+[ "$(printf '%s' "$vmm_passwd" | cut -d: -f7)" = "/usr/sbin/nologin" ] \
+  || fail_security "Cloud Hypervisor account has an interactive shell"
+printf '%s' "$vmm_name" | grep -Eq '^awfvmm-[0-9a-f]{20}$' \
+  || fail_security "Cloud Hypervisor account name is not a random per-run name: $vmm_name"
+[ "$(id -G "$vmm_name")" = "$vmm_gid" ] \
+  || fail_security "Cloud Hypervisor account inherited supplementary groups"
+getfacl --absolute-names --numeric /dev/kvm 2>/dev/null \
+  | grep -q "^user:$proc_uid:rw-" \
+  || fail_security "Cloud Hypervisor uid lacks its scoped /dev/kvm ACL"
 
 # Every capability set is empty. In particular, a zero CapBnd prevents the
 # non-root VMM from regaining CAP_NET_ADMIN after exec.
@@ -858,6 +876,14 @@ node -e '
 # `ip netns exec`, not a bare `ip link show`.
 sudo ip netns exec "$expected_namespace" ip link show "$expected_tap" >/dev/null 2>&1 \
   || fail_security "expected TAP interface $expected_tap not found in namespace $expected_namespace"
+tap_details=$(sudo ip netns exec "$expected_namespace" \
+  ip -details tuntap show dev "$expected_tap" 2>/dev/null || true)
+tap_line=$(printf '%s\n' "$tap_details" | grep "^${expected_tap}: " || true)
+[ -n "$tap_line" ] || fail_security "expected TAP details were not found: $tap_details"
+printf '%s' "$tap_line" | grep -Eq "(^|[[:space:]])user[[:space:]]+$proc_uid([[:space:]]|$)" \
+  || fail_security "TAP is not owned by the per-run VMM uid $proc_uid: $tap_details"
+printf '%s' "$tap_line" | grep -Eq "(^|[[:space:]])group[[:space:]]+$vmm_gid([[:space:]]|$)" \
+  || fail_security "TAP is not owned by the per-run VMM gid $vmm_gid: $tap_details"
 
 kill -TERM "$sec_pid"
 set +e
@@ -869,5 +895,14 @@ set -e
   exit 1
 }
 assert_no_residue
+if getent passwd "$vmm_name" >/dev/null; then
+  fail_security "per-run Cloud Hypervisor account remains after cancellation: $vmm_name"
+fi
+if getfacl --absolute-names --numeric /dev/kvm 2>/dev/null | grep -q "^user:$proc_uid:"; then
+  fail_security "per-run /dev/kvm ACL remains after cancellation for uid $proc_uid"
+fi
+if getfacl --absolute-names --numeric /dev/net/tun 2>/dev/null | grep -q "^user:$proc_uid:"; then
+  fail_security "per-run /dev/net/tun ACL remains after cancellation for uid $proc_uid"
+fi
 
 echo "Cloud Hypervisor live smoke/security suite passed."

src/cloud-hypervisor-runtime-backend.test.ts

@@ -99,6 +99,10 @@ const preflightResult = {
   cgroupVersion: 2 as const,
   kvmGid: 978,
   tools: {
+    getfacl: '/usr/bin/getfacl',
+    getent: '/usr/bin/getent',
+    groupdel: '/usr/sbin/groupdel',
+    id: '/usr/bin/id',
     ip: '/usr/bin/ip',
     nft: '/usr/sbin/nft',
     sysctl: '/usr/sbin/sysctl',
@@ -110,6 +114,9 @@ const preflightResult = {
     mount: '/usr/bin/mount',
     umount: '/usr/bin/umount',
     setpriv: '/usr/bin/setpriv',
+    setfacl: '/usr/bin/setfacl',
+    useradd: '/usr/sbin/useradd',
+    userdel: '/usr/sbin/userdel',
   },
 };
 
@@ -243,6 +250,8 @@ describe('Cloud Hypervisor runtime backend', () => {
         infrastructure(),
         [{ tag: 'workspace', source: '/workspace', target: '/workspace', mode: 'rw' }],
         { uid: 1000, gid: 1000 },
+        undefined,
+        preflightResult,
       )).toBeDefined();
       expect(createCloudHypervisorRuntimeBackend(config(), startInfrastructure))
         .toEqual(expect.objectContaining({ runtime: 'cloud-hypervisor' }));
@@ -302,6 +311,7 @@ describe('Cloud Hypervisor runtime backend', () => {
             },
           ],
         },
+        preflightResult,
       );
       expect(deps.logger.info).toHaveBeenCalledWith(
         '[cloud-hypervisor] stage=filesystem-write-policy boundary /workspace=ro ' +
@@ -325,7 +335,8 @@ describe('Cloud Hypervisor runtime backend', () => {
       { tag: 'workspace', source: '/workspace-host', target: '/workspace', mode: 'rw' },
     ]);
     expect(call[5]).toBeUndefined();
-    expect(call).toHaveLength(6);
+    expect(call[6]).toBe(preflightResult);
+    expect(call).toHaveLength(7);
     expect(deps.logger.info).not.toHaveBeenCalledWith(
       expect.stringContaining('stage=filesystem-write-policy'),
     );

src/cloud-hypervisor-runtime-backend.ts

@@ -113,7 +113,8 @@ export interface CloudHypervisorRuntimeBackendDependencies {
     infrastructure: MicrovmInfrastructureSnapshot,
     exports: readonly CloudHypervisorDirectoryExport[],
     identity: { uid: number; gid: number },
-    mountEnforcement?: VirtiofsdMountEnforcement,
+    mountEnforcement: VirtiofsdMountEnforcement | undefined,
+    verifiedArtifacts: CloudHypervisorPreflightResult,
   ): CloudHypervisorManagerAdapter;
   resolveExports(mountPolicy: CloudHypervisorOptions['mountPolicy']): Promise<CloudHypervisorDirectoryExport[]>;
   identity(): { uid: number; gid: number };
@@ -133,7 +134,15 @@ function defaultDependencies(
     preflight: runCloudHypervisorPreflight,
     resolveInfrastructure: (enableApiProxy, ipPath, topologyPeerNames) =>
       resolveMicrovmInfrastructure(enableApiProxy, undefined, ipPath, topologyPeerNames),
-    createManager: (config, workDir, infrastructure, exports, identity, mountEnforcement) =>
+    createManager: (
+      config,
+      workDir,
+      infrastructure,
+      exports,
+      identity,
+      mountEnforcement,
+      verifiedArtifacts,
+    ) =>
       new CloudHypervisorManager(
         config,
         workDir,
@@ -156,6 +165,7 @@ function defaultDependencies(
           supervisorSha256: config.sha256!.supervisor!,
           identity,
         },
+        verifiedArtifacts,
       ),
     resolveExports: (mountPolicy) => resolveCloudHypervisorExports(
       process.env,
@@ -316,6 +326,7 @@ class CloudHypervisorRuntimeBackend implements ExternalAgentRuntimeBackend {
           exports,
           this.identity,
           mountEnforcement,
+          this.preflightResult!,
         );
         try {
           stage = 'vmm-configuration';

src/cloud-hypervisor/confinement-verifier.test.ts

@@ -22,14 +22,14 @@ function launchPolicy(): CloudHypervisorLaunchConfinementPolicy {
   };
 }
 
-function status(name: string, seccomp: number, taskId = PID): string {
+function status(name: string, seccomp: number, taskId = PID, groups = '978'): string {
   return [
     `Name:\t${name}`,
     `Pid:\t${taskId}`,
     `Tgid:\t${PID}`,
     'Uid:\t1000\t1000\t1000\t1000',
     'Gid:\t1001\t1001\t1001\t1001',
-    'Groups:\t978',
+    `Groups:\t${groups}`,
     `CapInh:\t${CAPABILITY_MASK}`,
     `CapPrm:\t${CAPABILITY_MASK}`,
     `CapEff:\t${CAPABILITY_MASK}`,
@@ -50,13 +50,14 @@ function dependencies(overrides: {
   executable?: string;
   workerStatus?: string;
   cgroupProcs?: string;
+  groups?: string;
 } = {}): CloudHypervisorConfinementVerifierDependencies {
   const statReads = [...(overrides.statReads ?? [procStat('98765'), procStat('98765')])];
   const files: Record<string, string> = {
-    [`/proc/${PID}/task/${PID}/status`]: status('cloud-hypervis', 0),
+    [`/proc/${PID}/task/${PID}/status`]: status('cloud-hypervis', 0, PID, overrides.groups),
     [`/proc/${PID}/task/${PID + 1}/status`]:
-      overrides.workerStatus ?? status('vmm', 2, PID + 1),
-    [`/proc/${PID}/task/${PID + 2}/status`]: status('http-server', 2, PID + 2),
+      overrides.workerStatus ?? status('vmm', 2, PID + 1, overrides.groups),
+    [`/proc/${PID}/task/${PID + 2}/status`]: status('http-server', 2, PID + 2, overrides.groups),
     [`/proc/${PID}/cgroup`]: '0::/awf-cloud-hypervisor/run-1\n',
     [`${CGROUP}/cgroup.procs`]: overrides.cgroupProcs ?? `${PID}\n`,
     [`${CGROUP}/memory.max`]: '805306368\n',
@@ -141,6 +142,21 @@ describe('verifyCloudHypervisorConfinement', () => {
     }));
   });
 
+  it('accepts an empty Groups field when no supplementary groups are expected', async () => {
+    const verificationOptions = options();
+    verificationOptions.launchPolicy = {
+      ...verificationOptions.launchPolicy,
+      supplementaryGroups: [],
+    };
+
+    const result = await verifyCloudHypervisorConfinement(
+      verificationOptions,
+      dependencies({ groups: '' }),
+    );
+
+    expect(result.identity.supplementaryGroups).toEqual([]);
+  });
+
   it('fails closed when PID identity changes while evidence is collected', async () => {
     await expect(verifyCloudHypervisorConfinement(
       options(),

src/cloud-hypervisor/confinement-verifier.ts

@@ -242,7 +242,11 @@ function verifyThreadStatus(
   }
   assertIdentityField(status.Uid, options.identity.uid, 'UID', taskId);
   assertIdentityField(status.Gid, options.identity.gid, 'GID', taskId);
-  const groups = parseNumericFields(requiredStatus(status, 'Groups', taskId), 'Groups');
+  const groupStatus = status.Groups;
+  if (groupStatus === undefined) {
+    throw new Error(`Cloud Hypervisor thread ${taskId} is missing /proc status field Groups`);
+  }
+  const groups = parseNumericFields(groupStatus, 'Groups');
   if (groups.join(',') !== options.launchPolicy.supplementaryGroups.join(',')) {
     throw new Error(
       `Cloud Hypervisor thread ${taskId} has supplementary groups ${groups.join(',') || 'none'}, ` +

src/cloud-hypervisor/launcher.test.ts

@@ -10,21 +10,20 @@ describe('buildCloudHypervisorLaunchCommand', () => {
     tools: { ip: '/usr/sbin/ip', setpriv: '/usr/bin/setpriv' },
     namespaceName: 'awfch-abc123',
     identity: { uid: 1000, gid: 1000 },
-    kvmGid: 978,
     cloudHypervisorBinary: '/opt/cloud-hypervisor',
     apiSocketPath: '/run/awf/api.socket',
     logFilePath: '/run/awf/cloud-hypervisor.log',
   };
 
-  it('joins the namespace, retains only the kvm group, empties all capability sets, and execs Cloud Hypervisor with no shell', () => {
+  it('joins the namespace, drops privileges and groups, empties capabilities, then execs without a shell', () => {
     const result = buildCloudHypervisorLaunchCommand(baseOptions);
     expect(result.command).toBe('/usr/sbin/ip');
     expect(result.args).toEqual([
       'netns', 'exec', 'awfch-abc123',
       '/usr/bin/setpriv',
       '--reuid=1000',
       '--regid=1000',
-      '--groups=978',
+      '--clear-groups',
       '--no-new-privs',
       '--inh-caps=-all',
       '--bounding-set=-all',
@@ -36,9 +35,9 @@ describe('buildCloudHypervisorLaunchCommand', () => {
       '-v',
       '--seccomp', 'true',
     ]);
-    expect(result.args).not.toContain('--clear-groups');
+    expect(result.args).toContain('--clear-groups');
     expect(result.confinementPolicy).toEqual({
-      supplementaryGroups: [978],
+      supplementaryGroups: [],
       capabilities: {
         inheritable: '0000000000000000',
         permitted: '0000000000000000',
@@ -59,17 +58,12 @@ describe('buildCloudHypervisorLaunchCommand', () => {
     ['unsafe namespace name', { namespaceName: '../etc' }, /Unsafe Cloud Hypervisor network namespace name/],
     ['zero uid', { identity: { uid: 0, gid: 1000 } }, /uid must be a positive integer/],
     ['negative gid', { identity: { uid: 1000, gid: -1 } }, /gid must be a positive integer/],
-    ['negative kvm gid', { kvmGid: -1 }, /\/dev\/kvm group id must be a non-negative integer/],
     ['relative binary path', { cloudHypervisorBinary: 'cloud-hypervisor' }, /binary path must be absolute/],
     ['relative socket path', { apiSocketPath: 'api.socket' }, /API socket path must be absolute/],
   ])('rejects %s', (_label, overrides, error) => {
     expect(() => buildCloudHypervisorLaunchCommand({ ...baseOptions, ...overrides }))
       .toThrow(error);
   });
-
-  it('accepts a kvm gid of 0 (root-owned /dev/kvm on unusual hosts)', () => {
-    expect(() => buildCloudHypervisorLaunchCommand({ ...baseOptions, kvmGid: 0 })).not.toThrow();
-  });
 });
 
 describe('computeCloudHypervisorLandlockRules', () => {