diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 04b99d63c97..12fde33332b 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -874,6 +874,22 @@ After all portable TCP probe attempts fail, onboarding prints commands for the u Onboarding prints the same commands when the portable probe cannot reach the user-scoped Podman service. The printed rerun command keeps the portable experimental profile selected. +Portable commands reconstruct the current user's rootless Podman socket authority from NemoClaw state before they use the Docker-compatible API. +They do not select an endpoint from ambient Docker or Podman runtime variables or named connections. +When `podman.service` reports inactive and the recorded socket exists, NemoClaw first makes one 10-second API request through the guarded recorded authority. +A valid server version classifies the endpoint as warm and avoids starting another socket service. +A missing socket or a response without a valid server version enters bounded cold activation. +Any socket authority change during this precheck fails at the socket authority stage. +When the user-scoped socket-backed service needs activation, NemoClaw activates it and waits through a bounded startup period for a real Podman API response. +During cold activation, the first API probe can cause systemd to replace the socket inode. +NemoClaw requalifies one such replacement and repeats the probe only when the socket path, device, mode, owner, and complete directory authority remain unchanged. +Any other authority change or a second inode replacement fails the readiness check. +After cold activation succeeds, later API health checks use the fixed 10-second steady-state deadline. +Onboarding and portable sandbox lifecycle commands use this same readiness contract. +Failures identify socket authority, service activation, startup API health, or steady-state API health without reporting credentials. +NemoClaw does not fall back to Docker or report an absent or unreachable endpoint as healthy. +A successful cold path uses the `cold` timing label and reports activation, API, and total time in milliseconds. +A successful warm path uses the `warm` timing label and reports steady-state API and total time in milliseconds. To tune the existing-gateway HTTP health poll, use `NEMOCLAW_REUSE_HEALTH_POLL_COUNT` (default `6`) and `NEMOCLAW_REUSE_HEALTH_POLL_INTERVAL` (default `5` seconds). The poll count is clamped to a minimum of `1` so the health probe always runs at least once, and the interval is clamped to a minimum of `0` (no sleep between attempts). @@ -4968,6 +4984,7 @@ Defaults are sized for typical hardware; override only if you see false-positive | Variable | Default | Effect | |----------|---------|--------| | `NEMOCLAW_MCP_PROVIDER_SYNC_TIMEOUT_SECONDS` | `30` | Maximum time to wait for an OpenShell MCP provider credential revision to become active or fully revoked inside the sandbox. Integer seconds; raise only when provider synchronization is unusually slow. | +| `NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS` | `60000` | Maximum cold-start time for portable rootless Podman socket activation and the first real API response. Set an integer from `15000` through `300000` milliseconds. This setting does not change the fixed 10,000 ms steady-state API deadline. | | `NEMOCLAW_SANDBOX_EXEC_TIMEOUT_MS` | per call site (typically `15000`) | Overrides the default timeout for `openshell sandbox exec` calls issued by recovery and lifecycle helpers. Integer milliseconds; non-positive or non-numeric values fall back to the per-call-site default. | | `NEMOCLAW_STATUS_PROBE_TIMEOUT_MS` | built-in default | Overrides the timeout for the OpenShell status probe used by `$$nemoclaw status`. Integer milliseconds; non-positive or non-numeric values fall back to the default. | | `NEMOCLAW_WSL_GPU_PROOF_TIMEOUT_MS` | `180000` | Maximum time for the bounded Docker CUDA workload on an eligible ARM64 Linux host. A positive finite number of milliseconds overrides the default. Invalid, infinite, zero, and negative values use the default. | diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index a699d9c56d7..9b7191e621c 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -3329,6 +3329,71 @@ Then rerun portable onboarding: $$nemoclaw onboard --experimental-profile portable ``` +### Portable Podman Readiness Fails + +Portable commands use the current user's rootless Podman socket authority recorded in NemoClaw state. +They ignore ambient Docker and Podman runtime selectors, including named connections. +Do not export another `DOCKER_HOST`, `DOCKER_CONTEXT`, `CONTAINER_HOST`, or `CONTAINER_CONNECTION` to bypass a readiness failure. + +When `podman.service` reports inactive and the recorded socket exists, NemoClaw first makes one 10-second API request through the guarded recorded authority. +A valid server version classifies the endpoint as warm and avoids starting another socket service. +A missing socket or a response without a valid server version enters bounded cold activation. +Any socket authority change during this precheck fails at the socket authority stage and is not eligible for inode requalification. + +During cold activation, the first API probe can cause systemd to replace the socket inode. +NemoClaw requalifies one such replacement and repeats the probe only when the socket path, device, mode, owner, and complete directory authority remain unchanged. +Any other authority change or a second inode replacement fails at the socket authority stage. + +A portable readiness failure identifies the stage that did not complete: + +| Stage | Meaning | Recovery | +| --- | --- | --- | +| Socket authority | The portable lifecycle receipt is unsafe or invalid, its schema is `1` through `3` and predates recorded portable Podman authority, the recorded authority does not match the current Linux user, the recorded socket is unsafe, another authority field changed, or its inode was replaced more than once during cold activation. | If the error says that the receipt is unsafe or invalid, or that it predates recorded authority, run `$$nemoclaw onboard --experimental-profile portable`. If the recorded authority does not match the current Linux user, run NemoClaw as the user who created the portable state or rerun portable onboarding as the current user. Otherwise, stop and inspect the reported path. Restore the recorded current-user runtime instead of selecting another endpoint. | +| Service activation | The current user's systemd manager could not activate the Podman socket-backed service, or activation did not create the recorded socket within the startup period. | Inspect `podman.socket` and `podman.service` with the commands below. Correct the reported user-unit failure, then rerun the NemoClaw command. | +| Startup API health | The service was activated, but the recorded endpoint did not return a real Podman API response within the startup period. | Inspect the user-unit logs and run the explicit API request below against the reported socket path. Raise `NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS` only when valid cold activation needs more than 60,000 ms. | +| Steady-state API health | An endpoint that completed activation did not answer the later shorter health check. | Inspect host load and the user-unit logs, then rerun the NemoClaw command. | + +Inspect the current user's units without changing them: + +```bash +systemctl --user status podman.socket podman.service --no-pager +journalctl --user -u podman.socket -u podman.service --since -10m --no-pager +``` + +If the units need activation, restart the active service when present and start the socket for the current user session: + +```bash +systemctl --user try-restart podman.service +systemctl --user start podman.socket +``` + +These commands do not enable the socket for later user sessions. + +Use the exact socket path from the NemoClaw failure to require a real server response: + +```bash +podman --remote \ + --url unix:// \ + version \ + --format 'Server Version: {{.Server.Version}}' +``` + +Continue only when the command exits with status `0` and prints a nonempty server version. +The request and the readiness report contain no credentials. +Rerun the original NemoClaw command without exporting a Docker or Podman runtime selector. + +If valid cold activation needs a larger budget, set an integer from `15000` through `300000` milliseconds: + +```bash +export NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS=120000 +$$nemoclaw +``` + +The default cold-start budget is 60,000 ms. +The later steady-state API deadline is fixed at 10,000 ms and does not use this setting. +A successful cold path uses the `cold` timing label and reports activation, API, and total time in milliseconds. +A successful warm path uses the `warm` timing label and reports steady-state API and total time in milliseconds. + ### Portable Host Gateway Is Unreachable The portable experimental profile maps `host.openshell.internal` to the OpenShell Podman host gateway. @@ -3344,15 +3409,15 @@ Portable onboarding reports output like this: ``` If `podman.service` is active, restart it. -Then enable and start the user-scoped Podman socket: +Then start the user-scoped Podman socket for the current user session: ```bash systemctl --user try-restart podman.service -systemctl --user enable --now podman.socket +systemctl --user start podman.socket ``` The first command does not start an inactive service. -The second command enables and starts the current user's Podman API socket. +The second command starts the current user's Podman API socket without enabling it for later user sessions. These commands affect only the current user's Podman units. They do not read or write credentials. @@ -3368,6 +3433,9 @@ Expected output: active ``` +An active socket alone does not establish API health. +Run the explicit Podman API request from [Portable Podman Readiness Fails](#portable-podman-readiness-fails) before you rerun onboarding. + Then rerun portable onboarding: ```bash diff --git a/src/lib/actions/sandbox/doctor-lifecycle-registration.test.ts b/src/lib/actions/sandbox/doctor-lifecycle-registration.test.ts index 0b6e045d422..0d013f312e8 100644 --- a/src/lib/actions/sandbox/doctor-lifecycle-registration.test.ts +++ b/src/lib/actions/sandbox/doctor-lifecycle-registration.test.ts @@ -1,9 +1,45 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it } from "vitest"; +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const receiptReadinessMocks = vi.hoisted(() => ({ + inspect: vi.fn(), +})); + +vi.mock("../../onboard/experimental/portable-runtime-receipt-readiness", () => ({ + inspectPortableRuntimeReceiptReadiness: receiptReadinessMocks.inspect, +})); + +import type { PortablePodmanReadinessResult } from "../../onboard/experimental/portable-runtime-readiness"; import type { SandboxEntry } from "../../state/registry"; -import { buildLifecycleRegistrationCheck } from "./doctor-lifecycle-registration"; +import { + buildLifecycleRegistrationCheck, + buildPortableRuntimeCheck, +} from "./doctor-lifecycle-registration"; + +const READY_PORTABLE_RUNTIME = { + ok: true, + authority: { + directoryChain: [], + device: "1", + inode: "2", + mode: String(0o140600), + ownerUid: "1001", + socketPath: "/run/user/1001/podman/podman.sock", + }, + dockerHost: "unix:///run/user/1001/podman/podman.sock", + serverVersion: "5.6.1", + timing: { mode: "warm", activationMs: 0, apiMs: 7, totalMs: 7 }, +} satisfies PortablePodmanReadinessResult; + +const FAILED_PORTABLE_RUNTIME = { + ok: false, + stage: "startup API health", + detail: "Podman did not report a server version.", + socketPath: "/run/user/1001/podman/podman.sock", + timing: { mode: "cold", activationMs: 21, apiMs: 9, totalMs: 30 }, +} satisfies PortablePodmanReadinessResult; function sandbox(overrides: Partial = {}): SandboxEntry { return { @@ -24,6 +60,36 @@ function sandbox(overrides: Partial = {}): SandboxEntry { } describe("doctor lifecycle registration checks", () => { + beforeEach(() => { + receiptReadinessMocks.inspect.mockReset(); + }); + + it("renders server and timing detail for a ready portable Podman API", () => { + receiptReadinessMocks.inspect.mockReturnValue(READY_PORTABLE_RUNTIME); + + expect(buildPortableRuntimeCheck("alpha")).toEqual({ + group: "Host", + label: "Portable Podman API", + status: "ok", + detail: "server 5.6.1; warm; activation 0 ms; API 7 ms; total 7 ms", + }); + expect(receiptReadinessMocks.inspect).toHaveBeenCalledWith("alpha"); + }); + + it("renders the failure stage, recorded socket, and recovery hint", () => { + receiptReadinessMocks.inspect.mockReturnValue(FAILED_PORTABLE_RUNTIME); + + expect(buildPortableRuntimeCheck("alpha")).toEqual({ + group: "Host", + label: "Portable Podman API", + status: "fail", + detail: + "startup API health: Podman did not report a server version. Recorded socket: /run/user/1001/podman/podman.sock.", + hint: "repair the recorded current-user Podman endpoint, then retry", + }); + expect(receiptReadinessMocks.inspect).toHaveBeenCalledWith("alpha"); + }); + it("reports a complete managed sandbox registration as ok", () => { expect(buildLifecycleRegistrationCheck("alpha", sandbox(), "nemoclaw")).toMatchObject({ group: "Sandbox", diff --git a/src/lib/actions/sandbox/doctor-lifecycle-registration.ts b/src/lib/actions/sandbox/doctor-lifecycle-registration.ts index d93072e90a1..6f049f683b7 100644 --- a/src/lib/actions/sandbox/doctor-lifecycle-registration.ts +++ b/src/lib/actions/sandbox/doctor-lifecycle-registration.ts @@ -5,6 +5,7 @@ import { collectLifecycleRegistrationIssues, type LifecycleRegistrationIssue, } from "../../domain/lifecycle-registration"; +import { inspectPortableRuntimeReceiptReadiness } from "../../onboard/experimental/portable-runtime-receipt-readiness"; import type { SandboxEntry } from "../../state/registry"; import type { DoctorCheck } from "./doctor-report"; @@ -23,6 +24,28 @@ function formatFieldList( .join(", "); } +export function buildPortableRuntimeCheck(sandboxName: string): DoctorCheck | null { + const portable = inspectPortableRuntimeReceiptReadiness(sandboxName); + if (!portable) return null; + const recordedSocket = !portable.ok && portable.socketPath + ? ` Recorded socket: ${portable.socketPath}.` + : ""; + return portable.ok + ? { + group: "Host", + label: "Portable Podman API", + status: "ok", + detail: `server ${portable.serverVersion}; ${portable.timing.mode}; activation ${String(portable.timing.activationMs)} ms; API ${String(portable.timing.apiMs)} ms; total ${String(portable.timing.totalMs)} ms`, + } + : { + group: "Host", + label: "Portable Podman API", + status: "fail", + detail: `${portable.stage}: ${portable.detail}${recordedSocket}`, + hint: "repair the recorded current-user Podman endpoint, then retry", + }; +} + export function buildLifecycleRegistrationCheck( sandboxName: string, entry: SandboxEntry, diff --git a/src/lib/actions/sandbox/doctor.ts b/src/lib/actions/sandbox/doctor.ts index f3ab3846452..a3d4d1f3eba 100644 --- a/src/lib/actions/sandbox/doctor.ts +++ b/src/lib/actions/sandbox/doctor.ts @@ -49,7 +49,10 @@ import { type DoctorInferenceRoute, resolveDoctorReasoningEffort, } from "./doctor-inference"; -import { buildLifecycleRegistrationCheck } from "./doctor-lifecycle-registration"; +import { + buildLifecycleRegistrationCheck, + buildPortableRuntimeCheck, +} from "./doctor-lifecycle-registration"; import { collectMessagingDoctorChecks } from "./doctor-messaging"; import { buildDoctorReport, @@ -136,25 +139,25 @@ function cliBuildCheck(): DoctorCheck { }; } -function collectHostChecks(sb: SandboxEntry | null | undefined): { - checks: DoctorCheck[]; - openshellBin: ReturnType; -} { - const cli = cliBuildCheck(); - const openshellBin = resolveOpenshell(); - let runtimeCheck: DoctorCheck; +function inspectRuntimeHost(sb: SandboxEntry | null | undefined): DoctorCheck { + const portable = sb ? buildPortableRuntimeCheck(sb.name) : null; + if (portable) return portable; + const recorded = sb?.openshellDriver?.trim(); + const provider = recorded + ? requireRuntimeProviderBundle(recorded, CURRENT_RUNTIME_PROVIDER_BUNDLES) + : resolveCurrentRuntimeProviderBundle(); + return provider.preflightDoctor.inspectHost(); +} + +function runtimeHostCheck(sb: SandboxEntry | null | undefined): DoctorCheck { try { - const recorded = sb?.openshellDriver?.trim(); - const provider = recorded - ? requireRuntimeProviderBundle(recorded, CURRENT_RUNTIME_PROVIDER_BUNDLES) - : resolveCurrentRuntimeProviderBundle(); - runtimeCheck = provider.preflightDoctor.inspectHost(); + return inspectRuntimeHost(sb); } catch (error) { const detail = error instanceof RuntimeProviderSelectionError ? error.message : `Runtime provider inspection failed: ${error instanceof Error ? error.message : String(error)}`; - runtimeCheck = { + return { group: "Host", label: "Runtime provider", status: "fail", @@ -162,10 +165,18 @@ function collectHostChecks(sb: SandboxEntry | null | undefined): { hint: "restore a supported durable runtime provider identity before retrying", }; } +} + +function collectHostChecks(sb: SandboxEntry | null | undefined): { + checks: DoctorCheck[]; + openshellBin: ReturnType; +} { + const cli = cliBuildCheck(); + const openshellBin = resolveOpenshell(); return { checks: [ cli, - runtimeCheck, + runtimeHostCheck(sb), { group: "Host", label: "OpenShell CLI", diff --git a/src/lib/actions/sandbox/gateway-failure-classifier.ts b/src/lib/actions/sandbox/gateway-failure-classifier.ts index 41e76cbe9cc..3d8070af156 100644 --- a/src/lib/actions/sandbox/gateway-failure-classifier.ts +++ b/src/lib/actions/sandbox/gateway-failure-classifier.ts @@ -9,11 +9,20 @@ import { CLI_NAME } from "../../cli/branding"; import { GATEWAY_PORT } from "../../core/ports"; import { resolveSandboxContainerOwner } from "../../domain/sandbox/container-owner"; import { resolveGatewayPortFromName } from "../../onboard/gateway-binding"; +import type { PortablePodmanReadinessResult } from "../../onboard/experimental/portable-runtime-readiness"; +import { + inspectPortableRuntimeReceiptReadiness, + type PortableRuntimeReceiptReadinessDeps, +} from "../../onboard/experimental/portable-runtime-receipt-readiness"; import * as registry from "../../state/registry"; import { getSandboxTargetGatewayName } from "./gateway-target"; const DOCKER_TIMEOUT_MS = 3000; const PORT_PROBE_TIMEOUT_MS = 2000; +const portableRuntimeFailures = new Map< + string, + Extract +>(); export type GatewayFailureLayer = | "docker_unreachable" @@ -281,8 +290,22 @@ export function isDockerRuntimeDown( opts?: { runners?: Pick; getSandbox?: SandboxDriverLookup; + portableLifecycle?: PortableRuntimeReceiptReadinessDeps; }, ): boolean { + const portable = inspectPortableRuntimeReceiptReadiness(sandboxName, opts?.portableLifecycle); + if (portable) { + if (portable.ok) { + portableRuntimeFailures.delete(sandboxName); + console.log( + ` Portable Podman readiness: ${portable.timing.mode}; activation ${String(portable.timing.activationMs)} ms; API ${String(portable.timing.apiMs)} ms; total ${String(portable.timing.totalMs)} ms.`, + ); + return false; + } + portableRuntimeFailures.set(sandboxName, portable); + return true; + } + portableRuntimeFailures.delete(sandboxName); const getSandbox = opts?.getSandbox ?? registry.getSandbox; if (!isDockerBackedSandbox(sandboxName, getSandbox)) return false; const probe = opts?.runners?.dockerInfo ?? defaultRunners.dockerInfo; @@ -302,6 +325,25 @@ export function printDockerRuntimeDownGuidance( ): void { const writer = opts.writer ?? console.error; const retryCommand = opts.retryCommand ?? "status"; + const portable = portableRuntimeFailures.get(sandboxName); + portableRuntimeFailures.delete(sandboxName); + if (portable) { + writer(` Failure stage: ${portable.stage} — ${portable.detail}`); + if (portable.socketPath) writer(` Recorded socket: ${portable.socketPath}`); + writer( + ` Portable Podman readiness (${portable.timing.mode}): activation ${String(portable.timing.activationMs)} ms; API ${String(portable.timing.apiMs)} ms; total ${String(portable.timing.totalMs)} ms.`, + ); + writer( + ` The receipt-owned Podman endpoint for sandbox '${sandboxName}' is not ready; no Docker or named-connection fallback was used.`, + ); + writer(" Recovery:"); + writer( + " 1. Check the reported readiness stage and the current user's Podman socket service.", + ); + writer(" 2. Confirm the recorded endpoint returns a real Podman server version."); + writer(` 3. Retry: ${CLI_NAME} ${sandboxName} ${retryCommand}`); + return; + } writer(` ${getLayerHeader("docker_unreachable")}`); writer( ` The Docker daemon is not reachable, so sandbox '${sandboxName}' cannot be verified or started.`, diff --git a/src/lib/actions/sandbox/start.test.ts b/src/lib/actions/sandbox/start.test.ts index c56daff1e85..fb47e9dad6c 100644 --- a/src/lib/actions/sandbox/start.test.ts +++ b/src/lib/actions/sandbox/start.test.ts @@ -43,6 +43,12 @@ function harness(overrides: Partial = {}) { running: false, }, ]); + const hasPortableLifecycleReceipt = vi.fn< + DockerRuntimeProviderDependencies["hasPortableLifecycleReceipt"] + >(() => false); + const recoverPortableSandbox = vi.fn< + DockerRuntimeProviderDependencies["recoverPortableSandbox"] + >(() => ({ kind: "not-installed" })); const recoverDockerDriverSandbox = vi.fn( () => ({ recovered: true, @@ -68,9 +74,11 @@ function harness(overrides: Partial = {}) { "docker", createDockerRuntimeProviderBundle({ findLabeledSandboxContainers, + hasPortableLifecycleReceipt, isRuntimeDown: isDockerRuntimeDown, printRuntimeDownGuidance: printDockerRuntimeDownGuidance, recoverSandbox: recoverDockerDriverSandbox, + recoverPortableSandbox, unpauseContainer: dockerUnpause, }), ], @@ -90,10 +98,12 @@ function harness(overrides: Partial = {}) { dockerUnpause, findLabeledSandboxContainers, getSandbox, + hasPortableLifecycleReceipt, isDockerRuntimeDown, log, printDockerRuntimeDownGuidance, recoverDockerDriverSandbox, + recoverPortableSandbox, restoreStartupState, waitForManagedGatewaySupervisor, verifyGateway, @@ -408,6 +418,31 @@ describe("startSandbox", () => { expect(output).toContain("openshell-my-sandbox"); }); + it("uses recorded Podman authority instead of ambient Docker for a portable receipt (#9070)", async () => { + const h = harness(); + h.getSandbox.mockReturnValue( + sandbox({ + agent: "openclaw", + gatewayName: "nemoclaw", + lifecycleGeneration: "generation-alpha", + openshellDriver: "docker", + }), + ); + h.hasPortableLifecycleReceipt.mockReturnValue(true); + h.recoverPortableSandbox.mockReturnValue({ kind: "recovered" }); + + await expect(startSandbox("my-sandbox", h.deps)).resolves.toEqual({ exitCode: 0 }); + + expect(h.isDockerRuntimeDown).not.toHaveBeenCalled(); + expect(h.recoverPortableSandbox).toHaveBeenCalledWith( + "my-sandbox", + expect.objectContaining({ lifecycleGeneration: "generation-alpha" }), + expect.objectContaining({ env: process.env }), + ); + expect(h.findLabeledSandboxContainers).not.toHaveBeenCalled(); + expect(h.recoverDockerDriverSandbox).not.toHaveBeenCalled(); + }); + it("still probes when the container was already running (#6026)", async () => { const h = harness(); h.findLabeledSandboxContainers.mockReturnValue([ diff --git a/src/lib/actions/sandbox/stop.test.ts b/src/lib/actions/sandbox/stop.test.ts index 9561018cd53..21ecfd7f786 100644 --- a/src/lib/actions/sandbox/stop.test.ts +++ b/src/lib/actions/sandbox/stop.test.ts @@ -46,6 +46,12 @@ function harness(overrides: StopHarnessOverrides = {}) { const findLabeledSandboxContainers = vi.fn< DockerRuntimeProviderDependencies["findLabeledSandboxContainers"] >(findContainersOverride ?? (() => [container("openshell-my-sandbox", true)])); + const hasPortableLifecycleReceipt = vi.fn< + DockerRuntimeProviderDependencies["hasPortableLifecycleReceipt"] + >(() => false); + const stopPortableSandbox = vi.fn< + DockerRuntimeProviderDependencies["stopPortableSandbox"] + >(() => ({ kind: "not-installed" })); const stopSandboxChannels = vi.fn>(); const dockerStop = vi.fn( dockerStopOverride ?? (() => ({ status: 0 })), @@ -59,9 +65,11 @@ function harness(overrides: StopHarnessOverrides = {}) { "docker", createDockerRuntimeProviderBundle({ findLabeledSandboxContainers, + hasPortableLifecycleReceipt, isRuntimeDown: isDockerRuntimeDown, printRuntimeDownGuidance: printDockerRuntimeDownGuidance, stopContainer: dockerStop, + stopPortableSandbox, }), ], ["kubernetes", createKubernetesRuntimeProviderBundle()], @@ -83,10 +91,12 @@ function harness(overrides: StopHarnessOverrides = {}) { teardownSandboxDashboardForward, findLabeledSandboxContainers, getSandbox, + hasPortableLifecycleReceipt, isDockerRuntimeDown, log, printDockerRuntimeDownGuidance, stopSandboxChannels, + stopPortableSandbox, warn, }; } @@ -262,6 +272,39 @@ describe("stopSandbox", () => { expect(output).toContain("nemoclaw my-sandbox start"); }); + it("uses recorded Podman authority instead of ambient Docker for a portable receipt (#9070)", () => { + const h = harness(); + h.getSandbox.mockReturnValue( + sandbox({ + agent: "openclaw", + gatewayName: "nemoclaw", + lifecycleGeneration: "generation-alpha", + openshellDriver: "docker", + }), + ); + h.hasPortableLifecycleReceipt.mockReturnValue(true); + h.stopPortableSandbox.mockImplementation((_name, _context, beforeStop) => { + beforeStop(); + return { kind: "stopped" }; + }); + + expect(stopSandbox("my-sandbox", h.deps)).toEqual({ exitCode: 0 }); + + expect(h.isDockerRuntimeDown).not.toHaveBeenCalled(); + expect(h.stopPortableSandbox).toHaveBeenCalledWith( + "my-sandbox", + expect.objectContaining({ lifecycleGeneration: "generation-alpha" }), + expect.any(Function), + expect.objectContaining({ env: process.env }), + ); + expect(h.stopSandboxChannels).toHaveBeenCalledExactlyOnceWith( + "my-sandbox", + expect.any(Object), + ); + expect(h.findLabeledSandboxContainers).not.toHaveBeenCalled(); + expect(h.dockerStop).not.toHaveBeenCalled(); + }); + it("succeeds idempotently when the container is already stopped (#6026)", () => { const h = harness(); h.findLabeledSandboxContainers.mockReturnValue([container("openshell-my-sandbox", false)]); diff --git a/src/lib/onboard.ts b/src/lib/onboard.ts index e1c4b5041a2..d266f5d3ab1 100644 --- a/src/lib/onboard.ts +++ b/src/lib/onboard.ts @@ -1518,11 +1518,9 @@ const { getSandboxRuntimeRegistryFields, hasSandboxGpuDrift, updateReusedSandbox getInstalledOpenshellVersion, runCaptureOpenshell, }); - -// ── Step 5: Sandbox ────────────────────────────────────────────── - async function createSandboxWithBaseImageResolution( baseImageResolutionContext: import("./onboard/base-image-resolution-flow").BaseImageResolutionContext, + portableRuntimeAuthority: import("./state/onboard-checkpoint-types").CheckpointPortableRuntimeAuthority | null, computePlan: import("./onboard/compute/plan").OpenShellComputePlan, managedWorkloadRebuild: import("./onboard/workload/rebuild").ManagedWorkloadRebuildHandoff | null, tempManagedRuntime: boolean, @@ -1945,6 +1943,7 @@ async function createSandboxWithBaseImageResolution( sandboxEnv, sandboxStartupCommand, lifecycleGeneration: createdSandboxLifecycle.generation, + portableRuntimeAuthority, prebuild, restoreBackupPath, terminalAgent: agentDefs.isTerminalAgent(agent), @@ -2102,25 +2101,16 @@ async function createSandboxWithBaseImageResolution( return sandboxName; } -type CreateSandboxArgs = - Parameters extends [ - unknown, - unknown, - unknown, - unknown, - unknown, - unknown, - unknown, - ...infer Args, - ] - ? Args - : never; - const { createSandbox, createSandboxWithTemporaryManagedRuntime } = agentOnboard.createHermesApiPortScopedSandboxEntryPoints({ createBaseImageResolutionContext: () => baseImageResolutionFlow.createBaseImageResolutionContext({ fresh: false }), createSandboxWithBaseImageResolution, + resolvePortableRuntimeAuthority: () => + sandboxGpuCreateFlow.resolveExportedPortableRuntimeAuthority( + process.env, + onboardSession.loadSession, + ), resolveComputePlan: dockerDriverPlatform.resolveCurrentOpenShellComputePlan, }); @@ -3556,6 +3546,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise { withSandboxPortReservationScope((dashboardPortReservationScope) => createSandboxWithBaseImageResolution( baseImageResolutionContext, + lockedRuntime.preparedPortableAuthority, onboardingComputePlan, opts.managedWorkloadRebuild ?? null, opts.tempManagedRuntime === true, diff --git a/src/lib/onboard/dashboard-port.test.ts b/src/lib/onboard/dashboard-port.test.ts index 6bdf4940fe4..f170156ea66 100644 --- a/src/lib/onboard/dashboard-port.test.ts +++ b/src/lib/onboard/dashboard-port.test.ts @@ -261,6 +261,7 @@ describe("dashboard port reservation", () => { const createSandboxWithBaseImageResolution = vi.fn( async ( baseImageResolutionContext: { fresh: boolean }, + portableRuntimeAuthority: { socketPath: string } | null, computePlan: { sequence: number }, managedWorkloadRebuild: null, temporaryManagedRuntime: boolean, @@ -271,6 +272,7 @@ describe("dashboard port reservation", () => { events.push("create sandbox"); return { baseImageResolutionContext, + portableRuntimeAuthority, computePlan, managedWorkloadRebuild, temporaryManagedRuntime, @@ -287,6 +289,7 @@ describe("dashboard port reservation", () => { return { fresh: false }; }, createSandboxWithBaseImageResolution, + resolvePortableRuntimeAuthority: () => ({ socketPath: "/run/user/1001/podman.sock" }), resolveComputePlan: () => { events.push("resolve compute plan"); return { sequence: ++sequence }; @@ -295,6 +298,7 @@ describe("dashboard port reservation", () => { await expect(entryPoints.createSandbox("standard")).resolves.toMatchObject({ baseImageResolutionContext: { fresh: false }, + portableRuntimeAuthority: { socketPath: "/run/user/1001/podman.sock" }, computePlan: { sequence: 1 }, managedWorkloadRebuild: null, temporaryManagedRuntime: false, @@ -306,6 +310,7 @@ describe("dashboard port reservation", () => { entryPoints.createSandboxWithTemporaryManagedRuntime("temporary"), ).resolves.toMatchObject({ baseImageResolutionContext: { fresh: false }, + portableRuntimeAuthority: { socketPath: "/run/user/1001/podman.sock" }, computePlan: { sequence: 2 }, managedWorkloadRebuild: null, temporaryManagedRuntime: true, @@ -314,8 +319,8 @@ describe("dashboard port reservation", () => { sandboxName: "temporary", }); expect(createSandboxWithBaseImageResolution).toHaveBeenCalledTimes(2); - expect(createSandboxWithBaseImageResolution.mock.calls[0]?.[5]).not.toBe( - createSandboxWithBaseImageResolution.mock.calls[1]?.[5], + expect(createSandboxWithBaseImageResolution.mock.calls[0]?.[6]).not.toBe( + createSandboxWithBaseImageResolution.mock.calls[1]?.[6], ); expect(events).toEqual([ "resolve compute plan", @@ -333,10 +338,12 @@ describe("dashboard port reservation", () => { [string], string, { fresh: boolean }, + null, { sequence: number } >({ createBaseImageResolutionContext: () => ({ fresh: false }), createSandboxWithBaseImageResolution: async () => "unreachable", + resolvePortableRuntimeAuthority: () => null, resolveComputePlan: () => { throw setupFailure; }, diff --git a/src/lib/onboard/dashboard-port.ts b/src/lib/onboard/dashboard-port.ts index a0d9acdc41b..41ff60cd968 100644 --- a/src/lib/onboard/dashboard-port.ts +++ b/src/lib/onboard/dashboard-port.ts @@ -627,11 +627,13 @@ interface DashboardPortScopedSandboxEntryPointDeps< Args extends unknown[], Result, BaseImageResolutionContext, + PortableRuntimeAuthority, ComputePlan, > { createBaseImageResolutionContext(): BaseImageResolutionContext; createSandboxWithBaseImageResolution( baseImageResolutionContext: BaseImageResolutionContext, + portableRuntimeAuthority: PortableRuntimeAuthority, computePlan: ComputePlan, managedWorkloadRebuild: null, temporaryManagedRuntime: boolean, @@ -639,6 +641,7 @@ interface DashboardPortScopedSandboxEntryPointDeps< dashboardPortReservationScope: DashboardPortReservationScope, ...args: Args ): Promise; + resolvePortableRuntimeAuthority(): PortableRuntimeAuthority; resolveComputePlan(): ComputePlan; } @@ -646,12 +649,14 @@ export function createDashboardPortScopedSandboxEntryPoints< Args extends unknown[], Result, BaseImageResolutionContext, + PortableRuntimeAuthority, ComputePlan, >( deps: DashboardPortScopedSandboxEntryPointDeps< Args, Result, BaseImageResolutionContext, + PortableRuntimeAuthority, ComputePlan >, ): { @@ -663,6 +668,7 @@ export function createDashboardPortScopedSandboxEntryPoints< return withDashboardPortReservationScope((dashboardPortReservationScope) => deps.createSandboxWithBaseImageResolution( deps.createBaseImageResolutionContext(), + deps.resolvePortableRuntimeAuthority(), computePlan, null, temporaryManagedRuntime, diff --git a/src/lib/onboard/experimental/portable-demo-lifecycle-authority.test.ts b/src/lib/onboard/experimental/portable-demo-lifecycle-authority.test.ts index d6a5fb64ea5..4946605b421 100644 --- a/src/lib/onboard/experimental/portable-demo-lifecycle-authority.test.ts +++ b/src/lib/onboard/experimental/portable-demo-lifecycle-authority.test.ts @@ -7,6 +7,7 @@ import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; import type { PodmanSocketAuthorityDeps } from "../../adapters/podman"; +import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; import { installPortableDemoSandboxLifecycle, portableDemoLifecycleInternals, @@ -15,6 +16,16 @@ import { const CONTAINER_ID = "a".repeat(64); const SANDBOX_ID = "sandbox-id-alpha"; const SOCKET_PATH = "/run/user/1001/podman/podman.sock"; +const RUNTIME_AUTHORITY: CheckpointPortableRuntimeAuthority = { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid: 1001, + homeDir: "/home/tester", + configHome: "/home/tester/.config", + runtimeDir: "/run/user/1001", + socketPath: SOCKET_PATH, +}; const STARTUP_ARGV = [ "env", "CHAT_UI_URL=http://127.0.0.1:18789", @@ -37,8 +48,8 @@ function createPodman() { const podman = vi.fn((args: readonly string[], _env?: NodeJS.ProcessEnv) => { const command = args[0] === "--url" ? args.slice(2) : args; switch (command[0]) { - case "info": - return { status: 0, stdout: `${SOCKET_PATH}\n` }; + case "version": + return { status: 0, stdout: JSON.stringify({ Server: { Version: "5.6.1" } }) }; case "ps": return { status: 0, stdout: `${CONTAINER_ID}\n` }; case "inspect": @@ -90,6 +101,21 @@ function socketAuthorityDeps(socketInode: () => bigint = () => 9001n): PodmanSoc }; } +function lifecycleDeps(authorityDeps: PodmanSocketAuthorityDeps) { + return { + platform: "linux" as const, + runtimeAuthority: RUNTIME_AUTHORITY, + podmanSocketAuthorityDeps: authorityDeps, + hardenSocketDirectory: vi.fn(), + runtimeReadiness: { + uid: RUNTIME_AUTHORITY.uid, + home: RUNTIME_AUTHORITY.homeDir, + systemctl: () => ({ status: 0 }), + }, + log: vi.fn(), + }; +} + afterEach(() => { for (const directory of temporaryDirectories.splice(0)) { fs.rmSync(directory, { force: true, recursive: true }); @@ -106,16 +132,14 @@ describe("portable demo lifecycle authority", () => { STARTUP_ARGV, { NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }, { - platform: "linux", podman, stateDir, - podmanSocketAuthorityDeps: socketAuthorityDeps(), - hardenSocketDirectory: vi.fn(), + ...lifecycleDeps(socketAuthorityDeps()), }, ); expect(podman.mock.calls.map(([args]) => args)).toEqual([ - ["info", "--format", "{{.Host.RemoteSocket.Path}}"], + ["--url", `unix://${SOCKET_PATH}`, "version", "--format", "json"], [ "--url", `unix://${SOCKET_PATH}`, @@ -137,12 +161,13 @@ describe("portable demo lifecycle authority", () => { const filePath = portableDemoLifecycleInternals.receiptPath("alpha", stateDir); const receipt = JSON.parse(fs.readFileSync(filePath, "utf-8")); expect(receipt).toEqual({ - schemaVersion: 3, + schemaVersion: 4, sandboxName: "alpha", sandboxId: SANDBOX_ID, containerId: CONTAINER_ID, dashboardPort: 18789, registryGeneration: CONTAINER_ID, + runtimeAuthority: RUNTIME_AUTHORITY, }); expect(fs.statSync(filePath).mode & 0o777).toBe(0o600); }); @@ -161,16 +186,21 @@ describe("portable demo lifecycle authority", () => { CONTAINER_SSHKEY: "/tmp/attacker-key", }, { - platform: "linux", podman: runtime.podman, stateDir, - podmanSocketAuthorityDeps: socketAuthorityDeps(), - hardenSocketDirectory: vi.fn(), + ...lifecycleDeps(socketAuthorityDeps()), }, ); for (const [, env] of runtime.podman.mock.calls) { - expect(env).toEqual({ NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }); + expect(env).toMatchObject({ + HOME: RUNTIME_AUTHORITY.homeDir, + XDG_CONFIG_HOME: RUNTIME_AUTHORITY.configHome, + XDG_RUNTIME_DIR: RUNTIME_AUTHORITY.runtimeDir, + }); + expect(env).not.toHaveProperty("CONTAINER_CONNECTION"); + expect(env).not.toHaveProperty("CONTAINER_HOST"); + expect(env).not.toHaveProperty("CONTAINER_SSHKEY"); } }); @@ -191,11 +221,9 @@ describe("portable demo lifecycle authority", () => { STARTUP_ARGV, { NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }, { - platform: "linux", podman, stateDir, - podmanSocketAuthorityDeps: socketAuthorityDeps(() => inode), - hardenSocketDirectory: vi.fn(), + ...lifecycleDeps(socketAuthorityDeps(() => inode)), }, ), ).toThrow("changed after it was qualified"); diff --git a/src/lib/onboard/experimental/portable-demo-lifecycle-identity.test.ts b/src/lib/onboard/experimental/portable-demo-lifecycle-identity.test.ts index 5932eff76e3..91d85afc2ec 100644 --- a/src/lib/onboard/experimental/portable-demo-lifecycle-identity.test.ts +++ b/src/lib/onboard/experimental/portable-demo-lifecycle-identity.test.ts @@ -6,7 +6,8 @@ import os from "node:os"; import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; -import type { PodmanSocketAuthorityDeps } from "../../adapters/podman"; +import type { PodmanSocketAuthority, PodmanSocketAuthorityDeps } from "../../adapters/podman"; +import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; import { installPortableDemoSandboxLifecycle, type PortableDemoLifecycleDeps, @@ -16,6 +17,32 @@ import { const CONTAINER_ID = "a".repeat(64); const SANDBOX_ID = "sandbox-id-alpha"; const SOCKET_PATH = "/run/user/1001/podman/podman.sock"; +const RUNTIME_AUTHORITY: CheckpointPortableRuntimeAuthority = { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid: 1001, + homeDir: "/home/tester", + configHome: "/home/tester/.config", + runtimeDir: "/run/user/1001", + socketPath: SOCKET_PATH, +}; +const SOCKET_AUTHORITY: PodmanSocketAuthority = { + directoryChain: [], + device: "1", + inode: "2", + mode: String(0o140600), + ownerUid: "1001", + socketPath: SOCKET_PATH, +}; +const READINESS = { + uid: 1001, + home: RUNTIME_AUTHORITY.homeDir, + systemctl: () => ({ status: 0 }), + hardenSocketDirectory: vi.fn(), + captureSocketAuthority: () => SOCKET_AUTHORITY, + assertSocketAuthority: vi.fn(), +}; const STARTUP_ARGV = [ "env", "CHAT_UI_URL=http://127.0.0.1:18789", @@ -65,7 +92,9 @@ function createPodman() { const command = args[0] === "--url" ? args.slice(2) : args; switch (command[0]) { case "info": - return { status: 0, stdout: "/run/user/1001/podman/podman.sock\n" }; + return { status: 0, stdout: `${SOCKET_PATH}\n` }; + case "version": + return { status: 0, stdout: JSON.stringify({ Server: { Version: "5.6.1" } }) }; case "ps": return { status: 0, stdout: `${CONTAINER_ID}\n` }; case "inspect": @@ -128,7 +157,9 @@ function installReceipt(stateDir: string, podman: ReturnType { }); describe("portable lifecycle legacy generation migration", () => { - it("claims a schema-2 receipt before privileged cleanup and upgrades it (#8584)", async () => { + it("refuses privileged cleanup without receipt-owned runtime authority (#9070)", async () => { const stateDir = legacyStateDir(); const { backfill, registry } = await legacyRegistryEntry(stateDir); + const podman = createPodman(); - expect( - resolvePortableDemoPrivilegedExecTarget( - "alpha", - migrationDeps(stateDir, createPodman(), backfill), - ), - ).toMatchObject({ containerId: CONTAINER_ID, dockerHost: `unix://${SOCKET_PATH}` }); - expect(backfill).toHaveBeenCalledWith(CONTAINER_ID); + expect(() => + resolvePortableDemoPrivilegedExecTarget("alpha", migrationDeps(stateDir, podman, backfill)), + ).toThrow("predates recorded portable Podman authority"); + expect(podman).not.toHaveBeenCalled(); + expect(backfill).not.toHaveBeenCalled(); expect( JSON.parse( fs.readFileSync(portableDemoLifecycleInternals.receiptPath("alpha", stateDir), "utf8"), ), - ).toMatchObject({ schemaVersion: 3, registryGeneration: CONTAINER_ID }); - expect(registry.getSandbox("alpha")?.lifecycleGeneration).toBe(CONTAINER_ID); + ).toMatchObject({ schemaVersion: 2 }); + expect(registry.getSandbox("alpha")?.lifecycleGeneration).toBeUndefined(); }); - it("claims a schema-2 receipt before retained-sandbox recovery (#8584)", async () => { + it("refuses retained-sandbox recovery without receipt-owned runtime authority (#9070)", async () => { const stateDir = legacyStateDir(); const { backfill, registry } = await legacyRegistryEntry(stateDir); + const podman = createPodman(); - expect( + expect(() => recoverPortableDemoSandboxLifecycle( "alpha", { agent: "openclaw", gatewayName: "nemoclaw", openshellDriver: "docker" }, { - ...migrationDeps(stateDir, createPodman(), backfill), + ...migrationDeps(stateDir, podman, backfill), captureOpenshell: (args) => args.includes("curl") ? { status: 0, stdout: "200" } : { status: 0 }, }, ), - ).toEqual({ kind: "already-running" }); - expect(backfill).toHaveBeenCalledWith(CONTAINER_ID); + ).toThrow("predates recorded portable Podman authority"); + expect(podman).not.toHaveBeenCalled(); + expect(backfill).not.toHaveBeenCalled(); expect( JSON.parse( fs.readFileSync(portableDemoLifecycleInternals.receiptPath("alpha", stateDir), "utf8"), ), - ).toMatchObject({ schemaVersion: 3, registryGeneration: CONTAINER_ID }); - expect(registry.getSandbox("alpha")?.lifecycleGeneration).toBe(CONTAINER_ID); + ).toMatchObject({ schemaVersion: 2 }); + expect(registry.getSandbox("alpha")?.lifecycleGeneration).toBeUndefined(); }); - it("finishes a schema-2 receipt upgrade after its registry claim already committed (#8584)", async () => { + it("still requires receipt authority after a registry generation claim (#9070)", async () => { const stateDir = legacyStateDir(); const { backfill, registry } = await legacyRegistryEntry(stateDir); expect(backfill(CONTAINER_ID)).toBe(true); backfill.mockClear(); const podman = createPodman(); - expect( + expect(() => recoverPortableDemoSandboxLifecycle( "alpha", { @@ -194,21 +194,18 @@ describe("portable lifecycle legacy generation migration", () => { args.includes("curl") ? { status: 0, stdout: "200" } : { status: 0 }, }, ), - ).toEqual({ kind: "already-running" }); + ).toThrow("predates recorded portable Podman authority"); expect(backfill).not.toHaveBeenCalled(); - expect(podman).toHaveBeenCalledWith( - ["info", "--format", "{{.Host.RemoteSocket.Path}}"], - expect.any(Object), - ); + expect(podman).not.toHaveBeenCalled(); expect( JSON.parse( fs.readFileSync(portableDemoLifecycleInternals.receiptPath("alpha", stateDir), "utf8"), ), - ).toMatchObject({ schemaVersion: 3, registryGeneration: CONTAINER_ID }); + ).toMatchObject({ schemaVersion: 2 }); expect(registry.getSandbox("alpha")?.lifecycleGeneration).toBe(CONTAINER_ID); }); - it("does not claim an ambiguous legacy portable identity (#8584)", () => { + it("does not inspect an ambiguous legacy identity through ambient Podman (#9070)", () => { const stateDir = legacyStateDir(); const backfill = vi.fn(() => true); @@ -217,7 +214,7 @@ describe("portable lifecycle legacy generation migration", () => { "alpha", migrationDeps(stateDir, createPodman([CONTAINER_ID, "b".repeat(64)]), backfill), ), - ).toThrow("found 2"); + ).toThrow("predates recorded portable Podman authority"); expect(backfill).not.toHaveBeenCalled(); const receipt = JSON.parse( fs.readFileSync(portableDemoLifecycleInternals.receiptPath("alpha", stateDir), "utf8"), @@ -226,7 +223,7 @@ describe("portable lifecycle legacy generation migration", () => { expect(receipt).not.toHaveProperty("registryGeneration"); }); - it("does not upgrade the receipt when the registry row changes before its claim (#8584)", async () => { + it("does not use a changed registry row to synthesize runtime authority (#9070)", async () => { const stateDir = legacyStateDir(); const { backfill, registry } = await legacyRegistryEntry(stateDir); registry.updateSandbox("alpha", { model: "replacement" }); @@ -236,7 +233,8 @@ describe("portable lifecycle legacy generation migration", () => { "alpha", migrationDeps(stateDir, createPodman(), backfill), ), - ).toThrow("could not claim the current registry generation"); + ).toThrow("predates recorded portable Podman authority"); + expect(backfill).not.toHaveBeenCalled(); const receipt = JSON.parse( fs.readFileSync(portableDemoLifecycleInternals.receiptPath("alpha", stateDir), "utf8"), ); diff --git a/src/lib/onboard/experimental/portable-demo-lifecycle.test.ts b/src/lib/onboard/experimental/portable-demo-lifecycle.test.ts index 8b668e3c8f2..c777faddcb7 100644 --- a/src/lib/onboard/experimental/portable-demo-lifecycle.test.ts +++ b/src/lib/onboard/experimental/portable-demo-lifecycle.test.ts @@ -7,6 +7,7 @@ import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; import type { PodmanSocketAuthorityDeps } from "../../adapters/podman"; +import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; import type { SandboxEntry } from "../../state/registry"; import { recordUserLocalOllamaOwnership } from "./ollama-user-local-runtime"; import { @@ -16,11 +17,22 @@ import { recoverPortableDemoSandboxLifecycle as recoverPortableDemoSandboxLifecycleUnchecked, removePortableDemoSandboxLifecycleReceipt, resolvePortableDemoPrivilegedExecTarget, + stopPortableDemoSandboxLifecycle, } from "./portable-demo-lifecycle"; const CONTAINER_ID = "a".repeat(64); const SANDBOX_ID = "sandbox-id-alpha"; const SOCKET_PATH = "/run/user/1001/podman/podman.sock"; +const RUNTIME_AUTHORITY: CheckpointPortableRuntimeAuthority = { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid: 1001, + homeDir: "/home/tester", + configHome: "/home/tester/.config", + runtimeDir: "/run/user/1001", + socketPath: SOCKET_PATH, +}; const STARTUP_ARGV = [ "env", "CHAT_UI_URL=http://127.0.0.1:18789", @@ -78,12 +90,11 @@ function createPodman( let containerId = CONTAINER_ID; let containerName = `openshell-default--alpha-${sandboxId}`; let matches = [...(options.discoveredContainerIds ?? [CONTAINER_ID])]; - let socketPath = SOCKET_PATH; const podman = vi.fn((args: readonly string[], _env?: NodeJS.ProcessEnv) => { const command = args[0] === "--url" ? args.slice(2) : args; switch (command[0]) { - case "info": - return { status: 0, stdout: `${socketPath}\n` }; + case "version": + return { status: 0, stdout: JSON.stringify({ Server: { Version: "5.6.1" } }) }; case "ps": return { status: 0, stdout: matches.length > 0 ? `${matches.join("\n")}\n` : "" }; case "inspect": @@ -109,6 +120,9 @@ function createPodman( case "start": running = true; return { status: 0 }; + case "stop": + running = false; + return { status: 0 }; case "update": return { status: options.updateStatus ?? 0 }; default: @@ -145,9 +159,6 @@ function createPodman( setRunning(value: boolean) { running = value; }, - setSocketPath(value: string) { - socketPath = value; - }, }; } @@ -201,6 +212,17 @@ function resolveTarget( podman: runtime.podman, podmanSocketAuthorityDeps: socketAuthorityDeps(), hardenSocketDirectory: vi.fn(), + runtimeReadiness: { + uid: 1001, + home: RUNTIME_AUTHORITY.homeDir, + systemctl: () => ({ status: 0 }), + podmanCapture: () => ({ + status: 0, + stdout: JSON.stringify({ Server: { Version: "5.6.1" } }), + stderr: "", + }), + }, + log: vi.fn(), ...overrides, }); } @@ -214,8 +236,20 @@ function installReceipt(stateDir: string, podman: ReturnType ({ status: 0 }), + podmanCapture: () => ({ + status: 0, + stdout: JSON.stringify({ Server: { Version: "5.6.1" } }), + stderr: "", + }), + }, + log: vi.fn(), }, ); } @@ -225,12 +259,6 @@ function recoverPortableDemoSandboxLifecycle( context: Parameters[1], deps: PortableDemoLifecycleDeps = {}, ) { - const authorityDeps = deps.podman - ? { - podmanSocketAuthorityDeps: socketAuthorityDeps(), - hardenSocketDirectory: vi.fn(), - } - : {}; return recoverPortableDemoSandboxLifecycleUnchecked( sandboxName, { @@ -238,7 +266,23 @@ function recoverPortableDemoSandboxLifecycle( openshellDriver: "docker", ...context, }, - { ...authorityDeps, ...deps }, + { + platform: "linux", + podmanSocketAuthorityDeps: socketAuthorityDeps(), + hardenSocketDirectory: vi.fn(), + runtimeReadiness: { + uid: 1001, + home: RUNTIME_AUTHORITY.homeDir, + systemctl: () => ({ status: 0 }), + podmanCapture: () => ({ + status: 0, + stdout: JSON.stringify({ Server: { Version: "5.6.1" } }), + stderr: "", + }), + }, + log: vi.fn(), + ...deps, + }, ); } @@ -277,6 +321,50 @@ describe("portable demo sandbox lifecycle", () => { expect(runtime.podman).not.toHaveBeenCalled(); }); + it("stops the receipt-owned container through qualified Podman authority (#9070)", () => { + const stateDir = temporaryStateDir(); + const runtime = createPodman(); + installReceipt(stateDir, runtime.podman); + runtime.podman.mockClear(); + const beforeStop = vi.fn(); + + expect( + stopPortableDemoSandboxLifecycle( + "alpha", + { + agent: "openclaw", + gatewayName: "nemoclaw", + lifecycleGeneration: CONTAINER_ID, + openshellDriver: "docker", + }, + beforeStop, + { + platform: "linux", + podman: runtime.podman, + podmanSocketAuthorityDeps: socketAuthorityDeps(), + stateDir, + hardenSocketDirectory: vi.fn(), + runtimeReadiness: { + uid: 1001, + home: RUNTIME_AUTHORITY.homeDir, + systemctl: () => ({ status: 0 }), + podmanCapture: () => ({ + status: 0, + stdout: JSON.stringify({ Server: { Version: "5.6.1" } }), + stderr: "", + }), + }, + log: vi.fn(), + }, + ), + ).toEqual({ kind: "stopped" }); + expect(beforeStop).toHaveBeenCalledExactlyOnceWith(); + expect(runtime.podman).toHaveBeenCalledWith( + expect.arrayContaining(["stop", CONTAINER_ID]), + expect.any(Object), + ); + }); + it("removes a stale receipt for another startup contract (#8584)", () => { const stateDir = temporaryStateDir(); const runtime = createPodman(); @@ -383,10 +471,9 @@ describe("portable demo sandbox lifecycle", () => { containerId: CONTAINER_ID, dockerHost: "unix:///run/user/1001/podman/podman.sock", }); - expect(hardenSocketDirectory).toHaveBeenCalledWith(SOCKET_PATH); + expect(hardenSocketDirectory).toHaveBeenCalledWith(SOCKET_PATH, 1001); expect(socketEvents.slice(0, 2)).toEqual(["harden", "capture"]); expect(runtime.podman.mock.calls.map(([args]) => args)).toEqual([ - ["info", "--format", "{{.Host.RemoteSocket.Path}}"], [ "--url", "unix:///run/user/1001/podman/podman.sock", @@ -407,7 +494,6 @@ describe("portable demo sandbox lifecycle", () => { expect(runtime.podman.mock.calls.map(([, env]) => env)).toEqual([ expect.not.objectContaining({ CONTAINER_HOST: expect.anything() }), expect.not.objectContaining({ CONTAINER_HOST: expect.anything() }), - expect.not.objectContaining({ CONTAINER_HOST: expect.anything() }), ]); }); @@ -455,6 +541,7 @@ describe("portable demo sandbox lifecycle", () => { const receiptPath = portableDemoLifecycleInternals.receiptPath("alpha", stateDir); const receipt = JSON.parse(fs.readFileSync(receiptPath, "utf8")); delete receipt.registryGeneration; + delete receipt.runtimeAuthority; fs.writeFileSync(receiptPath, `${JSON.stringify({ ...receipt, schemaVersion: 2 })}\n`, { mode: 0o600, }); @@ -526,15 +613,6 @@ describe("portable demo sandbox lifecycle", () => { expect(() => resolveTarget(stateDir, runtime)).toThrow("is not running"); }); - it("refuses a non-local portable Podman socket before privileged exec (#8584)", () => { - const stateDir = temporaryStateDir(); - const runtime = createPodman(); - installReceipt(stateDir, runtime.podman); - runtime.setSocketPath("tcp://example.test:1234"); - - expect(() => resolveTarget(stateDir, runtime)).toThrow("socket path is invalid"); - }); - it.each([ ["foreign owner", socketAuthorityDeps({ socketUid: 2000n }), "owned by uid 2000"], ["world-writable socket", socketAuthorityDeps({ socketMode: 0o666n }), "writable by another"], @@ -545,14 +623,14 @@ describe("portable demo sandbox lifecycle", () => { ], ["writable parent", socketAuthorityDeps({ directoryMode: 0o770n }), "writable by another"], ["symlinked parent", socketAuthorityDeps({ directory: false }), "not a real directory"], - ])("refuses a %s for portable privileged exec (#8584)", (_case, authority, message) => { + ])("refuses a %s for portable privileged exec (#8584)", (_case, authority, _message) => { const stateDir = temporaryStateDir(); const runtime = createPodman(); installReceipt(stateDir, runtime.podman); expect(() => resolveTarget(stateDir, runtime, { podmanSocketAuthorityDeps: authority }), - ).toThrow(message); + ).toThrow("socket authority"); }); it("ignores ambient Podman remote selection for portable privileged exec (#8584)", () => { @@ -569,7 +647,16 @@ describe("portable demo sandbox lifecycle", () => { }, }); - expect(runtime.podman.mock.calls.map(([, env]) => env)).toEqual([{}, {}, {}]); + for (const [, commandEnv] of runtime.podman.mock.calls) { + expect(commandEnv).toMatchObject({ + HOME: RUNTIME_AUTHORITY.homeDir, + XDG_CONFIG_HOME: RUNTIME_AUTHORITY.configHome, + XDG_RUNTIME_DIR: RUNTIME_AUTHORITY.runtimeDir, + }); + expect(commandEnv).not.toHaveProperty("CONTAINER_CONNECTION"); + expect(commandEnv).not.toHaveProperty("CONTAINER_HOST"); + expect(commandEnv).not.toHaveProperty("CONTAINER_SSHKEY"); + } }); it("refuses socket replacement after portable workload inspection (#8584)", () => { @@ -601,8 +688,20 @@ describe("portable demo sandbox lifecycle", () => { platform: "linux", podman, stateDir, + runtimeAuthority: RUNTIME_AUTHORITY, podmanSocketAuthorityDeps: socketAuthorityDeps(), hardenSocketDirectory: vi.fn(), + runtimeReadiness: { + uid: 1001, + home: RUNTIME_AUTHORITY.homeDir, + systemctl: () => ({ status: 0 }), + podmanCapture: () => ({ + status: 0, + stdout: JSON.stringify({ Server: { Version: "5.6.1" } }), + stderr: "", + }), + }, + log: vi.fn(), }, ); @@ -682,7 +781,7 @@ describe("portable demo sandbox lifecycle", () => { expect(result).toEqual({ kind: "recovered" }); expect(runtime.podman).toHaveBeenCalledWith( - ["--url", "unix:///run/user/1001/podman/podman.sock", "start", CONTAINER_ID], + ["--url", `unix://${SOCKET_PATH}`, "start", CONTAINER_ID], expect.any(Object), ); expect(launchOpenshell).toHaveBeenCalledWith([ @@ -1326,13 +1425,14 @@ describe("portable demo sandbox lifecycle", () => { expect(launchHost).toHaveBeenCalledOnce(); }); - it("restarts the managed startup process once when recovery upgrades a schema-1 receipt (#8441)", () => { + it("refuses schema-1 recovery without recorded runtime authority (#9070)", () => { const stateDir = temporaryStateDir(); const runtime = createPodman(); installReceipt(stateDir, runtime.podman); const receiptPath = portableDemoLifecycleInternals.receiptPath("alpha", stateDir); const receipt = JSON.parse(fs.readFileSync(receiptPath, "utf-8")); delete receipt.registryGeneration; + delete receipt.runtimeAuthority; fs.writeFileSync( receiptPath, `${JSON.stringify({ ...receipt, schemaVersion: 1 }, null, 2)}\n`, @@ -1340,75 +1440,15 @@ describe("portable demo sandbox lifecycle", () => { mode: 0o600, }, ); - let startupRunning = true; - let gatewayRunning = true; - const launchOpenshell = vi.fn(() => { - startupRunning = true; - gatewayRunning = true; - }); - const captureOpenshell = vi.fn((args: readonly string[]) => { - const command = args.find((arg) => ["true", "pgrep", "pkill", "curl"].includes(arg)); - switch (command) { - case "true": - return { status: 0 }; - case "pgrep": - return { status: startupRunning ? 0 : 1 }; - case "pkill": - startupRunning = false; - gatewayRunning = false; - return { status: 0 }; - case "curl": - return { status: 0, stdout: gatewayRunning ? "200" : "000" }; - default: - throw new Error(`Unexpected OpenShell command: ${args.join(" ")}`); - } - }); - const deps = { - platform: "linux" as const, - stateDir, - podman: runtime.podman, - captureOpenshell, - launchOpenshell, - }; - - expect( - recoverPortableDemoSandboxLifecycle( - "alpha", - { agent: sandboxEntry().agent, gatewayName: "nemoclaw" }, - deps, - ), - ).toEqual({ kind: "recovered" }); - expect(captureOpenshell).toHaveBeenCalledWith( - [ - "sandbox", - "exec", - "-g", - "nemoclaw", - "--name", - "alpha", - "--no-tty", - "--", - "pkill", - "-TERM", - "-f", - "^(/usr/local/bin/nemoclaw-start|(bash|/bin/bash|/usr/bin/bash) /usr/local/bin/nemoclaw-start)( |$)", - ], - 5000, - ); - expect(launchOpenshell).toHaveBeenCalledOnce(); - expect(JSON.parse(fs.readFileSync(receiptPath, "utf-8"))).toMatchObject({ - schemaVersion: 3, - registryGeneration: CONTAINER_ID, - }); - - expect( + runtime.podman.mockClear(); + expect(() => recoverPortableDemoSandboxLifecycle( "alpha", { agent: sandboxEntry().agent, gatewayName: "nemoclaw" }, - deps, + { platform: "linux", stateDir, podman: runtime.podman }, ), - ).toEqual({ kind: "already-running" }); - expect(launchOpenshell).toHaveBeenCalledOnce(); + ).toThrow("predates recorded portable Podman authority"); + expect(runtime.podman).not.toHaveBeenCalled(); }); it("fails closed for a schema-1 receipt when the gateway is healthy without its managed startup process (#8441)", () => { @@ -1418,6 +1458,7 @@ describe("portable demo sandbox lifecycle", () => { const receiptPath = portableDemoLifecycleInternals.receiptPath("alpha", stateDir); const receipt = JSON.parse(fs.readFileSync(receiptPath, "utf-8")); delete receipt.registryGeneration; + delete receipt.runtimeAuthority; fs.writeFileSync( receiptPath, `${JSON.stringify({ ...receipt, schemaVersion: 1 }, null, 2)}\n`, @@ -1449,7 +1490,7 @@ describe("portable demo sandbox lifecycle", () => { launchOpenshell, }, ), - ).toThrow("agent gateway without its managed startup process"); + ).toThrow("predates recorded portable Podman authority"); expect(launchOpenshell).not.toHaveBeenCalled(); expect(JSON.parse(fs.readFileSync(receiptPath, "utf-8"))).toMatchObject({ schemaVersion: 1 }); }); diff --git a/src/lib/onboard/experimental/portable-demo-lifecycle.ts b/src/lib/onboard/experimental/portable-demo-lifecycle.ts index 3df813b1baa..64f3164b764 100644 --- a/src/lib/onboard/experimental/portable-demo-lifecycle.ts +++ b/src/lib/onboard/experimental/portable-demo-lifecycle.ts @@ -2,7 +2,6 @@ // SPDX-License-Identifier: Apache-2.0 import { spawn, spawnSync } from "node:child_process"; -import { createHash } from "node:crypto"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -11,14 +10,14 @@ import type { ContainerEngineCommandCapture } from "../../adapters/container-eng import { openRegularFileNoFollow } from "../../adapters/fs/regular-file"; import { assertPodmanSocketAuthority, - capturePodmanSocketAuthority, createPodmanContainerEngine, hardenPodmanSocketDirectory, - localPodmanEnvironment, type PodmanSocketAuthorityDeps, } from "../../adapters/podman"; import { ensureConfigDir } from "../../state/config-io"; -import { isPortableExperimentalProfile } from "../docker-driver-platform"; +import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; +import { parsePortableRuntimeAuthority } from "../../state/onboard/portable-runtime-authority"; +import { isPortableExperimentalProfile } from "./portable-profile"; import { PODMAN_MANAGED_LABEL, PODMAN_SANDBOX_CONTAINER_PREFIX, @@ -34,8 +33,19 @@ import { OLLAMA_PORT, recordUserLocalOllamaOwnership, } from "./ollama-user-local-runtime"; +import { + inspectPortablePodmanReadiness, + portablePodmanCommandEnvironment, + portablePodmanReadinessError, + type PortablePodmanReadinessDeps, + type PortablePodmanReadinessResult, +} from "./portable-runtime-readiness"; +import { + defaultPortableDemoStateDir, + inspectPortableRuntimeReceiptReadiness, + portableDemoReceiptPath, +} from "./portable-runtime-receipt-readiness"; -const RECEIPT_DIRECTORY = "portable-demo-lifecycle"; const MAX_RECEIPT_BYTES = 4096; const COMMAND_TIMEOUT_MS = 30_000; const PROBE_TIMEOUT_MS = 5_000; @@ -50,7 +60,7 @@ const CONTAINER_ID_PATTERN = /^[a-f0-9]{64}$/u; const SANDBOX_ID_PATTERN = /^[A-Za-z0-9._:-]{1,256}$/u; const OPENSHELL_RUNTIME_CA_CERT = "/etc/openshell-tls/openshell-ca.pem"; const OPENSHELL_RUNTIME_CA_BUNDLE = "/etc/openshell-tls/ca-bundle.pem"; -const CURRENT_RECEIPT_SCHEMA_VERSION = 3; +const CURRENT_RECEIPT_SCHEMA_VERSION = 4; const STARTUP_PROCESS_PATTERN = "^(/usr/local/bin/nemoclaw-start|(bash|/bin/bash|/usr/bin/bash) /usr/local/bin/nemoclaw-start)( |$)"; const SLEEP_BUFFER = new Int32Array(new SharedArrayBuffer(4)); @@ -63,12 +73,13 @@ type CommandResult = { }; interface PortableDemoLifecycleReceipt { - schemaVersion: 1 | 2 | 3; + schemaVersion: 1 | 2 | 3 | 4; sandboxName: string; sandboxId: string; containerId: string; dashboardPort: number; registryGeneration?: string; + runtimeAuthority?: CheckpointPortableRuntimeAuthority; } interface PodmanContainerInspection { @@ -90,7 +101,9 @@ export interface PortableDemoLifecycleDeps { openshellBinary?: string; podman?: (args: readonly string[], env?: NodeJS.ProcessEnv) => CommandResult; podmanSocketAuthorityDeps?: PodmanSocketAuthorityDeps; - hardenSocketDirectory?: (socketPath: string) => void; + runtimeAuthority?: CheckpointPortableRuntimeAuthority | null; + runtimeReadiness?: PortablePodmanReadinessDeps; + hardenSocketDirectory?: (socketPath: string, uid: number) => void; registryGeneration?: string; backfillRegistryGeneration?: (registryGeneration: string) => boolean; captureOpenshell?: (args: readonly string[], timeoutMs: number) => CommandResult; @@ -113,6 +126,11 @@ export type PortableDemoLifecycleRecoveryResult = | { kind: "already-running" } | { kind: "recovered" }; +export type PortableDemoLifecycleStopResult = + | { kind: "not-installed" } + | { kind: "already-stopped" } + | { kind: "stopped" }; + export interface PortableDemoLifecycleContext { agent?: string | null; gatewayName: string; @@ -121,13 +139,21 @@ export interface PortableDemoLifecycleContext { provider?: string | null; } -function defaultPodman(args: readonly string[], env: NodeJS.ProcessEnv): CommandResult { - return spawnSync("podman", [...args], { - encoding: "utf-8", - env, - stdio: ["ignore", "pipe", "pipe"], - timeout: COMMAND_TIMEOUT_MS, - }); +function defaultPodmanCapture(env: NodeJS.ProcessEnv): ContainerEngineCommandCapture { + return (_executable, args, timeoutMs) => { + const result = spawnSync("podman", [...args], { + encoding: "utf-8", + env, + stdio: ["ignore", "pipe", "pipe"], + timeout: timeoutMs, + }); + return { + status: result.status ?? 1, + stdout: String(result.stdout ?? ""), + stderr: String(result.stderr ?? ""), + ...(result.error ? { error: result.error } : {}), + }; + }; } function defaultCaptureOpenshell( @@ -209,14 +235,8 @@ function isRecord(value: unknown): value is Record { return typeof value === "object" && value !== null && !Array.isArray(value); } -function receiptPath(sandboxName: string, stateDir: string): string { - const fileName = `${createHash("sha256").update(sandboxName).digest("hex")}.json`; - return path.join(stateDir, RECEIPT_DIRECTORY, fileName); -} - -function defaultStateDir(env: NodeJS.ProcessEnv): string { - return path.join(env.HOME ?? os.homedir(), ".nemoclaw"); -} +const receiptPath = portableDemoReceiptPath; +const defaultStateDir = defaultPortableDemoStateDir; function writeReceipt(receipt: PortableDemoLifecycleReceipt, stateDir: string): void { const filePath = receiptPath(receipt.sandboxName, stateDir); @@ -243,14 +263,17 @@ function parseReceipt(value: unknown, sandboxName: string): PortableDemoLifecycl const keys = Object.keys(receipt).sort(); const expectedKeys = receipt.schemaVersion === CURRENT_RECEIPT_SCHEMA_VERSION - ? "containerId,dashboardPort,registryGeneration,sandboxId,sandboxName,schemaVersion" - : "containerId,dashboardPort,sandboxId,sandboxName,schemaVersion"; + ? "containerId,dashboardPort,registryGeneration,runtimeAuthority,sandboxId,sandboxName,schemaVersion" + : receipt.schemaVersion === 3 + ? "containerId,dashboardPort,registryGeneration,sandboxId,sandboxName,schemaVersion" + : "containerId,dashboardPort,sandboxId,sandboxName,schemaVersion"; if (keys.join(",") !== expectedKeys) { throw new Error("Portable demo lifecycle receipt fields are invalid"); } if ( (receipt.schemaVersion !== 1 && receipt.schemaVersion !== 2 && + receipt.schemaVersion !== 3 && receipt.schemaVersion !== CURRENT_RECEIPT_SCHEMA_VERSION) || receipt.sandboxName !== sandboxName || typeof receipt.containerId !== "string" || @@ -260,12 +283,20 @@ function parseReceipt(value: unknown, sandboxName: string): PortableDemoLifecycl !Number.isInteger(receipt.dashboardPort) || Number(receipt.dashboardPort) < 1024 || Number(receipt.dashboardPort) > 65535 || - (receipt.schemaVersion === CURRENT_RECEIPT_SCHEMA_VERSION && + ((receipt.schemaVersion === 3 || receipt.schemaVersion === CURRENT_RECEIPT_SCHEMA_VERSION) && (typeof receipt.registryGeneration !== "string" || - !SANDBOX_ID_PATTERN.test(receipt.registryGeneration))) + !SANDBOX_ID_PATTERN.test(receipt.registryGeneration))) || + (receipt.schemaVersion === CURRENT_RECEIPT_SCHEMA_VERSION && + parsePortableRuntimeAuthority(receipt.runtimeAuthority) === null) ) { throw new Error("Portable demo lifecycle receipt values are invalid"); } + if (receipt.schemaVersion === CURRENT_RECEIPT_SCHEMA_VERSION) { + return { + ...(receipt as unknown as PortableDemoLifecycleReceipt), + runtimeAuthority: parsePortableRuntimeAuthority(receipt.runtimeAuthority)!, + }; + } return receipt as unknown as PortableDemoLifecycleReceipt; } @@ -277,8 +308,8 @@ function requireCurrentRegistryGeneration( // container ID may claim a missing registry generation only after exact // local runtime validation; an existing generation must already match. const receiptGeneration = - receipt.schemaVersion === 3 ? receipt.registryGeneration : receipt.containerId; - if (registryGeneration === undefined && receipt.schemaVersion !== 3) return true; + receipt.schemaVersion >= 3 ? receipt.registryGeneration : receipt.containerId; + if (registryGeneration === undefined && receipt.schemaVersion < 3) return true; if (receiptGeneration !== registryGeneration) { throw new Error( `Portable demo lifecycle receipt for sandbox '${receipt.sandboxName}' does not belong to the current registry generation`, @@ -422,23 +453,6 @@ function discoverPodmanContainer( return inspectPodmanContainer(matches[0]!, sandboxName, podman); } -function podmanSocketPath( - podman: NonNullable, - env: NodeJS.ProcessEnv, -): string { - const result = podman(["info", "--format", "{{.Host.RemoteSocket.Path}}"], env); - requireCommand(result, "Resolving the portable Podman socket"); - const socket = String(result.stdout ?? "").trim(); - if (/[\u0000-\u001f\u007f-\u009f]/u.test(socket)) { - throw new Error("The portable Podman socket path is invalid"); - } - const socketPath = socket.startsWith("unix://") ? socket.slice("unix://".length) : socket; - if (!path.posix.isAbsolute(socketPath)) { - throw new Error("The portable Podman socket path is invalid"); - } - return socketPath; -} - function podmanCapture( podman: NonNullable, env: NodeJS.ProcessEnv, @@ -454,22 +468,35 @@ function podmanCapture( }; } -function qualifiedPodmanAuthority(commandEnv: NodeJS.ProcessEnv, deps: PortableDemoLifecycleDeps) { - const podman = deps.podman ?? ((args, env = commandEnv) => defaultPodman(args, env)); - const podmanEnv = localPodmanEnvironment(commandEnv); - const socketPath = podmanSocketPath(podman, podmanEnv); - (deps.hardenSocketDirectory ?? hardenPodmanSocketDirectory)(socketPath); - const socketAuthority = capturePodmanSocketAuthority(socketPath, deps.podmanSocketAuthorityDeps); +function qualifiedPodmanAuthority( + receipt: PortableDemoLifecycleReceipt, + commandEnv: NodeJS.ProcessEnv, + deps: PortableDemoLifecycleDeps, +) { + const readiness = inspectReceiptRuntimeReadiness(receipt, commandEnv, deps); + if (!readiness.ok) throw portablePodmanReadinessError(readiness); + if (!receipt.runtimeAuthority) { + throw new Error("Portable Podman readiness did not retain its recorded authority"); + } + const podmanEnv = portablePodmanCommandEnvironment(receipt.runtimeAuthority, commandEnv); + const capture = deps.podman + ? podmanCapture(deps.podman, podmanEnv) + : defaultPodmanCapture(podmanEnv); + (deps.log ?? console.log)( + ` Portable Podman readiness: ${readiness.timing.mode}; activation ${String(readiness.timing.activationMs)} ms; API ${String(readiness.timing.apiMs)} ms; total ${String(readiness.timing.totalMs)} ms.`, + ); + const socketAuthority = readiness.authority; const provider = createPodmanContainerEngine({ operation: "sandbox-lifecycle", socketAuthority, authorityDeps: deps.podmanSocketAuthorityDeps, - ...(deps.podman ? { capture: podmanCapture(podman, podmanEnv) } : {}), + capture, + assertAuthority: deps.runtimeReadiness?.assertSocketAuthority, }); return { assertRuntimeAuthority: () => assertPodmanSocketAuthority(socketAuthority, deps.podmanSocketAuthorityDeps), - dockerHost: `unix://${socketAuthority.socketPath}`, + dockerHost: readiness.dockerHost, podman: (args: readonly string[]) => provider.capture(args, COMMAND_TIMEOUT_MS), }; } @@ -490,29 +517,48 @@ function requireReceiptOwnedInspection( } } -function backfillLegacyReceiptGeneration( +function inspectReceiptRuntimeReadiness( receipt: PortableDemoLifecycleReceipt, - stateDir: string, - backfillRequired: boolean, + commandEnv: NodeJS.ProcessEnv, deps: PortableDemoLifecycleDeps, -): PortableDemoLifecycleReceipt { - if (receipt.schemaVersion === 3) return receipt; - if ( - backfillRequired && - (!deps.backfillRegistryGeneration || !deps.backfillRegistryGeneration(receipt.containerId)) - ) { - throw new Error( - `Portable demo lifecycle receipt for sandbox '${receipt.sandboxName}' could not claim the current registry generation`, - ); +): PortablePodmanReadinessResult { + if (receipt.schemaVersion !== CURRENT_RECEIPT_SCHEMA_VERSION || !receipt.runtimeAuthority) { + return { + ok: false, + stage: "socket authority", + detail: + "The lifecycle receipt predates recorded portable Podman authority; rerun onboarding.", + timing: { mode: "warm", activationMs: 0, apiMs: 0, totalMs: 0 }, + }; } - if (receipt.schemaVersion === 1) return receipt; - const migrated: PortableDemoLifecycleReceipt = { - ...receipt, - schemaVersion: CURRENT_RECEIPT_SCHEMA_VERSION, - registryGeneration: receipt.containerId, - }; - writeReceipt(migrated, stateDir); - return migrated; + const podmanEnv = portablePodmanCommandEnvironment(receipt.runtimeAuthority, commandEnv); + const capture = deps.podman + ? podmanCapture(deps.podman, podmanEnv) + : defaultPodmanCapture(podmanEnv); + return inspectPortablePodmanReadiness(receipt.runtimeAuthority, { + platform: deps.platform, + env: commandEnv, + socketAuthorityDeps: deps.podmanSocketAuthorityDeps, + hardenSocketDirectory: deps.hardenSocketDirectory ?? hardenPodmanSocketDirectory, + podmanCapture: capture, + ...deps.runtimeReadiness, + }); +} + +/** Inspect the receipt-owned portable runtime, or return null for an ordinary sandbox. */ +export function inspectPortableDemoRuntimeReadiness( + sandboxName: string, + deps: PortableDemoLifecycleDeps = {}, +): PortablePodmanReadinessResult | null { + return inspectPortableRuntimeReceiptReadiness(sandboxName, deps); +} + +/** Whether the sandbox has a receipt that requires authority-bound lifecycle operations. */ +export function hasPortableDemoSandboxLifecycleReceipt( + sandboxName: string, + env: NodeJS.ProcessEnv = process.env, +): boolean { + return loadReceipt(sandboxName, defaultStateDir(env)) !== null; } /** Resolve the receipt-owned portable container for a host-side privileged exec. */ @@ -527,15 +573,14 @@ export function resolvePortableDemoPrivilegedExecTarget( if ((deps.platform ?? process.platform) !== "linux") { throw new Error("Portable demo lifecycle receipt is only valid on Linux"); } - const backfillRequired = requireCurrentRegistryGeneration(receipt, deps.registryGeneration); - const authority = qualifiedPodmanAuthority(commandEnv, deps); + requireCurrentRegistryGeneration(receipt, deps.registryGeneration); + const authority = qualifiedPodmanAuthority(receipt, commandEnv, deps); const inspection = discoverPodmanContainer(sandboxName, authority.podman); requireReceiptOwnedInspection(receipt, inspection); if (!inspection.running) { throw new Error(`Portable sandbox '${sandboxName}' is not running`); } authority.assertRuntimeAuthority(); - backfillLegacyReceiptGeneration(receipt, stateDir, backfillRequired, deps); return { assertRuntimeAuthority: authority.assertRuntimeAuthority, containerId: inspection.containerId, @@ -804,8 +849,37 @@ export function installPortableDemoSandboxLifecycle( throw new Error("Portable demo lifecycle requires Linux"); } const commandEnv = deps.env ?? env; - const authority = qualifiedPodmanAuthority(commandEnv, deps); - const inspection = discoverPodmanContainer(sandboxName, authority.podman); + const runtimeAuthority = deps.runtimeAuthority; + if (!runtimeAuthority || !parsePortableRuntimeAuthority(runtimeAuthority)) { + throw new Error( + "Portable demo lifecycle requires the checkpoint-owned Podman runtime authority", + ); + } + const podmanEnv = portablePodmanCommandEnvironment(runtimeAuthority, commandEnv); + const readinessCapture = deps.podman + ? podmanCapture(deps.podman, podmanEnv) + : defaultPodmanCapture(podmanEnv); + const readiness = inspectPortablePodmanReadiness(runtimeAuthority, { + platform: deps.platform, + env: commandEnv, + socketAuthorityDeps: deps.podmanSocketAuthorityDeps, + hardenSocketDirectory: deps.hardenSocketDirectory ?? hardenPodmanSocketDirectory, + podmanCapture: readinessCapture, + ...deps.runtimeReadiness, + }); + if (!readiness.ok) throw portablePodmanReadinessError(readiness); + (deps.log ?? console.log)( + ` Portable Podman readiness: ${readiness.timing.mode}; activation ${String(readiness.timing.activationMs)} ms; API ${String(readiness.timing.apiMs)} ms; total ${String(readiness.timing.totalMs)} ms.`, + ); + const provider = createPodmanContainerEngine({ + operation: "sandbox-lifecycle", + socketAuthority: readiness.authority, + authorityDeps: deps.podmanSocketAuthorityDeps, + capture: readinessCapture, + assertAuthority: deps.runtimeReadiness?.assertSocketAuthority, + }); + const podman = (args: readonly string[]) => provider.capture(args, COMMAND_TIMEOUT_MS); + const inspection = discoverPodmanContainer(sandboxName, podman); const registryGeneration = deps.registryGeneration ?? inspection.containerId; if (!SANDBOX_ID_PATTERN.test(registryGeneration)) { throw new Error("Portable demo lifecycle registry generation is invalid"); @@ -817,13 +891,12 @@ export function installPortableDemoSandboxLifecycle( containerId: inspection.containerId, dashboardPort: parseDashboardPort(createdStartupArgv, sandboxName), registryGeneration, + runtimeAuthority, }; - authority.assertRuntimeAuthority(); requireCommand( - authority.podman(["update", "--restart=unless-stopped", inspection.containerId]), + podman(["update", "--restart=unless-stopped", inspection.containerId]), `Setting the portable restart policy for sandbox '${sandboxName}'`, ); - authority.assertRuntimeAuthority(); writeReceipt(receipt, stateDir); return registryGeneration; } @@ -849,20 +922,15 @@ export function recoverPortableDemoSandboxLifecycle( if (context.openshellDriver !== "docker") return { kind: "not-installed" }; const commandEnv = deps.env ?? process.env; const stateDir = deps.stateDir ?? defaultStateDir(commandEnv); - let receipt = loadReceipt(sandboxName, stateDir); + const receipt = loadReceipt(sandboxName, stateDir); if (!receipt) return { kind: "not-installed" }; if ((deps.platform ?? process.platform) !== "linux") { throw new Error("Portable demo lifecycle receipt is only valid on Linux"); } - const backfillRequired = requireCurrentRegistryGeneration(receipt, context.lifecycleGeneration); - const authority = qualifiedPodmanAuthority(commandEnv, deps); - if (backfillRequired || receipt.schemaVersion === 2) { - const migrationInspection = discoverPodmanContainer(sandboxName, authority.podman); - requireReceiptOwnedInspection(receipt, migrationInspection); - authority.assertRuntimeAuthority(); - receipt = backfillLegacyReceiptGeneration(receipt, stateDir, backfillRequired, deps); - } - const initialInspection = authority.podman(["inspect", receipt.containerId]); + requireCurrentRegistryGeneration(receipt, context.lifecycleGeneration); + const authority = qualifiedPodmanAuthority(receipt, commandEnv, deps); + const podman = authority.podman; + const initialInspection = podman(["inspect", receipt.containerId]); if (isMissingPodmanContainer(initialInspection)) { removeReceipt(sandboxName, stateDir); return { kind: "not-installed" }; @@ -870,7 +938,7 @@ export function recoverPortableDemoSandboxLifecycle( let inspection = inspectPodmanContainer( receipt.containerId, sandboxName, - authority.podman, + podman, initialInspection, ); if (inspection.sandboxId !== receipt.sandboxId) { @@ -879,13 +947,11 @@ export function recoverPortableDemoSandboxLifecycle( ); } if (!inspection.running) { - authority.assertRuntimeAuthority(); requireCommand( - authority.podman(["start", receipt.containerId]), + podman(["start", receipt.containerId]), `Starting portable sandbox '${sandboxName}'`, ); - authority.assertRuntimeAuthority(); - inspection = inspectPodmanContainer(receipt.containerId, sandboxName, authority.podman); + inspection = inspectPodmanContainer(receipt.containerId, sandboxName, podman); if (!inspection.running) { throw new Error(`Portable sandbox '${sandboxName}' did not enter the running state`); } @@ -976,18 +1042,54 @@ export function recoverPortableDemoSandboxLifecycle( `Portable sandbox '${sandboxName}' startup did not start its agent gateway; inspect /tmp/nemoclaw-start.log inside the sandbox`, ); } - if (refreshStartup) { - writeReceipt( - { - ...receipt, - schemaVersion: CURRENT_RECEIPT_SCHEMA_VERSION, - registryGeneration: context.lifecycleGeneration ?? receipt.containerId, - }, - deps.stateDir ?? defaultStateDir(commandEnv), - ); - } (deps.log ?? console.log)(` Portable demo lifecycle recovered sandbox '${sandboxName}'.`); return { kind: "recovered" }; } +/** Stop the exact receipt-owned portable container through its recorded Podman authority. */ +export function stopPortableDemoSandboxLifecycle( + sandboxName: string, + context: PortableDemoLifecycleContext, + beforeStop: () => void, + deps: PortableDemoLifecycleDeps = {}, +): PortableDemoLifecycleStopResult { + if ((context.agent ?? "openclaw") !== "openclaw") return { kind: "not-installed" }; + if (context.openshellDriver !== "docker") return { kind: "not-installed" }; + const commandEnv = deps.env ?? process.env; + const stateDir = deps.stateDir ?? defaultStateDir(commandEnv); + const receipt = loadReceipt(sandboxName, stateDir); + if (!receipt) return { kind: "not-installed" }; + if ((deps.platform ?? process.platform) !== "linux") { + throw new Error("Portable demo lifecycle receipt is only valid on Linux"); + } + requireCurrentRegistryGeneration(receipt, context.lifecycleGeneration); + const authority = qualifiedPodmanAuthority(receipt, commandEnv, deps); + const initialInspection = authority.podman(["inspect", receipt.containerId]); + if (isMissingPodmanContainer(initialInspection)) { + throw new Error( + `Portable sandbox '${sandboxName}' no longer has its recorded Podman container`, + ); + } + const inspection = inspectPodmanContainer( + receipt.containerId, + sandboxName, + authority.podman, + initialInspection, + ); + requireReceiptOwnedInspection(receipt, inspection); + if (!inspection.running) return { kind: "already-stopped" }; + + beforeStop(); + requireCommand( + authority.podman(["stop", receipt.containerId]), + `Stopping portable sandbox '${sandboxName}'`, + ); + const stopped = inspectPodmanContainer(receipt.containerId, sandboxName, authority.podman); + requireReceiptOwnedInspection(receipt, stopped); + if (stopped.running) { + throw new Error(`Portable sandbox '${sandboxName}' did not enter the stopped state`); + } + return { kind: "stopped" }; +} + export const portableDemoLifecycleInternals = { receiptPath }; diff --git a/src/lib/onboard/experimental/portable-host-preparation.test.ts b/src/lib/onboard/experimental/portable-host-preparation.test.ts index f6a1ecf7ed7..007b319f0bc 100644 --- a/src/lib/onboard/experimental/portable-host-preparation.test.ts +++ b/src/lib/onboard/experimental/portable-host-preparation.test.ts @@ -12,7 +12,7 @@ import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-che import { createPortableOnboardEnvironmentScope } from "../session-bootstrap"; import { portableHostPreparationInternals, - preparePortableExperimentalHost, + preparePortableExperimentalHost as preparePortableExperimentalHostUnchecked, } from "./portable-host-preparation"; type SpawnResult = ReturnType; @@ -48,6 +48,38 @@ function socketAuthority(socketPath = "/run/user/1001/podman/podman.sock"): Podm }; } +function successfulReadiness(home: string) { + return { + uid: 1001, + home, + hardenSocketDirectory: vi.fn(), + captureSocketAuthority: (socketPath: string) => socketAuthority(socketPath), + assertSocketAuthority: vi.fn(), + podmanCapture: () => ({ + status: 0, + stdout: JSON.stringify({ Server: { Version: "5.6.1" } }), + stderr: "", + }), + }; +} + +function preparePortableExperimentalHost( + env: NodeJS.ProcessEnv, + deps: Parameters[1] = {}, + expectedAuthority?: CheckpointPortableRuntimeAuthority | null, +) { + return preparePortableExperimentalHostUnchecked( + env, + { + ...deps, + runtimeReadiness: + deps.runtimeReadiness ?? + successfulReadiness(deps.home ?? expectedAuthority?.homeDir ?? os.userInfo().homedir), + }, + expectedAuthority, + ); +} + describe("preparePortableExperimentalHost", () => { const tempDirs: string[] = []; @@ -71,9 +103,9 @@ describe("preparePortableExperimentalHost", () => { it("prepares the rootless socket and managed loopback registry deterministically", () => { const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-portable-")); tempDirs.push(home); - const systemctl = vi.fn<(args: readonly string[], env: NodeJS.ProcessEnv) => SpawnResult>(() => - result(), - ); + const systemctl = vi.fn< + (args: readonly string[], env: NodeJS.ProcessEnv, timeoutMs?: number) => SpawnResult + >(() => result()); const docker = vi .fn<(args: readonly string[], env: NodeJS.ProcessEnv) => SpawnResult>() .mockReturnValueOnce(result()) // --version probe: docker-compatible CLI present @@ -95,16 +127,20 @@ describe("preparePortableExperimentalHost", () => { NEMOCLAW_EXPERIMENTAL_PROFILE: "portable", }; - preparePortableExperimentalHost(env, { - platform: "linux", - home, - uid: 1001, - systemctl, - podman, - docker, - hardenSocketDirectory, - validateConfigAuthority: vi.fn(), - }); + preparePortableExperimentalHost( + env, + { + platform: "linux", + home, + uid: 1001, + systemctl, + podman, + docker, + hardenSocketDirectory, + validateConfigAuthority: vi.fn(), + }, + runtimeAuthority(home, "/run/user/1001/custom/podman.sock"), + ); expect(env).toMatchObject({ CONTAINERS_CONF: path.join(home, ".config/nemoclaw/portable/containers.conf"), @@ -121,17 +157,10 @@ describe("preparePortableExperimentalHost", () => { `CONTAINERS_CONF=${path.join(home, ".config/nemoclaw/portable/containers.conf")}`, ], ["--user", "try-restart", "podman.service"], - ["--user", "enable", "--now", "podman.socket"], + ["--user", "is-active", "--quiet", "podman.service"], ]); - expect(podman).toHaveBeenCalledTimes(2); - for (const [args, commandEnv] of podman.mock.calls) { - expect(args).toEqual(["info", "--format", "{{.Host.RemoteSocket.Path}}"]); - expect(commandEnv).not.toMatchObject({ - CONTAINER_CONNECTION: expect.anything(), - CONTAINER_HOST: expect.anything(), - CONTAINER_SSHKEY: expect.anything(), - }); - } + expect(systemctl.mock.calls[2]?.[2]).toBe(10_000); + expect(podman).not.toHaveBeenCalled(); for (const [, commandEnv] of docker.mock.calls) { expect(commandEnv).not.toHaveProperty("CONTAINER_CONNECTION"); expect(commandEnv).not.toHaveProperty("CONTAINER_HOST"); @@ -170,6 +199,48 @@ describe("preparePortableExperimentalHost", () => { expect(fs.statSync(containersConf).mode & 0o777).toBe(0o600); }); + it("forwards readiness deadlines to injected host command adapters (#9070)", () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-portable-")); + tempDirs.push(home); + const systemctl = vi.fn< + (args: readonly string[], env: NodeJS.ProcessEnv, timeoutMs?: number) => SpawnResult + >(() => result()); + const podman = vi.fn((_args: readonly string[], _env: NodeJS.ProcessEnv, _timeoutMs?: number) => + result(0, JSON.stringify({ Server: { Version: "5.6.1" } })), + ); + const docker = vi + .fn<(args: readonly string[], env: NodeJS.ProcessEnv) => SpawnResult>() + .mockReturnValueOnce(result()) + .mockReturnValueOnce(result(1)) + .mockReturnValueOnce(result()); + + preparePortableExperimentalHost( + { NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }, + { + platform: "linux", + home, + uid: 1001, + systemctl, + podman, + docker, + captureSocketAuthority: (socketPath) => socketAuthority(socketPath), + validateConfigAuthority: vi.fn(), + runtimeReadiness: { + uid: 1001, + home, + now: () => 0, + hardenSocketDirectory: vi.fn(), + captureSocketAuthority: (socketPath) => socketAuthority(socketPath), + assertSocketAuthority: vi.fn(), + }, + }, + runtimeAuthority(home), + ); + + expect(systemctl.mock.calls[2]?.[2]).toBe(10_000); + expect(podman.mock.calls[0]?.[2]).toBe(10_000); + }); + it("keeps the portable firewall driver in the Podman default search path (#8441)", () => { const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-portable-")); tempDirs.push(home); @@ -264,7 +335,7 @@ describe("preparePortableExperimentalHost", () => { expect(docker).toHaveBeenCalledTimes(2); }); - it("fails closed when Podman does not report an absolute local socket", () => { + it("fails closed when the recorded authority is not an absolute local socket", () => { const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-portable-")); tempDirs.push(home); @@ -276,12 +347,13 @@ describe("preparePortableExperimentalHost", () => { home, uid: 1001, systemctl: () => result(), - podman: () => result(0, "tcp://127.0.0.1:1234"), + podman: vi.fn(), docker: vi.fn(), validateConfigAuthority: vi.fn(), }, + runtimeAuthority(home, "tcp://127.0.0.1:1234"), ), - ).toThrow(/invalid socket path/); + ).toThrow(/socket path/); }); it("names podman-docker and creates the registry only after a successful retry (#8453)", () => { @@ -480,7 +552,7 @@ describe("preparePortableExperimentalHost", () => { expect(fs.existsSync(path.join(home, ".config"))).toBe(false); }); - it("rejects a missing mismatched endpoint before portable effects (#9083)", () => { + it("keeps a missing recorded endpoint bound through activation (#9070)", () => { const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-portable-")); tempDirs.push(home); const staleSocket = "/run/user/1001/stale/podman.sock"; @@ -518,34 +590,24 @@ describe("preparePortableExperimentalHost", () => { }, runtimeAuthority(home, staleSocket), ), - ).toThrow(/socket path does not match the onboarding checkpoint/); + ).toThrow(/socket authority/); expect(captureSocketAuthority).toHaveBeenCalledWith(staleSocket, 1001); - expect(podman).toHaveBeenCalledOnce(); - expect(podman).toHaveBeenCalledWith( - ["info", "--format", "{{.Host.RemoteSocket.Path}}"], - expect.not.objectContaining({ - CONTAINER_CONNECTION: expect.anything(), - CONTAINER_HOST: expect.anything(), - CONTAINER_SSHKEY: expect.anything(), - }), - ); - expect(systemctl).not.toHaveBeenCalled(); + expect(podman).not.toHaveBeenCalled(); + expect(systemctl).toHaveBeenCalledTimes(3); expect(docker).not.toHaveBeenCalled(); - expect(hardenSocketDirectory).not.toHaveBeenCalled(); + expect(hardenSocketDirectory).toHaveBeenCalledWith(staleSocket, 1001); expect(qualifyPodman).not.toHaveBeenCalled(); - expect(env.NETAVARK_FW).toBeUndefined(); - expect(env.CONTAINERS_CONF).toBeUndefined(); - expect(fs.existsSync(path.join(home, ".config"))).toBe(false); + expect(env.NETAVARK_FW).toBe("iptables"); + expect(env.CONTAINERS_CONF).toBe(path.join(home, ".config/nemoclaw/portable/containers.conf")); + expect(fs.existsSync(path.join(home, ".config"))).toBe(true); }); - it("stops a failed admission discovery before portable effects (#9083)", () => { + it("stops an authority outside the current-user runtime before portable effects (#9083)", () => { const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-portable-")); tempDirs.push(home); const systemctl = vi.fn(() => result()); const docker = vi.fn(() => result()); - const podman = vi.fn( - () => ({ status: 1, stdout: "", stderr: "Podman discovery failed" }) as SpawnResult, - ); + const podman = vi.fn(); expect(() => preparePortableExperimentalHost( @@ -562,15 +624,15 @@ describe("preparePortableExperimentalHost", () => { }), validateConfigAuthority: vi.fn(), }, - runtimeAuthority(home), + runtimeAuthority(home, "/run/user/2002/podman/podman.sock"), ), - ).toThrow(/Resolving the rootless Podman API socket failed: Podman discovery failed/); + ).toThrow(/outside the current user runtime directory/); expect(systemctl).not.toHaveBeenCalled(); expect(docker).not.toHaveBeenCalled(); expect(fs.existsSync(path.join(home, ".config"))).toBe(false); }); - it("rejects a post-activation endpoint mismatch before qualification or registry work (#9083)", () => { + it("rejects an unavailable recorded endpoint after activation (#9070)", () => { const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-portable-")); tempDirs.push(home); const expectedSocket = "/run/user/1001/custom/podman.sock"; @@ -602,10 +664,10 @@ describe("preparePortableExperimentalHost", () => { }, runtimeAuthority(home, expectedSocket), ), - ).toThrow(/socket path does not match the onboarding checkpoint/); + ).toThrow(/socket authority/); expect(systemctl).toHaveBeenCalledTimes(3); - expect(podman).toHaveBeenCalledTimes(2); - expect(hardenSocketDirectory).not.toHaveBeenCalled(); + expect(podman).not.toHaveBeenCalled(); + expect(hardenSocketDirectory).toHaveBeenCalledWith(expectedSocket, 1001); expect(qualifyPodman).not.toHaveBeenCalled(); expect(docker).not.toHaveBeenCalled(); }); @@ -743,7 +805,7 @@ describe("preparePortableExperimentalHost", () => { socketPath: null, uid: process.getuid?.() ?? -1, }), - ).toThrow(/not a real directory/); + ).toThrow(/not a real directory|unsafe write permissions/); }); it("rejects writable portable configuration authority (#9035)", () => { diff --git a/src/lib/onboard/experimental/portable-host-preparation.ts b/src/lib/onboard/experimental/portable-host-preparation.ts index 7d233a99581..c6b8a8950b9 100644 --- a/src/lib/onboard/experimental/portable-host-preparation.ts +++ b/src/lib/onboard/experimental/portable-host-preparation.ts @@ -11,15 +11,18 @@ import { openRegularFileNoFollow } from "../../adapters/fs/regular-file"; import { assertPodmanSocketAuthority, capturePodmanSocketAuthority, - createPodmanContainerEngine, hardenPodmanSocketDirectory, - localPodmanEnvironment, type PodmanSocketAuthority, } from "../../adapters/podman"; import { ensureConfigDir } from "../../state/config-io"; import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; import { isPortableExperimentalProfile, PORTABLE_LOCAL_REGISTRY } from "../docker-driver-platform"; -import { qualifyPodmanHost } from "../runtime-provider/podman-preflight"; +import { + inspectPortablePodmanReadiness, + portablePodmanCommandEnvironment, + portablePodmanReadinessError, + type PortablePodmanReadinessDeps, +} from "./portable-runtime-readiness"; const REGISTRY_CONTAINER = "nemoclaw-portable-registry"; const REGISTRY_LABEL = "com.nvidia.nemoclaw.portable=1"; @@ -50,13 +53,22 @@ export interface PortableHostPreparationDeps { platform?: NodeJS.Platform; home?: string; uid?: number; - systemctl?: (args: readonly string[], env: NodeJS.ProcessEnv) => SpawnResult; - podman?: (args: readonly string[], env: NodeJS.ProcessEnv) => SpawnResult; + systemctl?: ( + args: readonly string[], + env: NodeJS.ProcessEnv, + timeoutMs?: number, + ) => SpawnResult; + podman?: ( + args: readonly string[], + env: NodeJS.ProcessEnv, + timeoutMs?: number, + ) => SpawnResult; docker?: (args: readonly string[], env: NodeJS.ProcessEnv) => SpawnResult; hardenSocketDirectory?: (socketPath: string, uid: number) => void; captureSocketAuthority?: (socketPath: string, uid: number) => PodmanSocketAuthority; assertSocketAuthority?: (authority: PodmanSocketAuthority) => void; qualifyPodman?: (authority: PodmanSocketAuthority) => void; + runtimeReadiness?: PortablePodmanReadinessDeps; validateConfigAuthority?: (input: { homeDir: string; configHome: string; @@ -392,17 +404,13 @@ export function preparePortableExperimentalHost( const podman = deps.podman ?? - ((args, childEnv) => + ((args, childEnv, timeoutMs = HOST_COMMAND_TIMEOUT_MS) => spawnSync("podman", [...args], { encoding: "utf-8", env: childEnv, - timeout: HOST_COMMAND_TIMEOUT_MS, + timeout: timeoutMs, })); - const discoverSocket = (childEnv: NodeJS.ProcessEnv): string => - resolvePodmanDockerHost( - podman(["info", "--format", "{{.Host.RemoteSocket.Path}}"], childEnv), - ).slice("unix://".length); - const admissionSocketPath = discoverSocket(localPodmanEnvironment(env)); + const admissionSocketPath = expectedSocketPath ?? path.join(runtimeDir, "podman", "podman.sock"); assertSocketInsideRuntime(runtimeDir, admissionSocketPath); if (expectedAuthority && admissionSocketPath !== expectedAuthority.socketPath) { throw new Error("Portable Podman socket path does not match the onboarding checkpoint."); @@ -410,14 +418,25 @@ export function preparePortableExperimentalHost( env.NETAVARK_FW = "iptables"; env.CONTAINERS_CONF = writePortableRuntimeConfig(configHome); + const runtimeAuthority: CheckpointPortableRuntimeAuthority = { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid: Number(uid), + homeDir: home, + configHome, + runtimeDir, + socketPath: admissionSocketPath, + }; + const serviceEnv = portablePodmanCommandEnvironment(runtimeAuthority, env); const systemctl = deps.systemctl ?? - ((args, childEnv) => + ((args, childEnv, timeoutMs = HOST_COMMAND_TIMEOUT_MS) => spawnSync("systemctl", [...args], { encoding: "utf-8", env: childEnv, - timeout: HOST_COMMAND_TIMEOUT_MS, + timeout: timeoutMs, })); requireCommand( systemctl( @@ -427,42 +446,67 @@ export function preparePortableExperimentalHost( "NETAVARK_FW=iptables", `CONTAINERS_CONF=${env.CONTAINERS_CONF}`, ], - env, + serviceEnv, ), "Configuring the rootless container service environment", ); requireCommand( - systemctl(["--user", "try-restart", "podman.service"], env), + systemctl(["--user", "try-restart", "podman.service"], serviceEnv), "Refreshing the rootless container service", ); - requireCommand( - systemctl(["--user", "enable", "--now", "podman.socket"], env), - "Starting the rootless container socket", + const podmanEnv = portablePodmanCommandEnvironment(runtimeAuthority, env); + const readiness = inspectPortablePodmanReadiness(runtimeAuthority, { + ...deps.runtimeReadiness, + platform: deps.platform, + uid: Number(uid), + home, + env, + systemctl: (args, childEnv, timeoutMs) => systemctl(args, childEnv, timeoutMs), + hardenSocketDirectory: + deps.hardenSocketDirectory ?? + deps.runtimeReadiness?.hardenSocketDirectory ?? + hardenPodmanSocketDirectory, + captureSocketAuthority: deps.captureSocketAuthority + ? (socketPath) => deps.captureSocketAuthority!(socketPath, Number(uid)) + : (deps.runtimeReadiness?.captureSocketAuthority ?? capturePodmanSocketAuthority), + assertSocketAuthority: + deps.assertSocketAuthority ?? + deps.runtimeReadiness?.assertSocketAuthority ?? + assertPodmanSocketAuthority, + podmanCapture: deps.runtimeReadiness?.podmanCapture + ? deps.runtimeReadiness.podmanCapture + : deps.podman + ? (_executable, args, timeoutMs) => { + const result = podman(args, podmanEnv, timeoutMs); + return { + status: result.status ?? 1, + stdout: String(result.stdout ?? ""), + stderr: String(result.stderr ?? ""), + ...(result.error ? { error: result.error } : {}), + }; + } + : (_executable, args, timeoutMs) => { + const result = spawnSync("podman", [...args], { + encoding: "utf-8", + env: podmanEnv, + stdio: ["ignore", "pipe", "pipe"], + timeout: timeoutMs, + }); + return { + status: result.status ?? 1, + stdout: String(result.stdout ?? ""), + stderr: String(result.stderr ?? ""), + ...(result.error ? { error: result.error } : {}), + }; + }, + }); + if (!readiness.ok) throw portablePodmanReadinessError(readiness); + const socketAuthority = readiness.authority; + deps.qualifyPodman?.(socketAuthority); + const dockerHost = readiness.dockerHost; + console.log( + ` Portable Podman readiness: ${readiness.timing.mode}; activation ${String(readiness.timing.activationMs)} ms; API ${String(readiness.timing.apiMs)} ms; total ${String(readiness.timing.totalMs)} ms.`, ); - - const podmanEnv = localPodmanEnvironment(env); - const socketPath = discoverSocket(podmanEnv); - const dockerHost = `unix://${socketPath}`; - if (expectedAuthority && socketPath !== expectedAuthority.socketPath) { - throw new Error("Portable Podman socket path does not match the onboarding checkpoint."); - } - assertSocketInsideRuntime(runtimeDir, socketPath); - (deps.hardenSocketDirectory ?? hardenPodmanSocketDirectory)(socketPath, Number(uid)); - const socketAuthority = deps.captureSocketAuthority - ? deps.captureSocketAuthority(socketPath, Number(uid)) - : deps.hardenSocketDirectory - ? null - : capturePodmanSocketAuthority(socketPath, { uid: Number(uid) }); - if (socketAuthority) { - ( - deps.qualifyPodman ?? - ((authority) => { - qualifyPodmanHost( - createPodmanContainerEngine({ operation: "host-doctor", socketAuthority: authority }), - ); - }) - )(socketAuthority); - } env.DOCKER_HOST = dockerHost; podmanEnv.DOCKER_HOST = dockerHost; @@ -477,19 +521,14 @@ export function preparePortableExperimentalHost( requireDockerCompatibleCli(docker, podmanEnv); ensureRegistryContainer(podmanEnv, docker); if (socketAuthority) { - (deps.assertSocketAuthority ?? assertPodmanSocketAuthority)(socketAuthority); + ( + deps.assertSocketAuthority ?? + deps.runtimeReadiness?.assertSocketAuthority ?? + assertPodmanSocketAuthority + )(socketAuthority); } return { - authority: { - schemaVersion: 1, - kind: "podman", - ownership: "current-user", - uid: Number(uid), - homeDir: home, - configHome, - runtimeDir, - socketPath, - }, + authority: runtimeAuthority, socketAuthority, containersConf: env.CONTAINERS_CONF, }; diff --git a/src/lib/onboard/experimental/portable-runtime-readiness.test.ts b/src/lib/onboard/experimental/portable-runtime-readiness.test.ts new file mode 100644 index 00000000000..be913c3dc7e --- /dev/null +++ b/src/lib/onboard/experimental/portable-runtime-readiness.test.ts @@ -0,0 +1,361 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; + +import type { ContainerEngineCommandCapture } from "../../adapters/container-engine"; +import type { PodmanSocketAuthority } from "../../adapters/podman"; +import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; +import { + DEFAULT_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS, + inspectPortablePodmanReadiness, + PORTABLE_PODMAN_STARTUP_TIMEOUT_ENV, + portablePodmanReadinessError, + resolvePortablePodmanStartupTimeout, +} from "./portable-runtime-readiness"; + +const AUTHORITY: CheckpointPortableRuntimeAuthority = { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid: 1001, + homeDir: "/home/tester", + configHome: "/home/tester/.config", + runtimeDir: "/run/user/1001", + socketPath: "/run/user/1001/podman/podman.sock", +}; + +const SOCKET_AUTHORITY: PodmanSocketAuthority = { + directoryChain: [], + device: "1", + inode: "2", + mode: String(0o140600), + ownerUid: "1001", + socketPath: AUTHORITY.socketPath, +}; + +const ROTATED_SOCKET_AUTHORITY: PodmanSocketAuthority = { + ...SOCKET_AUTHORITY, + inode: "3", +}; + +function capture(status: number, stdout = ""): ReturnType { + return { status, stdout, stderr: "" }; +} + +function harness( + options: { + active?: boolean; + captureSocket?: (socketPath: string) => PodmanSocketAuthority; + hardenSocket?: (socketPath: string, uid: number) => void; + podmanCapture?: ContainerEngineCommandCapture; + startStatus?: number; + assertSocket?: (authority: PodmanSocketAuthority) => void; + env?: NodeJS.ProcessEnv; + } = {}, +) { + let clock = 0; + const systemctl = vi.fn((args: readonly string[], _env: NodeJS.ProcessEnv, _timeoutMs: number) => + args.includes("is-active") + ? { status: options.active === false ? 3 : 0 } + : { status: options.startStatus ?? 0 }, + ); + const podmanCapture = + options.podmanCapture ?? + vi.fn(() => + capture(0, JSON.stringify({ Server: { Version: "5.6.1" } })), + ); + return { + systemctl, + podmanCapture, + deps: { + platform: "linux" as const, + uid: 1001, + home: AUTHORITY.homeDir, + env: options.env ?? {}, + now: () => clock, + sleep: (milliseconds: number) => { + clock += milliseconds; + }, + systemctl, + hardenSocketDirectory: options.hardenSocket ?? vi.fn(), + captureSocketAuthority: options.captureSocket ?? (() => SOCKET_AUTHORITY), + assertSocketAuthority: options.assertSocket ?? vi.fn(), + podmanCapture, + }, + }; +} + +describe("portable Podman activation readiness", () => { + it("activates a cold user socket and waits for a real API response (#9070)", () => { + const hardenSocket = vi.fn().mockImplementationOnce(() => { + throw Object.assign(new Error("missing"), { code: "ENOENT" }); + }); + const h = harness({ + active: false, + hardenSocket, + }); + + const result = inspectPortablePodmanReadiness(AUTHORITY, h.deps); + + expect(result).toMatchObject({ + ok: true, + serverVersion: "5.6.1", + timing: { mode: "cold", activationMs: 0, apiMs: 0, totalMs: 0 }, + }); + expect(h.systemctl.mock.calls.map(([args]) => args)).toEqual([ + ["--user", "is-active", "--quiet", "podman.service"], + ["--user", "start", "podman.socket"], + ]); + expect(h.podmanCapture).toHaveBeenCalledWith( + "podman", + ["--url", `unix://${AUTHORITY.socketPath}`, "version", "--format", "json"], + 60_000, + ); + expect(hardenSocket).toHaveBeenCalledTimes(2); + }); + + it("keeps polling after the socket directory is hardened (#9070)", () => { + const captureSocket = vi + .fn<() => PodmanSocketAuthority>() + .mockImplementationOnce(() => { + throw Object.assign(new Error("missing"), { code: "ENOENT" }); + }) + .mockReturnValue(SOCKET_AUTHORITY); + const hardenSocket = vi.fn(); + const h = harness({ active: false, captureSocket, hardenSocket }); + + const result = inspectPortablePodmanReadiness(AUTHORITY, h.deps); + + expect(result).toMatchObject({ + ok: true, + serverVersion: "5.6.1", + timing: { mode: "cold", apiMs: 0, totalMs: 0 }, + }); + expect(hardenSocket).toHaveBeenCalledTimes(1); + expect(captureSocket).toHaveBeenCalledTimes(2); + }); + + it("requalifies one socket inode rotation during cold API activation (#9070)", () => { + const hardenSocket = vi.fn().mockImplementationOnce(() => { + throw Object.assign(new Error("missing"), { code: "ENOENT" }); + }); + const captureSocket = vi + .fn() + .mockReturnValueOnce(SOCKET_AUTHORITY) + .mockReturnValue(ROTATED_SOCKET_AUTHORITY); + const assertSocket = vi + .fn() + .mockImplementationOnce(() => undefined) + .mockImplementationOnce(() => { + throw new Error("systemd activated the service socket"); + }) + .mockImplementationOnce(() => { + throw new Error("the initial socket is no longer current"); + }); + const h = harness({ active: false, captureSocket, hardenSocket, assertSocket }); + + const result = inspectPortablePodmanReadiness(AUTHORITY, h.deps); + + expect(result).toMatchObject({ ok: true, authority: ROTATED_SOCKET_AUTHORITY }); + expect(captureSocket).toHaveBeenCalledTimes(2); + expect(h.podmanCapture).toHaveBeenCalledTimes(2); + }); + + it("rejects a cold socket replacement that changes its device identity (#9070)", () => { + const hardenSocket = vi.fn().mockImplementationOnce(() => { + throw Object.assign(new Error("missing"), { code: "ENOENT" }); + }); + const captureSocket = vi + .fn() + .mockReturnValueOnce(SOCKET_AUTHORITY) + .mockReturnValue({ ...ROTATED_SOCKET_AUTHORITY, device: "9" }); + const assertSocket = vi + .fn() + .mockImplementationOnce(() => undefined) + .mockImplementationOnce(() => { + throw new Error("socket changed after the API request"); + }) + .mockImplementationOnce(() => { + throw new Error("the initial socket is no longer current"); + }); + const h = harness({ active: false, captureSocket, hardenSocket, assertSocket }); + + expect(inspectPortablePodmanReadiness(AUTHORITY, h.deps)).toMatchObject({ + ok: false, + stage: "socket authority", + socketPath: AUTHORITY.socketPath, + }); + expect(captureSocket).toHaveBeenCalledTimes(2); + expect(h.podmanCapture).toHaveBeenCalledOnce(); + }); + + it("uses the shorter steady-state deadline when the service reports active (#9070)", () => { + const h = harness({ active: true }); + + const result = inspectPortablePodmanReadiness(AUTHORITY, h.deps); + + expect(result).toMatchObject({ ok: true, timing: { mode: "warm" } }); + expect(h.systemctl).toHaveBeenCalledTimes(1); + expect(h.podmanCapture).toHaveBeenCalledWith("podman", expect.any(Array), 10_000); + }); + + it("uses an already healthy pinned API without starting another user socket (#9070)", () => { + const h = harness({ active: false }); + + const result = inspectPortablePodmanReadiness(AUTHORITY, h.deps); + + expect(result).toMatchObject({ ok: true, timing: { mode: "warm" } }); + expect(h.systemctl).toHaveBeenCalledTimes(1); + expect(h.podmanCapture).toHaveBeenCalledWith("podman", expect.any(Array), 10_000); + }); + + it("classifies socket activation failure without command output (#9070)", () => { + const h = harness({ + active: false, + hardenSocket: () => { + throw Object.assign(new Error("missing"), { code: "ENOENT" }); + }, + startStatus: 1, + }); + + expect(inspectPortablePodmanReadiness(AUTHORITY, h.deps)).toMatchObject({ + ok: false, + stage: "service activation", + detail: "The current user's Podman socket service could not be activated.", + socketPath: AUTHORITY.socketPath, + }); + expect(h.podmanCapture).not.toHaveBeenCalled(); + }); + + it("classifies a startup timeout before the socket appears (#9070)", () => { + const missing = () => { + throw Object.assign(new Error("secret-bearing path detail"), { code: "ENOENT" }); + }; + const h = harness({ + active: false, + hardenSocket: missing, + env: { [PORTABLE_PODMAN_STARTUP_TIMEOUT_ENV]: "15000" }, + }); + + const result = inspectPortablePodmanReadiness(AUTHORITY, h.deps); + + expect(result).toMatchObject({ ok: false, stage: "service activation" }); + expect(result).toMatchObject({ socketPath: AUTHORITY.socketPath }); + expect(result.ok ? "" : result.detail).not.toContain("secret-bearing"); + }); + + it("distinguishes cold and warm API health failures (#9070)", () => { + const unavailable = vi.fn(() => capture(1)); + const cold = harness({ active: false, podmanCapture: unavailable }); + const warm = harness({ active: true, podmanCapture: unavailable }); + + expect(inspectPortablePodmanReadiness(AUTHORITY, cold.deps)).toMatchObject({ + ok: false, + stage: "startup API health", + socketPath: AUTHORITY.socketPath, + timing: { mode: "cold" }, + }); + expect(inspectPortablePodmanReadiness(AUTHORITY, warm.deps)).toMatchObject({ + ok: false, + stage: "steady-state API health", + socketPath: AUTHORITY.socketPath, + timing: { mode: "warm" }, + }); + }); + + it("rejects unsafe and replaced sockets at the authority stage (#9070)", () => { + const unsafe = harness({ + captureSocket: () => { + throw new Error("unsafe"); + }, + }); + const replaced = harness({ + active: false, + assertSocket: () => { + throw new Error("replaced"); + }, + }); + + expect(inspectPortablePodmanReadiness(AUTHORITY, unsafe.deps)).toMatchObject({ + ok: false, + stage: "socket authority", + socketPath: AUTHORITY.socketPath, + }); + expect(inspectPortablePodmanReadiness(AUTHORITY, replaced.deps)).toMatchObject({ + ok: false, + stage: "socket authority", + socketPath: AUTHORITY.socketPath, + }); + expect(replaced.systemctl).toHaveBeenCalledOnce(); + }); + + it("reports only a validated recorded socket path (#9070)", () => { + const validFailure = inspectPortablePodmanReadiness(AUTHORITY, { + ...harness({ + active: false, + hardenSocket: () => { + throw Object.assign(new Error("missing"), { code: "ENOENT" }); + }, + startStatus: 1, + }).deps, + }); + const invalidFailure = inspectPortablePodmanReadiness(AUTHORITY, { + ...harness().deps, + uid: 2002, + }); + + expect(validFailure).toMatchObject({ ok: false, socketPath: AUTHORITY.socketPath }); + expect(validFailure.ok ? "" : portablePodmanReadinessError(validFailure).message).toContain( + `Recorded socket: ${AUTHORITY.socketPath}.`, + ); + expect(invalidFailure).toMatchObject({ ok: false, stage: "socket authority" }); + expect(invalidFailure).not.toHaveProperty("socketPath"); + }); + + it("ignores ambient engine selectors and uses the recorded user authority (#9070)", () => { + const h = harness({ + env: { + HOME: "/attacker", + XDG_RUNTIME_DIR: "/attacker/run", + CONTAINER_CONNECTION: "attacker", + CONTAINER_HOST: "tcp://attacker.invalid:9999", + DOCKER_CONTEXT: "attacker", + DOCKER_HOST: "tcp://attacker.invalid:2375", + }, + }); + + inspectPortablePodmanReadiness(AUTHORITY, h.deps); + + const childEnv = h.systemctl.mock.calls[0]?.[1] as NodeJS.ProcessEnv; + expect(childEnv).toMatchObject({ + HOME: AUTHORITY.homeDir, + XDG_CONFIG_HOME: AUTHORITY.configHome, + XDG_RUNTIME_DIR: AUTHORITY.runtimeDir, + DBUS_SESSION_BUS_ADDRESS: `unix:path=${AUTHORITY.runtimeDir}/bus`, + }); + expect(childEnv).not.toHaveProperty("CONTAINER_CONNECTION"); + expect(childEnv).not.toHaveProperty("CONTAINER_HOST"); + expect(childEnv).not.toHaveProperty("DOCKER_CONTEXT"); + expect(childEnv).not.toHaveProperty("DOCKER_HOST"); + }); + + it("bounds configurable startup timeouts and falls back safely (#9070)", () => { + expect(resolvePortablePodmanStartupTimeout({})).toBe( + DEFAULT_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS, + ); + expect( + resolvePortablePodmanStartupTimeout({ [PORTABLE_PODMAN_STARTUP_TIMEOUT_ENV]: "15000" }), + ).toBe(15_000); + expect( + resolvePortablePodmanStartupTimeout({ [PORTABLE_PODMAN_STARTUP_TIMEOUT_ENV]: "300000" }), + ).toBe(300_000); + for (const invalid of ["1", "300001", "1.5", "not-a-number"]) { + expect( + resolvePortablePodmanStartupTimeout({ + [PORTABLE_PODMAN_STARTUP_TIMEOUT_ENV]: invalid, + }), + ).toBe(DEFAULT_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS); + } + }); +}); diff --git a/src/lib/onboard/experimental/portable-runtime-readiness.ts b/src/lib/onboard/experimental/portable-runtime-readiness.ts new file mode 100644 index 00000000000..1d9351dca4e --- /dev/null +++ b/src/lib/onboard/experimental/portable-runtime-readiness.ts @@ -0,0 +1,512 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import os from "node:os"; +import path from "node:path"; + +import type { ContainerEngineCommandCapture } from "../../adapters/container-engine"; +import { + assertPodmanSocketAuthority, + capturePodmanSocketAuthority, + createPodmanContainerEngine, + hardenPodmanSocketDirectory, + type PodmanSocketAuthority, + type PodmanSocketAuthorityDeps, +} from "../../adapters/podman"; +import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; +import { parsePortableRuntimeAuthority } from "../../state/onboard/portable-runtime-authority"; + +export const PORTABLE_PODMAN_STARTUP_TIMEOUT_ENV = "NEMOCLAW_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS"; +export const DEFAULT_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS = 60_000; +export const MIN_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS = 15_000; +export const MAX_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS = 300_000; +export const PORTABLE_PODMAN_STEADY_STATE_TIMEOUT_MS = 10_000; +const POLL_INTERVAL_MS = 100; +const SLEEP_BUFFER = new Int32Array(new SharedArrayBuffer(4)); +const USER_COMMAND_ENV_NAMES = new Set([ + "USER", + "LOGNAME", + "SHELL", + "PATH", + "TERM", + "HOSTNAME", + "LANG", + "TMPDIR", + "TMP", + "TEMP", + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "http_proxy", + "https_proxy", + "no_proxy", + "SSL_CERT_FILE", + "SSL_CERT_DIR", + "CURL_CA_BUNDLE", +]); + +type CommandResult = { + readonly status: number | null; + readonly stdout?: string | Buffer | null; + readonly stderr?: string | Buffer | null; + readonly error?: Error; +}; + +export type PortablePodmanReadinessStage = + | "socket authority" + | "service activation" + | "startup API health" + | "steady-state API health"; + +export interface PortablePodmanReadinessTiming { + readonly mode: "cold" | "warm"; + readonly activationMs: number; + readonly apiMs: number; + readonly totalMs: number; +} + +export type PortablePodmanReadinessResult = + | { + readonly ok: true; + readonly authority: PodmanSocketAuthority; + readonly dockerHost: string; + readonly serverVersion: string; + readonly timing: PortablePodmanReadinessTiming; + } + | { + readonly ok: false; + readonly stage: PortablePodmanReadinessStage; + readonly detail: string; + readonly socketPath?: string; + readonly timing: PortablePodmanReadinessTiming; + }; + +export interface PortablePodmanReadinessDeps { + readonly platform?: NodeJS.Platform; + readonly uid?: number; + readonly home?: string; + readonly env?: NodeJS.ProcessEnv; + readonly now?: () => number; + readonly sleep?: (milliseconds: number) => void; + readonly systemctl?: ( + args: readonly string[], + env: NodeJS.ProcessEnv, + timeoutMs: number, + ) => CommandResult; + readonly podmanCapture?: ContainerEngineCommandCapture; + readonly socketAuthorityDeps?: PodmanSocketAuthorityDeps; + readonly hardenSocketDirectory?: (socketPath: string, uid: number) => void; + readonly captureSocketAuthority?: ( + socketPath: string, + deps?: PodmanSocketAuthorityDeps, + ) => PodmanSocketAuthority; + readonly assertSocketAuthority?: ( + authority: PodmanSocketAuthority, + deps?: PodmanSocketAuthorityDeps, + ) => void; +} + +function sleep(milliseconds: number): void { + if (milliseconds > 0) Atomics.wait(SLEEP_BUFFER, 0, 0, milliseconds); +} + +function elapsed(now: () => number, startedAt: number): number { + return Math.max(0, Math.round(now() - startedAt)); +} + +function timing( + mode: "cold" | "warm", + activationMs: number, + apiMs: number, + totalMs: number, +): PortablePodmanReadinessTiming { + return { mode, activationMs, apiMs, totalMs }; +} + +export function resolvePortablePodmanStartupTimeout(env: NodeJS.ProcessEnv): number { + const raw = env[PORTABLE_PODMAN_STARTUP_TIMEOUT_ENV]; + if (raw === undefined || raw.trim() === "") return DEFAULT_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS; + if (!/^\d+$/u.test(raw.trim())) return DEFAULT_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS; + const value = Number(raw); + return Number.isSafeInteger(value) && + value >= MIN_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS && + value <= MAX_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS + ? value + : DEFAULT_PORTABLE_PODMAN_STARTUP_TIMEOUT_MS; +} + +export function portablePodmanCommandEnvironment( + authority: CheckpointPortableRuntimeAuthority, + source: NodeJS.ProcessEnv, +): NodeJS.ProcessEnv { + const env: NodeJS.ProcessEnv = {}; + for (const [name, value] of Object.entries(source)) { + if (value !== undefined && (USER_COMMAND_ENV_NAMES.has(name) || name.startsWith("LC_"))) { + env[name] = value; + } + } + Object.assign(env, { + HOME: authority.homeDir, + XDG_CONFIG_HOME: authority.configHome, + XDG_RUNTIME_DIR: authority.runtimeDir, + DBUS_SESSION_BUS_ADDRESS: `unix:path=${authority.runtimeDir}/bus`, + }); + const containersConf = path.join(authority.configHome, "nemoclaw", "portable", "containers.conf"); + if (source.CONTAINERS_CONF === containersConf) env.CONTAINERS_CONF = containersConf; + if (source.NETAVARK_FW === "iptables") env.NETAVARK_FW = "iptables"; + if (source.GIT_SSL_CAINFO) env.GIT_SSL_CAINFO = source.GIT_SSL_CAINFO; + if (source.GIT_SSL_CAPATH) env.GIT_SSL_CAPATH = source.GIT_SSL_CAPATH; + if (source.GIT_CONFIG_NOSYSTEM === "1") { + env.GIT_CONFIG_NOSYSTEM = "1"; + } + return env; +} + +function failure( + stage: PortablePodmanReadinessStage, + detail: string, + mode: "cold" | "warm", + activationMs: number, + apiMs: number, + totalMs: number, + socketPath?: string, +): PortablePodmanReadinessResult { + return { + ok: false, + stage, + detail, + ...(socketPath ? { socketPath } : {}), + timing: timing(mode, activationMs, apiMs, totalMs), + }; +} + +function parseServerVersion(stdout: string): string | null { + try { + const value = JSON.parse(stdout) as { Server?: { Version?: unknown } }; + const version = value.Server?.Version; + return typeof version === "string" && version.trim() !== "" ? version.trim() : null; + } catch { + return null; + } +} + +function isMissingSocket(error: unknown): boolean { + return (error as NodeJS.ErrnoException | undefined)?.code === "ENOENT"; +} + +function sameDirectoryAuthority( + expected: PodmanSocketAuthority, + actual: PodmanSocketAuthority, +): boolean { + return ( + actual.directoryChain.length === expected.directoryChain.length && + actual.directoryChain.every((component, index) => { + const pinned = expected.directoryChain[index]; + return ( + pinned !== undefined && + component.device === pinned.device && + component.inode === pinned.inode && + component.mode === pinned.mode && + component.ownerUid === pinned.ownerUid && + component.path === pinned.path + ); + }) + ); +} + +function recaptureColdActivationSocket( + expected: PodmanSocketAuthority, + socketPath: string, + authorityDeps: PodmanSocketAuthorityDeps, + capture: (socketPath: string, deps?: PodmanSocketAuthorityDeps) => PodmanSocketAuthority, + assert: (authority: PodmanSocketAuthority, deps?: PodmanSocketAuthorityDeps) => void, +): PodmanSocketAuthority | null { + try { + assert(expected, authorityDeps); + return null; + } catch { + // A cold systemd activation can replace only the socket inode after the + // first successful connection. Re-capture the same secured path and keep + // every other authority field pinned before retrying the API probe. + } + try { + const replacement = capture(socketPath, authorityDeps); + return replacement.socketPath === expected.socketPath && + replacement.device === expected.device && + replacement.inode !== expected.inode && + replacement.mode === expected.mode && + replacement.ownerUid === expected.ownerUid && + sameDirectoryAuthority(expected, replacement) + ? replacement + : null; + } catch { + return null; + } +} + +function probePodmanServerVersion( + socketAuthority: PodmanSocketAuthority, + timeoutMs: number, + authorityDeps: PodmanSocketAuthorityDeps, + assertAuthority: NonNullable, + podmanCapture?: ContainerEngineCommandCapture, +): string | null { + const provider = createPodmanContainerEngine({ + operation: "host-doctor", + socketAuthority, + authorityDeps, + assertAuthority, + ...(podmanCapture ? { capture: podmanCapture } : {}), + }); + const result = provider.capture(["version", "--format", "json"], timeoutMs); + return result.status === 0 ? parseServerVersion(result.stdout) : null; +} + +/** + * Verify one receipt-owned rootless Podman API. Cold activation and warm health + * use separate budgets; no process-global engine selector participates. + */ +export function inspectPortablePodmanReadiness( + recordedAuthority: CheckpointPortableRuntimeAuthority, + deps: PortablePodmanReadinessDeps = {}, +): PortablePodmanReadinessResult { + const startedAt = (deps.now ?? Date.now)(); + const now = deps.now ?? Date.now; + const env = deps.env ?? process.env; + const uid = deps.uid ?? process.geteuid?.() ?? process.getuid?.(); + const home = deps.home ?? os.userInfo().homedir; + const normalized = parsePortableRuntimeAuthority(recordedAuthority); + if ( + (deps.platform ?? process.platform) !== "linux" || + !normalized || + !Number.isSafeInteger(uid) || + normalized.uid !== uid || + normalized.homeDir !== home + ) { + return failure( + "socket authority", + "The recorded portable Podman authority does not match the current Linux user.", + "warm", + 0, + 0, + elapsed(now, startedAt), + ); + } + + const commandEnv = portablePodmanCommandEnvironment(normalized, env); + const startupTimeoutMs = resolvePortablePodmanStartupTimeout(env); + const systemctl = + deps.systemctl ?? + ((args, childEnv, timeoutMs) => + spawnSync("systemctl", [...args], { + encoding: "utf-8", + env: childEnv, + stdio: ["ignore", "pipe", "pipe"], + timeout: timeoutMs, + })); + const capture = deps.captureSocketAuthority ?? capturePodmanSocketAuthority; + const assert = deps.assertSocketAuthority ?? assertPodmanSocketAuthority; + const authorityDeps = { ...deps.socketAuthorityDeps, uid: normalized.uid }; + let socketSeen = false; + let socketHardened = false; + let socketAuthority: PodmanSocketAuthority | null = null; + let coldActivationSocketRecaptured = false; + const active = systemctl( + ["--user", "is-active", "--quiet", "podman.service"], + commandEnv, + PORTABLE_PODMAN_STEADY_STATE_TIMEOUT_MS, + ); + const serviceActive = active.status === 0 && !active.error; + if (!serviceActive) { + try { + (deps.hardenSocketDirectory ?? hardenPodmanSocketDirectory)( + normalized.socketPath, + normalized.uid, + ); + socketHardened = true; + socketAuthority = capture(normalized.socketPath, authorityDeps); + if (socketAuthority.socketPath !== normalized.socketPath) { + throw new Error("The captured Podman socket does not match the recorded endpoint."); + } + socketSeen = true; + } catch (error) { + if (!isMissingSocket(error)) { + return failure( + "socket authority", + "The recorded portable Podman socket is absent, unsafe, or no longer has its recorded authority.", + "cold", + 0, + 0, + elapsed(now, startedAt), + normalized.socketPath, + ); + } + socketAuthority = null; + } + if (socketAuthority) { + const apiStartedAt = now(); + let serverVersion: string | null; + try { + serverVersion = probePodmanServerVersion( + socketAuthority, + PORTABLE_PODMAN_STEADY_STATE_TIMEOUT_MS, + authorityDeps, + assert, + deps.podmanCapture, + ); + } catch { + return failure( + "socket authority", + "The recorded portable Podman socket changed while its API was being verified.", + "cold", + 0, + elapsed(now, apiStartedAt), + elapsed(now, startedAt), + normalized.socketPath, + ); + } + if (serverVersion) { + return { + ok: true, + authority: socketAuthority, + dockerHost: `unix://${normalized.socketPath}`, + serverVersion, + timing: timing("warm", 0, elapsed(now, apiStartedAt), elapsed(now, startedAt)), + }; + } + } + } + const mode = serviceActive ? "warm" : "cold"; + const deadlineMs = mode === "warm" ? PORTABLE_PODMAN_STEADY_STATE_TIMEOUT_MS : startupTimeoutMs; + const activationStartedAt = now(); + if (mode === "cold") { + const activated = systemctl( + ["--user", "start", "podman.socket"], + commandEnv, + Math.max(1, deadlineMs - elapsed(now, startedAt)), + ); + if (activated.status !== 0 || activated.error) { + return failure( + "service activation", + "The current user's Podman socket service could not be activated.", + mode, + elapsed(now, activationStartedAt), + 0, + elapsed(now, startedAt), + normalized.socketPath, + ); + } + } + const activationMs = elapsed(now, activationStartedAt); + const apiStartedAt = now(); + const budgetStartedAt = mode === "warm" ? apiStartedAt : startedAt; + + while (elapsed(now, budgetStartedAt) < deadlineMs) { + if (!socketAuthority) { + try { + if (!socketHardened) { + (deps.hardenSocketDirectory ?? hardenPodmanSocketDirectory)( + normalized.socketPath, + normalized.uid, + ); + socketHardened = true; + } + socketAuthority = capture(normalized.socketPath, authorityDeps); + if (socketAuthority.socketPath !== normalized.socketPath) { + throw new Error("The captured Podman socket does not match the recorded endpoint."); + } + socketSeen = true; + } catch (error) { + if (mode === "cold" && isMissingSocket(error)) { + (deps.sleep ?? sleep)( + Math.min(POLL_INTERVAL_MS, Math.max(0, deadlineMs - elapsed(now, startedAt))), + ); + continue; + } + return failure( + "socket authority", + "The recorded portable Podman socket is absent, unsafe, or no longer has its recorded authority.", + mode, + activationMs, + elapsed(now, apiStartedAt), + elapsed(now, startedAt), + normalized.socketPath, + ); + } + } + + const remainingMs = Math.max(1, deadlineMs - elapsed(now, budgetStartedAt)); + try { + const serverVersion = probePodmanServerVersion( + socketAuthority, + remainingMs, + authorityDeps, + assert, + deps.podmanCapture, + ); + if (serverVersion) { + return { + ok: true, + authority: socketAuthority, + dockerHost: `unix://${normalized.socketPath}`, + serverVersion, + timing: timing(mode, activationMs, elapsed(now, apiStartedAt), elapsed(now, startedAt)), + }; + } + } catch { + const replacement = + mode === "cold" && !coldActivationSocketRecaptured + ? recaptureColdActivationSocket( + socketAuthority, + normalized.socketPath, + authorityDeps, + capture, + assert, + ) + : null; + if (replacement) { + socketAuthority = replacement; + coldActivationSocketRecaptured = true; + continue; + } + return failure( + "socket authority", + "The recorded portable Podman socket changed while its API was being verified.", + mode, + activationMs, + elapsed(now, apiStartedAt), + elapsed(now, startedAt), + normalized.socketPath, + ); + } + if (mode === "warm") break; + (deps.sleep ?? sleep)( + Math.min(POLL_INTERVAL_MS, Math.max(0, deadlineMs - elapsed(now, startedAt))), + ); + } + + const stage = mode === "warm" ? "steady-state API health" : "startup API health"; + return failure( + socketSeen ? stage : "service activation", + socketSeen + ? mode === "warm" + ? "The recorded portable Podman API did not return a server version within 10000 ms." + : `The activated portable Podman API did not return a server version within ${String(startupTimeoutMs)} ms.` + : `The activated portable Podman service did not create its recorded socket within ${String(startupTimeoutMs)} ms.`, + mode, + activationMs, + elapsed(now, apiStartedAt), + elapsed(now, startedAt), + normalized.socketPath, + ); +} + +export function portablePodmanReadinessError( + result: Extract, +): Error { + const socket = result.socketPath ? ` Recorded socket: ${result.socketPath}.` : ""; + return new Error( + `Portable Podman readiness failed at ${result.stage}: ${result.detail}${socket}`, + ); +} diff --git a/src/lib/onboard/experimental/portable-runtime-receipt-readiness.ts b/src/lib/onboard/experimental/portable-runtime-receipt-readiness.ts new file mode 100644 index 00000000000..7f8d34c05f8 --- /dev/null +++ b/src/lib/onboard/experimental/portable-runtime-receipt-readiness.ts @@ -0,0 +1,203 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import os from "node:os"; +import path from "node:path"; + +import type { ContainerEngineCommandCapture } from "../../adapters/container-engine"; +import { openRegularFileNoFollow } from "../../adapters/fs/regular-file"; +import { hardenPodmanSocketDirectory, type PodmanSocketAuthorityDeps } from "../../adapters/podman"; +import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; +import { parsePortableRuntimeAuthority } from "../../state/onboard/portable-runtime-authority"; +import { + inspectPortablePodmanReadiness, + portablePodmanCommandEnvironment, + type PortablePodmanReadinessDeps, + type PortablePodmanReadinessResult, +} from "./portable-runtime-readiness"; + +const RECEIPT_DIRECTORY = "portable-demo-lifecycle"; +const MAX_RECEIPT_BYTES = 4096; +const CURRENT_RECEIPT_SCHEMA_VERSION = 4; +const CONTAINER_ID_PATTERN = /^[a-f0-9]{64}$/u; +const SANDBOX_ID_PATTERN = /^[A-Za-z0-9._:-]{1,256}$/u; + +type CommandResult = { + readonly status: number | null; + readonly stdout?: string | Buffer | null; + readonly stderr?: string | Buffer | null; + readonly error?: Error; +}; + +export interface PortableRuntimeReceiptReadinessDeps { + readonly platform?: NodeJS.Platform; + readonly stateDir?: string; + readonly env?: NodeJS.ProcessEnv; + readonly podman?: (args: readonly string[], env?: NodeJS.ProcessEnv) => CommandResult; + readonly podmanSocketAuthorityDeps?: PodmanSocketAuthorityDeps; + readonly runtimeReadiness?: PortablePodmanReadinessDeps; + readonly hardenSocketDirectory?: (socketPath: string, uid: number) => void; +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +export function portableDemoReceiptPath(sandboxName: string, stateDir: string): string { + const fileName = `${createHash("sha256").update(sandboxName).digest("hex")}.json`; + return path.join(stateDir, RECEIPT_DIRECTORY, fileName); +} + +export function defaultPortableDemoStateDir(env: NodeJS.ProcessEnv): string { + if ( + env.VITEST === "true" && + (env.HOME ?? "") === env.NEMOCLAW_TEST_BASE_HOME && + env.NEMOCLAW_TEST_STATE_DIR && + path.isAbsolute(env.NEMOCLAW_TEST_STATE_DIR) + ) { + return env.NEMOCLAW_TEST_STATE_DIR; + } + return path.join(env.HOME ?? os.homedir(), ".nemoclaw"); +} + +function exactReceiptKeys(receipt: Record): boolean { + const expected = + receipt.schemaVersion === CURRENT_RECEIPT_SCHEMA_VERSION + ? "containerId,dashboardPort,registryGeneration,runtimeAuthority,sandboxId,sandboxName,schemaVersion" + : receipt.schemaVersion === 3 + ? "containerId,dashboardPort,registryGeneration,sandboxId,sandboxName,schemaVersion" + : "containerId,dashboardPort,sandboxId,sandboxName,schemaVersion"; + return Object.keys(receipt).sort().join(",") === expected; +} + +function parseReceiptAuthority( + value: unknown, + sandboxName: string, +): CheckpointPortableRuntimeAuthority | "legacy" { + if (!isRecord(value) || !exactReceiptKeys(value)) { + throw new Error("Portable demo lifecycle receipt fields are invalid"); + } + const schemaVersion = value.schemaVersion; + if ( + (schemaVersion !== 1 && + schemaVersion !== 2 && + schemaVersion !== 3 && + schemaVersion !== CURRENT_RECEIPT_SCHEMA_VERSION) || + value.sandboxName !== sandboxName || + typeof value.containerId !== "string" || + !CONTAINER_ID_PATTERN.test(value.containerId) || + typeof value.sandboxId !== "string" || + !SANDBOX_ID_PATTERN.test(value.sandboxId) || + !Number.isInteger(value.dashboardPort) || + Number(value.dashboardPort) < 1024 || + Number(value.dashboardPort) > 65535 || + ((schemaVersion === 3 || schemaVersion === CURRENT_RECEIPT_SCHEMA_VERSION) && + (typeof value.registryGeneration !== "string" || + !SANDBOX_ID_PATTERN.test(value.registryGeneration))) + ) { + throw new Error("Portable demo lifecycle receipt values are invalid"); + } + if (schemaVersion !== CURRENT_RECEIPT_SCHEMA_VERSION) return "legacy"; + const authority = parsePortableRuntimeAuthority(value.runtimeAuthority); + if (!authority) throw new Error("Portable demo lifecycle receipt values are invalid"); + return authority; +} + +function loadReceiptAuthority( + sandboxName: string, + stateDir: string, +): CheckpointPortableRuntimeAuthority | "legacy" | null { + let file; + try { + file = openRegularFileNoFollow(portableDemoReceiptPath(sandboxName, stateDir)); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; + throw error; + } + try { + return parseReceiptAuthority(JSON.parse(file.readUtf8(MAX_RECEIPT_BYTES)), sandboxName); + } catch (error) { + if (error instanceof SyntaxError) { + throw new Error("Portable demo lifecycle receipt is malformed"); + } + throw error; + } finally { + file.close(); + } +} + +function podmanCapture( + podman: NonNullable, + env: NodeJS.ProcessEnv, +): ContainerEngineCommandCapture { + return (_executable, args) => { + const result = podman(args, env); + return { + status: result.status ?? 1, + stdout: String(result.stdout ?? ""), + stderr: String(result.stderr ?? ""), + ...(result.error ? { error: result.error } : {}), + }; + }; +} + +function defaultPodmanCapture(env: NodeJS.ProcessEnv): ContainerEngineCommandCapture { + return (_executable, args, timeoutMs) => { + const result = spawnSync("podman", [...args], { + encoding: "utf-8", + env, + stdio: ["ignore", "pipe", "pipe"], + timeout: timeoutMs, + }); + return { + status: result.status ?? 1, + stdout: String(result.stdout ?? ""), + stderr: String(result.stderr ?? ""), + ...(result.error ? { error: result.error } : {}), + }; + }; +} + +/** Inspect the receipt-owned portable runtime, or return null for an ordinary sandbox. */ +export function inspectPortableRuntimeReceiptReadiness( + sandboxName: string, + deps: PortableRuntimeReceiptReadinessDeps = {}, +): PortablePodmanReadinessResult | null { + const commandEnv = deps.env ?? process.env; + const stateDir = deps.stateDir ?? defaultPortableDemoStateDir(commandEnv); + let authority: CheckpointPortableRuntimeAuthority | "legacy" | null; + try { + authority = loadReceiptAuthority(sandboxName, stateDir); + } catch { + return { + ok: false, + stage: "socket authority", + detail: "The portable lifecycle receipt is unsafe or invalid; rerun onboarding.", + timing: { mode: "warm", activationMs: 0, apiMs: 0, totalMs: 0 }, + }; + } + if (!authority) return null; + if (authority === "legacy") { + return { + ok: false, + stage: "socket authority", + detail: + "The lifecycle receipt predates recorded portable Podman authority; rerun onboarding.", + timing: { mode: "warm", activationMs: 0, apiMs: 0, totalMs: 0 }, + }; + } + const podmanEnv = portablePodmanCommandEnvironment(authority, commandEnv); + const capture = deps.podman + ? podmanCapture(deps.podman, podmanEnv) + : defaultPodmanCapture(podmanEnv); + return inspectPortablePodmanReadiness(authority, { + platform: deps.platform, + env: commandEnv, + socketAuthorityDeps: deps.podmanSocketAuthorityDeps, + hardenSocketDirectory: deps.hardenSocketDirectory ?? hardenPodmanSocketDirectory, + podmanCapture: capture, + ...deps.runtimeReadiness, + }); +} diff --git a/src/lib/onboard/gateway-sandbox-reachability.test.ts b/src/lib/onboard/gateway-sandbox-reachability.test.ts index 003ea1f68ae..d3ed09bd87e 100644 --- a/src/lib/onboard/gateway-sandbox-reachability.test.ts +++ b/src/lib/onboard/gateway-sandbox-reachability.test.ts @@ -592,7 +592,7 @@ describe("formatSandboxBridgeUnreachableMessage", () => { }); expect(msg).toContain("OpenShell Podman host gateway"); expect(msg).toContain("systemctl --user try-restart podman.service"); - expect(msg).toContain("systemctl --user enable --now podman.socket"); + expect(msg).toContain("systemctl --user start podman.socket"); expect(msg).toContain("nemoclaw onboard --experimental-profile portable"); expect(msg).not.toContain("Restart Docker"); expect(msg).not.toContain("ufw allow"); @@ -607,7 +607,7 @@ describe("formatSandboxBridgeUnreachableMessage", () => { }); expect(msg).toContain("Podman service is not reachable"); expect(msg).toContain("systemctl --user try-restart podman.service"); - expect(msg).toContain("systemctl --user enable --now podman.socket"); + expect(msg).toContain("systemctl --user start podman.socket"); expect(msg).toContain("nemoclaw onboard --experimental-profile portable"); expect(msg).not.toContain("Restart the Docker daemon"); }); diff --git a/src/lib/onboard/gateway-sandbox-reachability.ts b/src/lib/onboard/gateway-sandbox-reachability.ts index ac70ac5df7d..9eb4c1f1c28 100644 --- a/src/lib/onboard/gateway-sandbox-reachability.ts +++ b/src/lib/onboard/gateway-sandbox-reachability.ts @@ -513,8 +513,8 @@ export function formatSandboxBridgeUnreachableMessage( result.detail ? ` ${result.detail}` : undefined, " If the user-scoped Podman service is active, restart it:", " systemctl --user try-restart podman.service", - " Enable and start the user-scoped Podman socket:", - " systemctl --user enable --now podman.socket", + " Start the current user's Podman socket for this session; this does not enable it for later sessions:", + " systemctl --user start podman.socket", ` Then rerun \`${cliName()} onboard --experimental-profile portable\`.`, ] .filter((line): line is string => Boolean(line)) @@ -539,8 +539,8 @@ export function formatSandboxBridgeUnreachableMessage( ` The probe mapped ${HOST_INTERNAL_NAME} to the OpenShell Podman host gateway.`, " If the user-scoped Podman service is active, restart it:", " systemctl --user try-restart podman.service", - " Enable and start the user-scoped Podman socket:", - " systemctl --user enable --now podman.socket", + " Start the current user's Podman socket for this session; this does not enable it for later sessions:", + " systemctl --user start podman.socket", ` Then rerun \`${cliName()} onboard --experimental-profile portable\`.`, ].join("\n"); } diff --git a/src/lib/onboard/hermes-api-port.test.ts b/src/lib/onboard/hermes-api-port.test.ts index 6346ba62fb2..316faacdb3a 100644 --- a/src/lib/onboard/hermes-api-port.test.ts +++ b/src/lib/onboard/hermes-api-port.test.ts @@ -28,6 +28,7 @@ describe("Hermes API and dashboard port creation scopes", () => { const createSandboxWithBaseImageResolution = vi.fn( async ( _baseImageResolutionContext: { fresh: boolean }, + portableRuntimeAuthority: { socketPath: string }, _computePlan: { sequence: number }, _managedWorkloadRebuild: null, temporaryManagedRuntime: boolean, @@ -38,6 +39,7 @@ describe("Hermes API and dashboard port creation scopes", () => { ) => ({ dashboardPortReservationScope, hermesApiPortReservationScope, + portableRuntimeAuthority, sandboxName, temporaryManagedRuntime, }), @@ -46,6 +48,7 @@ describe("Hermes API and dashboard port creation scopes", () => { const entryPoints = createHermesApiPortScopedSandboxEntryPoints({ createBaseImageResolutionContext: () => ({ fresh: false }), createSandboxWithBaseImageResolution, + resolvePortableRuntimeAuthority: () => ({ socketPath: "/run/user/1001/podman.sock" }), resolveComputePlan: () => ({ sequence: ++sequence }), }); @@ -53,6 +56,9 @@ describe("Hermes API and dashboard port creation scopes", () => { const temporary = await entryPoints.createSandboxWithTemporaryManagedRuntime("temporary"); expect(standard).toMatchObject({ sandboxName: "standard", temporaryManagedRuntime: false }); + expect(standard.portableRuntimeAuthority).toEqual({ + socketPath: "/run/user/1001/podman.sock", + }); expect(temporary).toMatchObject({ sandboxName: "temporary", temporaryManagedRuntime: true }); expect(standard.dashboardPortReservationScope).not.toBe( temporary.dashboardPortReservationScope, diff --git a/src/lib/onboard/hermes-api-port.ts b/src/lib/onboard/hermes-api-port.ts index 68add4243cc..99a2318b980 100644 --- a/src/lib/onboard/hermes-api-port.ts +++ b/src/lib/onboard/hermes-api-port.ts @@ -51,11 +51,13 @@ interface HermesApiPortScopedSandboxEntryPointDeps< Args extends unknown[], Result, BaseImageResolutionContext, + PortableRuntimeAuthority, ComputePlan, > { createBaseImageResolutionContext(): BaseImageResolutionContext; createSandboxWithBaseImageResolution( baseImageResolutionContext: BaseImageResolutionContext, + portableRuntimeAuthority: PortableRuntimeAuthority, computePlan: ComputePlan, managedWorkloadRebuild: null, temporaryManagedRuntime: boolean, @@ -64,6 +66,7 @@ interface HermesApiPortScopedSandboxEntryPointDeps< hermesApiPortReservationScope: HermesApiPortReservationScope, ...args: Args ): Promise; + resolvePortableRuntimeAuthority(): PortableRuntimeAuthority; resolveComputePlan(): ComputePlan; } @@ -72,12 +75,14 @@ export function createHermesApiPortScopedSandboxEntryPoints< Args extends unknown[], Result, BaseImageResolutionContext, + PortableRuntimeAuthority, ComputePlan, >( deps: HermesApiPortScopedSandboxEntryPointDeps< Args, Result, BaseImageResolutionContext, + PortableRuntimeAuthority, ComputePlan >, ): { @@ -88,6 +93,7 @@ export function createHermesApiPortScopedSandboxEntryPoints< createBaseImageResolutionContext: deps.createBaseImageResolutionContext, createSandboxWithBaseImageResolution: ( baseImageResolutionContext, + portableRuntimeAuthority, computePlan, managedWorkloadRebuild, temporaryManagedRuntime, @@ -98,6 +104,7 @@ export function createHermesApiPortScopedSandboxEntryPoints< withHermesApiPortReservationScope((hermesApiPortReservationScope) => deps.createSandboxWithBaseImageResolution( baseImageResolutionContext, + portableRuntimeAuthority, computePlan, managedWorkloadRebuild, temporaryManagedRuntime, @@ -108,6 +115,7 @@ export function createHermesApiPortScopedSandboxEntryPoints< ), ), resolveComputePlan: deps.resolveComputePlan, + resolvePortableRuntimeAuthority: deps.resolvePortableRuntimeAuthority, }); } diff --git a/src/lib/onboard/portable-resume-lock-boundary.test.ts b/src/lib/onboard/portable-resume-lock-boundary.test.ts index 05fb4e1ac0c..0c22d2b2d17 100644 --- a/src/lib/onboard/portable-resume-lock-boundary.test.ts +++ b/src/lib/onboard/portable-resume-lock-boundary.test.ts @@ -9,6 +9,8 @@ import path from "node:path"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { testTimeoutOptions } from "../../../test/helpers/timeouts"; + const originalEnv = { ...process.env }; const STOP_AFTER_PREPARATION = "stop after observed portable preparation"; let tempHome: string; @@ -89,54 +91,58 @@ function runWithObservedPreparation( } describe("portable resume command lock boundary", () => { - it("rejects a losing CLI before portable config writes or socket activation (#9035)", async () => { - const { command, onboardModule, session } = boundaryModules; - const childScript = ` - const fs = require("node:fs"); - const path = require("node:path"); - const lockFile = process.argv[1]; - fs.mkdirSync(path.dirname(lockFile), { recursive: true }); - const fd = fs.openSync(lockFile, "wx", 0o600); - fs.writeSync(fd, JSON.stringify({ - pid: process.pid, - startedAt: new Date().toISOString(), - command: "separate nemoclaw onboard process", - })); - process.stdout.write("locked\\n"); - setInterval(() => {}, 1000); - `; - const child = spawn(process.execPath, ["-e", childScript, session.LOCK_FILE], { - stdio: ["ignore", "pipe", "inherit"], - }); - await once(child.stdout, "data"); - vi.spyOn(console, "error").mockImplementation(() => {}); - vi.spyOn(process, "exit").mockImplementation(((code?: number) => { - throw new Error(`exit:${String(code ?? 0)}`); - }) as typeof process.exit); - - try { - await expect( - command.runOnboardCommand({ - flags: { - fresh: true, - "experimental-profile": "portable", - "yes-i-accept-third-party-software": true, - }, - env: process.env, - resolveResumeIntent: () => ({ effectiveResume: false, snapshot: null }), - runOnboard: (options) => runWithObservedPreparation(onboardModule, options), - }), - ).rejects.toThrow("exit:1"); - expect(preparePortableHost).not.toHaveBeenCalled(); - expect(fs.existsSync(configWriteMarker)).toBe(false); - expect(fs.existsSync(socketActivationMarker)).toBe(false); - } finally { - const exited = once(child, "exit"); - child.kill(); - await exited; - fs.rmSync(session.LOCK_FILE, { force: true }); - } - }, 15_000); + it( + "rejects a losing CLI before portable config writes or socket activation (#9035)", + testTimeoutOptions(30_000), + async () => { + const { command, onboardModule, session } = boundaryModules; + const childScript = ` + const fs = require("node:fs"); + const path = require("node:path"); + const lockFile = process.argv[1]; + fs.mkdirSync(path.dirname(lockFile), { recursive: true }); + const fd = fs.openSync(lockFile, "wx", 0o600); + fs.writeSync(fd, JSON.stringify({ + pid: process.pid, + startedAt: new Date().toISOString(), + command: "separate nemoclaw onboard process", + })); + process.stdout.write("locked\\n"); + setInterval(() => {}, 1000); + `; + const child = spawn(process.execPath, ["-e", childScript, session.LOCK_FILE], { + stdio: ["ignore", "pipe", "inherit"], + }); + await once(child.stdout, "data"); + vi.spyOn(console, "error").mockImplementation(() => {}); + vi.spyOn(process, "exit").mockImplementation(((code?: number) => { + throw new Error(`exit:${String(code ?? 0)}`); + }) as typeof process.exit); + + try { + await expect( + command.runOnboardCommand({ + flags: { + fresh: true, + "experimental-profile": "portable", + "yes-i-accept-third-party-software": true, + }, + env: process.env, + resolveResumeIntent: () => ({ effectiveResume: false, snapshot: null }), + runOnboard: (options) => runWithObservedPreparation(onboardModule, options), + }), + ).rejects.toThrow("exit:1"); + expect(preparePortableHost).not.toHaveBeenCalled(); + expect(fs.existsSync(configWriteMarker)).toBe(false); + expect(fs.existsSync(socketActivationMarker)).toBe(false); + } finally { + const exited = once(child, "exit"); + child.kill(); + await exited; + fs.rmSync(session.LOCK_FILE, { force: true }); + } + }, + ); it("releases the first lock before one bounded pre-read retry and preparation (#9035)", async () => { const { command, onboardModule, session, checkpointMigration, resumeIntent } = boundaryModules; diff --git a/src/lib/onboard/runtime-provider/docker.ts b/src/lib/onboard/runtime-provider/docker.ts index dacc02615a9..353e1dfd440 100644 --- a/src/lib/onboard/runtime-provider/docker.ts +++ b/src/lib/onboard/runtime-provider/docker.ts @@ -13,6 +13,11 @@ import { recoverDockerDriverSandbox, } from "../docker-driver-sandbox-recovery"; import { createDockerManagedBootstrapSurface } from "../managed-bootstrap/docker-runtime"; +import { + hasPortableDemoSandboxLifecycleReceipt, + recoverPortableDemoSandboxLifecycle, + stopPortableDemoSandboxLifecycle, +} from "../experimental/portable-demo-lifecycle"; import { MANAGED_IMAGE_CAPABILITY_CONTRACT_VERSION, MANAGED_IMAGE_PLATFORMS, @@ -53,12 +58,15 @@ export interface DockerRuntimeProviderDependencies { timeout?: number, ) => RuntimeProviderCommandCapture; readonly findLabeledSandboxContainers: typeof findLabeledSandboxContainers; + readonly hasPortableLifecycleReceipt: typeof hasPortableDemoSandboxLifecycleReceipt; readonly isRuntimeDown: typeof isDockerRuntimeDown; readonly printRuntimeDownGuidance: typeof printDockerRuntimeDownGuidance; readonly recoverSandbox: typeof recoverDockerDriverSandbox; + readonly recoverPortableSandbox: typeof recoverPortableDemoSandboxLifecycle; readonly queryRuntimeSnapshot: typeof queryOpenShellDockerSandboxRuntimeSnapshot; readonly removeImage: DockerRemoveImage; readonly stopContainer: DockerStop; + readonly stopPortableSandbox: typeof stopPortableDemoSandboxLifecycle; readonly unpauseContainer: DockerUnpause; } @@ -86,15 +94,21 @@ function resolveDependencies( ((command, args, timeout) => captureHostCommand(command, args, timeout)), findLabeledSandboxContainers: overrides.findLabeledSandboxContainers ?? findLabeledSandboxContainers, + hasPortableLifecycleReceipt: + overrides.hasPortableLifecycleReceipt ?? hasPortableDemoSandboxLifecycleReceipt, isRuntimeDown: overrides.isRuntimeDown ?? isDockerRuntimeDown, printRuntimeDownGuidance: overrides.printRuntimeDownGuidance ?? printDockerRuntimeDownGuidance, recoverSandbox: overrides.recoverSandbox ?? recoverDockerDriverSandbox, + recoverPortableSandbox: + overrides.recoverPortableSandbox ?? recoverPortableDemoSandboxLifecycle, queryRuntimeSnapshot: overrides.queryRuntimeSnapshot ?? queryOpenShellDockerSandboxRuntimeSnapshot, removeImage: overrides.removeImage ?? ((reference, options) => loadDockerRemoveImage()(reference, options)), stopContainer: overrides.stopContainer ?? ((name, options) => loadDockerStop()(name, options)), + stopPortableSandbox: + overrides.stopPortableSandbox ?? stopPortableDemoSandboxLifecycle, unpauseContainer: overrides.unpauseContainer ?? ((name, options) => loadDockerUnpause()(name, options)), }; @@ -124,6 +138,14 @@ function dockerLifecyclePreflight( input: RuntimeProviderLifecycleInput, deps: DockerRuntimeProviderDependencies, ): RuntimeProviderLifecycleResult | null { + try { + if (deps.hasPortableLifecycleReceipt(input.sandboxName, input.environment)) return null; + } catch (error) { + return { + exitCode: 1, + message: error instanceof Error ? error.message : String(error), + }; + } if (!deps.isRuntimeDown(input.sandboxName)) return null; deps.printRuntimeDownGuidance(input.sandboxName, { retryCommand: action }); return { exitCode: 1 }; @@ -141,6 +163,22 @@ function startDockerSandbox( input: RuntimeProviderLifecycleInput, deps: DockerRuntimeProviderDependencies, ): RuntimeProviderLifecycleResult { + try { + const portable = deps.recoverPortableSandbox( + input.sandboxName, + { + agent: input.sandbox.agent, + gatewayName: input.sandbox.gatewayName ?? "nemoclaw", + lifecycleGeneration: input.sandbox.lifecycleGeneration, + openshellDriver: input.sandbox.openshellDriver, + provider: input.sandbox.provider, + }, + { env: input.environment, log: input.log }, + ); + if (portable.kind !== "not-installed") return { exitCode: 0 }; + } catch (error) { + return { exitCode: 1, message: error instanceof Error ? error.message : String(error) }; + } const containers = deps.findLabeledSandboxContainers(input.sandboxName); const paused = containers.find((container) => isPausedStatus(container.status)); if (paused) { @@ -186,6 +224,26 @@ function stopDockerSandbox( hooks: RuntimeProviderLifecycleStopHooks, deps: DockerRuntimeProviderDependencies, ): RuntimeProviderLifecycleStopOutcome { + try { + const portable = deps.stopPortableSandbox( + input.sandboxName, + { + agent: input.sandbox.agent, + gatewayName: input.sandbox.gatewayName ?? "nemoclaw", + lifecycleGeneration: input.sandbox.lifecycleGeneration, + openshellDriver: input.sandbox.openshellDriver, + provider: input.sandbox.provider, + }, + hooks.beforeStop, + { env: input.environment, log: input.log }, + ); + if (portable.kind === "already-stopped") { + return { exitCode: 0, state: "already-stopped" }; + } + if (portable.kind === "stopped") return { exitCode: 0, state: "stopped" }; + } catch (error) { + return { exitCode: 1, message: error instanceof Error ? error.message : String(error) }; + } const containers = deps.findLabeledSandboxContainers(input.sandboxName); if (containers.length === 0) { return { diff --git a/src/lib/onboard/sandbox-gpu-create-flow.test.ts b/src/lib/onboard/sandbox-gpu-create-flow.test.ts index 163293b4359..ab40558f90a 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.test.ts @@ -53,6 +53,7 @@ vi.mock("./openshell-docker-sandbox-containers", async (importOriginal) => ({ })); import type { AgentDefinition } from "../agent/defs"; +import type { CheckpointPortableRuntimeAuthority } from "../state/onboard-checkpoint-types"; import type { SandboxGpuProofResult } from "../state/registry"; import { createGpuFlowDeps as createDeps, @@ -80,6 +81,7 @@ import type { import { createRuntimeProviderBundleRegistry } from "./runtime-provider/registry"; import { prepareSandboxCreateLaunch } from "./sandbox-create-launch"; import { + resolveExportedPortableRuntimeAuthority, resolveAgentCreateInput, resolvePortableLifecycleMode, runSandboxGpuCreateFlow, @@ -114,6 +116,17 @@ const DEFAULT_RUNTIME_SNAPSHOT = { containerId: "container-a", }; +const PORTABLE_RUNTIME_AUTHORITY: CheckpointPortableRuntimeAuthority = { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid: 1001, + homeDir: "/home/tester", + configHome: "/home/tester/.config", + runtimeDir: "/run/user/1001", + socketPath: "/run/user/1001/podman/podman.sock", +}; + type OpenShellResult = ReturnType; function readySandboxGetResult(): OpenShellResult { @@ -209,6 +222,36 @@ describe("resolveAgentCreateInput", () => { }); }); +describe("resolveExportedPortableRuntimeAuthority", () => { + it("passes checkpoint-owned authority to exported portable creation (#9070)", () => { + expect( + resolveExportedPortableRuntimeAuthority( + { NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }, + () => ({ + checkpoint: { + profile: { kind: "selected", value: "portable" }, + runtimeAuthority: { kind: "selected", value: PORTABLE_RUNTIME_AUTHORITY }, + }, + }), + ), + ).toEqual(PORTABLE_RUNTIME_AUTHORITY); + }); + + it("rejects exported portable creation before effects when authority is absent (#9070)", () => { + expect(() => + resolveExportedPortableRuntimeAuthority( + { NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }, + () => ({ + checkpoint: { + profile: { kind: "selected", value: "portable" }, + runtimeAuthority: { kind: "unset" }, + }, + }), + ), + ).toThrow("requires checkpoint-owned Podman runtime authority before creation begins"); + }); +}); + describe("runSandboxGpuCreateFlow provider-owned managed create", () => { it("recovers before an MXC-style create without a Docker branch in central orchestration", async () => { const input = createInput(); @@ -435,7 +478,6 @@ describe("runSandboxGpuCreateFlow provider-owned managed create", () => { expect(errorOutput()).not.toContain(recoverySecret); }); }); - describe("runSandboxGpuCreateFlow proof authorization", () => { it("does not retry compatibility when the native proof throws an exec/policy error (#6110)", async () => { const deps = createDeps(); @@ -938,6 +980,7 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { it("uses the provided lifecycle generation for portable setup and registration (#8942)", async () => { const input = createInput(); input.lifecycleGeneration = "current-generation"; + input.portableRuntimeAuthority = PORTABLE_RUNTIME_AUTHORITY; const deps = createDeps(); deps.installPortableDemoLifecycle = vi.fn( (_sandboxName, _startupCommand, _env, options) => options.registryGeneration ?? null, @@ -954,13 +997,17 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { input.sandboxName, input.sandboxStartupCommand, process.env, - { registryGeneration: "current-generation" }, + { + registryGeneration: "current-generation", + runtimeAuthority: PORTABLE_RUNTIME_AUTHORITY, + }, ); }); it("preserves the provided lifecycle generation when portable setup is unavailable (#8942)", async () => { const input = createInput(); input.lifecycleGeneration = "fresh-generation"; + input.portableRuntimeAuthority = PORTABLE_RUNTIME_AUTHORITY; const deps = createDeps(); deps.installPortableDemoLifecycle = vi.fn(() => null); @@ -973,7 +1020,10 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { input.sandboxName, input.sandboxStartupCommand, process.env, - { registryGeneration: "fresh-generation" }, + { + registryGeneration: "fresh-generation", + runtimeAuthority: PORTABLE_RUNTIME_AUTHORITY, + }, ); }); @@ -998,6 +1048,7 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { input.gpuRoutePlan = "native-only"; input.hostEnv = { NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }; input.portableLifecycle = true; + input.portableRuntimeAuthority = PORTABLE_RUNTIME_AUTHORITY; input.lifecycleGeneration = "checkpoint-generation"; input.persistStartupCommand = true; const deps = createDeps(); @@ -1014,7 +1065,10 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { input.sandboxName, input.sandboxStartupCommand, input.hostEnv, - { registryGeneration: "checkpoint-generation" }, + { + registryGeneration: "checkpoint-generation", + runtimeAuthority: PORTABLE_RUNTIME_AUTHORITY, + }, ); expect(mocks.waitForCreatedSandboxReadyWithTrace.mock.invocationCallOrder[0]).toBeLessThan( vi.mocked(deps.installPortableDemoLifecycle).mock.invocationCallOrder[0]!, @@ -1035,6 +1089,7 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { input.gpuRoutePlan = "native-only"; input.hostEnv = { NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }; input.portableLifecycle = true; + input.portableRuntimeAuthority = PORTABLE_RUNTIME_AUTHORITY; const deps = createDeps(); deps.installPortableDemoLifecycle = vi.fn(() => { throw new Error("portable authority changed"); @@ -1056,6 +1111,7 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { input.gpuRoutePlan = "native-only"; input.hostEnv = { NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }; input.portableLifecycle = true; + input.portableRuntimeAuthority = PORTABLE_RUNTIME_AUTHORITY; const deps = createDeps(); deps.installPortableDemoLifecycle = vi.fn(() => "current-generation"); vi.mocked(deps.verifyDirectSandboxGpu).mockImplementation(() => { diff --git a/src/lib/onboard/sandbox-gpu-create-flow.ts b/src/lib/onboard/sandbox-gpu-create-flow.ts index d53e4e192eb..b850951fdfc 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.ts @@ -4,6 +4,8 @@ import type { AgentDefinition } from "../agent/defs"; import type { StreamSandboxCreateResult } from "../sandbox/create-stream"; import { redactFull } from "../security/redact"; +import type { CheckpointPortableRuntimeAuthority } from "../state/onboard-checkpoint-types"; +import { parsePortableRuntimeAuthority } from "../state/onboard/portable-runtime-authority"; import type { SandboxEntry, SandboxGpuProofResult } from "../state/registry"; import * as dockerGpuLocalInference from "./docker-gpu-local-inference"; import { collectDockerGpuPatchDiagnostics } from "./docker-gpu-patch"; @@ -44,6 +46,32 @@ export function resolvePortableLifecycleMode( return isPortableExperimentalProfile(env) && (agent?.name ?? "openclaw") === "openclaw"; } +/** Resolve the checkpoint-owned authority required by exported portable creation helpers. */ +export function resolveExportedPortableRuntimeAuthority( + env: NodeJS.ProcessEnv, + loadSession: () => + | { + checkpoint?: { + profile: { kind: "selected"; value: "default" | "portable" }; + runtimeAuthority: + | { kind: "unset" } + | { kind: "selected"; value: CheckpointPortableRuntimeAuthority }; + } | null; + } + | null, +): CheckpointPortableRuntimeAuthority | null { + if (!isPortableExperimentalProfile(env)) return null; + const checkpoint = loadSession()?.checkpoint; + const authority = + checkpoint?.profile.value === "portable" && checkpoint.runtimeAuthority.kind === "selected" + ? parsePortableRuntimeAuthority(checkpoint.runtimeAuthority.value) + : null; + if (authority) return authority; + throw new Error( + "Portable sandbox creation requires checkpoint-owned Podman runtime authority before creation begins.", + ); +} + export function resolveAgentCreateInput( agent: AgentDefinition | null, dockerDriverGateway: boolean, @@ -107,6 +135,7 @@ export interface SandboxGpuCreateFlowInput { sandboxEnv: NodeJS.ProcessEnv; sandboxStartupCommand: string[]; lifecycleGeneration?: SandboxEntry["lifecycleGeneration"]; + portableRuntimeAuthority?: CheckpointPortableRuntimeAuthority | null; prebuild: SandboxPrebuildResult; restoreBackupPath: string | null; terminalAgent: boolean; @@ -292,6 +321,7 @@ export async function runSandboxGpuCreateFlow( process.env, { ...(input.lifecycleGeneration ? { registryGeneration: input.lifecycleGeneration } : {}), + runtimeAuthority: input.portableRuntimeAuthority ?? null, }, ) ?? null; } catch (error) { diff --git a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts index 916f76ca967..94a6c75b35f 100644 --- a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts +++ b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts @@ -78,6 +78,7 @@ function createPortableRuntimePatch( input.hostEnv ?? process.env, { ...(input.lifecycleGeneration ? { registryGeneration: input.lifecycleGeneration } : {}), + runtimeAuthority: input.portableRuntimeAuthority ?? null, }, ); if (!generation) { diff --git a/src/lib/state/onboard-checkpoint.ts b/src/lib/state/onboard-checkpoint.ts index d4bb280b8aa..fdf63fbd09b 100644 --- a/src/lib/state/onboard-checkpoint.ts +++ b/src/lib/state/onboard-checkpoint.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import path from "node:path"; + import { SUPPORTED_GATEWAY_CAPABILITIES } from "../core/gateway-capabilities"; import { isObjectRecord } from "../core/json-types"; import { DEFAULT_GATEWAY_PORT } from "../core/ports"; @@ -20,7 +21,6 @@ import { type CheckpointLoadResult, type CheckpointMessagingSelection, type CheckpointOnboardProfile, - type CheckpointPortableRuntimeAuthority, type CheckpointProfileDecision, type CheckpointProviderBinding, type CheckpointResourceProfile, @@ -31,6 +31,7 @@ import { type CheckpointSandboxRecreateTransaction, type OnboardCheckpoint, } from "./onboard-checkpoint-types"; +import { parsePortableRuntimeAuthority } from "./onboard/portable-runtime-authority"; const EFFECT_GROUP_NAMES: readonly CheckpointEffectGroupName[] = [ "web_search_provider", @@ -72,22 +73,6 @@ function hasExactKeys(value: Record, expected: readonly string[ return actual.length === wanted.length && actual.every((key, index) => key === wanted[index]); } -function readCanonicalAbsolutePath(value: unknown): string | null { - if (typeof value !== "string" || value === "" || /[\0\r\n]/u.test(value)) return null; - if (!path.isAbsolute(value) || path.normalize(value) !== value) return null; - return value; -} - -function isStrictDescendant(root: string, candidate: string): boolean { - const relative = path.relative(root, candidate); - return ( - relative !== "" && - !path.isAbsolute(relative) && - relative !== ".." && - !relative.startsWith(`..${path.sep}`) - ); -} - function parseProfile(value: unknown): CheckpointProfileDecision | null { if (!isObjectRecord(value) || !hasExactKeys(value, ["kind", "value"])) return null; if (value.kind !== "selected" || (value.value !== "default" && value.value !== "portable")) { @@ -96,51 +81,6 @@ function parseProfile(value: unknown): CheckpointProfileDecision | null { return { kind: "selected", value: value.value as CheckpointOnboardProfile }; } -function parsePortableRuntimeAuthority(value: unknown): CheckpointPortableRuntimeAuthority | null { - if ( - !isObjectRecord(value) || - !hasExactKeys(value, [ - "schemaVersion", - "kind", - "ownership", - "uid", - "homeDir", - "configHome", - "runtimeDir", - "socketPath", - ]) - ) { - return null; - } - if ( - value.schemaVersion !== 1 || - value.kind !== "podman" || - value.ownership !== "current-user" || - !Number.isSafeInteger(value.uid) || - Number(value.uid) < 0 - ) { - return null; - } - const homeDir = readCanonicalAbsolutePath(value.homeDir); - const configHome = readCanonicalAbsolutePath(value.configHome); - const runtimeDir = readCanonicalAbsolutePath(value.runtimeDir); - const socketPath = readCanonicalAbsolutePath(value.socketPath); - if (!homeDir || !configHome || !runtimeDir || !socketPath) return null; - if (configHome !== path.join(homeDir, ".config")) return null; - if (runtimeDir !== path.join("/run/user", String(value.uid))) return null; - if (!isStrictDescendant(runtimeDir, socketPath)) return null; - return { - schemaVersion: 1, - kind: "podman", - ownership: "current-user", - uid: Number(value.uid), - homeDir, - configHome, - runtimeDir, - socketPath, - }; -} - function parseRuntimeAuthority(value: unknown): CheckpointRuntimeAuthorityDecision | null { if (!isObjectRecord(value)) return null; if (hasExactKeys(value, ["kind"]) && value.kind === "unset") return { kind: "unset" }; diff --git a/src/lib/state/onboard/portable-runtime-authority.ts b/src/lib/state/onboard/portable-runtime-authority.ts new file mode 100644 index 00000000000..e0f8b4a5cf5 --- /dev/null +++ b/src/lib/state/onboard/portable-runtime-authority.ts @@ -0,0 +1,80 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import path from "node:path"; + +import type { CheckpointPortableRuntimeAuthority } from "../onboard-checkpoint-types"; + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function hasExactKeys(value: Record, expected: readonly string[]): boolean { + const actual = Object.keys(value).sort(); + const wanted = [...expected].sort(); + return actual.length === wanted.length && actual.every((key, index) => key === wanted[index]); +} + +function readCanonicalAbsolutePath(value: unknown): string | null { + if (typeof value !== "string" || value === "" || /[\0\r\n]/u.test(value)) return null; + if (!path.isAbsolute(value) || path.normalize(value) !== value) return null; + return value; +} + +function isStrictDescendant(root: string, candidate: string): boolean { + const relative = path.relative(root, candidate); + return ( + relative !== "" && + !path.isAbsolute(relative) && + relative !== ".." && + !relative.startsWith(`..${path.sep}`) + ); +} + +/** Parse the secret-free, current-user Podman authority shared by checkpoints and receipts. */ +export function parsePortableRuntimeAuthority( + value: unknown, +): CheckpointPortableRuntimeAuthority | null { + if ( + !isRecord(value) || + !hasExactKeys(value, [ + "schemaVersion", + "kind", + "ownership", + "uid", + "homeDir", + "configHome", + "runtimeDir", + "socketPath", + ]) + ) { + return null; + } + if ( + value.schemaVersion !== 1 || + value.kind !== "podman" || + value.ownership !== "current-user" || + !Number.isSafeInteger(value.uid) || + Number(value.uid) < 0 + ) { + return null; + } + const homeDir = readCanonicalAbsolutePath(value.homeDir); + const configHome = readCanonicalAbsolutePath(value.configHome); + const runtimeDir = readCanonicalAbsolutePath(value.runtimeDir); + const socketPath = readCanonicalAbsolutePath(value.socketPath); + if (!homeDir || !configHome || !runtimeDir || !socketPath) return null; + if (configHome !== path.join(homeDir, ".config")) return null; + if (runtimeDir !== path.join("/run/user", String(value.uid))) return null; + if (!isStrictDescendant(runtimeDir, socketPath)) return null; + return { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid: Number(value.uid), + homeDir, + configHome, + runtimeDir, + socketPath, + }; +} diff --git a/test/e2e/live/podman-cpu-lifecycle.test.ts b/test/e2e/live/podman-cpu-lifecycle.test.ts index d8ff495aacc..65db0df75bc 100644 --- a/test/e2e/live/podman-cpu-lifecycle.test.ts +++ b/test/e2e/live/podman-cpu-lifecycle.test.ts @@ -19,6 +19,7 @@ import { installPortableDemoSandboxLifecycle, portableDemoLifecycleInternals, } from "../../../src/lib/onboard/experimental/portable-demo-lifecycle"; +import { inspectPortablePodmanReadiness } from "../../../src/lib/onboard/experimental/portable-runtime-readiness"; import type { RuntimeProviderBundle, RuntimeProviderLifecycleInput, @@ -60,6 +61,7 @@ const SUPERVISOR_IMAGE = const E2E_PHASES = [ "pin the exact rootless Podman endpoint", "qualify the Podman 5 host contract", + "prove cold activation and warm API readiness", "start the pinned OpenShell Podman gateway", "activate registered-agent identities through the pinned OpenShell CLI", "exercise exact-container stop and start", @@ -96,7 +98,7 @@ test("activates pinned OpenShell sandboxes and preserves registered-agent Podman expect(process.platform).toBe("linux"); expect(process.getuid?.()).not.toBe(0); expect(ARTIFACT_DIR).not.toBe(""); - const runtimeEngines = engines(); + let runtimeEngines = engines(); const bundle = createPodmanRuntimeProviderBundle({ engines: runtimeEngines }); progress.phase("qualify the Podman 5 host contract"); @@ -122,6 +124,69 @@ test("activates pinned OpenShell sandboxes and preserves registered-agent Podman ).toContain(OPENSHELL_VERSION); } + const uid = process.getuid?.() ?? -1; + expect(uid, "Rootless portable lifecycle evidence requires a non-root Linux UID").toBeGreaterThan( + 0, + ); + const runtimeAuthority = { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid, + homeDir: os.homedir(), + configHome: path.join(os.homedir(), ".config"), + runtimeDir: path.join("/run/user", String(uid)), + socketPath: SOCKET_PATH, + } as const; + + progress.phase("prove cold activation and warm API readiness"); + const proofServicePid = process.env.E2E_PODMAN_SERVICE_PID ?? ""; + expect(proofServicePid).toMatch(/^[1-9][0-9]*$/u); + await runCommand( + shellProbe, + "bash", + [ + "-ceu", + ` +pid="$1" +kill "$pid" +for _attempt in $(seq 1 100); do + if ! kill -0 "$pid" 2>/dev/null; then + exit 0 + fi + sleep 0.1 +done +printf 'Podman proof service %s did not stop\n' "$pid" >&2 +exit 1 +`, + "podman-proof-service-stop", + proofServicePid, + ], + { artifactName: "podman-lifecycle-stop-proof-service", timeoutMs: 60_000 }, + ); + expect(fs.existsSync(`/proc/${proofServicePid}`)).toBe(false); + fs.rmSync(SOCKET_PATH, { force: true }); + await runCommand( + shellProbe, + "systemctl", + ["--user", "stop", "podman.service", "podman.socket"], + { artifactName: "podman-lifecycle-stop-user-units", timeoutMs: 10_000 }, + ); + for (const unit of ["podman.service", "podman.socket"]) { + expect( + await runCommand(shellProbe, "systemctl", ["--user", "is-active", unit], { + allowFailure: true, + artifactName: `podman-lifecycle-cold-${unit}`, + timeoutMs: 10_000, + }), + ).not.toBe("active"); + } + const coldReadiness = inspectPortablePodmanReadiness(runtimeAuthority); + expect(coldReadiness).toMatchObject({ ok: true, timing: { mode: "cold" } }); + const warmReadiness = inspectPortablePodmanReadiness(runtimeAuthority); + expect(warmReadiness).toMatchObject({ ok: true, timing: { mode: "warm" } }); + runtimeEngines = engines(); + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-podman-openshell-")); const stateDir = path.join(root, "gateway-state"); const cliEnv: NodeJS.ProcessEnv = { @@ -253,6 +318,7 @@ test("activates pinned OpenShell sandboxes and preserves registered-agent Podman const openclawSandbox = AGENTS[0].sandboxName; const portableStateDir = path.join(root, "portable-lifecycle"); + const readinessLogs: string[] = []; installPortableDemoSandboxLifecycle( openclawSandbox, [ @@ -268,21 +334,29 @@ test("activates pinned OpenShell sandboxes and preserves registered-agent Podman { ...process.env, NEMOCLAW_EXPERIMENTAL_PROFILE: "portable" }, { platform: "linux", - podman: (args) => - runtimeEngines.sandboxLifecycle.capture(args[0] === "--url" ? args.slice(2) : args), + log: (message) => readinessLogs.push(message), stateDir: portableStateDir, + runtimeAuthority, }, ); + expect(readinessLogs).toContainEqual(expect.stringContaining("readiness: warm")); + runtimeEngines = engines(); const portableReceipt = JSON.parse( fs.readFileSync( portableDemoLifecycleInternals.receiptPath(openclawSandbox, portableStateDir), "utf-8", ), - ) as { containerId: string; sandboxName: string; schemaVersion: number }; + ) as { + containerId: string; + runtimeAuthority: { socketPath: string }; + sandboxName: string; + schemaVersion: number; + }; expect(portableReceipt).toMatchObject({ containerId: exactContainerId(runtimeEngines.sandboxLifecycle, openclawSandbox), sandboxName: openclawSandbox, - schemaVersion: 3, + runtimeAuthority: { socketPath: SOCKET_PATH }, + schemaVersion: 4, }); progress.phase("exercise exact-container stop and start"); diff --git a/test/gateway-failure-classifier.test.ts b/test/gateway-failure-classifier.test.ts index c8d0d0ce211..70c85e6ec1c 100644 --- a/test/gateway-failure-classifier.test.ts +++ b/test/gateway-failure-classifier.test.ts @@ -1,7 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it } from "vitest"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { describe, expect, it, vi } from "vitest"; import { classifyGatewayFailure, @@ -12,6 +16,45 @@ import { printDockerRuntimeDownGuidance, type SandboxContainerFailureRunners, } from "../src/lib/actions/sandbox/gateway-failure-classifier.js"; +import type { PodmanSocketAuthority } from "../src/lib/adapters/podman/index.js"; +import { portableDemoLifecycleInternals } from "../src/lib/onboard/experimental/portable-demo-lifecycle.js"; + +const PORTABLE_SOCKET = "/run/user/1001/podman/podman.sock"; +const PORTABLE_SOCKET_AUTHORITY: PodmanSocketAuthority = { + directoryChain: [], + device: "1", + inode: "2", + mode: String(0o140600), + ownerUid: "1001", + socketPath: PORTABLE_SOCKET, +}; + +function writePortableReceipt(stateDir: string): void { + const receiptPath = portableDemoLifecycleInternals.receiptPath("alpha", stateDir); + fs.mkdirSync(path.dirname(receiptPath), { recursive: true }); + fs.writeFileSync( + receiptPath, + `${JSON.stringify({ + schemaVersion: 4, + sandboxName: "alpha", + sandboxId: "sandbox-id-alpha", + containerId: "a".repeat(64), + dashboardPort: 18789, + registryGeneration: "a".repeat(64), + runtimeAuthority: { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid: 1001, + homeDir: "/home/tester", + configHome: "/home/tester/.config", + runtimeDir: "/run/user/1001", + socketPath: PORTABLE_SOCKET, + }, + })}\n`, + { mode: 0o600 }, + ); +} function makeRunners(overrides: Partial = {}): GatewayFailureRunners { return { @@ -144,6 +187,67 @@ describe("isDockerRuntimeDown", () => { ).toBe(false); }); + it("uses receipt-owned Podman readiness without probing ordinary Docker (#9070)", () => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-runtime-readiness-")); + writePortableReceipt(stateDir); + const dockerInfo = vi.fn(() => true); + try { + expect( + isDockerRuntimeDown("alpha", { + runners: { dockerInfo }, + getSandbox: dockerSandbox, + portableLifecycle: { + platform: "linux", + stateDir, + runtimeReadiness: { + uid: 1001, + home: "/home/tester", + systemctl: () => ({ status: 0 }), + hardenSocketDirectory: vi.fn(), + captureSocketAuthority: () => PORTABLE_SOCKET_AUTHORITY, + assertSocketAuthority: vi.fn(), + podmanCapture: () => ({ status: 1, stdout: "", stderr: "" }), + }, + }, + }), + ).toBe(true); + expect(dockerInfo).not.toHaveBeenCalled(); + const out: string[] = []; + printDockerRuntimeDownGuidance("alpha", { writer: (line) => out.push(line) }); + expect(out.join("\n")).toContain("steady-state API health"); + expect(out.join("\n")).toContain( + `Recorded socket: ${PORTABLE_SOCKET_AUTHORITY.socketPath}`, + ); + expect(out.join("\n")).toContain("no Docker or named-connection fallback was used"); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } + }); + + it("classifies an unsafe portable receipt without exposing its contents (#9070)", () => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-runtime-readiness-")); + const receiptPath = portableDemoLifecycleInternals.receiptPath("alpha", stateDir); + fs.mkdirSync(path.dirname(receiptPath), { recursive: true }); + fs.writeFileSync(receiptPath, '{"credential":"super-secret"}\n', { mode: 0o600 }); + const dockerInfo = vi.fn(() => true); + try { + expect( + isDockerRuntimeDown("alpha", { + runners: { dockerInfo }, + getSandbox: dockerSandbox, + portableLifecycle: { stateDir }, + }), + ).toBe(true); + expect(dockerInfo).not.toHaveBeenCalled(); + const out: string[] = []; + printDockerRuntimeDownGuidance("alpha", { writer: (line) => out.push(line) }); + expect(out.join("\n")).toContain("socket authority"); + expect(out.join("\n")).not.toContain("super-secret"); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } + }); + it("returns false for the vm driver even when docker info fails", () => { // A vm sandbox runs in a real VM with no local Docker daemon, so a failing // `docker info` must not be misclassified as a runtime outage. diff --git a/test/onboard.test.ts b/test/onboard.test.ts index 6c4fbedfb2d..9e91e4625f4 100644 --- a/test/onboard.test.ts +++ b/test/onboard.test.ts @@ -938,6 +938,9 @@ const { createSandbox } = require(${onboardPath}); const fakeBin = path.join(tmpDir, "bin"); const scriptPath = path.join(tmpDir, "portable-managed-recreate.js"); const onboardPath = JSON.stringify(path.join(repoRoot, "src", "lib", "onboard.ts")); + const onboardSessionPath = JSON.stringify( + path.join(repoRoot, "src", "lib", "state", "onboard-session.ts"), + ); const runnerPath = JSON.stringify(path.join(repoRoot, "src", "lib", "runner.ts")); const registryPath = JSON.stringify(path.join(repoRoot, "src", "lib", "state", "registry.ts")); const catalogPath = JSON.stringify( @@ -962,6 +965,25 @@ const { createSandbox } = require(${onboardPath}); const script = String.raw` const runner = require(${runnerPath}); require(${scriptMocksPath}).mockStandaloneGatewayTeardownAuthority(); +const onboardSession = require(${onboardSessionPath}); +onboardSession.loadSession = () => ({ + checkpoint: { + profile: { kind: "selected", value: "portable" }, + runtimeAuthority: { + kind: "selected", + value: { + schemaVersion: 1, + kind: "podman", + ownership: "current-user", + uid: 1001, + homeDir: "/home/tester", + configHome: "/home/tester/.config", + runtimeDir: "/run/user/1001", + socketPath: "/run/user/1001/podman/podman.sock", + }, + }, + }, +}); const events = []; const normalize = (command) => (Array.isArray(command) ? command.join(" ") : String(command)).replace(/'/g, "");