diff --git a/ci/source-shape-test-budget.json b/ci/source-shape-test-budget.json index 8e473b6d06a..c5f7af36e78 100644 --- a/ci/source-shape-test-budget.json +++ b/ci/source-shape-test-budget.json @@ -528,7 +528,7 @@ }, { "file": "test/runtime-provider-source-shape.test.ts", - "test": "keeps production activation paths disconnected from managed bootstrap", + "test": "keeps production activation paths disconnected from driver bootstrap adapters", "category": "security" }, { diff --git a/docs/inference/verify-inference-route.mdx b/docs/inference/verify-inference-route.mdx index b13874e8184..b260592517c 100644 --- a/docs/inference/verify-inference-route.mdx +++ b/docs/inference/verify-inference-route.mdx @@ -41,11 +41,15 @@ Restore the configured endpoint or proxy, run `$$nemoclaw onboard --resume` to c ## Understand Local Provider Post-Ready Checks -For local Ollama and vLLM on Docker GPU sandboxes using the compatibility route, onboarding performs an additional check after the sandbox becomes ready. +For local Ollama, local vLLM, and local NVIDIA NIM on Docker GPU sandboxes using the compatibility route, onboarding performs an additional check after the sandbox becomes ready. +Local NIM uses the `vllm-local` route, so it receives the same reversible post-ready check as local vLLM. It requests `https://inference.local/v1/models` from inside the sandbox and accepts only a 2xx response. -When this check fails, onboarding reports the endpoint and local-provider recovery steps before the first agent prompt. +If this check fails after compatibility recreation, onboarding prints failure diagnostics and attempts to restore the pre-patch container before it exits. +If that rollback fails, onboarding reports that the pre-patch container was not restored and prints container-cleanup guidance. +The local-provider failure output includes the endpoint and recovery steps before the first agent prompt. +GPU-proof diagnostics are captured before rollback and can also print cleanup guidance before the final container state is known, so inspect the sandbox and its labeled Docker containers before running a deletion command. -NVIDIA NIM and other compatible endpoints receive their provider validation during onboarding but do not receive this local-provider post-ready check. +Remote NVIDIA NIM and other compatible endpoints receive their provider validation during onboarding but do not receive this local-provider post-ready check. For those routes, continue to the final route check, then use the status command and a short agent request after onboarding. ## Understand Final Route Checks diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 7d231cb4331..ceb977e9b7b 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -813,7 +813,10 @@ On ordinary native Linux, the compatibility path uses an available NVIDIA CDI sp On Docker Desktop WSL, the compatibility path skips CDI and tries Docker `--gpus all` before the NVIDIA runtime. On Jetson/Tegra hosts, the compatibility path uses the NVIDIA runtime and adds eligible host group IDs for the supported GPU device nodes. These include selected `/dev/nvmap`, `/dev/nvhost-*`, and `/dev/nvgpu/igpu0/*` nodes plus real `/dev/dri/renderD*` character devices. -If the compatibility attempt fails, onboarding keeps its diagnostics and the failed sandbox in place and prints a manual cleanup command. +After compatibility recreation starts, onboarding keeps the pre-patch container as a rollback backup until the replacement passes the Ready, GPU, and applicable local-inference checks. +If a later check fails, onboarding prints failure diagnostics and attempts to restore the pre-patch container. +If rollback fails, onboarding reports that the pre-patch container was not restored and prints container-cleanup guidance. +GPU-proof diagnostics are captured before rollback and can print that guidance before the final container state is known, so inspect the sandbox and its labeled Docker containers before running a deletion command. Prerequisites: diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index e8d7228b931..51fb92cd978 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -2506,7 +2506,7 @@ Identify the matching path before applying the recovery guidance. | --- | --- | --- | | Native `--gpu` is rejected, host runtime evidence identifies GPU injection failure, or an explicit driver proof fails and host configuration confirms no GPU attachment | Ordinary Linux native attempt | The default native-only route stops. Retry with `NEMOCLAW_DOCKER_GPU_PATCH=fallback` only if you explicitly accept one bounded compatibility retry, or use `=1` to select compatibility before creation. | | `Cleanup could not be proven safe` | Native-to-compatibility handoff | Run the printed sandbox deletion command, verify both the gateway row and OpenShell-managed Docker containers labeled for that sandbox are absent, then rerun onboarding. | -| The patched container exits or the compatibility attempt fails | Compatibility recreation | Inspect the saved diagnostics, repair the NVIDIA Container Toolkit/CDI configuration, clean up the failed sandbox, and rerun onboarding. | +| The patched container exits or the compatibility attempt fails | Compatibility recreation | Inspect the saved diagnostics and the rollback outcome, then repair the NVIDIA Container Toolkit/CDI configuration. Keep the sandbox when the pre-patch container was restored; delete it only after inspection confirms restoration failed. Then rerun onboarding. | | A recreated container inherits only a loopback DNS stub and no usable upstream | Compatibility DNS fallback | Repair the host's `systemd-resolved` upstream configuration, then rerun onboarding. | For bridge-networked compatibility recreation without an explicit container DNS setting, NemoClaw selects a usable IPv4 upstream from `systemd-resolved` and probes that exact `--dns` path before it stops the original container. @@ -2552,20 +2552,26 @@ Use `NEMOCLAW_DOCKER_GPU_PATCH=0` only for troubleshooting because it bypasses t #### Common compatibility-path recovery -If the compatibility attempt fails on any host, onboarding leaves the failed sandbox and diagnostic bundle in place so you can inspect the OpenShell and Docker state. +After compatibility recreation starts, onboarding keeps the pre-patch container as a rollback backup until the replacement passes the Ready, direct GPU, and applicable local-inference checks. +If a later check fails, onboarding prints failure diagnostics and attempts to restore the pre-patch container before it exits. +When rollback succeeds, the pre-patch sandbox remains available. +When rollback fails, onboarding reports that the pre-patch container was not restored and prints container-cleanup guidance. +GPU-proof diagnostics are captured before rollback and can print that guidance before the final container state is known. +Inspect the sandbox and its labeled Docker containers before running a deletion command. Starting with NemoClaw v0.0.43, the standard installer handles the `/proc//task//comm` permission case during this patch path. If an older release fails direct GPU proof with that path and `Permission denied`, upgrade NemoClaw and rerun onboarding. -The output includes a cleanup command such as: +When inspection confirms that the failed sandbox remains, delete it with a command such as: ```bash openshell sandbox delete ``` -Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics, clean up the failed sandbox, then rerun onboarding. +Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics. +Run the deletion command only after confirming that the pre-patch sandbox was not restored, then rerun onboarding. If you do not need GPU access inside the sandbox, rerun with `--no-sandbox-gpu`. If sandbox creation fails with `CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all`, the OpenShell gateway tried `docker create --device nvidia.com/gpu=all` and Docker could not resolve the CDI spec. diff --git a/src/lib/adapters/openshell/sandbox-identity.ts b/src/lib/adapters/openshell/sandbox-identity.ts index 1820a8f8f7d..dbcc47696c4 100644 --- a/src/lib/adapters/openshell/sandbox-identity.ts +++ b/src/lib/adapters/openshell/sandbox-identity.ts @@ -14,3 +14,19 @@ export function parseOpenShellSandboxId(output: string): string | null { ? (matches[0] as string) : null; } + +export function resolveOpenShellSandboxId( + sandboxName: string, + runCaptureOpenshell: (args: string[], options?: Record) => string, +): string { + const output = runCaptureOpenshell(["sandbox", "get", sandboxName], { + ignoreError: false, + }); + const sandboxId = parseOpenShellSandboxId(output); + if (!sandboxId) { + throw new Error( + `OpenShell sandbox '${sandboxName}' did not return one exact durable sandbox ID.`, + ); + } + return sandboxId; +} diff --git a/src/lib/onboard.ts b/src/lib/onboard.ts index d2c07e3e537..b625740adb2 100644 --- a/src/lib/onboard.ts +++ b/src/lib/onboard.ts @@ -2708,7 +2708,7 @@ async function createSandboxWithBaseImageResolution( recreateRuntime.advance("creating"); const { createResult, - dockerGpuCreatePatch, + runtimePatch, route: selectedGpuRoute, firstCreateOutput, registryImageRef, @@ -2764,7 +2764,7 @@ async function createSandboxWithBaseImageResolution( } if (effectiveSandboxGpuConfig.sandboxGpuEnabled) { - dockerGpuLocalInference.verifyGpuSandboxLocalInferenceAfterReady( + await dockerGpuLocalInference.verifyGpuSandboxLocalInferenceAndCommitAfterReady( effectiveSandboxGpuConfig, provider, { @@ -2772,11 +2772,10 @@ async function createSandboxWithBaseImageResolution( dockerDriverGateway, selectedRoute: selectedGpuRoute, verifyDirectSandboxGpu, - verifyGpuOrExit: dockerGpuCreatePatch.verifyGpuOrExit, - selectedMode: dockerGpuCreatePatch.selectedMode, runCaptureOpenshell, log: console.log, }, + runtimePatch, ); } diff --git a/src/lib/onboard/__test-helpers__/sandbox-gpu-create-flow.ts b/src/lib/onboard/__test-helpers__/sandbox-gpu-create-flow.ts index 7061252f12c..eebacbacd3d 100644 --- a/src/lib/onboard/__test-helpers__/sandbox-gpu-create-flow.ts +++ b/src/lib/onboard/__test-helpers__/sandbox-gpu-create-flow.ts @@ -64,8 +64,10 @@ export function createGpuPatchFixture() { maybeApplyDuringCreate: vi.fn(), createFailureMessage: vi.fn(() => null), exitOnPatchError: vi.fn(), + rollbackManagedStartupAfterCreateFailure: vi.fn(), ensureApplied: vi.fn(), waitForSupervisorReconnectIfNeeded: vi.fn(), + commitAfterReady: vi.fn(), selectedMode: vi.fn(() => null), printReadinessFailureIfEnabled: vi.fn(), verifyGpuOrExit: vi.fn(() => VERIFIED_GPU_PROOF), diff --git a/src/lib/onboard/docker-gpu-local-inference.test.ts b/src/lib/onboard/docker-gpu-local-inference.test.ts index 60dfe28c639..3f0f6180ec4 100644 --- a/src/lib/onboard/docker-gpu-local-inference.test.ts +++ b/src/lib/onboard/docker-gpu-local-inference.test.ts @@ -11,6 +11,7 @@ import { shouldUseDockerGpuPatchHostNetwork, verifyDockerGpuSandboxLocalInference, verifyGpuSandboxAfterReady, + verifyGpuSandboxLocalInferenceAndCommitAfterReady, } from "./docker-gpu-local-inference"; const HOST_NETWORK_ENV = { @@ -311,10 +312,10 @@ describe("verifyGpuSandboxAfterReady", () => { }; } - it("runs the GPU proof and the runtime inference gate when the patch is active", () => { + it("runs the GPU proof and the runtime inference gate when the patch is active", async () => { const log = vi.fn(); const verifyDirectSandboxGpu = vi.fn(); - verifyGpuSandboxAfterReady( + await verifyGpuSandboxAfterReady( GPU_CONFIG, "vllm-local", baseOptions({ @@ -327,12 +328,12 @@ describe("verifyGpuSandboxAfterReady", () => { expect(log).toHaveBeenCalledWith(expect.stringContaining("reached local inference")); }); - it("captures the CUDA-usability proof onto the config for status persistence (#4231)", () => { + it("captures the CUDA-usability proof onto the config for status persistence (#4231)", async () => { const proof = { status: "verified" as const, cudaVerified: true, at: "t" }; const config: { sandboxGpuEnabled: boolean; sandboxGpuProof?: typeof proof | null } = { sandboxGpuEnabled: true, }; - verifyGpuSandboxAfterReady( + await verifyGpuSandboxAfterReady( config, "vllm-local", baseOptions({ @@ -343,45 +344,86 @@ describe("verifyGpuSandboxAfterReady", () => { expect(config.sandboxGpuProof).toEqual(proof); }); - it("does not duplicate proof diagnostics when Docker GPU patch verifier handles them", () => { + it("does not duplicate proof diagnostics when Docker GPU patch verifier handles them", async () => { const proofError = new Error("process.exit"); const verifyGpuOrExit = vi.fn(() => { throw proofError; }); const logError = vi.fn(); - expect(() => + await expect( verifyGpuSandboxAfterReady( GPU_CONFIG, "ollama-local", baseOptions({ verifyGpuOrExit, logError }), ), - ).toThrow(proofError); + ).rejects.toBe(proofError); expect(logError).not.toHaveBeenCalled(); }); - it("routes failure diagnostics through the provided error sink and exits", () => { + it("routes failure diagnostics through the provided error sink and throws for rollback", async () => { const logError = vi.fn(); - const exitSpy = vi.spyOn(process, "exit").mockImplementation((() => { - throw new Error("process.exit"); - }) as never); - try { - expect(() => - verifyGpuSandboxAfterReady( - GPU_CONFIG, - "ollama-local", - baseOptions({ - logError, - deps: { execInSandbox: execEmitting("HTTP_000"), sleep: vi.fn() }, - }), - ), - ).toThrow("process.exit"); - expect(exitSpy).toHaveBeenCalledWith(1); - expect(logError).toHaveBeenCalledWith( - expect.stringContaining("Local inference reachability check failed"), - ); - } finally { - exitSpy.mockRestore(); - } + await expect( + verifyGpuSandboxAfterReady( + GPU_CONFIG, + "ollama-local", + baseOptions({ + logError, + deps: { execInSandbox: execEmitting("HTTP_000"), sleep: vi.fn() }, + }), + ), + ).rejects.toThrow("GPU sandbox local inference reachability failed"); + expect(logError).toHaveBeenCalledWith( + expect.stringContaining("Local inference reachability check failed"), + ); + }); +}); + +describe("verifyGpuSandboxLocalInferenceAndCommitAfterReady", () => { + function options() { + return { + ...gpuPatchOptions(), + verifyDirectSandboxGpu: vi.fn(), + runCaptureOpenshell: vi.fn(() => ""), + log: vi.fn(), + }; + } + + it("commits only after local-inference reachability returns HTTP 2xx", async () => { + const runtimePatch = { + commitAfterReady: vi.fn(), + rollbackManagedStartupAfterCreateFailure: vi.fn(), + }; + await verifyGpuSandboxLocalInferenceAndCommitAfterReady( + GPU_CONFIG, + "ollama-local", + { + ...options(), + deps: { execInSandbox: execEmitting("HTTP_200"), sleep: vi.fn() }, + }, + runtimePatch, + ); + expect(runtimePatch.commitAfterReady).toHaveBeenCalledOnce(); + expect(runtimePatch.rollbackManagedStartupAfterCreateFailure).not.toHaveBeenCalled(); + }); + + it("rolls back before propagating an inference verification failure", async () => { + const runtimePatch = { + commitAfterReady: vi.fn(), + rollbackManagedStartupAfterCreateFailure: vi.fn(), + }; + await expect( + verifyGpuSandboxLocalInferenceAndCommitAfterReady( + GPU_CONFIG, + "ollama-local", + { + ...options(), + deps: { execInSandbox: execEmitting("HTTP_000"), sleep: vi.fn() }, + }, + runtimePatch, + ), + ).rejects.toThrow("GPU sandbox local inference reachability failed"); + expect(runtimePatch.rollbackManagedStartupAfterCreateFailure).toHaveBeenCalledOnce(); + expect(runtimePatch.commitAfterReady).not.toHaveBeenCalled(); }); }); diff --git a/src/lib/onboard/docker-gpu-local-inference.ts b/src/lib/onboard/docker-gpu-local-inference.ts index da58f8c44d8..496ee830448 100644 --- a/src/lib/onboard/docker-gpu-local-inference.ts +++ b/src/lib/onboard/docker-gpu-local-inference.ts @@ -11,6 +11,7 @@ import { import type { DockerGpuPatchMode } from "./docker-gpu-patch-types"; import type { SelectedDockerGpuRoute } from "./docker-gpu-route"; import { adaptDockerGpuRouteForPatch } from "./docker-gpu-route-patch-adapter"; +import type { ManagedBootstrapRuntimePatch } from "./managed-bootstrap/runtime-create"; import { executeSandboxCommandForVerification } from "./sandbox-verification-exec"; const { @@ -385,9 +386,9 @@ export type GpuSandboxAfterReadyOptions = { verifyDirectSandboxGpu: (sandboxName: string) => SandboxGpuProofResult; verifyGpuOrExit?: ( verifyDirectSandboxGpu: (sandboxName: string) => SandboxGpuProofResult, - ) => SandboxGpuProofResult; + ) => Promise; reportGpuProofFailure?: boolean; - selectedMode: () => DockerGpuPatchMode | null; + selectedMode: ManagedBootstrapRuntimePatch["selectedMode"]; runCaptureOpenshell: (args: string[], opts?: Record) => string; env?: NodeJS.ProcessEnv; platform?: NodeJS.Platform; @@ -396,33 +397,47 @@ export type GpuSandboxAfterReadyOptions = { deps?: DockerGpuSandboxInferenceVerifyDeps; }; +function asDockerGpuPatchMode( + selected: ReturnType, +): DockerGpuPatchMode | null { + if (!selected || !["gpus", "nvidia-runtime", "cdi", "startup-command"].includes(selected.kind)) { + return null; + } + return { + kind: selected.kind as DockerGpuPatchMode["kind"], + label: selected.label, + device: selected.device, + args: [...selected.args], + }; +} + /** * Post-readiness GPU sandbox verification orchestrator (kept out of the * ~12k-line onboard.ts entrypoint per the codebase-growth guardrail). Runs the * direct GPU proof, then — only when the Docker GPU patch is active for a local * inference provider — gates on local inference reachability from the sandbox - * runtime (#4509). Exits the process with actionable output if either proof - * fails. + * runtime (#4509). Throws with actionable output if either proof fails so the + * caller can complete rollback before selecting a terminal exit status. */ -export function verifyGpuSandboxAfterReady( +export async function verifyGpuSandboxAfterReady( config: DockerGpuLocalInferenceConfig, provider: string | null | undefined, options: GpuSandboxAfterReadyOptions, -): void { - verifyGpuSandboxAccessAfterReady(config, options); +): Promise { + await verifyGpuSandboxAccessAfterReady(config, options); verifyGpuSandboxLocalInferenceAfterReady(config, provider, options); } -export function verifyGpuSandboxAccessAfterReady( +export async function verifyGpuSandboxAccessAfterReady( config: DockerGpuLocalInferenceConfig, options: GpuSandboxAfterReadyOptions, -): SandboxGpuProofResult { +): Promise { try { // Capture the CUDA-usability proof result and write it back onto the shared // config so onboarding can persist it to the registry and `status` can // report proven usability rather than mere configuration (#4231). const proof = options.verifyGpuOrExit - ? options.verifyGpuOrExit(options.verifyDirectSandboxGpu) + ? await options.verifyGpuOrExit(options.verifyDirectSandboxGpu) : options.verifyDirectSandboxGpu(options.sandboxName); config.sandboxGpuProof = proof; return proof; @@ -431,11 +446,16 @@ export function verifyGpuSandboxAccessAfterReady( // prints the richer Error-phase / patched-container diagnostics before // rethrowing. Avoid a second generic proof-failure block in that path. if (!options.verifyGpuOrExit && options.reportGpuProofFailure !== false) { - printDockerGpuProofFailure(options.sandboxName, error, options.selectedMode(), { - runCaptureOpenshell: options.runCaptureOpenshell, - additionalSummaryLines: adaptDockerGpuRouteForPatch(options.selectedRoute) - .additionalSummaryLines, - }); + printDockerGpuProofFailure( + options.sandboxName, + error, + asDockerGpuPatchMode(options.selectedMode()), + { + runCaptureOpenshell: options.runCaptureOpenshell, + additionalSummaryLines: adaptDockerGpuRouteForPatch(options.selectedRoute) + .additionalSummaryLines, + }, + ); } throw error; } @@ -444,7 +464,7 @@ export function verifyGpuSandboxAccessAfterReady( export function verifyGpuSandboxLocalInferenceAfterReady( config: DockerGpuLocalInferenceConfig, provider: string | null | undefined, - options: GpuSandboxAfterReadyOptions, + options: Omit, ): void { if (options.selectedRoute !== "compatibility") return; const verification = verifyDockerGpuSandboxLocalInference(config, provider, { @@ -469,6 +489,39 @@ export function verifyGpuSandboxLocalInferenceAfterReady( verification, options.logError ?? ((message) => console.error(message)), ); - process.exit(1); + throw new Error( + `GPU sandbox local inference reachability failed for ${verification.endpoint}.`, + ); + } +} + +/** + * Keep the managed create transaction reversible until the sandbox's real + * local-inference reachability check returns HTTP 2xx. Rollback failures are + * attached to the original verification failure so callers retain both pieces + * of evidence. + */ +export async function verifyGpuSandboxLocalInferenceAndCommitAfterReady( + config: DockerGpuLocalInferenceConfig, + provider: string | null | undefined, + options: Omit, + runtimePatch: Pick< + ManagedBootstrapRuntimePatch, + "commitAfterReady" | "rollbackManagedStartupAfterCreateFailure" + >, +): Promise { + try { + verifyGpuSandboxLocalInferenceAfterReady(config, provider, options); + await runtimePatch.commitAfterReady(); + } catch (error) { + const failure = error instanceof Error ? error : new Error(String(error)); + try { + await runtimePatch.rollbackManagedStartupAfterCreateFailure(); + } catch (rollbackError) { + ( + failure as Error & { managedBootstrapRollbackError?: unknown } + ).managedBootstrapRollbackError = rollbackError; + } + throw failure; } } diff --git a/src/lib/onboard/docker-gpu-route-consumers.test.ts b/src/lib/onboard/docker-gpu-route-consumers.test.ts index f2454e5d45d..b11215d340e 100644 --- a/src/lib/onboard/docker-gpu-route-consumers.test.ts +++ b/src/lib/onboard/docker-gpu-route-consumers.test.ts @@ -137,7 +137,7 @@ describe("selected route consumers", () => { expect(reverifyBridgeReachability).not.toHaveBeenCalled(); }); - it("skips compatibility-only inference gates after native wins", () => { + it("skips compatibility-only inference gates after native wins", async () => { const execInSandbox = vi.fn(); expect( verifyDockerGpuSandboxLocalInference(GPU_CONFIG, "ollama-local", { @@ -149,7 +149,7 @@ describe("selected route consumers", () => { ).toEqual({ status: "skipped", reason: "not-docker-gpu-patch" }); const verifyDirectSandboxGpu = vi.fn(); - verifyGpuSandboxAfterReady(GPU_CONFIG, "ollama-local", { + await verifyGpuSandboxAfterReady(GPU_CONFIG, "ollama-local", { sandboxName: "alpha", dockerDriverGateway: true, selectedRoute: "native", @@ -162,11 +162,11 @@ describe("selected route consumers", () => { expect(execInSandbox).not.toHaveBeenCalled(); }); - it("defers native proof diagnostics while automatic fallback owns recovery", () => { + it("defers native proof diagnostics while automatic fallback owns recovery", async () => { const proofError = new Error("native CUDA proof failed"); const consoleError = vi.spyOn(console, "error").mockImplementation(() => {}); try { - expect(() => + await expect( verifyGpuSandboxAfterReady(GPU_CONFIG, "ollama-local", { sandboxName: "alpha", dockerDriverGateway: true, @@ -178,7 +178,7 @@ describe("selected route consumers", () => { selectedMode: () => null, runCaptureOpenshell: vi.fn(() => ""), }), - ).toThrow(proofError); + ).rejects.toThrow(proofError); expect(consoleError).not.toHaveBeenCalled(); } finally { consoleError.mockRestore(); diff --git a/src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts b/src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts index fb94eba175e..6ddea611aa9 100644 --- a/src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts +++ b/src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts @@ -42,7 +42,7 @@ describe("createDockerGpuSandboxCreatePatch composed flow", () => { vi.restoreAllMocks(); }); - it("defers backup removal until waitForSupervisorReconnectIfNeeded sees supervisorReady=true", () => { + it("retains the backup after reconnect and removes it only after post-Ready commit", async () => { const deps = makeDeps(); const result = deferredCreateResult(); const recreatePatch = vi.fn(() => result); @@ -84,13 +84,57 @@ describe("createDockerGpuSandboxCreatePatch composed flow", () => { patch.waitForSupervisorReconnectIfNeeded(); expect(waitForSupervisor).toHaveBeenCalledTimes(1); + expect(finalizeBackup).not.toHaveBeenCalled(); + + await patch.commitAfterReady(); expect(finalizeBackup).toHaveBeenCalledTimes(1); expect(finalizeBackup).toHaveBeenCalledWith({ result, supervisorReady: true }, deps); expect(capturePreRollbackDiagnostics).not.toHaveBeenCalled(); expect(onPatchFailureExit).not.toHaveBeenCalled(); }); - it("refuses compatibility success when the backup container cannot be removed", () => { + it("reports a failed post-Ready rollback instead of treating it as restored", async () => { + const deps = makeDeps(); + const result = deferredCreateResult(); + const finalizeBackup = vi.fn(() => ({ + backupRemoved: false, + rolledBack: false, + })); + const onPatchFailureExit = vi.fn(); + const patch = createDockerGpuSandboxCreatePatch({ + route: "compatibility", + sandboxName: "alpha", + timeoutSecs: 60, + deps, + overrides: { + findContainerIds: vi.fn(() => ["existing-container"]), + recreatePatch: vi.fn(() => result), + waitForSupervisor: vi.fn(() => true), + finalizeBackup, + onPatchFailureExit, + }, + }); + + patch.maybeApplyDuringCreate(); + patch.waitForSupervisorReconnectIfNeeded(); + await patch.rollbackManagedStartupAfterCreateFailure(); + + expect(finalizeBackup).toHaveBeenCalledWith({ result, supervisorReady: false }, deps); + expect(onPatchFailureExit).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ + message: expect.stringContaining("pre-patch container was not restored"), + }), + expect.objectContaining({ + context: expect.objectContaining({ + backupContainerName: result.backupContainerName, + rolledBack: false, + }), + }), + ); + }); + + it("refuses compatibility success when the backup container cannot be removed", async () => { const deps = makeDeps(); const result = deferredCreateResult(); const onPatchFailureExit = vi.fn(); @@ -113,11 +157,14 @@ describe("createDockerGpuSandboxCreatePatch composed flow", () => { patch.maybeApplyDuringCreate(); patch.waitForSupervisorReconnectIfNeeded(); + expect(onPatchFailureExit).not.toHaveBeenCalled(); + + await patch.commitAfterReady(); expect(onPatchFailureExit).toHaveBeenCalledOnce(); expect(onPatchFailureExit.mock.calls[0]?.[1]).toEqual( expect.objectContaining({ - message: expect.stringContaining("backup container"), + message: expect.stringContaining("rollback backup"), }), ); expect(onPatchFailureExit.mock.calls[0]?.[2]).toEqual( @@ -245,7 +292,7 @@ describe("createDockerGpuSandboxCreatePatch composed flow", () => { expect(onPatchFailureExit).not.toHaveBeenCalled(); }); - it("records patchError when recreate throws and exitOnPatchError reports it via printDockerGpuPatchFailureAndExit", () => { + it("records patchError when recreate throws and exitOnPatchError reports it via printDockerGpuPatchFailureAndExit", async () => { const deps = makeDeps(); const recreatePatch = vi.fn(() => { throw new Error("docker rename failed"); @@ -271,7 +318,7 @@ describe("createDockerGpuSandboxCreatePatch composed flow", () => { patch.maybeApplyDuringCreate(); expect(patch.createFailureMessage()).toMatch(/Docker GPU patch failed/); - patch.exitOnPatchError(); + await patch.exitOnPatchError(); expect(onPatchFailureExit).toHaveBeenCalledTimes(1); // Supervisor wait must be skipped because needsSupervisorWait stayed false. patch.waitForSupervisorReconnectIfNeeded(); @@ -279,7 +326,7 @@ describe("createDockerGpuSandboxCreatePatch composed flow", () => { expect(finalizeBackup).not.toHaveBeenCalled(); }); - it("hard-stops a structured failed GPU proof on the compatibility route", () => { + it("hard-stops a structured failed GPU proof on the compatibility route", async () => { const deps = makeDeps(); const patch = createDockerGpuSandboxCreatePatch({ route: "compatibility", @@ -291,7 +338,7 @@ describe("createDockerGpuSandboxCreatePatch composed flow", () => { }, }); - expect(() => + await expect( patch.verifyGpuOrExit(() => ({ status: "failed", cudaVerified: false, @@ -299,6 +346,38 @@ describe("createDockerGpuSandboxCreatePatch composed flow", () => { detail: "No devices were found", at: "2026-07-07T00:00:00.000Z", })), - ).toThrow("Sandbox GPU proof returned failed status: nvidia-smi when available"); + ).rejects.toThrow("Sandbox GPU proof returned failed status: nvidia-smi when available"); + }); + + it("reports a failed rollback after GPU-proof diagnostics", async () => { + const deps = makeDeps(); + const result = deferredCreateResult(); + const patch = createDockerGpuSandboxCreatePatch({ + route: "compatibility", + sandboxName: "alpha", + timeoutSecs: 60, + deps, + overrides: { + findContainerIds: vi.fn(() => ["existing-container"]), + recreatePatch: vi.fn(() => result), + waitForSupervisor: vi.fn(() => true), + finalizeBackup: vi.fn(() => ({ + backupRemoved: false, + rolledBack: false, + })), + }, + }); + + patch.maybeApplyDuringCreate(); + patch.waitForSupervisorReconnectIfNeeded(); + + await expect( + patch.verifyGpuOrExit(() => { + throw new Error("nvidia-smi failed"); + }), + ).rejects.toThrow("nvidia-smi failed"); + expect(console.error).toHaveBeenCalledWith( + expect.stringContaining("pre-patch container was not restored"), + ); }); }); diff --git a/src/lib/onboard/docker-gpu-sandbox-create.ts b/src/lib/onboard/docker-gpu-sandbox-create.ts index b683cbb1c77..52c15d0b5be 100644 --- a/src/lib/onboard/docker-gpu-sandbox-create.ts +++ b/src/lib/onboard/docker-gpu-sandbox-create.ts @@ -42,7 +42,7 @@ export { type DockerGpuSandboxCreateDeps = Pick< DockerGpuPatchDeps, - "runOpenshell" | "runCaptureOpenshell" | "sleep" | "dockerCapture" + "runOpenshell" | "runCaptureOpenshell" | "sleep" | "dockerCapture" | "dockerRun" | "dockerStop" >; type WaitSupervisorFn = typeof waitForOpenShellSupervisorReconnect; @@ -61,6 +61,11 @@ type PatchFailureExitFn = ( type DockerGpuSandboxCreatePatchOptions = { route: SelectedDockerGpuRoute; persistStartupCommand?: boolean; + /** + * A managed bootstrap owns the one permitted recreation after Ready. Keep + * route diagnostics/proof active without running the legacy recreator. + */ + externalRecreation?: boolean; sandboxName: string; gpuDevice?: string | null; openshellSandboxCommand?: readonly string[] | null; @@ -91,12 +96,27 @@ type DockerGpuSandboxCreatePatchOptions = { }; }; +export interface DockerManagedBootstrapDeferredCutover { + readonly selectedMode: DockerGpuPatchMode; + readonly failureContext: DockerGpuPatchFailureContext; + rollback(): Promise; + commit(): Promise; +} + export type DockerGpuSandboxCreatePatch = { maybeApplyDuringCreate: () => void; createFailureMessage: () => string | null; - exitOnPatchError: () => void; - ensureApplied: () => void; + exitOnPatchError: () => Promise; + attachManagedBootstrapCutover: (cutover: DockerManagedBootstrapDeferredCutover) => void; + rollbackManagedStartupAfterCreateFailure: () => Promise; + ensureApplied: () => Promise; waitForSupervisorReconnectIfNeeded: () => void; + /** + * Commit an attached managed cutover or remove a legacy recreation backup. + * Call only after authoritative Ready and the required GPU and applicable + * local-inference checks pass. + */ + commitAfterReady: () => Promise; selectedMode: () => DockerGpuPatchMode | null; /** * Print the Docker GPU readiness-failure block (including the Error-phase @@ -106,14 +126,14 @@ export type DockerGpuSandboxCreatePatch = { printReadinessFailureIfEnabled: () => void; /** * Run the GPU proof while distinguishing "sandbox in terminal phase" from - * "proof failed inside a live sandbox". Calls `process.exit(1)` for the - * former and rethrows after printing diagnostics for the latter so the - * onboarding flow surfaces the right failure cause (#4316). Returns the - * CUDA-usability proof result on success so callers can persist it (#4231). + * "proof failed inside a live sandbox". Awaits rollback and throws after + * printing diagnostics so the onboarding flow can select the terminal exit + * status without racing the rollback (#4316). Returns the CUDA-usability + * proof result on success so callers can persist it (#4231). */ verifyGpuOrExit: ( verifyDirectSandboxGpu: (sandboxName: string) => SandboxGpuProofResult, - ) => SandboxGpuProofResult; + ) => Promise; }; export function createDockerGpuSandboxCreatePatch( @@ -121,8 +141,12 @@ export function createDockerGpuSandboxCreatePatch( ): DockerGpuSandboxCreatePatch { const routeAdapter = adaptDockerGpuRouteForPatch(options.route); let result: DockerGpuPatchResult | null = null; + let managedBootstrapCutover: DockerManagedBootstrapDeferredCutover | null = null; let patchError: unknown = null; let needsSupervisorWait = false; + let cutoverFinalized = false; + let cutoverFinalization: Promise | null = null; + let cutoverFinalizationOutcome: "commit" | "rollback" | null = null; const findContainerIds = options.overrides?.findContainerIds ?? findOpenShellDockerSandboxContainerIds; @@ -145,7 +169,10 @@ export function createDockerGpuSandboxCreatePatch( backend: options.backend, dockerDesktopWsl: options.dockerDesktopWsl ?? isDockerDesktopWslRuntime(), }; - const patchEnabled = routeAdapter.enabled || options.persistStartupCommand === true; + const recreationEnabled = + options.externalRecreation !== true && + (routeAdapter.enabled || options.persistStartupCommand === true); + const patchEnabled = recreationEnabled; const patchTarget = routeAdapter.enabled ? "NVIDIA GPU access" : "restart-safe startup"; const recreateSelectedPatch = createDockerSandboxRecreator({ gpuEnabled: routeAdapter.enabled, @@ -156,21 +183,92 @@ export function createDockerGpuSandboxCreatePatch( recreateStartup: recreateStartupPatch, }); + const applyPatch = (deps: DockerGpuPatchDeps): void => { + if (!recreationEnabled) return; + result = recreateSelectedPatch(false, deps); + needsSupervisorWait = true; + console.log(` ✓ Docker container mode selected: ${result.mode.label}`); + }; + + const rollbackAfterFailure = async (): Promise => { + if (cutoverFinalized || (!managedBootstrapCutover && !result)) return null; + if (cutoverFinalization) { + try { + if (cutoverFinalizationOutcome !== "rollback") { + throw new Error("Managed startup rollback raced an in-progress commit finalization."); + } + await cutoverFinalization; + return null; + } catch (error) { + return error instanceof Error ? error : new Error(String(error)); + } + } + const finalization = (async () => { + await managedBootstrapCutover?.rollback(); + const finalizeOutcome = result + ? finalizeBackup({ result, supervisorReady: false }, options.deps) + : null; + cutoverFinalized = true; + needsSupervisorWait = false; + if (finalizeOutcome && !finalizeOutcome.rolledBack) { + throw new Error( + "Docker container rollback failed; the pre-patch container was not restored.", + ); + } + })(); + cutoverFinalization = finalization; + cutoverFinalizationOutcome = "rollback"; + try { + await finalization; + return null; + } catch (error) { + return error instanceof Error ? error : new Error(String(error)); + } finally { + if (!cutoverFinalized) { + cutoverFinalization = null; + cutoverFinalizationOutcome = null; + } + } + }; + + const reportPatchErrorAndExit = async (): Promise => { + if (!patchError) return; + const rollbackError = await rollbackAfterFailure(); + if (rollbackError) { + patchError = new Error( + `${patchError instanceof Error ? patchError.message : String(patchError)}; managed startup rollback failed: ${rollbackError.message}`, + ); + } + onPatchFailureExit(options.sandboxName, patchError, { + runCaptureOpenshell: options.deps.runCaptureOpenshell, + dockerCapture: options.deps.dockerCapture, + additionalSummaryLines: routeAdapter.additionalSummaryLines, + }); + }; + const selectedMode = (): DockerGpuPatchMode | null => + managedBootstrapCutover?.selectedMode ?? result?.mode ?? null; + const failureContext = (): DockerGpuPatchFailureContext => + managedBootstrapCutover?.failureContext ?? buildFailureContext(options.sandboxName, result); + return { maybeApplyDuringCreate() { if (!patchEnabled || result || patchError) return; const containerIds = findContainerIds(options.sandboxName); if (containerIds.length === 0) return; + if (containerIds.length !== 1) { + patchError = new Error( + `Docker recreation observed ${String(containerIds.length)} matching containers; refusing an ambiguous replacement.`, + ); + return; + } console.log( ` OpenShell Docker container detected; recreating it with ${patchTarget} before readiness wait...`, ); try { - result = recreateSelectedPatch(false, { + applyPatch({ runCaptureOpenshell: options.deps.runCaptureOpenshell, sleep: options.deps.sleep, }); - needsSupervisorWait = true; - console.log(` ✓ Docker container mode selected: ${result.mode.label}`); } catch (error) { patchError = error; } @@ -183,33 +281,44 @@ export function createDockerGpuSandboxCreatePatch( : "Docker startup-command patch failed while OpenShell sandbox create was still waiting."; }, - exitOnPatchError() { - if (!patchError) return; - onPatchFailureExit(options.sandboxName, patchError, { + async exitOnPatchError() { + await reportPatchErrorAndExit(); + }, + + attachManagedBootstrapCutover(cutover) { + if (managedBootstrapCutover || result || cutoverFinalized) { + throw new Error("Managed bootstrap cutover may be attached exactly once."); + } + managedBootstrapCutover = cutover; + }, + + async rollbackManagedStartupAfterCreateFailure() { + const rollbackError = await rollbackAfterFailure(); + if (!rollbackError) return; + onPatchFailureExit(options.sandboxName, rollbackError, { runCaptureOpenshell: options.deps.runCaptureOpenshell, dockerCapture: options.deps.dockerCapture, additionalSummaryLines: routeAdapter.additionalSummaryLines, + context: { + ...failureContext(), + rolledBack: false, + }, }); }, - ensureApplied() { + async ensureApplied() { if (!patchEnabled || result) return; console.log(` Recreating OpenShell Docker sandbox container with ${patchTarget}...`); try { - result = recreateSelectedPatch(false, options.deps); - needsSupervisorWait = true; - console.log(` ✓ Docker container mode selected: ${result.mode.label}`); + applyPatch(options.deps); } catch (error) { - onPatchFailureExit(options.sandboxName, error, { - runCaptureOpenshell: options.deps.runCaptureOpenshell, - dockerCapture: options.deps.dockerCapture, - additionalSummaryLines: routeAdapter.additionalSummaryLines, - }); + patchError = error; + await reportPatchErrorAndExit(); } }, waitForSupervisorReconnectIfNeeded() { - if (!needsSupervisorWait) return; + if (!needsSupervisorWait || cutoverFinalized) return; const supervisorReconnectTimeoutSecs = getDockerGpuSupervisorReconnectTimeoutSecs( options.timeoutSecs, ); @@ -221,14 +330,17 @@ export function createDockerGpuSandboxCreatePatch( supervisorReconnectTimeoutSecs, { runOpenshell: options.deps.runOpenshell, - // Pass `runCaptureOpenshell` so the supervisor-reconnect wait can - // short-circuit on a terminal sandbox phase instead of burning - // the full reconnect timeout window when the patched container - // crashed on startup (#4316). runCaptureOpenshell: options.deps.runCaptureOpenshell, sleep: options.deps.sleep, }, ); + if (supervisorReady) { + // Reconnect completes the legacy recreation check. Keep its rollback + // backup until the caller accepts authoritative Ready and the required + // GPU checks. + needsSupervisorWait = false; + return; + } if (!supervisorReady && result) { try { captureFailedClone(options.sandboxName, result, options.deps); @@ -239,40 +351,13 @@ export function createDockerGpuSandboxCreatePatch( } } const finalizeOutcome = result - ? finalizeBackup({ result, supervisorReady }, options.deps) + ? finalizeBackup({ result, supervisorReady: false }, options.deps) : null; - if (supervisorReady) { - if (finalizeOutcome && !finalizeOutcome.backupRemoved) { - onPatchFailureExit( - options.sandboxName, - new Error( - "OpenShell supervisor reconnected, but the recreated backup container could not be removed.", - ), - { - runCaptureOpenshell: options.deps.runCaptureOpenshell, - dockerCapture: options.deps.dockerCapture, - additionalSummaryLines: routeAdapter.additionalSummaryLines, - context: { - sandboxName: options.sandboxName, - oldContainerId: result?.oldContainerId, - newContainerId: result?.newContainerId, - backupContainerName: result?.backupContainerName, - selectedMode: result?.mode ?? null, - rolledBack: false, - }, - }, - ); - } - return; - } - const failureMessage = (() => { - if (!finalizeOutcome) { - return "OpenShell supervisor did not reconnect to the recreated container."; - } - return finalizeOutcome.rolledBack - ? "OpenShell supervisor did not reconnect to the recreated container; pre-patch sandbox restored." - : "OpenShell supervisor did not reconnect to the recreated container and rollback failed; pre-patch sandbox was NOT restored."; - })(); + cutoverFinalized = true; + needsSupervisorWait = false; + const failureMessage = finalizeOutcome?.rolledBack + ? "OpenShell supervisor did not reconnect to the recreated container; pre-patch sandbox restored." + : "OpenShell supervisor did not reconnect to the recreated container and rollback failed; pre-patch sandbox was NOT restored."; onPatchFailureExit(options.sandboxName, new Error(failureMessage), { runCaptureOpenshell: options.deps.runCaptureOpenshell, dockerCapture: options.deps.dockerCapture, @@ -288,21 +373,108 @@ export function createDockerGpuSandboxCreatePatch( }); }, + async commitAfterReady() { + if (cutoverFinalized || (!managedBootstrapCutover && !result)) return; + if (needsSupervisorWait) { + const error = new Error( + "Managed startup cannot commit before the recreated OpenShell supervisor reconnects.", + ); + const rollbackError = await rollbackAfterFailure(); + onPatchFailureExit( + options.sandboxName, + rollbackError + ? new Error(`${error.message} Rollback failed: ${rollbackError.message}`) + : error, + { + runCaptureOpenshell: options.deps.runCaptureOpenshell, + dockerCapture: options.deps.dockerCapture, + additionalSummaryLines: routeAdapter.additionalSummaryLines, + }, + ); + return; + } + if (cutoverFinalization) { + if (cutoverFinalizationOutcome !== "commit") { + throw new Error("Managed startup commit raced an in-progress rollback finalization."); + } + await cutoverFinalization; + return; + } + const finalization = (async () => { + if (managedBootstrapCutover) { + try { + await managedBootstrapCutover.commit(); + } catch (error) { + const failure = error instanceof Error ? error : new Error(String(error)); + let rollbackError: Error | null = null; + try { + await managedBootstrapCutover.rollback(); + cutoverFinalized = true; + needsSupervisorWait = false; + } catch (rollbackFailure) { + rollbackError = + rollbackFailure instanceof Error + ? rollbackFailure + : new Error(String(rollbackFailure)); + ( + failure as Error & { managedBootstrapRollbackError?: unknown } + ).managedBootstrapRollbackError = rollbackError; + } + onPatchFailureExit(options.sandboxName, failure, { + runCaptureOpenshell: options.deps.runCaptureOpenshell, + dockerCapture: options.deps.dockerCapture, + additionalSummaryLines: routeAdapter.additionalSummaryLines, + context: { + ...failureContext(), + rolledBack: rollbackError === null, + }, + }); + return; + } + } + const finalizeOutcome = result + ? finalizeBackup({ result, supervisorReady: true }, options.deps) + : null; + cutoverFinalized = true; + if (!finalizeOutcome || finalizeOutcome.backupRemoved) return; + onPatchFailureExit( + options.sandboxName, + new Error("Managed startup passed Ready, but its rollback backup could not be removed."), + { + runCaptureOpenshell: options.deps.runCaptureOpenshell, + dockerCapture: options.deps.dockerCapture, + additionalSummaryLines: routeAdapter.additionalSummaryLines, + context: failureContext(), + }, + ); + })(); + cutoverFinalization = finalization; + cutoverFinalizationOutcome = "commit"; + try { + await finalization; + } finally { + if (!cutoverFinalized) { + cutoverFinalization = null; + cutoverFinalizationOutcome = null; + } + } + }, + selectedMode() { - return result?.mode ?? null; + return selectedMode(); }, printReadinessFailureIfEnabled() { if (!routeAdapter.enabled) return; - printDockerGpuReadinessFailure(options.sandboxName, result?.mode ?? null, { + printDockerGpuReadinessFailure(options.sandboxName, selectedMode(), { runCaptureOpenshell: options.deps.runCaptureOpenshell, dockerCapture: options.deps.dockerCapture, - context: buildFailureContext(options.sandboxName, result), + context: failureContext(), additionalSummaryLines: routeAdapter.additionalSummaryLines, }); }, - verifyGpuOrExit(verifyDirectSandboxGpu) { + async verifyGpuOrExit(verifyDirectSandboxGpu) { // Before issuing GPU proof commands through `openshell sandbox exec`, // confirm the sandbox is still in a live phase. A sandbox that // transitioned to Error after the readiness wait succeeded (e.g. the @@ -312,7 +484,7 @@ export function createDockerGpuSandboxCreatePatch( // container/Error-phase classification instead of running the proof // (#4316). const sandboxName = options.sandboxName; - const failureContext = buildFailureContext(sandboxName, result); + const currentFailureContext = failureContext(); if (routeAdapter.enabled && options.deps.runCaptureOpenshell) { const list = options.deps.runCaptureOpenshell(["sandbox", "list"], { ignoreError: true, @@ -321,20 +493,23 @@ export function createDockerGpuSandboxCreatePatch( if (phase) { console.error(""); console.error(` Skipping GPU proof: sandbox '${sandboxName}' is in ${phase} phase.`); - printDockerGpuProofFailure( - sandboxName, - new Error( - `Sandbox '${sandboxName}' entered ${phase} phase after readiness; GPU proof skipped.`, - ), - result?.mode ?? null, - { - runCaptureOpenshell: options.deps.runCaptureOpenshell, - dockerCapture: options.deps.dockerCapture, - context: failureContext, - additionalSummaryLines: routeAdapter.additionalSummaryLines, - }, + const failure = new Error( + `Sandbox '${sandboxName}' entered ${phase} phase after readiness; GPU proof skipped.`, ); - process.exit(1); + printDockerGpuProofFailure(sandboxName, failure, selectedMode(), { + runCaptureOpenshell: options.deps.runCaptureOpenshell, + dockerCapture: options.deps.dockerCapture, + context: currentFailureContext, + additionalSummaryLines: routeAdapter.additionalSummaryLines, + }); + const rollbackError = await rollbackAfterFailure(); + if (rollbackError) { + console.error(` ${rollbackError.message}`); + ( + failure as Error & { managedBootstrapRollbackError?: unknown } + ).managedBootstrapRollbackError = rollbackError; + } + throw failure; } } try { @@ -346,13 +521,21 @@ export function createDockerGpuSandboxCreatePatch( } return proof; } catch (error) { - printDockerGpuProofFailure(sandboxName, error, result?.mode ?? null, { + const failure = error instanceof Error ? error : new Error(String(error)); + printDockerGpuProofFailure(sandboxName, failure, selectedMode(), { runCaptureOpenshell: options.deps.runCaptureOpenshell, dockerCapture: options.deps.dockerCapture, - context: routeAdapter.enabled ? failureContext : null, + context: routeAdapter.enabled ? currentFailureContext : null, additionalSummaryLines: routeAdapter.additionalSummaryLines, }); - throw error; + const rollbackError = await rollbackAfterFailure(); + if (rollbackError) { + console.error(` ${rollbackError.message}`); + ( + failure as Error & { managedBootstrapRollbackError?: unknown } + ).managedBootstrapRollbackError = rollbackError; + } + throw failure; } }, }; diff --git a/src/lib/onboard/docker-startup-command-sandbox-create.test.ts b/src/lib/onboard/docker-startup-command-sandbox-create.test.ts index 17cd02f6432..ec0ad3cb306 100644 --- a/src/lib/onboard/docker-startup-command-sandbox-create.test.ts +++ b/src/lib/onboard/docker-startup-command-sandbox-create.test.ts @@ -68,7 +68,7 @@ describe("Docker startup-command sandbox creation", () => { vi.restoreAllMocks(); }); - it("uses the startup-command recreation path with DCode's exact resource limits", () => { + it("uses the startup-command recreation path with DCode's exact resource limits", async () => { const dockerCaptureOutput: Record = { ps: "old-container-id\n", inspect: JSON.stringify([inspectFixture()]), @@ -102,7 +102,7 @@ describe("Docker startup-command sandbox creation", () => { }, }); - patch.ensureApplied(); + await patch.ensureApplied(); expect(recreatePatch).not.toHaveBeenCalled(); expect(dockerRunDetached.mock.calls[0]?.[0]).toEqual( @@ -156,7 +156,102 @@ describe("Docker startup-command sandbox creation", () => { expect(context.rolledBack).toBe(true); }); - it("reports startup-command creation failures through the composed patch boundary", () => { + it("defers a driver-owned managed cutover until the authoritative caller commits", async () => { + const deps = makeDeps(); + let releaseCommit = () => {}; + const commit = vi.fn( + () => + new Promise((resolve) => { + releaseCommit = resolve; + }), + ); + const rollback = vi.fn(async () => {}); + const patch = createDockerGpuSandboxCreatePatch({ + route: "native", + externalRecreation: true, + sandboxName: "alpha", + timeoutSecs: 60, + deps, + }); + patch.attachManagedBootstrapCutover({ + selectedMode: { + kind: "startup-command", + label: "managed bootstrap", + device: "", + args: [], + }, + failureContext: { sandboxName: "alpha" }, + commit, + rollback, + }); + patch.maybeApplyDuringCreate(); + await patch.ensureApplied(); + patch.waitForSupervisorReconnectIfNeeded(); + expect(commit).not.toHaveBeenCalled(); + const firstCommit = patch.commitAfterReady(); + const duplicateCommit = patch.commitAfterReady(); + expect(commit).toHaveBeenCalledOnce(); + expect(rollback).not.toHaveBeenCalled(); + releaseCommit(); + await Promise.all([firstCommit, duplicateCommit]); + }); + + it("rolls back a driver-owned cutover before reporting commit failure", async () => { + const deps = makeDeps(); + const events: string[] = []; + const commit = vi.fn(async () => { + events.push("commit"); + throw new Error("receipt validation failed"); + }); + const rollback = vi.fn(async () => { + events.push("rollback"); + }); + const onPatchFailureExit = vi.fn(() => { + events.push("exit"); + }); + const patch = createDockerGpuSandboxCreatePatch({ + route: "native", + externalRecreation: true, + sandboxName: "alpha", + timeoutSecs: 60, + deps, + overrides: { onPatchFailureExit }, + }); + patch.attachManagedBootstrapCutover({ + selectedMode: { + kind: "startup-command", + label: "managed bootstrap", + device: "", + args: [], + }, + failureContext: { + sandboxName: "alpha", + oldContainerId: "held-container", + newContainerId: "replacement-container", + }, + commit, + rollback, + }); + + await patch.commitAfterReady(); + + expect(events).toEqual(["commit", "rollback", "exit"]); + expect(onPatchFailureExit).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ message: "receipt validation failed" }), + expect.objectContaining({ + context: expect.objectContaining({ + oldContainerId: "held-container", + newContainerId: "replacement-container", + rolledBack: true, + }), + }), + ); + await patch.rollbackManagedStartupAfterCreateFailure(); + expect(rollback).toHaveBeenCalledOnce(); + }); + + it("reports startup-command creation failures through the composed patch boundary", async () => { const deps = makeDeps(); const onPatchFailureExit = vi.fn(); const patch = createDockerGpuSandboxCreatePatch({ @@ -177,7 +272,7 @@ describe("Docker startup-command sandbox creation", () => { patch.maybeApplyDuringCreate(); expect(patch.createFailureMessage()).toMatch(/startup-command patch failed/); - patch.exitOnPatchError(); + await patch.exitOnPatchError(); expect(onPatchFailureExit).toHaveBeenCalledWith( "alpha", expect.objectContaining({ message: "startup recreate failed" }), diff --git a/src/lib/onboard/managed-bootstrap/README.md b/src/lib/onboard/managed-bootstrap/README.md index a102dc031e7..159ab401e86 100644 --- a/src/lib/onboard/managed-bootstrap/README.md +++ b/src/lib/onboard/managed-bootstrap/README.md @@ -3,9 +3,11 @@ # Managed bootstrap protocol -This directory defines a dormant, driver-neutral transaction contract and its -first driver adapter. It does not register a runtime provider, activate managed -bootstrap, or change the current user-visible lifecycle paths. +This directory defines a dormant, driver-neutral transaction contract, its +first driver adapter, and an injectable sandbox-create lifecycle. Production +runtime bundles still report bootstrap as unsupported, so the candidate +lifecycle remains inert. The shared finalization path keeps existing Docker +recreation reversible through later Ready, GPU, and local-inference checks. The protocol binds one random bootstrap identity to: @@ -66,10 +68,12 @@ including its supervisor environment, to immutable prepared authority before activation. The native boundary introduces no driver-specific environment policy. -The first Docker-specific groundwork defines a private, monotonic cutover -journal and a canonical launch-spec normalizer. Each surface is independently -validated and remains dormant: no registered runtime provider imports either -module, and neither changes sandbox creation or lifecycle behavior. +The Docker-specific layers define a private, monotonic cutover journal, a +canonical launch-spec normalizer, and an injectable provider create lifecycle. +The candidate provider surface composes these layers without registering them +in a production runtime bundle. It remains inert, while the shared finalization +surface extends rollback ownership for the existing Docker compatibility and +startup recreation paths. The Docker adapter creates and validates a stopped replacement under an identity-derived staging name while the original remains running. It stages the @@ -89,35 +93,40 @@ Activation must also inject the selected gateway's canonical state root. ## Architectural disposition -The coordinator deliberately lands as a dormant trust-boundary slice before a -provider or image activates it. This keeps the driver-neutral transaction -authority review separate from the first driver implementation instead of -making that implementation the de facto central contract. The coordinator -module remains cohesive because its receipt shapes, normalization, state -transitions, and rollback proofs form one authority boundary; provider-specific -logic must live outside it rather than growing this file. +The runtime-provider bundle is the only bootstrap registration boundary. The +candidate Docker surface owns create routing, replacement construction, +native-to-compatibility fallback evidence, and deferred commit or rollback. +Central onboarding accepts that provider-neutral surface without a Docker or +Podman selection branch. Tests register an MXC-style surface through the same +bundle and render held launches for OpenClaw, Hermes, and LangChain Deep Agents +Code. + +The coordinator remains the driver-neutral transaction authority: its receipt +shapes, normalization, state transitions, and rollback proofs form one cohesive +boundary, while provider-specific routing and runtime operations stay outside +it. This is executable, bounded groundwork rather than an untested placeholder. `adapter.test.ts` drives prepare, durable record, activation, finalization, and failure rollback for OpenClaw, Hermes, and LangChain Deep Agents Code through an -MXC-named fake driver. `runtime-provider-source-shape.test.ts` separately -inventories the protocol, provider, and image-packaging surfaces and proves that -production activation does not import or install the protocol into a runtime -image yet. The later activation slice must add a registered-provider contract -test for the same transaction before removing those dormancy assertions. +MXC-named fake driver. `runtime-provider-contract.test.ts` registers both the +dormant Docker candidate and an MXC-style bootstrap surface through the same +provider bundle contract without changing the production registry. +`runtime-provider-source-shape.test.ts` separately inventories the protocol, +provider, and image-packaging surfaces and proves that production activation +does not select a driver-specific bootstrap implementation. The native entrypoint source is intentionally not compiled into production artifacts, and neither image-owned source is installed or selected in a runtime -image yet. No production activation or provider module outside this dormant -directory imports the protocol or Docker adapter. The current image definitions -do not package `nemoclaw-managed-startup-hold`, +image yet. Production onboarding imports only the provider-neutral create +contract; no activation path or registered provider imports or selects the +driver-specific Docker candidate. The current image definitions do not package +`nemoclaw-managed-startup-hold`, `managed-startup-image-runtime.cjs`, or the shared-state bootstrap modes consumed -by the adapter. A later provider integration must compile and verify the -freestanding entrypoint natively for amd64 and arm64 in every agent image. It -must add those prerequisites together with their image-runtime bootstrap modes -and wire the coordinator and Docker adapter into create as one boundary. The -same contract is exercised for OpenClaw, Hermes, and Deep Agents Code without a -provider-specific central switch. The remaining integration and qualification -work is tracked in [epic #7744](https://github.com/NVIDIA/NemoClaw/issues/7744) -and its linked implementation stack. Until that complete boundary lands, every -registered runtime provider keeps its bootstrap surface unsupported. +by the adapter. Later persistence and qualification slices must compile and +verify the freestanding entrypoint for amd64 and arm64 in every agent image, add +the image-runtime prerequisites, and provide the canonical durable authority +store. The remaining integration and qualification work is tracked in +[epic #7744](https://github.com/NVIDIA/NemoClaw/issues/7744). Until that complete +boundary passes protected E2E, every production runtime provider keeps +bootstrap unsupported. diff --git a/src/lib/onboard/managed-bootstrap/docker-runtime.ts b/src/lib/onboard/managed-bootstrap/docker-runtime.ts new file mode 100644 index 00000000000..6da0f754f1b --- /dev/null +++ b/src/lib/onboard/managed-bootstrap/docker-runtime.ts @@ -0,0 +1,294 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { detectTegraDeviceGroupGids } from "../docker-gpu-jetson-groups"; +import { buildDockerGpuMode, selectDockerGpuPatchMode } from "../docker-gpu-patch-mode"; +import type { DockerGpuPatchMode } from "../docker-gpu-patch-types"; +import { renderCompatibilityFallbackCreateArgs } from "../docker-gpu-route"; +import { + createDockerGpuSandboxCreatePatch, + isDockerDesktopWslRuntime, +} from "../docker-gpu-sandbox-create"; +import { + isImmutableDockerImageId, + queryOpenShellDockerSandboxContainers, + queryOpenShellDockerSandboxRuntimeSnapshot, +} from "../openshell-docker-sandbox-containers"; +import type { RuntimeProviderBootstrapSurface } from "../runtime-provider/contract"; +import * as sandboxGpuCreateAttempt from "../sandbox-gpu-create-attempt"; +import { + activateManagedBootstrapSequence, + finalizeManagedBootstrapSequence, + MANAGED_BOOTSTRAP_SCHEMA_VERSION, + prepareManagedBootstrapSequence, +} from "./adapter"; +import { createDockerManagedBootstrapAdapter } from "./docker"; +import type { + ManagedBootstrapRuntimeCompatibilityLaunchInput, + ManagedBootstrapRuntimeCreateLaunchResult, + ManagedBootstrapRuntimeCreateLifecycle, + ManagedBootstrapRuntimeCreateLifecycleInput, + ManagedBootstrapRuntimeOnboardRoutingInput, +} from "./runtime-create"; + +type SupportedBootstrapSurface = Extract< + RuntimeProviderBootstrapSurface, + { readonly supported: true } +>; + +function dockerReplacementOptions( + mode: DockerGpuPatchMode, + input: ManagedBootstrapRuntimeCreateLifecycleInput, +) { + const backend = input.sandboxGpuConfig.hostGpuPlatform === "jetson" ? "jetson" : "generic"; + return { + values: { + gpuModeArgs: [...mode.args], + gpuModeDevice: mode.device, + gpuModeKind: mode.kind, + gpuModeLabel: mode.label, + requiredUlimits: input.requiredLimits.map( + (limit) => `${limit.name}=${limit.soft}:${limit.hard}`, + ), + extraGroupGids: + backend === "jetson" && input.route === "compatibility" ? detectTegraDeviceGroupGids() : [], + }, + }; +} + +function selectedDockerMode( + input: ManagedBootstrapRuntimeCreateLifecycleInput, + dockerDesktopWsl: boolean | undefined, +): DockerGpuPatchMode { + const backend = input.sandboxGpuConfig.hostGpuPlatform === "jetson" ? "jetson" : "generic"; + if (input.route !== "compatibility" || !input.sandboxGpuConfig.sandboxGpuEnabled) { + return buildDockerGpuMode("startup-command"); + } + const selection = selectDockerGpuPatchMode( + { + image: `${input.image.repository}@${input.image.manifestDigest}`, + device: input.sandboxGpuConfig.sandboxGpuDevice, + backend, + dockerDesktopWsl, + }, + input.dependencies, + ); + if (selection.mode) return selection.mode; + throw new Error( + backend === "jetson" + ? "Docker did not accept the Jetson NVIDIA runtime GPU mode for managed bootstrap." + : "Docker did not accept a compatibility GPU mode for managed bootstrap.", + ); +} + +function createDockerLifecycle( + providerId: string, + input: ManagedBootstrapRuntimeCreateLifecycleInput, +): ManagedBootstrapRuntimeCreateLifecycle { + if (input.providerId !== providerId) { + throw new Error( + `Managed bootstrap provider '${providerId}' cannot run authority for '${input.providerId}'.`, + ); + } + const dockerDesktopWsl = + input.route === "compatibility" ? isDockerDesktopWslRuntime() : undefined; + const mode = selectedDockerMode(input, dockerDesktopWsl); + const backend = input.sandboxGpuConfig.hostGpuPlatform === "jetson" ? "jetson" : "generic"; + const persistStartupCommand = + input.persistStartupCommand && (input.route !== "native" || input.requiredLimits.length > 0); + const patch = createDockerGpuSandboxCreatePatch({ + route: input.route, + persistStartupCommand, + externalRecreation: true, + sandboxName: input.sandboxName, + gpuDevice: input.sandboxGpuConfig.sandboxGpuDevice, + openshellSandboxCommand: input.heldWorkloadArgv, + requiredUlimits: input.requiredLimits, + timeoutSecs: input.timeoutSecs, + backend, + dockerDesktopWsl, + deps: input.dependencies, + ...(input.onPatchFailure + ? { + overrides: { + onPatchFailureExit: (_sandboxName: string, error: unknown) => + input.onPatchFailure?.(error), + }, + } + : {}), + }); + const adapter = input.adapterOverride ?? createDockerManagedBootstrapAdapter(input.dependencies); + const createPlan = { + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + sandboxName: input.sandboxName, + driverId: providerId, + image: input.image, + profile: { + agent: input.request.agent, + fingerprint: input.request.profileFingerprint, + }, + agentIdentity: input.agentIdentity, + intendedWorkloadArgv: input.intendedWorkloadArgv, + expectedSupervisorArgv: input.expectedSupervisorArgv, + metadata: {}, + } as const; + const replacementOptions = dockerReplacementOptions(mode, input); + + return { + launchArgv: input.launchArgv, + patch, + async prepareNetwork() { + if (input.route !== "compatibility") return; + const { enforceDockerGpuPatchPreserveNetwork } = await import( + "../docker-gpu-local-inference" + ); + await enforceDockerGpuPatchPreserveNetwork( + input.network.inferenceProvider, + input.sandboxGpuConfig, + { + dockerDriverGateway: input.network.gatewayUsesContainerBridge, + selectedRoute: input.route, + gatewayPort: input.network.gatewayPort, + log: console.log, + }, + ); + }, + async runCreate( + launch: (input: { + readonly heldWorkloadArgv: readonly string[]; + readonly bootstrapIdentity: string; + }) => Promise>, + ): Promise { + const launchState: { value?: ManagedBootstrapRuntimeCreateLaunchResult } = {}; + const prepared = await prepareManagedBootstrapSequence(adapter, { + create: { + bootstrapIdentity: input.bootstrapIdentity, + plan: createPlan, + request: input.request, + launch: async (launchInput) => { + const launched = await launch(launchInput); + launchState.value = launched; + return launched.receipt; + }, + }, + request: input.request, + replacementOptions, + }); + const activated = await activateManagedBootstrapSequence(adapter, { + transaction: prepared, + authorityStore: input.authorityStore, + timeoutSecs: input.timeoutSecs, + }); + const launched = launchState.value; + if (!launched) { + await finalizeManagedBootstrapSequence(adapter, { + outcome: "rollback", + transaction: activated, + }); + throw new Error("Managed bootstrap did not return its OpenShell create receipt."); + } + let finalized = false; + patch.attachManagedBootstrapCutover({ + selectedMode: mode, + failureContext: { + sandboxName: input.sandboxName, + oldContainerId: activated.snapshot.runtimeId, + newContainerId: activated.replacement.replacementRuntimeId, + backupContainerName: null, + selectedMode: mode, + }, + async rollback() { + if (finalized) return; + await finalizeManagedBootstrapSequence(adapter, { + outcome: "rollback", + transaction: activated, + }); + finalized = true; + }, + async commit() { + if (finalized) return; + await finalizeManagedBootstrapSequence(adapter, { + outcome: "commit", + transaction: activated, + }); + finalized = true; + }, + }); + return launched.value; + }, + }; +} + +function createDockerOnboardRouting(input: ManagedBootstrapRuntimeOnboardRoutingInput) { + const baseline = input.nativeFallbackEnabled + ? queryOpenShellDockerSandboxContainers(input.sandboxName) + : null; + const inspectNativeRuntime = () => { + const snapshot = queryOpenShellDockerSandboxRuntimeSnapshot(input.sandboxName); + return snapshot.ok + ? { + imageId: snapshot.imageId, + bookkeepingImageRef: snapshot.bookkeepingImageRef, + stateError: snapshot.stateError, + nativeGpuAttachmentState: snapshot.nativeGpuAttachmentState, + } + : null; + }; + return { + nativeFallbackHasCleanBaseline: baseline?.ok === true && baseline.ids.length === 0, + inspectNativeRuntime, + isNativeCreateRoutingFailure: (output: string, sawProgress: boolean): boolean => + sandboxGpuCreateAttempt.isNativeGpuCreateRoutingFailure(output, { sawProgress }), + isTrustedNativeRuntimeError: (error: string): boolean => + sandboxGpuCreateAttempt.isTrustedNativeGpuRuntimeError(error), + isNativeReadinessRoutingFailure: (failure: { + readonly failurePhase: string | null; + readonly runtimeError: string; + }): boolean => sandboxGpuCreateAttempt.isNativeGpuReadinessRoutingFailure(failure), + prepareCompatibilityLaunch: ( + compatibility: ManagedBootstrapRuntimeCompatibilityLaunchInput, + ) => { + const runtime = compatibility.runtimeSnapshot; + const imageId = + runtime?.imageId ?? + (compatibility.prebuildImageId && isImmutableDockerImageId(compatibility.prebuildImageId) + ? compatibility.prebuildImageId.toLowerCase() + : null); + let registryImageRef = compatibility.currentRegistryImageRef; + if ( + !registryImageRef && + runtime?.bookkeepingImageRef && + !isImmutableDockerImageId(runtime.bookkeepingImageRef) + ) { + registryImageRef = runtime.bookkeepingImageRef; + } + const createArgs = renderCompatibilityFallbackCreateArgs(compatibility.createArgs, { + imageRef: imageId, + allowUnbuiltSource: compatibility.allowUnbuiltSource, + compatibilityPolicyPath: compatibility.compatibilityPolicyPath, + }); + return { + createArgv: input.openshellArgv([ + "sandbox", + "create", + ...createArgs, + "--", + ...compatibility.startupCommand, + ]), + registryImageRef, + }; + }, + }; +} + +/** Candidate Docker surface. Production activation remains a later qualification slice. */ +export function createDockerManagedBootstrapSurface( + providerId = "docker", +): SupportedBootstrapSurface { + return { + providerId, + supported: true, + createLifecycle: (input) => createDockerLifecycle(providerId, input), + createOnboardRouting: createDockerOnboardRouting, + }; +} diff --git a/src/lib/onboard/managed-bootstrap/index.ts b/src/lib/onboard/managed-bootstrap/index.ts index 17099572608..c55768afe02 100644 --- a/src/lib/onboard/managed-bootstrap/index.ts +++ b/src/lib/onboard/managed-bootstrap/index.ts @@ -20,3 +20,7 @@ export { serializeManagedBootstrapEnvelope, serializeManagedBootstrapImageCompletion, } from "./envelope"; +export type { + ManagedBootstrapRuntimeCreateLifecycle, + ManagedBootstrapRuntimePatch, +} from "./runtime-create"; diff --git a/src/lib/onboard/managed-bootstrap/runtime-create.ts b/src/lib/onboard/managed-bootstrap/runtime-create.ts new file mode 100644 index 00000000000..198d91710f0 --- /dev/null +++ b/src/lib/onboard/managed-bootstrap/runtime-create.ts @@ -0,0 +1,145 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import type { SandboxGpuProofResult } from "../../state/registry"; +import type { ManagedStartupRootApplyRequest } from "../managed-startup/root-apply"; +import type { SandboxGpuConfig } from "../sandbox-gpu-mode"; +import type { + ManagedBootstrapAdapter, + ManagedBootstrapAgentIdentity, + ManagedBootstrapAuthorityStore, + ManagedBootstrapCreateReceipt, + ManagedBootstrapImageIdentity, +} from "./adapter"; + +export interface ManagedBootstrapRuntimeCommandResult { + readonly status?: number | null; + readonly stdout?: string | Buffer | null; + readonly stderr?: string | Buffer | null; + readonly error?: Error | null; +} + +export interface ManagedBootstrapRuntimeDependencies { + readonly runCaptureOpenshell?: (args: string[], options?: Record) => string; + readonly runOpenshell?: ( + args: string[], + options?: Record, + ) => ManagedBootstrapRuntimeCommandResult; + readonly sleep?: (seconds: number) => void; +} + +export type ManagedBootstrapRuntimeRoute = "none" | "native" | "compatibility"; + +export interface ManagedBootstrapRuntimeLimit { + readonly name: string; + readonly soft: number; + readonly hard: number; +} + +/** Provider-neutral lifecycle surface consumed by sandbox-create coordinators. */ +export interface ManagedBootstrapRuntimePatch { + maybeApplyDuringCreate(): void | Promise; + createFailureMessage(): string | null; + exitOnPatchError(): void | Promise; + rollbackManagedStartupAfterCreateFailure(): void | Promise; + ensureApplied(): void | Promise; + waitForSupervisorReconnectIfNeeded(): void | Promise; + commitAfterReady(): void | Promise; + selectedMode(): { + readonly kind: string; + readonly label: string; + readonly device: string; + readonly args: readonly string[]; + } | null; + printReadinessFailureIfEnabled(): void; + verifyGpuOrExit( + verifyDirectSandboxGpu: (sandboxName: string) => SandboxGpuProofResult, + ): Promise; +} + +export interface ManagedBootstrapRuntimeCreateLifecycleInput { + readonly providerId: string; + readonly bootstrapIdentity: string; + readonly request: ManagedStartupRootApplyRequest; + readonly image: ManagedBootstrapImageIdentity; + readonly agentIdentity: ManagedBootstrapAgentIdentity; + readonly intendedWorkloadArgv: readonly string[]; + readonly expectedSupervisorArgv: readonly string[]; + readonly launchArgv: readonly string[]; + readonly heldWorkloadArgv: readonly string[]; + readonly authorityStore: ManagedBootstrapAuthorityStore; + readonly adapterOverride?: ManagedBootstrapAdapter; + readonly route: ManagedBootstrapRuntimeRoute; + readonly persistStartupCommand: boolean; + readonly sandboxName: string; + readonly sandboxGpuConfig: SandboxGpuConfig; + readonly requiredLimits: readonly ManagedBootstrapRuntimeLimit[]; + readonly timeoutSecs: number; + readonly onPatchFailure?: (error: unknown) => never; + readonly network: { + readonly inferenceProvider: string; + readonly gatewayUsesContainerBridge: boolean; + readonly gatewayPort: number; + }; + readonly dependencies: ManagedBootstrapRuntimeDependencies; +} + +export interface ManagedBootstrapRuntimeCreateLaunchResult { + readonly value: T; + readonly receipt: ManagedBootstrapCreateReceipt; +} + +export interface ManagedBootstrapRuntimeCreateLifecycle { + readonly launchArgv: readonly string[]; + readonly patch: ManagedBootstrapRuntimePatch; + prepareNetwork(): Promise; + runCreate( + launch: (input: { + readonly heldWorkloadArgv: readonly string[]; + readonly bootstrapIdentity: string; + }) => Promise>, + ): Promise; +} + +export interface ManagedBootstrapRuntimeSnapshot { + readonly imageId: string | null; + readonly bookkeepingImageRef: string | null; + readonly stateError: string; + readonly nativeGpuAttachmentState: "present" | "absent" | "unknown"; +} + +export interface ManagedBootstrapRuntimeCompatibilityLaunchInput { + readonly createArgs: readonly string[]; + readonly currentRegistryImageRef: string | null; + readonly prebuildImageId: string | null; + readonly allowUnbuiltSource: boolean; + readonly compatibilityPolicyPath: string; + readonly startupCommand: readonly string[]; + readonly runtimeSnapshot: ManagedBootstrapRuntimeSnapshot | null; +} + +export interface ManagedBootstrapRuntimeCompatibilityLaunch { + readonly createArgv: readonly string[]; + readonly registryImageRef: string | null; +} + +/** Provider-owned native-to-compatibility evidence and launch preparation. */ +export interface ManagedBootstrapRuntimeOnboardRouting { + readonly nativeFallbackHasCleanBaseline: boolean; + inspectNativeRuntime(): ManagedBootstrapRuntimeSnapshot | null; + isNativeCreateRoutingFailure(output: string, sawProgress: boolean): boolean; + isTrustedNativeRuntimeError(error: string): boolean; + isNativeReadinessRoutingFailure(input: { + readonly failurePhase: string | null; + readonly runtimeError: string; + }): boolean; + prepareCompatibilityLaunch( + input: ManagedBootstrapRuntimeCompatibilityLaunchInput, + ): ManagedBootstrapRuntimeCompatibilityLaunch; +} + +export interface ManagedBootstrapRuntimeOnboardRoutingInput { + readonly sandboxName: string; + readonly openshellArgv: (args: string[]) => string[]; + readonly nativeFallbackEnabled: boolean; +} diff --git a/src/lib/onboard/runtime-provider/contract.ts b/src/lib/onboard/runtime-provider/contract.ts index 8888d95f83a..5e806862404 100644 --- a/src/lib/onboard/runtime-provider/contract.ts +++ b/src/lib/onboard/runtime-provider/contract.ts @@ -2,6 +2,12 @@ // SPDX-License-Identifier: Apache-2.0 import type { SandboxEntry, SandboxWorkloadReceipt } from "../../state/registry/types"; +import type { + ManagedBootstrapRuntimeCreateLifecycle, + ManagedBootstrapRuntimeCreateLifecycleInput, + ManagedBootstrapRuntimeOnboardRouting, + ManagedBootstrapRuntimeOnboardRoutingInput, +} from "../managed-bootstrap/runtime-create"; import type { ManagedImageSelectionPolicy } from "../workload/source"; export const RUNTIME_PROVIDER_BUNDLE_CONTRACT_VERSION = 1 as const; @@ -261,7 +267,12 @@ export type RuntimeProviderMutationAuthoritySurface = export type RuntimeProviderBootstrapSurface = | RuntimeProviderSupportedSurface<{ - prepare(sandbox: SandboxEntry): unknown; + createLifecycle( + input: ManagedBootstrapRuntimeCreateLifecycleInput, + ): ManagedBootstrapRuntimeCreateLifecycle; + createOnboardRouting( + input: ManagedBootstrapRuntimeOnboardRoutingInput, + ): ManagedBootstrapRuntimeOnboardRouting; }> | RuntimeProviderUnsupportedSurface; diff --git a/src/lib/onboard/runtime-provider/registry.ts b/src/lib/onboard/runtime-provider/registry.ts index 332792d2dc2..1d0a31ae6aa 100644 --- a/src/lib/onboard/runtime-provider/registry.ts +++ b/src/lib/onboard/runtime-provider/registry.ts @@ -337,7 +337,8 @@ function validateMutationAuthoritySurface( function validateBootstrapSurface(surface: Record): void { if (surface.supported === true) { - requireFunction(surface, "prepare", "bootstrap"); + requireFunction(surface, "createLifecycle", "bootstrap"); + requireFunction(surface, "createOnboardRouting", "bootstrap"); } } diff --git a/src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts b/src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts index bf2a47fe4b4..b5628f99b78 100644 --- a/src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts +++ b/src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts @@ -20,6 +20,7 @@ import { stopSandbox } from "../../actions/sandbox/stop"; import { loadAgent } from "../../agent/defs"; import type { SandboxEntry, SandboxWorkloadReceipt } from "../../state/registry/types"; import { cloneSandboxWorkloadReceipt } from "../../state/registry/workload"; +import { createDockerManagedBootstrapSurface } from "../managed-bootstrap/docker-runtime"; import { MANAGED_IMAGE_REPOSITORIES } from "../managed-image/contract"; import { encodeManagedStartupProfile, @@ -145,6 +146,28 @@ describe("RuntimeProviderBundle registry contract", () => { } }); + it("validates the dormant Docker bootstrap candidate through the same bundle registry", () => { + const docker = createDockerRuntimeProviderBundle(); + const providers = createRuntimeProviderBundleRegistry([ + [ + "docker", + { + ...docker, + bootstrap: createDockerManagedBootstrapSurface(), + }, + ], + ]); + + expect(providers.docker?.bootstrap).toMatchObject({ + providerId: "docker", + supported: true, + }); + expect(CURRENT_RUNTIME_PROVIDER_BUNDLES.docker?.bootstrap).toMatchObject({ + providerId: "docker", + supported: false, + }); + }); + it("deeply clones and freezes every registered nested value", () => { const source = mxcBundle(); const registry = createRuntimeProviderBundleRegistry([["mxc", source]]); @@ -170,6 +193,59 @@ describe("RuntimeProviderBundle registry contract", () => { }).toThrow(TypeError); }); + it("registers an MXC-style managed-bootstrap provider through the bundle surface", () => { + const bundle = mxcBundle(); + const createLifecycle = vi.fn(() => ({ + launchArgv: ["mxc", "create"], + patch: { + maybeApplyDuringCreate: vi.fn(), + createFailureMessage: vi.fn(() => null), + exitOnPatchError: vi.fn(), + rollbackManagedStartupAfterCreateFailure: vi.fn(), + ensureApplied: vi.fn(), + waitForSupervisorReconnectIfNeeded: vi.fn(), + commitAfterReady: vi.fn(), + selectedMode: vi.fn(() => null), + printReadinessFailureIfEnabled: vi.fn(), + verifyGpuOrExit: vi.fn(async (verify) => verify("alpha")), + }, + prepareNetwork: vi.fn(async () => undefined), + runCreate: vi.fn(), + })); + const createOnboardRouting = vi.fn(() => ({ + nativeFallbackHasCleanBaseline: false, + inspectNativeRuntime: vi.fn(() => null), + isNativeCreateRoutingFailure: vi.fn(() => false), + isTrustedNativeRuntimeError: vi.fn(() => false), + isNativeReadinessRoutingFailure: vi.fn(() => false), + prepareCompatibilityLaunch: vi.fn(() => ({ createArgv: [], registryImageRef: null })), + })); + const providers = createRuntimeProviderBundleRegistry([ + [ + "mxc", + replaceSurface(bundle, "bootstrap", { + providerId: "mxc", + supported: true, + createLifecycle, + createOnboardRouting, + }), + ], + ]); + const registered = providers.mxc!; + expectSupportedSurface(registered.bootstrap); + + const routing = registered.bootstrap.createOnboardRouting({ + sandboxName: "alpha", + openshellArgv: (args) => args, + nativeFallbackEnabled: false, + }); + + expect(registered.identity.id).toBe("mxc"); + expect(routing.nativeFallbackHasCleanBaseline).toBe(false); + expect(createOnboardRouting).toHaveBeenCalledOnce(); + expect(createLifecycle).not.toHaveBeenCalled(); + }); + it("rejects an omitted managed platform without changing legacy receipt acceptance", () => { const { platform: _omittedPlatform, ...managedWithoutPlatform } = MANAGED_RECEIPT; const persistedManaged = cloneSandboxWorkloadReceipt(managedWithoutPlatform); diff --git a/src/lib/onboard/sandbox-create-launch.test.ts b/src/lib/onboard/sandbox-create-launch.test.ts index 3a650fc9c11..dec7913e208 100644 --- a/src/lib/onboard/sandbox-create-launch.test.ts +++ b/src/lib/onboard/sandbox-create-launch.test.ts @@ -8,8 +8,11 @@ import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; +import { managedStartupE2eProfile } from "../../../scripts/checks/generate-managed-startup-profile-fixture.mts"; import { loadAgent } from "../agent/defs"; import { SANDBOX_BUILD_CONTEXT_PREFIX } from "../sandbox/build-context"; +import { encodeManagedStartupProfile } from "./managed-startup/profile"; +import { createManagedStartupRootApplyRequest } from "./managed-startup/root-apply"; import { createOpenshellCliHelpers } from "./openshell-cli"; import { buildSandboxRuntimeEnvArgs, @@ -104,6 +107,49 @@ describe("buildSandboxRuntimeEnvArgs", () => { }); describe("prepareSandboxCreateLaunch", () => { + it.each([ + "openclaw", + "hermes", + "langchain-deepagents-code", + ] as const)("renders one identity-bound held launch for %s without exposing the startup profile", (agentName) => { + const request = createManagedStartupRootApplyRequest({ + agent: agentName, + encodedProfile: encodeManagedStartupProfile(managedStartupE2eProfile(agentName)), + }); + const result = prepareSandboxCreateLaunch({ + agent: loadAgent(agentName), + chatUiUrl: "", + createArgs: ["--name", `${agentName}-sandbox`], + env: {}, + extraPlaceholderKeys: [], + getDashboardForwardPort: () => "0", + hermesDashboardState: disabledHermesDashboardState, + manageDashboard: false, + openshellShellCommand: (args) => args.join(" "), + openshellArgv: (args) => ["openshell", ...args], + buildEnv: () => ({}), + managedStartupRootApplyRequest: request, + }); + + expect(result.intendedSandboxStartupCommand).toEqual([ + "env", + ...result.envArgs, + "nemoclaw-start", + ]); + expect(result.managedBootstrapIdentity).toMatch(/^[a-f0-9]{64}$/u); + expect(result.sandboxStartupCommand).toEqual([ + ...result.intendedSandboxStartupCommand.slice(0, -1), + "/usr/local/bin/nemoclaw-managed-startup-hold", + "--agent", + agentName, + "--profile-fingerprint", + request.profileFingerprint, + "--bootstrap-identity", + result.managedBootstrapIdentity, + ]); + expect(result.createArgv.join("\n")).not.toContain(request.encodedProfile); + }); + it("builds the sandbox create command and runtime env envelope", () => { const openshellShellCommand = vi.fn((args: string[]) => `openshell ${args.join(" ")}`); const result = prepareSandboxCreateLaunch({ diff --git a/src/lib/onboard/sandbox-create-launch.ts b/src/lib/onboard/sandbox-create-launch.ts index 1f3aa3cd01b..09e703f6d4b 100644 --- a/src/lib/onboard/sandbox-create-launch.ts +++ b/src/lib/onboard/sandbox-create-launch.ts @@ -9,6 +9,11 @@ import { appendExtraPlaceholderKeysEnvArg } from "./extra-placeholder-keys"; import type { HermesDashboardOnboardState } from "./hermes-dashboard"; import { appendHermesDashboardEnvArgs } from "./hermes-dashboard"; import { appendHostProxyEnvArgs } from "./host-proxy-env"; +import { + createManagedBootstrapIdentity, + renderManagedBootstrapHeldCommand, +} from "./managed-bootstrap/adapter"; +import type { ManagedStartupRootApplyRequest } from "./managed-startup/root-apply"; import { appendOpenClawRuntimeEnvArgs } from "./openclaw-runtime-env"; import { prebuildSandboxImageIfEligible, @@ -57,6 +62,8 @@ export interface SandboxCreateLaunchInput { openshellShellCommand: OpenshellShellCommand; openshellArgv?: OpenshellArgv; buildEnv?(): Record; + /** Dormant until a complete runtime bundle and durable authority store are selected. */ + managedStartupRootApplyRequest?: ManagedStartupRootApplyRequest | null; } export interface SandboxCreateLaunch { @@ -66,6 +73,9 @@ export interface SandboxCreateLaunch { envArgs: string[]; sandboxEnv: Record; sandboxStartupCommand: string[]; + intendedSandboxStartupCommand: string[]; + managedBootstrapIdentity: string | null; + managedStartupRootApplyRequest: ManagedStartupRootApplyRequest | null; } export interface SandboxCreateLaunchWithPrebuildInput extends SandboxCreateLaunchInput { @@ -203,7 +213,21 @@ export function prepareSandboxCreateLaunch(input: SandboxCreateLaunchInput): San // from openshell because bash returns the status of the last pipeline // command (awk, always 0) unless pipefail is set. Removing the pipe // lets the real exit code flow through to run(). - const sandboxStartupCommand = ["env", ...envArgs, "nemoclaw-start"]; + const intendedSandboxStartupCommand = ["env", ...envArgs, "nemoclaw-start"]; + const managedStartupRootApplyRequest = input.managedStartupRootApplyRequest ?? null; + const managedBootstrapIdentity = managedStartupRootApplyRequest + ? createManagedBootstrapIdentity() + : null; + const sandboxStartupCommand = + managedStartupRootApplyRequest && managedBootstrapIdentity + ? [ + ...renderManagedBootstrapHeldCommand( + managedStartupRootApplyRequest, + managedBootstrapIdentity, + intendedSandboxStartupCommand, + ), + ] + : intendedSandboxStartupCommand; const openshellArgs = ["sandbox", "create", ...input.createArgs, "--", ...sandboxStartupCommand]; const createCommand = renderSandboxCreateCommand( input.createArgs, @@ -221,6 +245,9 @@ export function prepareSandboxCreateLaunch(input: SandboxCreateLaunchInput): San envArgs, sandboxEnv, sandboxStartupCommand, + intendedSandboxStartupCommand, + managedBootstrapIdentity, + managedStartupRootApplyRequest, }; } diff --git a/src/lib/onboard/sandbox-gpu-create-flow.test.ts b/src/lib/onboard/sandbox-gpu-create-flow.test.ts index c7671ff4e60..8602f2e51f5 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.test.ts @@ -3,6 +3,9 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { managedStartupE2eProfile } from "../../../scripts/checks/generate-managed-startup-profile-fixture.mts"; +import { createInMemoryRuntimeProviderBundle } from "../../../test/helpers/runtime-provider-bundle"; + const mocks = vi.hoisted(() => ({ streamSandboxCreate: vi.fn(), waitForCreatedSandboxReadyWithTrace: vi.fn(), @@ -59,6 +62,18 @@ import { setupGpuFlowMocks, VERIFIED_GPU_PROOF as VERIFIED_PROOF, } from "./__test-helpers__/sandbox-gpu-create-flow"; +import type { + ManagedBootstrapRuntimeCreateLifecycleInput, + ManagedBootstrapRuntimePatch, +} from "./managed-bootstrap/runtime-create"; +import { encodeManagedStartupProfile } from "./managed-startup/profile"; +import { createManagedStartupRootApplyRequest } from "./managed-startup/root-apply"; +import type { + RuntimeProviderBootstrapSurface, + RuntimeProviderBundle, +} from "./runtime-provider/contract"; +import { createRuntimeProviderBundleRegistry } from "./runtime-provider/registry"; +import { prepareSandboxCreateLaunch } from "./sandbox-create-launch"; import { runSandboxGpuCreateFlow, type SandboxGpuCreateFlowDeps, @@ -150,6 +165,145 @@ function createSourceInput(): SandboxGpuCreateFlowInput { beforeEach(() => setupGpuFlowMocks(mocks)); afterEach(resetGpuFlowMocks); +describe("runSandboxGpuCreateFlow provider-owned managed create", () => { + it("runs an MXC-style bundle without a Docker branch in central orchestration", async () => { + const input = createInput(); + input.sandboxGpuConfig = { + mode: "0", + hostGpuDetected: false, + hostGpuPlatform: null, + sandboxGpuEnabled: false, + sandboxGpuDevice: null, + errors: [], + }; + input.gpuRoutePlan = "none"; + input.initialGpuRoute = "none"; + const request = createManagedStartupRootApplyRequest({ + agent: "openclaw", + encodedProfile: encodeManagedStartupProfile(managedStartupE2eProfile("openclaw")), + }); + const launch = prepareSandboxCreateLaunch({ + agent: null, + sandboxName: "alpha", + chatUiUrl: "", + createArgs: ["--name", "alpha"], + env: {}, + extraPlaceholderKeys: [], + getDashboardForwardPort: () => "0", + hermesDashboardState: { config: null, enabled: false }, + manageDashboard: false, + openshellShellCommand: (args) => args.join(" "), + openshellArgv: (args) => ["openshell", ...args], + buildEnv: () => ({}), + managedStartupRootApplyRequest: request, + }); + input.createArgv = launch.createArgv; + input.sandboxEnv = launch.sandboxEnv; + input.sandboxStartupCommand = launch.sandboxStartupCommand; + const patch = createPatch() as unknown as ManagedBootstrapRuntimePatch; + const createLifecycle = vi.fn( + (lifecycleInput: ManagedBootstrapRuntimeCreateLifecycleInput) => ({ + launchArgv: ["mxc-launch", ...lifecycleInput.launchArgv.slice(1)], + patch, + prepareNetwork: vi.fn(async () => undefined), + runCreate: async ( + start: (held: { + readonly heldWorkloadArgv: readonly string[]; + readonly bootstrapIdentity: string; + }) => Promise<{ readonly value: T }>, + ): Promise => + ( + await start({ + heldWorkloadArgv: lifecycleInput.heldWorkloadArgv, + bootstrapIdentity: lifecycleInput.bootstrapIdentity, + }) + ).value, + }), + ); + const source = createInMemoryRuntimeProviderBundle({ + providerId: "mxc", + workloadProfile: { + support: null, + hostArchitectures: [], + managedImageSelectionPolicy: "prefer-managed", + legacyDockerfileBuilds: true, + }, + }); + const registered = createRuntimeProviderBundleRegistry([ + [ + "mxc", + { + ...source, + bootstrap: { + providerId: "mxc", + supported: true, + createLifecycle, + createOnboardRouting: vi.fn(() => ({ + nativeFallbackHasCleanBaseline: false, + inspectNativeRuntime: vi.fn(() => null), + isNativeCreateRoutingFailure: vi.fn(() => false), + isTrustedNativeRuntimeError: vi.fn(() => false), + isNativeReadinessRoutingFailure: vi.fn(() => false), + prepareCompatibilityLaunch: vi.fn(() => ({ + createArgv: [], + registryImageRef: null, + })), + })), + }, + }, + ], + ]); + const runtimeProvider = registered.mxc as RuntimeProviderBundle & { + readonly bootstrap: Extract; + }; + input.managedBootstrap = { + bootstrapIdentity: launch.managedBootstrapIdentity!, + runtimeProvider, + authorityStore: { + async recordPreparedAuthority(authority) { + return { + schemaVersion: 1, + sandbox: authority.sandbox, + bootstrapIdentity: authority.bootstrapIdentity, + authorityFingerprint: authority.authorityFingerprint, + recordId: "mxc-record-alpha", + recordedAt: "2026-07-31T00:00:00.000Z", + }; + }, + }, + request, + image: { + repository: "registry.example/nemoclaw-openclaw", + manifestDigest: `sha256:${"d".repeat(64)}`, + }, + agentIdentity: { uid: 1000, gid: 1000, workdir: "/sandbox" }, + intendedWorkloadArgv: launch.intendedSandboxStartupCommand, + expectedSupervisorArgv: ["/mxc/supervisor"], + }; + const deps = createDeps(); + vi.mocked(deps.runCaptureOpenshell).mockImplementation((args) => + args[1] === "get" ? "ID: mxc-alpha\n" : "alpha Ready", + ); + + const result = await runSandboxGpuCreateFlow(input, deps); + + expect(result).toMatchObject({ route: "none", runtimePatch: patch }); + expect(createLifecycle).toHaveBeenCalledWith( + expect.objectContaining({ providerId: "mxc", route: "none" }), + ); + expect(mocks.streamSandboxCreate).toHaveBeenCalledWith( + "mxc-launch", + input.createArgv.slice(1), + input.sandboxEnv, + expect.anything(), + ); + expect(mocks.createDockerGpuSandboxCreatePatch).not.toHaveBeenCalled(); + expect(mocks.queryOpenShellDockerSandboxContainers).not.toHaveBeenCalled(); + expect(mocks.queryOpenShellDockerSandboxRuntimeSnapshot).not.toHaveBeenCalled(); + expect(mocks.enforceDockerGpuPatchPreserveNetwork).not.toHaveBeenCalled(); + }); +}); + describe("runSandboxGpuCreateFlow proof authorization", () => { it("does not retry compatibility when the native proof throws an exec/policy error (#6110)", async () => { const deps = createDeps(); diff --git a/src/lib/onboard/sandbox-gpu-create-flow.ts b/src/lib/onboard/sandbox-gpu-create-flow.ts index 98574891e0b..48acddb658d 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.ts @@ -10,8 +10,19 @@ import type { DockerGpuPatchDeps, DockerUlimit } from "./docker-gpu-patch-types" import type { SelectedDockerGpuRoute } from "./docker-gpu-route"; import { renderCompatibilityFallbackCreateArgs } from "./docker-gpu-route"; import { adaptDockerGpuRouteForPatch } from "./docker-gpu-route-patch-adapter"; -import type { DockerGpuSandboxCreatePatch } from "./docker-gpu-sandbox-create"; +import type { + ManagedBootstrapAdapter, + ManagedBootstrapAgentIdentity, + ManagedBootstrapAuthorityStore, + ManagedBootstrapImageIdentity, +} from "./managed-bootstrap/adapter"; +import type { ManagedBootstrapRuntimePatch } from "./managed-bootstrap/runtime-create"; +import type { ManagedStartupRootApplyRequest } from "./managed-startup/root-apply"; import { isImmutableDockerImageId } from "./openshell-docker-sandbox-containers"; +import type { + RuntimeProviderBootstrapSurface, + RuntimeProviderBundle, +} from "./runtime-provider/contract"; import * as sandboxGpuCreateAttempt from "./sandbox-gpu-create-attempt"; import { createSandboxGpuCreateAttemptRunner } from "./sandbox-gpu-create-run-attempt"; import type { SandboxGpuConfig } from "./sandbox-gpu-mode"; @@ -41,6 +52,18 @@ export interface SandboxGpuCreateFlowInput { restoreBackupPath: string | null; terminalAgent: boolean; persistStartupCommand?: boolean; + managedBootstrap?: { + readonly bootstrapIdentity: string; + readonly runtimeProvider: RuntimeProviderBundle & { + readonly bootstrap: Extract; + }; + readonly authorityStore: ManagedBootstrapAuthorityStore; + readonly request: ManagedStartupRootApplyRequest; + readonly image: ManagedBootstrapImageIdentity; + readonly agentIdentity: ManagedBootstrapAgentIdentity; + readonly intendedWorkloadArgv: readonly string[]; + readonly expectedSupervisorArgv: readonly string[]; + } | null; requiredUlimits?: readonly DockerUlimit[] | null; } @@ -50,11 +73,13 @@ export interface SandboxGpuCreateFlowDeps { sleep: Sleep; openshellArgv(args: string[]): string[]; verifyDirectSandboxGpu(sandboxName: string): SandboxGpuProofResult; + /** Production callers omit this factory and use the runtime provider's adapter. */ + createManagedBootstrapAdapter?: () => ManagedBootstrapAdapter; } export interface SandboxGpuCreateFlowResult { createResult: StreamSandboxCreateResult; - dockerGpuCreatePatch: DockerGpuSandboxCreatePatch; + runtimePatch: ManagedBootstrapRuntimePatch; route: SelectedDockerGpuRoute; firstCreateOutput: string; /** Mutable tag/reference retained only for registry and image-GC bookkeeping. */ @@ -105,46 +130,65 @@ export async function runSandboxGpuCreateFlow( throw new Error("Compatibility retry policy was not materialized."); } const nativeRuntimeSnapshot = attemptRunner.state.nativeRuntimeSnapshot; - const prebuildImageId = input.prebuild.imageId; - const imageId = - nativeRuntimeSnapshot?.imageId ?? - (prebuildImageId && isImmutableDockerImageId(prebuildImageId) - ? prebuildImageId.toLowerCase() - : null); - if ( - !registryImageRef && - nativeRuntimeSnapshot?.bookkeepingImageRef && - !isImmutableDockerImageId(nativeRuntimeSnapshot.bookkeepingImageRef) - ) { - registryImageRef = nativeRuntimeSnapshot.bookkeepingImageRef; + if (attemptRunner.managedRouting) { + const prepared = attemptRunner.managedRouting.prepareCompatibilityLaunch({ + createArgs: input.prebuild.createArgs, + currentRegistryImageRef: registryImageRef, + prebuildImageId: input.prebuild.imageId, + allowUnbuiltSource: attemptRunner.state.allowUnbuiltCompatibilitySource, + compatibilityPolicyPath: input.compatibilityPolicyPath, + startupCommand: input.sandboxStartupCommand, + runtimeSnapshot: nativeRuntimeSnapshot, + }); + attemptRunner.state.compatibilityArgv = [...prepared.createArgv]; + registryImageRef = prepared.registryImageRef; + } else { + const prebuildImageId = input.prebuild.imageId; + const imageId = + nativeRuntimeSnapshot?.imageId ?? + (prebuildImageId && isImmutableDockerImageId(prebuildImageId) + ? prebuildImageId.toLowerCase() + : null); + if ( + !registryImageRef && + nativeRuntimeSnapshot?.bookkeepingImageRef && + !isImmutableDockerImageId(nativeRuntimeSnapshot.bookkeepingImageRef) + ) { + registryImageRef = nativeRuntimeSnapshot.bookkeepingImageRef; + } + const compatibilityArgs = renderCompatibilityFallbackCreateArgs( + input.prebuild.createArgs, + { + imageRef: imageId, + allowUnbuiltSource: attemptRunner.state.allowUnbuiltCompatibilitySource, + compatibilityPolicyPath: input.compatibilityPolicyPath, + }, + ); + attemptRunner.state.compatibilityArgv = deps.openshellArgv([ + "sandbox", + "create", + ...compatibilityArgs, + "--", + ...input.sandboxStartupCommand, + ]); } - const compatibilityArgs = renderCompatibilityFallbackCreateArgs(input.prebuild.createArgs, { - imageRef: imageId, - allowUnbuiltSource: attemptRunner.state.allowUnbuiltCompatibilitySource, - compatibilityPolicyPath: input.compatibilityPolicyPath, - }); - attemptRunner.state.compatibilityArgv = deps.openshellArgv([ - "sandbox", - "create", - ...compatibilityArgs, - "--", - ...input.sandboxStartupCommand, - ]); if (attemptRunner.state.compatibilityArgv.length === 0) { throw new Error("Compatibility sandbox create executable is missing."); } }, activateCompatibilityAttempt: async () => { - await dockerGpuLocalInference.enforceDockerGpuPatchPreserveNetwork( - input.provider, - input.sandboxGpuConfig, - { - dockerDriverGateway: input.dockerDriverGateway, - selectedRoute: "compatibility", - gatewayPort: input.gatewayPort, - log: console.log, - }, - ); + if (!input.managedBootstrap) { + await dockerGpuLocalInference.enforceDockerGpuPatchPreserveNetwork( + input.provider, + input.sandboxGpuConfig, + { + dockerDriverGateway: input.dockerDriverGateway, + selectedRoute: "compatibility", + gatewayPort: input.gatewayPort, + log: console.log, + }, + ); + } input.sandboxGpuConfig.sandboxGpuProof = null; }, traceEvent: addTraceEvent, diff --git a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts index 726758cc0ac..8eeb4a8aac0 100644 --- a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts +++ b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { resolveOpenShellSandboxId } from "../adapters/openshell/sandbox-identity"; import { printSandboxCreateRecoveryHints } from "../build-context"; import { getSandboxDeleteOutcome } from "../domain/sandbox/destroy"; import { streamSandboxCreate } from "../sandbox/create-stream"; @@ -13,8 +14,8 @@ import { reportSandboxCreateFailure } from "./created-sandbox-failure"; import * as dockerGpuLocalInference from "./docker-gpu-local-inference"; import type { SelectedDockerGpuRoute } from "./docker-gpu-route"; import { createDockerGpuSandboxCreatePatch } from "./docker-gpu-sandbox-create"; +import type { ManagedBootstrapRuntimeSnapshot } from "./managed-bootstrap/runtime-create"; import { - type OpenShellDockerSandboxRuntimeSnapshotQuery, queryOpenShellDockerSandboxContainers, queryOpenShellDockerSandboxRuntimeSnapshot, } from "./openshell-docker-sandbox-containers"; @@ -28,7 +29,7 @@ import * as sandboxGpuPreflight from "./sandbox-gpu-preflight"; import * as sandboxReadinessTracing from "./sandbox-readiness-tracing"; import { addTraceEvent } from "./tracing"; -type NativeRuntimeSnapshot = Extract; +type NativeRuntimeSnapshot = ManagedBootstrapRuntimeSnapshot; export type SandboxGpuCreateAttemptState = { firstCreateOutput: string; @@ -41,6 +42,12 @@ export type SandboxGpuCreateAttemptState = { // Ready row. Require one confirmation poll before advancing to the GPU proof. const COMPATIBILITY_STABLE_READY_POLLS = 2; +class ManagedBootstrapCreateStreamFailure extends Error { + constructor(readonly result: Awaited>) { + super("Managed bootstrap held workload did not complete its create stream."); + } +} + export function createSandboxGpuCreateAttemptRunner( input: SandboxGpuCreateFlowInput, deps: SandboxGpuCreateFlowDeps, @@ -51,13 +58,26 @@ export function createSandboxGpuCreateAttemptRunner( allowUnbuiltCompatibilitySource: false, nativeRuntimeSnapshot: null, }; + const managedRouting = input.managedBootstrap?.runtimeProvider.bootstrap.createOnboardRouting({ + sandboxName: input.sandboxName, + openshellArgv: deps.openshellArgv, + nativeFallbackEnabled: + input.initialGpuRoute === "native" && input.gpuRoutePlan === "native-with-fallback", + }); const nativeFallbackBaseline = - input.initialGpuRoute === "native" && input.gpuRoutePlan === "native-with-fallback" + !managedRouting && + input.initialGpuRoute === "native" && + input.gpuRoutePlan === "native-with-fallback" ? queryOpenShellDockerSandboxContainers(input.sandboxName) : null; const nativeFallbackHasCleanBaseline = - nativeFallbackBaseline?.ok === true && nativeFallbackBaseline.ids.length === 0; - const inspectNativeRuntime = () => queryOpenShellDockerSandboxRuntimeSnapshot(input.sandboxName); + managedRouting?.nativeFallbackHasCleanBaseline ?? + (nativeFallbackBaseline?.ok === true && nativeFallbackBaseline.ids.length === 0); + const inspectNativeRuntime = (): NativeRuntimeSnapshot | null => { + if (managedRouting) return managedRouting.inspectNativeRuntime(); + const snapshot = queryOpenShellDockerSandboxRuntimeSnapshot(input.sandboxName); + return snapshot.ok ? snapshot : null; + }; const runAttempt = async (route: SelectedDockerGpuRoute) => { const compatibility = route === "compatibility"; @@ -70,73 +90,199 @@ export function createSandboxGpuCreateAttemptRunner( ); } const hasRequiredUlimits = (input.requiredUlimits?.length ?? 0) > 0; - const dockerGpuCreatePatch = createDockerGpuSandboxCreatePatch({ - route, - // The startup clone preserves native CDI devices, so DCode can apply its - // exact required limits without replacing the native GPU envelope. - // Other native routes are not swapped solely to persist a command. - persistStartupCommand: - input.persistStartupCommand === true && (route !== "native" || hasRequiredUlimits), - sandboxName: input.sandboxName, - gpuDevice: input.sandboxGpuConfig.sandboxGpuDevice, - openshellSandboxCommand: input.sandboxStartupCommand, - requiredUlimits: input.requiredUlimits, - timeoutSecs: input.sandboxReadyTimeoutSecs, - backend: input.sandboxGpuConfig.hostGpuPlatform === "jetson" ? "jetson" : "generic", - deps, - }); + const managedBootstrap = input.managedBootstrap ?? null; const attemptArgv = state.compatibilityArgv ?? input.createArgv; - const [createExecutable, ...createExecutableArgs] = attemptArgv; + const managedLifecycle = managedBootstrap + ? managedBootstrap.runtimeProvider.bootstrap.createLifecycle({ + providerId: managedBootstrap.runtimeProvider.identity.id, + bootstrapIdentity: managedBootstrap.bootstrapIdentity, + request: managedBootstrap.request, + image: managedBootstrap.image, + agentIdentity: managedBootstrap.agentIdentity, + intendedWorkloadArgv: managedBootstrap.intendedWorkloadArgv, + expectedSupervisorArgv: managedBootstrap.expectedSupervisorArgv, + launchArgv: attemptArgv, + heldWorkloadArgv: input.sandboxStartupCommand, + authorityStore: managedBootstrap.authorityStore, + ...(deps.createManagedBootstrapAdapter + ? { adapterOverride: deps.createManagedBootstrapAdapter() } + : {}), + route, + persistStartupCommand: input.persistStartupCommand === true, + sandboxName: input.sandboxName, + sandboxGpuConfig: input.sandboxGpuConfig, + requiredLimits: input.requiredUlimits ?? [], + timeoutSecs: input.sandboxReadyTimeoutSecs, + network: { + inferenceProvider: input.provider, + gatewayUsesContainerBridge: input.dockerDriverGateway, + gatewayPort: input.gatewayPort, + }, + dependencies: { + runCaptureOpenshell: deps.runCaptureOpenshell, + runOpenshell: deps.runOpenshell, + sleep: deps.sleep, + }, + }) + : null; + const runtimePatch = + managedLifecycle?.patch ?? + createDockerGpuSandboxCreatePatch({ + route, + // The startup clone preserves native CDI devices, so DCode can apply its + // exact required limits without replacing the native GPU envelope. + // Other native routes are not swapped solely to persist a command. + persistStartupCommand: + input.persistStartupCommand === true && (route !== "native" || hasRequiredUlimits), + externalRecreation: false, + sandboxName: input.sandboxName, + gpuDevice: input.sandboxGpuConfig.sandboxGpuDevice, + openshellSandboxCommand: input.sandboxStartupCommand, + requiredUlimits: input.requiredUlimits, + timeoutSecs: input.sandboxReadyTimeoutSecs, + backend: input.sandboxGpuConfig.hostGpuPlatform === "jetson" ? "jetson" : "generic", + deps, + }); + await managedLifecycle?.prepareNetwork(); + const [createExecutable, ...createExecutableArgs] = managedLifecycle?.launchArgv ?? attemptArgv; if (!createExecutable) throw new Error("Sandbox create executable is missing."); - const createResult = await streamSandboxCreate( - createExecutable, - createExecutableArgs, - input.sandboxEnv, - { + const streamCreate = () => + streamSandboxCreate(createExecutable, createExecutableArgs, input.sandboxEnv, { readyCheck: () => { const list = deps.runCaptureOpenshell(["sandbox", "list"], { ignoreError: true }); return isSandboxReady(list, input.sandboxName); }, - onPoll: () => dockerGpuCreatePatch.maybeApplyDuringCreate(), + onPoll: () => runtimePatch.maybeApplyDuringCreate(), readyCheckOutputPatterns: getReadyCheckOutputPatternsForAgent( input.terminalAgent, input.sandboxEnv, ), - failureCheck: dockerGpuCreatePatch.createFailureMessage, + failureCheck: runtimePatch.createFailureMessage, traceEvent: addTraceEvent, initialPhase: compatibility && (input.prebuild.imageRef || state.compatibilityArgv) ? "create" : undefined, - }, - ); + }); + let createResult: Awaited>; + let managedIncompleteCreateRecovered = false; + if (managedBootstrap && managedLifecycle) { + try { + createResult = await managedLifecycle.runCreate( + async ({ heldWorkloadArgv, bootstrapIdentity }) => { + if ( + bootstrapIdentity !== managedBootstrap.bootstrapIdentity || + heldWorkloadArgv.length !== input.sandboxStartupCommand.length || + heldWorkloadArgv.some((value, index) => value !== input.sandboxStartupCommand[index]) + ) { + throw new Error( + "Managed bootstrap launch does not match the rendered identity-bound hold.", + ); + } + const result = await streamCreate(); + const createFailure = + result.status === 0 ? null : classifySandboxCreateFailure(result.output); + if (result.status !== 0 && createFailure?.kind !== "sandbox_create_incomplete") { + throw new ManagedBootstrapCreateStreamFailure(result); + } + if (createFailure?.kind === "sandbox_create_incomplete") { + const readiness = sandboxReadinessTracing.waitForCreatedSandboxReadyWithTrace({ + sandboxName: input.sandboxName, + timeoutSecs: input.sandboxReadyTimeoutSecs, + runCaptureOpenshell: deps.runCaptureOpenshell, + isSandboxReady, + getSandboxFailurePhase, + stableReadyPolls: 1, + sleep: deps.sleep, + }); + if (!readiness.ready) { + throw new Error( + `Managed bootstrap incomplete create did not reach authoritative Ready state (${readiness.reason}).`, + ); + } + } else { + const list = deps.runCaptureOpenshell(["sandbox", "list"], { + ignoreError: true, + }); + if (!isSandboxReady(list, input.sandboxName)) { + throw new Error( + "Managed bootstrap create completed without an authoritative Ready sandbox.", + ); + } + } + let sandboxId: string; + try { + sandboxId = resolveOpenShellSandboxId(input.sandboxName, deps.runCaptureOpenshell); + } catch (error) { + throw new Error( + createFailure?.kind === "sandbox_create_incomplete" + ? "Managed bootstrap incomplete create did not return one exact durable sandbox identity after Ready." + : "Managed bootstrap create did not return one exact durable sandbox identity after Ready.", + { cause: error }, + ); + } + managedIncompleteCreateRecovered = createFailure?.kind === "sandbox_create_incomplete"; + return { + value: result, + receipt: { + sandbox: { + sandboxName: input.sandboxName, + sandboxId, + driverId: managedBootstrap.runtimeProvider.identity.id, + }, + ready: true, + readyAt: new Date().toISOString(), + }, + }; + }, + ); + } catch (error) { + if (!(error instanceof ManagedBootstrapCreateStreamFailure)) throw error; + createResult = error.result; + } + } else { + createResult = await streamCreate(); + } if (!state.firstCreateOutput) state.firstCreateOutput = createResult.output; - dockerGpuCreatePatch.exitOnPatchError(); + await runtimePatch.exitOnPatchError(); if (createResult.status !== 0) { const failure = classifySandboxCreateFailure(createResult.output); if (failure.kind === "sandbox_create_incomplete") { console.warn(""); - console.warn( - ` Create stream exited with code ${createResult.status} after sandbox was created.`, - ); - console.warn(" Checking whether the sandbox reaches Ready state..."); + if (managedIncompleteCreateRecovered) { + console.warn( + ` Create stream exited with code ${createResult.status}; the exact durable sandbox reached Ready, and onboarding is continuing with final checks.`, + ); + } else { + console.warn( + ` Create stream exited with code ${createResult.status} after sandbox was created.`, + ); + console.warn(" Checking whether the sandbox reaches Ready state..."); + } } else if ( route === "native" && input.gpuRoutePlan === "native-with-fallback" && nativeFallbackHasCleanBaseline && (() => { if ( - sandboxGpuCreateAttempt.isNativeGpuCreateRoutingFailure(createResult.output, { - sawProgress: createResult.sawProgress, - }) + managedRouting + ? managedRouting.isNativeCreateRoutingFailure( + createResult.output, + createResult.sawProgress, + ) + : sandboxGpuCreateAttempt.isNativeGpuCreateRoutingFailure(createResult.output, { + sawProgress: createResult.sawProgress, + }) ) { state.allowUnbuiltCompatibilitySource = input.prebuild.imageRef === null; return true; } const snapshot = inspectNativeRuntime(); if ( - snapshot.ok && - sandboxGpuCreateAttempt.isTrustedNativeGpuRuntimeError(snapshot.stateError) + snapshot && + (managedRouting + ? managedRouting.isTrustedNativeRuntimeError(snapshot.stateError) + : sandboxGpuCreateAttempt.isTrustedNativeGpuRuntimeError(snapshot.stateError)) ) { state.nativeRuntimeSnapshot = snapshot; return true; @@ -144,6 +290,7 @@ export function createSandboxGpuCreateAttemptRunner( return false; })() ) { + await runtimePatch.rollbackManagedStartupAfterCreateFailure(); return { ok: false, route, @@ -152,6 +299,7 @@ export function createSandboxGpuCreateAttemptRunner( fallbackEligible: true, } as const; } else { + await runtimePatch.rollbackManagedStartupAfterCreateFailure(); reportSandboxCreateFailure( { sandboxName: input.sandboxName, @@ -171,8 +319,8 @@ export function createSandboxGpuCreateAttemptRunner( ); } } - dockerGpuCreatePatch.ensureApplied(); - dockerGpuCreatePatch.waitForSupervisorReconnectIfNeeded(); + await runtimePatch.ensureApplied(); + await runtimePatch.waitForSupervisorReconnectIfNeeded(); console.log(" Waiting for sandbox to become ready..."); const readiness = sandboxReadinessTracing.waitForCreatedSandboxReadyWithTrace({ sandboxName: input.sandboxName, @@ -197,13 +345,19 @@ export function createSandboxGpuCreateAttemptRunner( const runtimeSnapshot = canClassifyNativeReadiness ? inspectNativeRuntime() : null; if ( canClassifyNativeReadiness && - runtimeSnapshot?.ok && - sandboxGpuCreateAttempt.isNativeGpuReadinessRoutingFailure({ - failurePhase: readiness.failurePhase, - runtimeError: runtimeSnapshot.stateError, - }) + runtimeSnapshot && + (managedRouting + ? managedRouting.isNativeReadinessRoutingFailure({ + failurePhase: readiness.failurePhase, + runtimeError: runtimeSnapshot.stateError, + }) + : sandboxGpuCreateAttempt.isNativeGpuReadinessRoutingFailure({ + failurePhase: readiness.failurePhase, + runtimeError: runtimeSnapshot.stateError, + })) ) { state.nativeRuntimeSnapshot = runtimeSnapshot; + await runtimePatch.rollbackManagedStartupAfterCreateFailure(); return { ok: false, route, @@ -214,10 +368,11 @@ export function createSandboxGpuCreateAttemptRunner( fallbackEligible: true, } as const; } + await runtimePatch.rollbackManagedStartupAfterCreateFailure(); printSandboxCreateFailureDiagnostics(input.sandboxName, { backupPath: input.restoreBackupPath, }); - if (compatibility) dockerGpuCreatePatch.printReadinessFailureIfEnabled(); + if (compatibility) runtimePatch.printReadinessFailureIfEnabled(); else { const deletion = deps.runOpenshell(["sandbox", "delete", input.sandboxName], { ignoreError: true, @@ -240,27 +395,32 @@ export function createSandboxGpuCreateAttemptRunner( route === "native" && input.gpuRoutePlan === "native-with-fallback" && nativeFallbackHasCleanBaseline; - const proof: SandboxGpuProofResult = dockerGpuLocalInference.verifyGpuSandboxAccessAfterReady( - input.sandboxGpuConfig, - { - sandboxName: input.sandboxName, - dockerDriverGateway: input.dockerDriverGateway, - selectedRoute: route, - verifyDirectSandboxGpu: deps.verifyDirectSandboxGpu, - verifyGpuOrExit: deferNativeProofFailure - ? undefined - : dockerGpuCreatePatch.verifyGpuOrExit, - reportGpuProofFailure: !deferNativeProofFailure, - selectedMode: dockerGpuCreatePatch.selectedMode, - runCaptureOpenshell: deps.runCaptureOpenshell, - log: console.log, - }, - ); + let proof: SandboxGpuProofResult; + try { + proof = await dockerGpuLocalInference.verifyGpuSandboxAccessAfterReady( + input.sandboxGpuConfig, + { + sandboxName: input.sandboxName, + dockerDriverGateway: input.dockerDriverGateway, + selectedRoute: route, + verifyDirectSandboxGpu: deps.verifyDirectSandboxGpu, + verifyGpuOrExit: deferNativeProofFailure ? undefined : runtimePatch.verifyGpuOrExit, + reportGpuProofFailure: !deferNativeProofFailure, + selectedMode: runtimePatch.selectedMode, + runCaptureOpenshell: deps.runCaptureOpenshell, + log: console.log, + }, + ); + } catch (error) { + await runtimePatch.rollbackManagedStartupAfterCreateFailure(); + throw error; + } if (deferNativeProofFailure && proof.status === "failed") { if (sandboxGpuPreflight.isExplicitNvidiaSmiDriverProofFailure(proof)) { const snapshot = inspectNativeRuntime(); - if (snapshot.ok && snapshot.nativeGpuAttachmentState === "absent") { + if (snapshot?.nativeGpuAttachmentState === "absent") { state.nativeRuntimeSnapshot = snapshot; + await runtimePatch.rollbackManagedStartupAfterCreateFailure(); return { ok: false, route, @@ -272,6 +432,7 @@ export function createSandboxGpuCreateAttemptRunner( } as const; } } + await runtimePatch.rollbackManagedStartupAfterCreateFailure(); console.error(""); console.error(" Native sandbox GPU proof failed."); console.error( @@ -283,15 +444,22 @@ export function createSandboxGpuCreateAttemptRunner( process.exit(1); } if (proof.status === "failed") { + await runtimePatch.rollbackManagedStartupAfterCreateFailure(); throw new Error("Sandbox GPU proof returned failed status."); } } + // GPU-enabled cutover stays reversible until the caller also proves the + // configured host-local inference path. Non-GPU workloads have completed + // their final authoritative Ready gate here. + if (!input.sandboxGpuConfig.sandboxGpuEnabled) { + await runtimePatch.commitAfterReady(); + } return { ok: true, route, - value: { createResult, dockerGpuCreatePatch }, + value: { createResult, runtimePatch }, } as const; }; - return { state, runAttempt }; + return { state, managedRouting, runAttempt }; } diff --git a/test/inference-options-docs.test.ts b/test/inference-options-docs.test.ts index 59bab6d063e..0a453be5640 100644 --- a/test/inference-options-docs.test.ts +++ b/test/inference-options-docs.test.ts @@ -428,7 +428,7 @@ describe("inference setup navigation", () => { expect(getSandboxRuntimeInferenceEndpoint("nvidia-nim")).toBeNull(); expect(getSandboxRuntimeInferenceEndpoint("compatible-endpoint")).toBeNull(); expect(section).toContain( - "For local Ollama and vLLM on Docker GPU sandboxes using the compatibility route", + "For local Ollama, local vLLM, and local NVIDIA NIM on Docker GPU sandboxes using the compatibility route", ); expect(section).toContain("NVIDIA NIM and other compatible endpoints"); }); diff --git a/test/onboard-prepared-build-context.test.ts b/test/onboard-prepared-build-context.test.ts index 0d91d612ae4..5babec1765c 100644 --- a/test/onboard-prepared-build-context.test.ts +++ b/test/onboard-prepared-build-context.test.ts @@ -98,12 +98,15 @@ let stageCalls = 0; dockerGpuSandboxCreate.createDockerGpuSandboxCreatePatch = () => ({ maybeApplyDuringCreate: () => {}, createFailureMessage: () => null, - exitOnPatchError: () => {}, - ensureApplied: () => {}, + exitOnPatchError: async () => {}, + attachManagedBootstrapCutover: () => {}, + rollbackManagedStartupAfterCreateFailure: async () => {}, + ensureApplied: async () => {}, waitForSupervisorReconnectIfNeeded: () => {}, + commitAfterReady: async () => {}, selectedMode: () => null, printReadinessFailureIfEnabled: () => {}, - verifyGpuOrExit: (verify) => verify(sandboxName), + verifyGpuOrExit: async (verify) => verify(sandboxName), }); buildContextStage.stageCreateSandboxBuildContext = () => { diff --git a/test/onboard-sandbox-recreation.test.ts b/test/onboard-sandbox-recreation.test.ts index 11b35a58b6e..c0a4eb2f6f3 100644 --- a/test/onboard-sandbox-recreation.test.ts +++ b/test/onboard-sandbox-recreation.test.ts @@ -1203,6 +1203,7 @@ const fs = require("node:fs"); const commands = []; let sandboxListCalls = 0; +let dockerPsCalls = 0; const keepAlive = setInterval(() => {}, 1000); runner.run = (command, opts = {}) => { _deleted = _deleted || _n(command).includes("sandbox delete"); @@ -1210,6 +1211,10 @@ runner.run = (command, opts = {}) => { return { status: 0 }; }; runner.runCapture = (command) => { + if (_n(command).startsWith("docker ps -a --no-trunc ")) { + dockerPsCalls += 1; + if (dockerPsCalls === 1) return "a".repeat(64); + } if (_n(command).includes("sandbox get") && _n(command).includes("my-assistant")) return ""; if (_n(command).includes("sandbox list")) { sandboxListCalls += 1; diff --git a/test/onboard-terminal-dashboard.test.ts b/test/onboard-terminal-dashboard.test.ts index 61baa83310a..0dd2686eadd 100644 --- a/test/onboard-terminal-dashboard.test.ts +++ b/test/onboard-terminal-dashboard.test.ts @@ -67,12 +67,15 @@ const _n = (c) => (Array.isArray(c) ? c.join(" ") : String(c)).replace(/'/g, "") dockerGpuSandboxCreate.createDockerGpuSandboxCreatePatch = () => ({ maybeApplyDuringCreate: () => {}, createFailureMessage: () => null, - exitOnPatchError: () => {}, - ensureApplied: () => {}, + exitOnPatchError: async () => {}, + attachManagedBootstrapCutover: () => {}, + rollbackManagedStartupAfterCreateFailure: async () => {}, + ensureApplied: async () => {}, waitForSupervisorReconnectIfNeeded: () => {}, + commitAfterReady: async () => {}, selectedMode: () => null, printReadinessFailureIfEnabled: () => {}, - verifyGpuOrExit: (verify) => verify(sandboxName), + verifyGpuOrExit: async (verify) => verify(sandboxName), }); agentOnboard.createAgentSandbox = () => { diff --git a/test/runtime-provider-source-shape.test.ts b/test/runtime-provider-source-shape.test.ts index 48e889d78c1..d4f3f74de5b 100644 --- a/test/runtime-provider-source-shape.test.ts +++ b/test/runtime-provider-source-shape.test.ts @@ -28,11 +28,19 @@ let activationPaths: string[] = []; let providerPaths: string[] = []; let dockerfilePaths: string[] = []; let packagingPaths: string[] = []; -const bootstrapLoad = - /(?:from\s*|import\s*|import\s*\(\s*|require\s*\(\s*)["'][^"']*managed-bootstrap/iu; +const managedBootstrapLoad = + /(?:from\s*|import\s*|import\s*\(\s*|require\s*\(\s*)["']([^"']*managed-bootstrap(?:\/[^"']*)?)["']/giu; +const allowedManagedBootstrapLoad = + /\/managed-bootstrap\/(?:adapter|runtime-create)(?:\.[cm]?[jt]s)?$/u; const packagedBootstrapAsset = /(?:nemoclaw-managed-bootstrap|managed-bootstrap-trampoline|managed-startup-image-runtime\.cjs|nemoclaw-managed-startup-hold)/u; +function disallowedManagedBootstrapLoads(source: string): string[] { + return [...source.matchAll(managedBootstrapLoad)] + .map((match) => match[1] ?? "") + .filter((specifier) => !allowedManagedBootstrapLoad.test(specifier)); +} + beforeAll(() => { productionPaths = trackedPaths( "src/lib/onboard.ts", @@ -70,7 +78,7 @@ beforeAll(() => { }); describe("runtime provider central source boundary", () => { - // source-shape-contract: compatibility -- Migrated lifecycle and mutation consumers must stay provider-neutral while production selection excludes unqualified future providers and managed-bootstrap dependencies + // source-shape-contract: compatibility -- Migrated lifecycle and mutation consumers must stay provider-neutral while production selection excludes unqualified future providers and driver-specific bootstrap dependencies it("keeps migrated provider identities and implementations behind the one bundle composition", () => { const driverNeutralActions = { "actions/inference-set.ts": read("src/lib/actions/inference-set.ts"), @@ -114,7 +122,12 @@ describe("runtime provider central source boundary", () => { expect(driverNeutralActions["actions/sandbox/start.ts"]).toMatch( /resolved\.lifecycle\.verifyStarted\(/u, ); - expect(Object.values(providerContract).join("\n")).not.toMatch(/managed-bootstrap/u); + expect(providerContract.contract).toMatch( + /import type[\s\S]*from ["']\.\.\/managed-bootstrap\/runtime-create["']/u, + ); + expect( + [providerContract.current, providerContract.docker, providerContract.registry].join("\n"), + ).not.toMatch(/managed-bootstrap/u); expect(providerContract.current).not.toMatch(/\b(?:podman|mxc)\b/iu); }); @@ -122,12 +135,14 @@ describe("runtime provider central source boundary", () => { expect(bootstrapProtocolPaths).toEqual([ "src/lib/onboard/managed-bootstrap/adapter.ts", "src/lib/onboard/managed-bootstrap/docker-journal.ts", + "src/lib/onboard/managed-bootstrap/docker-runtime.ts", "src/lib/onboard/managed-bootstrap/docker-shared-state.ts", "src/lib/onboard/managed-bootstrap/docker-spec.ts", "src/lib/onboard/managed-bootstrap/docker-test-fixture.ts", "src/lib/onboard/managed-bootstrap/docker.ts", "src/lib/onboard/managed-bootstrap/envelope.ts", "src/lib/onboard/managed-bootstrap/index.ts", + "src/lib/onboard/managed-bootstrap/runtime-create.ts", ]); }); @@ -160,26 +175,68 @@ describe("runtime provider central source boundary", () => { read("src/lib/onboard/managed-bootstrap/envelope.ts"), read("src/lib/onboard/managed-bootstrap/index.ts"), ].join("\n"); + const runtimeCreateContract = read("src/lib/onboard/managed-bootstrap/runtime-create.ts"); expect(bootstrapProtocolSource).not.toMatch(/from\s+["'][^"']*(?:docker|podman)[^"']*["']/iu); expect(bootstrapProtocolSource).not.toMatch( /(?:driverId|providerId)\s*(?:===|!==)\s*["'](?:docker|podman)["']/iu, ); expect(bootstrapProtocolSource).not.toMatch(/\b(?:docker|podman|openshell|mxc)\b/iu); + expect(runtimeCreateContract).not.toMatch(/\b(?:docker|podman|mxc)\b/iu); + expect(runtimeCreateContract).not.toMatch( + /(?:driverId|providerId)\s*(?:===|!==)\s*["'][^"']+["']/iu, + ); }); - // source-shape-contract: security -- Production onboarding must not activate managed bootstrap until a complete provider image and rollback implementation lands together - it("keeps production activation paths disconnected from managed bootstrap", () => { + // source-shape-contract: security -- Production onboarding may consume the provider-neutral create contract but cannot select a driver-specific bootstrap implementation + it("keeps production activation paths disconnected from driver bootstrap adapters", () => { const onboardEntry = read("src/lib/onboard.ts"); const activationSource = activationPaths.map(read).join("\n"); - expect(onboardEntry).not.toMatch(bootstrapLoad); - expect(activationSource).not.toMatch(bootstrapLoad); + expect(disallowedManagedBootstrapLoads(onboardEntry)).toEqual([]); + expect(disallowedManagedBootstrapLoads(activationSource)).toEqual([]); + expect( + disallowedManagedBootstrapLoads( + [ + 'import type { Contract } from "../managed-bootstrap/adapter";', + 'import type { Lifecycle } from "../../managed-bootstrap/runtime-create.mts";', + ].join("\n"), + ), + ).toEqual([]); + expect( + disallowedManagedBootstrapLoads( + [ + 'import "../managed-bootstrap";', + 'import "../managed-bootstrap/index";', + 'import "../managed-bootstrap/docker-runtime";', + 'await import("../managed-bootstrap/podman-runtime");', + 'require("../managed-bootstrap/mxc-runtime");', + 'export { provider } from "../managed-bootstrap/future-provider";', + 'import type { Fake } from "../fake-managed-bootstrap/adapter";', + 'import type { Nested } from "../managed-bootstrap/docker/adapter";', + ].join("\n"), + ), + ).toEqual([ + "../managed-bootstrap", + "../managed-bootstrap/index", + "../managed-bootstrap/docker-runtime", + "../managed-bootstrap/podman-runtime", + "../managed-bootstrap/mxc-runtime", + "../managed-bootstrap/future-provider", + "../fake-managed-bootstrap/adapter", + "../managed-bootstrap/docker/adapter", + ]); }); // source-shape-contract: security -- Registered runtime providers must remain bootstrap-unsupported until their complete transaction implementations are qualified it("keeps registered providers bootstrap-unsupported", () => { const dockerProvider = read("src/lib/onboard/runtime-provider/docker.ts"); - const providerSource = providerPaths.map(read).join("\n"); - expect(providerSource).not.toMatch(/managed-bootstrap/iu); + const providerImplementationSource = providerPaths + .filter((path) => path !== "src/lib/onboard/runtime-provider/contract.ts") + .map(read) + .join("\n"); + expect(dockerProvider).not.toMatch( + /(?:from\s+["'][^"']*managed-bootstrap|require\([^)]*managed-bootstrap)/u, + ); + expect(providerImplementationSource).not.toMatch(/managed-bootstrap/iu); expect(dockerProvider.match(/bootstrap:\s*unsupported\(/gu)).toHaveLength(2); expect(dockerProvider.match(/recovery:\s*unsupported\(/gu)).toHaveLength(2); });