diff --git a/docs/inference/set-up-ollama.mdx b/docs/inference/set-up-ollama.mdx index 3ec5e3281b7..72d7770f97d 100644 --- a/docs/inference/set-up-ollama.mdx +++ b/docs/inference/set-up-ollama.mdx @@ -219,6 +219,20 @@ It pulls missing models through the Ollama HTTP API without requiring an Ollama NemoClaw rejects a reachable Windows daemon when it accepts the untrusted `Host` probe. +### Recover After Unconfirmed Installer Cancellation + +If NemoClaw cannot confirm that the Windows installer stopped, it does not restore the previous Windows Ollama state because the installer might still change it. +Complete these steps before you rerun onboarding: + +1. In Windows PowerShell, stop the installer and every Ollama process. +2. Restore the previous User-scope `OLLAMA_HOST` value. + If the value was previously unset, remove it from User scope. +3. From WSL, rerun onboarding: + +```bash +$$nemoclaw onboard +``` + If the endpoint is not reachable, NemoClaw also checks the Windows `ollama.exe` process through PowerShell interop. When the daemon does not become reachable, onboarding prints PowerShell commands for inspecting the Windows process and port state. Choose the WSL-local Ollama install when Docker Desktop cannot reach the protected Windows loopback route. diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 443058f6292..c1f2be01366 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -1146,7 +1146,9 @@ $$nemoclaw dcode-sandbox agent -n "Summarize this repository" $$nemoclaw dcode-sandbox agent -n "Summarize this repository" --json ``` -For non-JSON OpenClaw turns, the wrapper captures `stdout` and `stderr` and replays them only after the in-sandbox command exits. The combined capture limit is `64 MiB`; exceeding it reports an OpenShell invocation error and exits with status `1`. If the captured output contains an embedded-fallback marker, the wrapper suppresses both streams, prints `recover`, `rebuild --yes`, and `onboard --resume` guidance to `stderr`, and exits with status `1`. Otherwise, it writes the captured output to the corresponding host streams and returns the OpenShell command's exit status. The in-sandbox NemoClaw plugin writes its registration banner to `stderr`, so the banner does not prefix the agent reply on `stdout` in non-JSON mode. Because a delivered turn always writes to one of the two streams, the wrapper reports a dispatch with status `0` and no output as a failure. The wrapper prints recovery guidance to `stderr` and exits with status `1`. Pressing `Ctrl+C` interrupts the OpenShell child, and sending `SIGTERM` to the host wrapper forwards `SIGTERM` to that child. NemoClaw waits for OpenShell to stop the in-sandbox turn, replays captured output, and returns status `130` for `SIGINT` or `143` for `SIGTERM`. When the forwarded argv sets `openclaw agent --timeout `, both captured paths bound the OpenShell command at that value plus 30 seconds. The extra seconds let the in-sandbox turn report its own timeout first, so the host bound catches only a turn that stops answering. +For non-JSON OpenClaw turns, the wrapper captures `stdout` and `stderr` and replays them only after the in-sandbox command exits. The combined capture limit is `64 MiB`; exceeding it reports an OpenShell invocation error and exits with status `1`. If the captured output contains an embedded-fallback marker, the wrapper suppresses both streams, prints `recover`, `rebuild --yes`, and `onboard --resume` guidance to `stderr`, and exits with status `1`. Otherwise, it writes the captured output to the corresponding host streams and returns the OpenShell command's exit status. The in-sandbox NemoClaw plugin writes its registration banner to `stderr`, so the banner does not prefix the agent reply on `stdout` in non-JSON mode. Because a delivered turn always writes to one of the two streams, the wrapper reports a dispatch with status `0` and no output as a failure. The wrapper prints recovery guidance to `stderr` and exits with status `1`. Pressing `Ctrl+C` interrupts the OpenShell child, and sending `SIGTERM` to the host wrapper forwards `SIGTERM` to that child. NemoClaw waits for OpenShell to stop the in-sandbox turn, replays captured output, and returns status `130` for `SIGINT` or `143` for `SIGTERM`. + +When the forwarded argv sets a positive whole-number `openclaw agent --timeout `, NemoClaw starts that deadline before its best-effort Ollama recovery. Recovery can use only the remaining time minus a reserved one second for the turn. Before dispatch, NemoClaw rewrites the forwarded timeout to the remaining time rounded up to a whole second, with a one-second minimum. Both captured paths then bound the OpenShell command at that forwarded remainder plus 30 seconds. The extra seconds let the in-sandbox turn report its own timeout first, so the host bound catches only a turn that stops answering. These leave the OpenShell wait unbounded: diff --git a/src/lib/actions/sandbox/agent/ollama-restart-recovery.test.ts b/src/lib/actions/sandbox/agent/ollama-restart-recovery.test.ts index 81ab7212c84..79f17c6995d 100644 --- a/src/lib/actions/sandbox/agent/ollama-restart-recovery.test.ts +++ b/src/lib/actions/sandbox/agent/ollama-restart-recovery.test.ts @@ -1,31 +1,96 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { EventEmitter } from "node:events"; +import type { StdioOptions } from "node:child_process"; import { chmodSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { delimiter, join } from "node:path"; -import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; + +import { describe, expect, it, vi } from "vitest"; import { OLLAMA_PORT, OLLAMA_PROXY_PORT } from "../../../core/ports"; -import { prepareOllamaApiExecution } from "../../../inference/local"; +import { CONTAINER_REACHABILITY_IMAGE } from "../../../inference/local"; +import type { SandboxExecSignalSource } from "../exec"; +import type { AgentDispatchChild } from "./passthrough-dispatch"; import { maybeWarmOllamaAfterDaemonRestart, + runOllamaRecoveryCapture, + type OllamaRecoveryCaptureFn, + type OllamaRecoveryCaptureResult, + type OllamaRecoverySpawner, type OllamaRestartRecoveryDeps, } from "./ollama-restart-recovery"; -const unloadedStatus = { - probed: true, - loaded: false, - cpuOnly: false, -}; - -function successfulWarmResult() { +function recoveryResult( + stdout: string, + overrides: Partial = {}, +): OllamaRecoveryCaptureResult { return { - stdout: JSON.stringify({ response: "Hello!", done: true }), + stdout, + stderr: "", exitCode: 0, timedOut: false, + ...overrides, }; } +function successfulWarmResult(): OllamaRecoveryCaptureResult { + return recoveryResult(JSON.stringify({ response: "Hello!", done: true })); +} + +function unloadedProbeResult(): OllamaRecoveryCaptureResult { + return recoveryResult(JSON.stringify({ models: [] })); +} + +function scriptedRecoveryCapture( + ...responses: OllamaRecoveryCaptureResult[] +): ReturnType> { + const pending = [...responses]; + return vi.fn(async () => + Promise.resolve(pending.shift() ?? Promise.reject(new Error("unexpected recovery request"))), + ); +} + +function failingRecoveryCapture( + error: Error, + ...responses: OllamaRecoveryCaptureResult[] +): ReturnType> { + const pending = [ + ...responses.map((response) => () => Promise.resolve(response)), + () => Promise.reject(error), + ]; + return vi.fn(async () => + (pending.shift() ?? (() => Promise.reject(new Error("unexpected recovery request"))))(), + ); +} + +function completingRecoverySpawner( + responses: readonly string[], +): ReturnType> { + const pending = [...responses]; + return vi.fn((_binary, _args, _stdio, _env) => { + const childEvents = new EventEmitter(); + const stderr = new EventEmitter(); + const stdout = new EventEmitter(); + const child: AgentDispatchChild = { + exitCode: null, + signalCode: null, + kill: vi.fn(() => true), + once: ((event: string, listener: (...args: unknown[]) => void) => + childEvents.once(event, listener)) as AgentDispatchChild["once"], + stderr, + stdout, + }; + const response = pending.shift() ?? ""; + queueMicrotask(() => { + stdout.emit("data", response); + child.exitCode = 0; + childEvents.emit("close", 0, null); + }); + return child; + }); +} + function getCommandUrl(command: readonly string[]): string { return command.find((arg) => arg.startsWith("http://")) ?? ""; } @@ -35,175 +100,437 @@ function getCommandBody(command: readonly string[]): Record { return JSON.parse(command[dataIndex + 1] ?? "null") as Record; } -function windowsRouteProtectionCapture(command: readonly string[]): string { - const rendered = command.join(" "); - return rendered.includes("Get-NetTCPConnection") - ? "127.0.0.1" - : command.includes("Host: rebinding.invalid") - ? "403" - : JSON.stringify({ models: [] }); -} - describe("maybeWarmOllamaAfterDaemonRestart", () => { - const originalPath = process.env.PATH; - let fakeDockerDir: string; - - beforeAll(() => { - fakeDockerDir = mkdtempSync(join(tmpdir(), "nemoclaw-restart-recovery-docker-")); - const fakeDockerPath = join(fakeDockerDir, "docker"); - writeFileSync(fakeDockerPath, "#!/bin/sh\nprintf '%s\\n' 'Operating System: Docker Desktop'\n"); - chmodSync(fakeDockerPath, 0o755); - process.env.PATH = `${fakeDockerDir}${delimiter}${originalPath ?? ""}`; + it("skips routes that are not local Ollama", async () => { + await expect( + maybeWarmOllamaAfterDaemonRestart({ provider: "vllm-local", model: "meta/llama" }), + ).resolves.toEqual({ kind: "skipped", reason: "not-ollama" }); }); - afterAll(() => { - Reflect.deleteProperty(process.env, "PATH"); - Object.assign(process.env, originalPath === undefined ? {} : { PATH: originalPath }); - rmSync(fakeDockerDir, { recursive: true, force: true }); + it("skips a local Ollama route without a registered model", async () => { + await expect(maybeWarmOllamaAfterDaemonRestart({ provider: "ollama-local" })).resolves.toEqual({ + kind: "skipped", + reason: "missing-model", + }); }); - beforeEach(() => { - vi.spyOn(process, "platform", "get").mockReturnValue("linux"); - vi.stubEnv("DOCKER_CONTEXT", "default"); - vi.stubEnv("WSL_DISTRO_NAME", "Ubuntu"); - }); + it("uses the resolved Windows host for an ambiguous direct bridge route", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + successfulWarmResult(), + ); - afterEach(() => { - vi.restoreAllMocks(); - vi.unstubAllEnvs(); - }); + await expect( + maybeWarmOllamaAfterDaemonRestart( + { + provider: "ollama-local", + model: "qwen3.6:35b", + endpointUrl: `http://host.openshell.internal:${OLLAMA_PORT}/v1`, + }, + { getOllamaHost: () => "host.docker.internal", + revalidateOllamaHost: () => "host.docker.internal", runRecoveryCaptureImpl }, + ), + ).resolves.toEqual({ kind: "warmed", ok: true }); - it("skips routes that are not local Ollama", () => { - expect( - maybeWarmOllamaAfterDaemonRestart({ provider: "vllm-local", model: "meta/llama" }), - ).toEqual({ kind: "skipped", reason: "not-ollama" }); + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[0]?.[0] ?? [])).toBe( + `http://host.docker.internal:${OLLAMA_PORT}/api/ps`, + ); + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[1]?.[0] ?? [])).toBe( + `http://host.docker.internal:${OLLAMA_PORT}/api/generate`, + ); + expect(getCommandBody(runRecoveryCaptureImpl.mock.calls[1]?.[0] ?? [])).toMatchObject({ + model: "qwen3.6:35b", + stream: false, + think: false, + }); + expect(runRecoveryCaptureImpl.mock.calls.map(([, options]) => options.host)).toEqual([ + "host.docker.internal", + "host.docker.internal", + ]); }); - it("skips a local Ollama route without a registered model", () => { - expect(maybeWarmOllamaAfterDaemonRestart({ provider: "ollama-local" })).toEqual({ + it("skips a stale raw Windows route before model probes or warm-up", async () => { + const runRecoveryCaptureImpl = vi.fn(); + + await expect( + maybeWarmOllamaAfterDaemonRestart( + { + provider: "ollama-local", + model: "qwen3.6:35b", + endpointUrl: `http://host.openshell.internal:${OLLAMA_PORT}/v1`, + }, + { + getOllamaHost: () => "host.docker.internal", + revalidateOllamaHost: () => null, + runRecoveryCaptureImpl, + }, + ), + ).resolves.toEqual({ kind: "skipped", - reason: "missing-model", + reason: "unreachable", + endpoint: `http://host.docker.internal:${OLLAMA_PORT}`, }); + expect(runRecoveryCaptureImpl).not.toHaveBeenCalled(); }); - it("uses the persisted direct bridge route for both the default probe and warm-up", () => { + it("uses the resolved WSL-local host for an ambiguous direct bridge route", async () => { + const prepareDockerEnvironment = vi.fn(() => { + throw new Error("unexpected Docker transport"); + }); + const spawnRecoveryChild = completingRecoverySpawner([ + unloadedProbeResult().stdout, + successfulWarmResult().stdout, + ]); + + await expect( + maybeWarmOllamaAfterDaemonRestart( + { + provider: "ollama-local", + model: "qwen3.6:35b", + endpointUrl: `http://host.openshell.internal:${OLLAMA_PORT}/v1`, + }, + { + getOllamaHost: () => "127.0.0.1", + prepareDockerEnvironment, + spawnRecoveryChild, + }, + ), + ).resolves.toEqual({ kind: "warmed", ok: true }); + + const commands = spawnRecoveryChild.mock.calls.map(([binary, args]) => [binary, ...args]); + expect(commands.map(getCommandUrl)).toEqual([ + `http://127.0.0.1:${OLLAMA_PORT}/api/ps`, + `http://127.0.0.1:${OLLAMA_PORT}/api/generate`, + ]); + expect(commands.map(([binary]) => binary)).toEqual(["curl", "curl"]); + expect(prepareDockerEnvironment).not.toHaveBeenCalled(); + }); + + it("runs the production status probe and warm-up through the async capture boundary", async () => { + const runRecoveryCaptureImpl = vi + .fn() + .mockResolvedValueOnce({ + stdout: JSON.stringify({ models: [] }), + stderr: "", + exitCode: 0, + timedOut: false, + }) + .mockResolvedValueOnce({ ...successfulWarmResult(), stderr: "" }); + + await expect( + maybeWarmOllamaAfterDaemonRestart( + { provider: "ollama-local", model: "qwen3.6:35b" }, + { runRecoveryCaptureImpl }, + ), + ).resolves.toEqual({ kind: "warmed", ok: true }); + expect(runRecoveryCaptureImpl).toHaveBeenCalledTimes(2); + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[0]?.[0] ?? [])).toBe( + `http://127.0.0.1:${OLLAMA_PORT}/api/ps`, + ); + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[1]?.[0] ?? [])).toBe( + `http://127.0.0.1:${OLLAMA_PORT}/api/generate`, + ); + }); + + it("runs Windows recovery through the isolated Docker transport", async () => { + const fakeDockerDirectory = mkdtempSync(join(tmpdir(), "nemoclaw-recovery-docker-")); + const fakeDockerPath = join(fakeDockerDirectory, "docker"); + writeFileSync(fakeDockerPath, "#!/bin/sh\nprintf '%s\\n' 'Operating System: Docker Desktop'\n"); + chmodSync(fakeDockerPath, 0o755); + const originalPath = process.env.PATH; + process.env.PATH = `${fakeDockerDirectory}${delimiter}${originalPath ?? ""}`; + const platformSpy = vi.spyOn(process, "platform", "get").mockReturnValue("linux"); + vi.stubEnv("DOCKER_CONTEXT", "default"); + vi.stubEnv("DOCKER_HOST", ""); + vi.stubEnv("WSL_DISTRO_NAME", "Ubuntu"); const cleanup = vi.fn(() => ({ ok: true as const })); - const prepareDockerEnvironment = () => ({ + const prepareDockerEnvironment = vi.fn(() => ({ env: { DOCKER_CONFIG: "/tmp/credential-free-docker" }, isolatedCredentialConfig: true, cleanup, + })); + const routeProtectionCapture = vi.fn((command: readonly string[]) => { + const rendered = command.join(" "); + return rendered.includes("Get-NetTCPConnection") + ? "127.0.0.1" + : command.includes("Host: rebinding.invalid") + ? "403" + : JSON.stringify({ models: [] }); }); - const runCaptureImpl = vi.fn( - (command: readonly string[], options?: { env?: NodeJS.ProcessEnv }) => { - const protection = windowsRouteProtectionCapture(command); - return protection !== JSON.stringify({ models: [] }) - ? protection - : options?.env?.DOCKER_CONFIG === "/tmp/credential-free-docker" - ? protection - : ""; - }, - ); - const runCaptureExImpl = vi.fn((_command: string[], options?: { env?: NodeJS.ProcessEnv }) => - options?.env?.DOCKER_CONFIG === "/tmp/credential-free-docker" - ? successfulWarmResult() - : { stdout: "", exitCode: 1, timedOut: false }, - ); + const spawnRecoveryChild = completingRecoverySpawner([ + unloadedProbeResult().stdout, + successfulWarmResult().stdout, + ]); - expect( - maybeWarmOllamaAfterDaemonRestart( + try { + const result = await maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b", endpointUrl: `http://host.openshell.internal:${OLLAMA_PORT}/v1`, }, { - runCaptureImpl, - runCaptureExImpl, - revalidateOllamaHost: () => "host.docker.internal", + dockerContextIsDefault: () => true, + getOllamaHost: () => "host.docker.internal", prepareDockerEnvironment, - prepareOllamaApiExecution: (command, host, options) => - prepareOllamaApiExecution(command, host, { - ...options, - prepareDockerEnvironment, - runCaptureImpl, - }), + revalidateOllamaHost: () => "host.docker.internal", + routeProtectionCapture, + spawnRecoveryChild, }, + ); + expect(result).toEqual({ kind: "warmed", ok: true }); + } finally { + Reflect.deleteProperty(process.env, "PATH"); + Object.assign(process.env, originalPath === undefined ? {} : { PATH: originalPath }); + platformSpy.mockRestore(); + vi.unstubAllEnvs(); + rmSync(fakeDockerDirectory, { recursive: true, force: true }); + } + + const commands = spawnRecoveryChild.mock.calls.map(([binary, args]) => [binary, ...args]); + expect(commands.map(([binary]) => binary)).toEqual(["docker", "docker"]); + expect(commands).toEqual([ + expect.arrayContaining([ + CONTAINER_REACHABILITY_IMAGE, + `http://host.docker.internal:${OLLAMA_PORT}/api/ps`, + ]), + expect.arrayContaining([ + CONTAINER_REACHABILITY_IMAGE, + `http://host.docker.internal:${OLLAMA_PORT}/api/generate`, + ]), + ]); + expect(spawnRecoveryChild.mock.calls.map(([, , , env]) => env.DOCKER_CONFIG)).toEqual([ + "/tmp/credential-free-docker", + "/tmp/credential-free-docker", + ]); + expect(prepareDockerEnvironment).toHaveBeenCalled(); + expect(cleanup).toHaveBeenCalled(); + }); + + it("stops recovery after an async status probe is cancelled", async () => { + const runRecoveryCaptureImpl = vi.fn().mockResolvedValue({ + stdout: "", + stderr: "", + exitCode: null, + timedOut: false, + signal: "SIGTERM", + }); + + await expect( + maybeWarmOllamaAfterDaemonRestart( + { provider: "ollama-local", model: "qwen3.6:35b" }, + { runRecoveryCaptureImpl }, ), - ).toEqual({ kind: "warmed", ok: true, timedOut: false }); + ).resolves.toEqual({ kind: "cancelled", signal: "SIGTERM" }); + expect(runRecoveryCaptureImpl).toHaveBeenCalledOnce(); + }); - const modelProbe = runCaptureImpl.mock.calls.find(([command]) => - getCommandUrl(command).endsWith("/api/ps"), - ); - expect(getCommandUrl(modelProbe?.[0] ?? [])).toBe( - `http://host.docker.internal:${OLLAMA_PORT}/api/ps`, + it("prepares and cleans one Docker environment for each recovery request", async () => { + const cleanups: Array> = []; + const prepareDockerEnvironment = vi.fn(() => { + const cleanup = vi.fn(() => ({ ok: true as const })); + cleanups.push(cleanup); + return { + env: { DOCKER_CONFIG: `/tmp/credential-free-docker-${cleanups.length}` }, + isolatedCredentialConfig: true, + cleanup, + }; + }); + const spawnRecoveryChild = completingRecoverySpawner(["one", "two", "three"]); + const command = ["docker", "run", "--rm", CONTAINER_REACHABILITY_IMAGE, "true"]; + const options = { + host: "127.0.0.1", + timeoutMilliseconds: 5_000, + spawnRecoveryChild, + prepareDockerEnvironment, + }; + await expect(runOllamaRecoveryCapture(command, options)).resolves.toMatchObject({ + exitCode: 0, + timedOut: false, + }); + await expect(runOllamaRecoveryCapture(command, options)).resolves.toMatchObject({ + exitCode: 0, + timedOut: false, + }); + await expect(runOllamaRecoveryCapture(command, options)).resolves.toMatchObject({ + exitCode: 0, + timedOut: false, + }); + + const commands = spawnRecoveryChild.mock.calls.map(([binary, args]) => [binary, ...args]); + expect(commands.every((command) => command[0] === "docker")).toBe(true); + expect(commands).toEqual([ + expect.arrayContaining([CONTAINER_REACHABILITY_IMAGE]), + expect.arrayContaining([CONTAINER_REACHABILITY_IMAGE]), + expect.arrayContaining([CONTAINER_REACHABILITY_IMAGE]), + ]); + expect(spawnRecoveryChild.mock.calls.map(([, , , env]) => env.DOCKER_CONFIG)).toEqual([ + "/tmp/credential-free-docker-1", + "/tmp/credential-free-docker-2", + "/tmp/credential-free-docker-3", + ]); + expect(prepareDockerEnvironment).toHaveBeenCalledTimes(3); + expect(cleanups).toHaveLength(3); + expect(cleanups[0]).toHaveBeenCalledOnce(); + expect(cleanups[1]).toHaveBeenCalledOnce(); + expect(cleanups[2]).toHaveBeenCalledOnce(); + }); + + it("forwards SIGTERM to an active recovery child and releases its Docker environment", async () => { + const childEvents = new EventEmitter(); + const signalEvents = new EventEmitter(); + const stderr = new EventEmitter(); + const stdout = new EventEmitter(); + const cleanup = vi.fn(() => ({ ok: true as const })); + const child: AgentDispatchChild = { + exitCode: null, + signalCode: null, + kill: vi.fn((signal) => { + child.signalCode = signal; + queueMicrotask(() => childEvents.emit("close", null, signal)); + return true; + }), + once: ((event: string, listener: (...args: unknown[]) => void) => + childEvents.once(event, listener)) as AgentDispatchChild["once"], + stderr, + stdout, + }; + const signalSource: SandboxExecSignalSource = { + add: (signal, listener) => signalEvents.on(signal, listener), + remove: (signal, listener) => signalEvents.off(signal, listener), + }; + const spawnRecoveryChild = vi.fn( + (_binary: string, _args: readonly string[], _stdio: StdioOptions, _env: NodeJS.ProcessEnv) => + child, ); - expect(modelProbe?.[0][0]).toBe("docker"); - expect(getCommandUrl(runCaptureExImpl.mock.calls[0][0])).toBe( - `http://host.docker.internal:${OLLAMA_PORT}/api/generate`, + + const pending = runOllamaRecoveryCapture( + ["docker", "run", "--rm", CONTAINER_REACHABILITY_IMAGE, "true"], + { + host: "127.0.0.1", + timeoutMilliseconds: 300_000, + prepareDockerEnvironment: () => ({ + env: { DOCKER_CONFIG: "/tmp/credential-free-docker" }, + isolatedCredentialConfig: true, + cleanup, + }), + signalSource, + spawnRecoveryChild, + }, ); - expect(runCaptureExImpl.mock.calls[0][0][0]).toBe("docker"); - expect(getCommandBody(runCaptureExImpl.mock.calls[0][0])).toMatchObject({ - model: "qwen3.6:35b", - stream: false, - think: false, + signalEvents.emit("SIGTERM"); + + await expect(pending).resolves.toMatchObject({ + exitCode: null, + signal: "SIGTERM", + timedOut: false, }); - expect(modelProbe?.[1]?.env?.DOCKER_CONFIG).toBe("/tmp/credential-free-docker"); - expect(runCaptureExImpl.mock.calls[0][1]?.env?.DOCKER_CONFIG).toBe( + expect(child.kill).toHaveBeenCalledOnce(); + expect(child.kill).toHaveBeenCalledWith("SIGTERM"); + expect(spawnRecoveryChild.mock.calls[0]?.[0]).toBe("docker"); + expect(spawnRecoveryChild.mock.calls[0]?.[3]?.DOCKER_CONFIG).toBe( "/tmp/credential-free-docker", ); - expect(cleanup).toHaveBeenCalledTimes(6); + expect(cleanup).toHaveBeenCalledOnce(); + expect(signalEvents.listenerCount("SIGTERM")).toBe(0); + expect(signalEvents.listenerCount("SIGINT")).toBe(0); }); - it("maps an auth-proxy route back to host loopback", () => { - const runCaptureImpl = vi.fn((_command: readonly string[]) => JSON.stringify({ models: [] })); - const runCaptureExImpl = vi.fn((_command: string[]) => successfulWarmResult()); + it("forces a timed-out recovery child to close and releases its Docker environment", async () => { + vi.useFakeTimers(); + const childEvents = new EventEmitter(); + const signalEvents = new EventEmitter(); + const cleanup = vi.fn(() => ({ ok: true as const })); + let child: AgentDispatchChild; + const kill = vi + .fn<(signal: NodeJS.Signals) => boolean>() + .mockReturnValueOnce(true) + .mockImplementationOnce((signal) => { + child.signalCode = signal; + queueMicrotask(() => childEvents.emit("close", null, signal)); + return true; + }); + child = { + exitCode: null, + signalCode: null, + kill, + once: ((event: string, listener: (...args: unknown[]) => void) => + childEvents.once(event, listener)) as AgentDispatchChild["once"], + stderr: new EventEmitter(), + stdout: new EventEmitter(), + }; + const signalSource: SandboxExecSignalSource = { + add: (signal, listener) => signalEvents.on(signal, listener), + remove: (signal, listener) => signalEvents.off(signal, listener), + }; + + try { + const pending = runOllamaRecoveryCapture( + ["docker", "run", "--rm", CONTAINER_REACHABILITY_IMAGE, "true"], + { + host: "127.0.0.1", + timeoutMilliseconds: 10, + prepareDockerEnvironment: () => ({ + env: { DOCKER_CONFIG: "/tmp/credential-free-docker" }, + isolatedCredentialConfig: true, + cleanup, + }), + signalSource, + spawnRecoveryChild: vi.fn(() => child), + }, + ); + + await vi.advanceTimersByTimeAsync(10); + expect(child.kill).toHaveBeenCalledTimes(1); + expect(child.kill).toHaveBeenLastCalledWith("SIGTERM"); + expect(cleanup).not.toHaveBeenCalled(); + + await vi.advanceTimersByTimeAsync(1_000); + await expect(pending).resolves.toMatchObject({ + exitCode: null, + signal: "SIGKILL", + timedOut: true, + }); + expect(child.kill).toHaveBeenCalledTimes(2); + expect(child.kill).toHaveBeenLastCalledWith("SIGKILL"); + expect(cleanup).toHaveBeenCalledOnce(); + expect(signalEvents.listenerCount("SIGTERM")).toBe(0); + expect(signalEvents.listenerCount("SIGINT")).toBe(0); + } finally { + vi.useRealTimers(); + } + }); - maybeWarmOllamaAfterDaemonRestart( + it("maps an auth-proxy route back to host loopback", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + successfulWarmResult(), + ); + + await maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b", endpointUrl: `http://host.openshell.internal:${OLLAMA_PROXY_PORT}/v1`, }, - { runCaptureImpl, runCaptureExImpl }, + { runRecoveryCaptureImpl }, ); - expect(getCommandUrl(runCaptureImpl.mock.calls[0][0])).toBe( + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[0]?.[0] ?? [])).toBe( `http://127.0.0.1:${OLLAMA_PORT}/api/ps`, ); - expect(runCaptureImpl.mock.calls[0][0][0]).toBe("curl"); - expect(getCommandUrl(runCaptureExImpl.mock.calls[0][0])).toBe( + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[1]?.[0] ?? [])).toBe( `http://127.0.0.1:${OLLAMA_PORT}/api/generate`, ); - expect(runCaptureExImpl.mock.calls[0][0][0]).toBe("curl"); }); - it("skips a stale raw Windows route before model probes or warm-up", () => { - const probeRuntimeModelStatus = vi.fn(() => unloadedStatus); - const runCaptureExImpl = vi.fn(() => successfulWarmResult()); - - expect( - maybeWarmOllamaAfterDaemonRestart( - { - provider: "ollama-local", - model: "qwen3.6:35b", - endpointUrl: `http://host.openshell.internal:${OLLAMA_PORT}/v1`, - }, - { - revalidateOllamaHost: () => null, - probeRuntimeModelStatus, - runCaptureExImpl, - }, - ), - ).toEqual({ kind: "skipped", reason: "unreachable" }); - expect(probeRuntimeModelStatus).not.toHaveBeenCalled(); - expect(runCaptureExImpl).not.toHaveBeenCalled(); - }); - - it("falls back to an allowlisted host instead of probing an arbitrary registry URL", () => { - const runCaptureImpl = vi.fn((_command: readonly string[]) => JSON.stringify({ models: [] })); - const runCaptureExImpl = vi.fn((_command: string[]) => successfulWarmResult()); + it("falls back to an allowlisted host instead of probing an arbitrary registry URL", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + successfulWarmResult(), + ); - maybeWarmOllamaAfterDaemonRestart( + await maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b", @@ -211,20 +538,25 @@ describe("maybeWarmOllamaAfterDaemonRestart", () => { }, { getOllamaHost: () => "also.example.com", - runCaptureImpl, - runCaptureExImpl, + runRecoveryCaptureImpl, }, ); - expect(getCommandUrl(runCaptureImpl.mock.calls[0][0])).toContain("http://127.0.0.1:"); - expect(getCommandUrl(runCaptureExImpl.mock.calls[0][0])).toContain("http://127.0.0.1:"); + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[0]?.[0] ?? [])).toContain( + "http://127.0.0.1:", + ); + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[1]?.[0] ?? [])).toContain( + "http://127.0.0.1:", + ); }); - it("does not map an unrecognized proxy-port host to host loopback (#6039)", () => { - const runCaptureImpl = vi.fn(windowsRouteProtectionCapture); - const runCaptureExImpl = vi.fn((_command: string[]) => successfulWarmResult()); + it("does not map an unrecognized proxy-port host to host loopback (#6039)", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + successfulWarmResult(), + ); - maybeWarmOllamaAfterDaemonRestart( + await maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b", @@ -232,173 +564,260 @@ describe("maybeWarmOllamaAfterDaemonRestart", () => { }, { getOllamaHost: () => "host.docker.internal", - revalidateOllamaHost: () => "host.docker.internal", - runCaptureImpl, - runCaptureExImpl, - prepareOllamaApiExecution: (command, host, options) => - prepareOllamaApiExecution(command, host, { ...options, runCaptureImpl }), + revalidateOllamaHost: () => "host.docker.internal", + runRecoveryCaptureImpl, }, ); - const modelProbe = runCaptureImpl.mock.calls.find(([command]) => - getCommandUrl(command).endsWith("/api/ps"), - ); - expect(getCommandUrl(modelProbe?.[0] ?? [])).toBe( + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[0]?.[0] ?? [])).toBe( `http://host.docker.internal:${OLLAMA_PORT}/api/ps`, ); - expect(getCommandUrl(runCaptureExImpl.mock.calls[0][0])).toBe( + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[1]?.[0] ?? [])).toBe( `http://host.docker.internal:${OLLAMA_PORT}/api/generate`, ); }); - it("skips the warm-up when the selected model is already loaded", () => { - const probeRuntimeModelStatus = vi.fn(() => ({ - probed: true, - loaded: true, - cpuOnly: false, - })); - const runCaptureExImpl = vi.fn(() => successfulWarmResult()); + it("skips the warm-up when the selected model is already loaded", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + recoveryResult(JSON.stringify({ models: [{ name: "qwen3.6:35b", size_vram: 1 }] })), + ); - expect( + await expect( maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b" }, - { probeRuntimeModelStatus, runCaptureExImpl }, + { runRecoveryCaptureImpl }, ), - ).toEqual({ kind: "skipped", reason: "already-loaded" }); - expect(runCaptureExImpl).not.toHaveBeenCalled(); + ).resolves.toEqual({ kind: "skipped", reason: "already-loaded" }); + expect(runRecoveryCaptureImpl).toHaveBeenCalledOnce(); }); - it("skips the warm-up when the daemon probe is unreachable", () => { - const runCaptureExImpl = vi.fn(() => successfulWarmResult()); + it("skips the warm-up when the daemon probe is unreachable", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture(recoveryResult("", { exitCode: 7 })); - expect( + await expect( maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b" }, - { runCaptureImpl: () => "", runCaptureExImpl }, + { runRecoveryCaptureImpl }, ), - ).toEqual({ kind: "skipped", reason: "unreachable" }); - expect(runCaptureExImpl).not.toHaveBeenCalled(); + ).resolves.toEqual({ + kind: "skipped", + reason: "unreachable", + endpoint: `http://127.0.0.1:${OLLAMA_PORT}`, + }); + expect(runRecoveryCaptureImpl).toHaveBeenCalledOnce(); }); - it("reports a bounded warm-up timeout", () => { - expect( + it("skips the warm-up when the daemon status response is malformed", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture(recoveryResult("not-json")); + + await expect( maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b" }, - { - probeRuntimeModelStatus: () => unloadedStatus, - runCaptureExImpl: () => ({ - stdout: "", - exitCode: 28, - timedOut: true, - }), - }, + { runRecoveryCaptureImpl }, ), - ).toEqual({ + ).resolves.toEqual({ + kind: "skipped", + reason: "unreachable", + endpoint: `http://127.0.0.1:${OLLAMA_PORT}`, + }); + expect(runRecoveryCaptureImpl).toHaveBeenCalledOnce(); + }); + + it("reports a bounded warm-up timeout", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + recoveryResult("", { exitCode: 28, timedOut: true }), + ); + await expect( + maybeWarmOllamaAfterDaemonRestart( + { provider: "ollama-local", model: "qwen3.6:35b" }, + { runRecoveryCaptureImpl }, + ), + ).resolves.toEqual({ kind: "warmed", ok: false, - timedOut: true, reason: "timeout", endpoint: "http://127.0.0.1:11434", detail: "warm-up exceeded 300 seconds", }); }); - it("does not treat an exit-zero Ollama error body as a successful warm-up", () => { - expect( + it("limits warm-up to the command timeout budget remaining after the probe", async () => { + let nowMs = 1_000; + const responses = [ + () => { + nowMs = 6_000; + return unloadedProbeResult(); + }, + () => successfulWarmResult(), + ]; + const runRecoveryCaptureImpl = vi.fn(async () => responses.shift()!()); + + await expect( maybeWarmOllamaAfterDaemonRestart( - { provider: "ollama-local", model: "missing:latest" }, + { provider: "ollama-local", model: "qwen3.6:35b" }, { - probeRuntimeModelStatus: () => unloadedStatus, - probeModelInventory: () => null, - runCaptureExImpl: () => ({ - stdout: JSON.stringify({ error: "model not found" }), - exitCode: 0, - timedOut: false, - }), + runRecoveryCaptureImpl, + timeoutSeconds: 30, + now: () => nowMs, }, ), - ).toMatchObject({ + ).resolves.toEqual({ kind: "warmed", ok: true }); + + const warmCommand = runRecoveryCaptureImpl.mock.calls[1]?.[0] ?? []; + expect(warmCommand[warmCommand.indexOf("--max-time") + 1]).toBe("25"); + expect(runRecoveryCaptureImpl.mock.calls[0]?.[1].timeoutMilliseconds).toBe(5_000); + expect(runRecoveryCaptureImpl.mock.calls[1]?.[1].timeoutMilliseconds).toBe(25_000); + }); + + it("skips warm-up when the probe consumes the command timeout budget", async () => { + let nowMs = 1_000; + const runRecoveryCaptureImpl = vi.fn(async () => { + nowMs = 31_000; + return unloadedProbeResult(); + }); + + await expect( + maybeWarmOllamaAfterDaemonRestart( + { provider: "ollama-local", model: "qwen3.6:35b" }, + { + runRecoveryCaptureImpl, + timeoutSeconds: 30, + now: () => nowMs, + }, + ), + ).resolves.toEqual({ + kind: "skipped", + reason: "deadline-exhausted", + endpoint: `http://127.0.0.1:${OLLAMA_PORT}`, + }); + expect(runRecoveryCaptureImpl).toHaveBeenCalledOnce(); + }); + + it("bounds the daemon probe and skips warm-up when a short timeout is consumed", async () => { + let nowMs = 1_000; + const runRecoveryCaptureImpl = vi.fn(async () => { + nowMs = 3_000; + return unloadedProbeResult(); + }); + + await expect( + maybeWarmOllamaAfterDaemonRestart( + { provider: "ollama-local", model: "qwen3.6:35b" }, + { runRecoveryCaptureImpl, timeoutSeconds: 2, now: () => nowMs }, + ), + ).resolves.toEqual({ + kind: "skipped", + reason: "deadline-exhausted", + endpoint: `http://127.0.0.1:${OLLAMA_PORT}`, + }); + expect(runRecoveryCaptureImpl.mock.calls[0]?.[1].timeoutMilliseconds).toBe(2_000); + }); + + it("does not treat an exit-zero Ollama error body as a successful warm-up", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + recoveryResult(JSON.stringify({ error: "model not found" })), + recoveryResult("not-json"), + ); + await expect( + maybeWarmOllamaAfterDaemonRestart( + { provider: "ollama-local", model: "missing:latest" }, + { runRecoveryCaptureImpl }, + ), + ).resolves.toMatchObject({ kind: "warmed", ok: false, - timedOut: false, reason: "ollama-error", endpoint: "http://127.0.0.1:11434", detail: expect.stringContaining("model not found"), }); }); - it("reports an endpoint that no longer holds the model instead of a warm failure (#9455)", () => { - const probeModelInventory = vi.fn(() => ["llama3.2:1b"]); - const runCaptureImpl = vi.fn(windowsRouteProtectionCapture); + it("keeps the warm-up error when the inventory probe throws", async () => { + const runRecoveryCaptureImpl = failingRecoveryCapture( + new Error("inventory unavailable"), + unloadedProbeResult(), + recoveryResult(JSON.stringify({ error: "runner stopped unexpectedly" })), + ); + await expect( + maybeWarmOllamaAfterDaemonRestart( + { provider: "ollama-local", model: "qwen3.6:35b" }, + { runRecoveryCaptureImpl }, + ), + ).resolves.toMatchObject({ + kind: "warmed", + ok: false, + reason: "ollama-error", + detail: expect.stringContaining("runner stopped unexpectedly"), + }); + }); + + it("reports an endpoint that no longer holds the model instead of a warm failure (#9455)", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + recoveryResult(JSON.stringify({ error: "model not found" })), + recoveryResult(JSON.stringify({ models: [{ name: "llama3.2:1b" }] })), + ); - expect( + await expect( maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "gemma4:26b", endpointUrl: `http://host.openshell.internal:${OLLAMA_PORT}/v1`, }, - { - revalidateOllamaHost: () => "host.docker.internal", - probeRuntimeModelStatus: () => unloadedStatus, - probeModelInventory, - runCaptureImpl, - prepareOllamaApiExecution: (command, host, options) => - prepareOllamaApiExecution(command, host, { ...options, runCaptureImpl }), - runCaptureExImpl: () => ({ - stdout: JSON.stringify({ error: "model not found" }), - exitCode: 0, - timedOut: false, - }), - }, + { getOllamaHost: () => "host.docker.internal", + revalidateOllamaHost: () => "host.docker.internal", runRecoveryCaptureImpl }, ), - ).toEqual({ + ).resolves.toEqual({ kind: "skipped", reason: "model-absent", endpoint: `http://host.docker.internal:${OLLAMA_PORT}`, inventoryLabel: "llama3.2:1b", }); - expect(probeModelInventory).toHaveBeenCalledWith("host.docker.internal", expect.any(Function)); + expect(runRecoveryCaptureImpl).toHaveBeenCalledTimes(3); + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[2]?.[0] ?? [])).toBe( + `http://host.docker.internal:${OLLAMA_PORT}/api/tags`, + ); + expect(runRecoveryCaptureImpl.mock.calls[2]?.[1].timeoutMilliseconds).toBe(5_000); }); - it("keeps the warm failure when the daemon does hold the model (#9455)", () => { - expect( + it("keeps the warm failure when the daemon does hold the model (#9455)", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + recoveryResult(JSON.stringify({ error: "runner stopped unexpectedly" })), + recoveryResult(JSON.stringify({ models: [{ name: "qwen3.6:35b" }] })), + ); + await expect( maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b" }, - { - probeRuntimeModelStatus: () => unloadedStatus, - probeModelInventory: () => ["qwen3.6:35b"], - runCaptureExImpl: () => ({ - stdout: JSON.stringify({ error: "runner stopped unexpectedly" }), - exitCode: 0, - timedOut: false, - }), - }, + { runRecoveryCaptureImpl }, ), - ).toMatchObject({ + ).resolves.toMatchObject({ kind: "warmed", ok: false, - timedOut: false, reason: "ollama-error", endpoint: "http://127.0.0.1:11434", detail: expect.stringContaining("runner stopped unexpectedly"), }); + expect(runRecoveryCaptureImpl).toHaveBeenCalledTimes(3); + expect(getCommandUrl(runRecoveryCaptureImpl.mock.calls[2]?.[0] ?? [])).toBe( + `http://127.0.0.1:${OLLAMA_PORT}/api/tags`, + ); }); - it("accepts a completed thinking-only response from a thinking model", () => { - expect( + it("accepts a completed thinking-only response from a thinking model", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + recoveryResult(JSON.stringify({ response: "", thinking: "The model is ready.", done: true })), + ); + await expect( maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b" }, - { - probeRuntimeModelStatus: () => unloadedStatus, - runCaptureExImpl: () => ({ - stdout: JSON.stringify({ response: "", thinking: "The model is ready.", done: true }), - exitCode: 0, - timedOut: false, - }), - }, + { runRecoveryCaptureImpl }, ), - ).toEqual({ kind: "warmed", ok: true, timedOut: false }); + ).resolves.toEqual({ kind: "warmed", ok: true }); }); it.each([ @@ -406,57 +825,56 @@ describe("maybeWarmOllamaAfterDaemonRestart", () => { ["malformed JSON", "not-json"], ["missing done marker", JSON.stringify({ response: "Hello!" })], ["empty response", JSON.stringify({ response: "", done: true })], - ])("rejects an invalid warm response: %s", (_name, stdout) => { - expect( + ])("rejects an invalid warm response: %s", async (_name, stdout) => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + recoveryResult(stdout), + ); + await expect( maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b" }, - { - probeRuntimeModelStatus: () => unloadedStatus, - runCaptureExImpl: () => ({ stdout, exitCode: 0, timedOut: false }), - }, + { runRecoveryCaptureImpl }, ), - ).toMatchObject({ + ).resolves.toMatchObject({ kind: "warmed", ok: false, - timedOut: false, reason: "invalid-response", endpoint: "http://127.0.0.1:11434", }); }); - it("reports a non-zero warm command exit", () => { - expect( + it("reports a non-zero warm command exit", async () => { + const runRecoveryCaptureImpl = scriptedRecoveryCapture( + unloadedProbeResult(), + recoveryResult("", { exitCode: 7 }), + ); + await expect( maybeWarmOllamaAfterDaemonRestart( { provider: "ollama-local", model: "qwen3.6:35b" }, - { - probeRuntimeModelStatus: () => unloadedStatus, - runCaptureExImpl: () => ({ stdout: "", exitCode: 7, timedOut: false }), - }, + { runRecoveryCaptureImpl }, ), - ).toEqual({ + ).resolves.toEqual({ kind: "warmed", ok: false, - timedOut: false, reason: "command-failed", endpoint: "http://127.0.0.1:11434", detail: "warm-up exited 7", }); }); - it("reports a warm process spawn failure without throwing", () => { + it("reports a warm process spawn failure without throwing", async () => { const deps: OllamaRestartRecoveryDeps = { - probeRuntimeModelStatus: () => unloadedStatus, - runCaptureExImpl: () => { - throw new Error("spawn failed"); - }, + runRecoveryCaptureImpl: failingRecoveryCapture( + new Error("spawn failed"), + unloadedProbeResult(), + ), }; - expect( + await expect( maybeWarmOllamaAfterDaemonRestart({ provider: "ollama-local", model: "qwen3.6:35b" }, deps), - ).toEqual({ + ).resolves.toEqual({ kind: "warmed", ok: false, - timedOut: false, reason: "spawn-failed", endpoint: "http://127.0.0.1:11434", detail: "spawn failed", diff --git a/src/lib/actions/sandbox/agent/ollama-restart-recovery.ts b/src/lib/actions/sandbox/agent/ollama-restart-recovery.ts index 3267fffd645..9fac386a4e8 100644 --- a/src/lib/actions/sandbox/agent/ollama-restart-recovery.ts +++ b/src/lib/actions/sandbox/agent/ollama-restart-recovery.ts @@ -14,27 +14,31 @@ // supported Ollama versions persist runners across restart, or when NemoClaw // manages daemon lifecycle and can warm the model at restart time instead. +import { spawn, type StdioOptions } from "node:child_process"; + import { buildValidatedCurlCommandArgs } from "../../../adapters/http/curl-args"; import { OLLAMA_PORT, OLLAMA_PROXY_PORT } from "../../../core/ports"; import { describeModelInventory, - createOllamaApiCapture, findReachableOllamaHost, - getOllamaApiCommand, getResolvedOllamaHost, ollamaInventoryContainsModel, OLLAMA_HOST_DOCKER_INTERNAL, OLLAMA_LOCALHOST, + parseOllamaModelInventory, prepareOllamaApiExecution, - probeOllamaEndpointInventory, - type RunCaptureFn, - type RunCaptureExFn, } from "../../../inference/local"; import { type OllamaRuntimeModelStatus, - probeOllamaRuntimeModelStatus, + parseOllamaRuntimeModelStatus, } from "../../../inference/ollama-runtime-context"; -import { runCaptureEx } from "../../../runner"; +import { buildSubprocessEnv, redact, redactFull } from "../../../runner"; +import type { SandboxExecSignalSource } from "../exec"; +import { + type AgentDispatchChild, + type AgentDispatchSpawner, + runAgentDispatch, +} from "./passthrough-dispatch"; export interface OllamaRestartRecoveryRoute { provider?: string | null; @@ -42,18 +46,24 @@ export interface OllamaRestartRecoveryRoute { endpointUrl?: string | null; } -export interface OllamaRestartRecoveryDeps { - probeRuntimeModelStatus?: ( - model: string, - getOllamaHost: () => string, - runCaptureImpl?: RunCaptureFn, - ) => OllamaRuntimeModelStatus; - probeModelInventory?: (host: string, runCaptureImpl?: RunCaptureFn) => string[] | null; - runCaptureExImpl?: RunCaptureExFn; +export interface OllamaRestartRecoveryOptions { + timeoutSeconds?: number; +} + +type PrepareOllamaDockerEnvironment = NonNullable< + Parameters[2] +>["prepareDockerEnvironment"]; +type OllamaExecutionOptions = NonNullable[2]>; + +export interface OllamaRestartRecoveryDeps extends OllamaRestartRecoveryOptions { getOllamaHost?: () => string; - runCaptureImpl?: RunCaptureFn; - prepareDockerEnvironment?: Parameters[2]; - prepareOllamaApiExecution?: typeof prepareOllamaApiExecution; + dockerContextIsDefault?: OllamaExecutionOptions["dockerContextIsDefault"]; + prepareDockerEnvironment?: PrepareOllamaDockerEnvironment; + routeProtectionCapture?: OllamaExecutionOptions["runCaptureImpl"]; + runRecoveryCaptureImpl?: OllamaRecoveryCaptureFn; + signalSource?: SandboxExecSignalSource; + spawnRecoveryChild?: OllamaRecoverySpawner; + now?: () => number; revalidateOllamaHost?: () => string | null; } @@ -65,13 +75,15 @@ export type OllamaRestartRecoveryFailureReason = | "spawn-failed"; export type OllamaRestartRecoveryResult = - | { kind: "skipped"; reason: "not-ollama" | "missing-model" | "already-loaded" | "unreachable" } + | { kind: "skipped"; reason: "not-ollama" | "missing-model" | "already-loaded" } + | { kind: "skipped"; reason: "unreachable"; endpoint: string } + | { kind: "skipped"; reason: "deadline-exhausted"; endpoint: string } | { kind: "skipped"; reason: "model-absent"; endpoint: string; inventoryLabel: string } - | { kind: "warmed"; ok: true; timedOut: false } + | { kind: "warmed"; ok: true } + | { kind: "cancelled"; signal: NodeJS.Signals } | { kind: "warmed"; ok: false; - timedOut: boolean; reason: OllamaRestartRecoveryFailureReason; endpoint: string; detail: string; @@ -79,6 +91,9 @@ export type OllamaRestartRecoveryResult = export const OLLAMA_LOCAL_PROVIDER = "ollama-local"; const OLLAMA_RESTART_RECOVERY_TIMEOUT_SECONDS = 300; +const OLLAMA_RESTART_RECOVERY_PROBE_TIMEOUT_MILLISECONDS = 5_000; +const OLLAMA_RESTART_RECOVERY_MAX_BUFFER_BYTES = 1024 * 1024; +const OLLAMA_RESTART_RECOVERY_TERMINATION_GRACE_MILLISECONDS = 1_000; const OPENSHELL_HOST_BRIDGE = "host.openshell.internal"; const ALLOWED_RAW_OLLAMA_HOSTS = new Set([ OLLAMA_LOCALHOST, @@ -124,7 +139,7 @@ function resolveRawOllamaHost( hostname === OPENSHELL_HOST_BRIDGE && port === OLLAMA_PORT ) { - return OLLAMA_HOST_DOCKER_INTERNAL; + return getAllowedFallbackHost(getOllamaHost); } if ( endpoint.protocol === "http:" && @@ -147,7 +162,124 @@ function resolveRawOllamaHost( return getAllowedFallbackHost(getOllamaHost); } -function buildWarmCommand(model: string, hostname: string): string[] { +export type OllamaRecoveryCaptureResult = { + stdout: string; + stderr: string; + exitCode: number | null; + timedOut: boolean; + signal?: NodeJS.Signals | null; + error?: Error; +}; + +export type OllamaRecoveryCaptureFn = ( + command: readonly string[], + options: { + host: string; + timeoutMilliseconds: number; + dockerContextIsDefault?: OllamaExecutionOptions["dockerContextIsDefault"]; + prepareDockerEnvironment?: PrepareOllamaDockerEnvironment; + routeProtectionCapture?: OllamaExecutionOptions["runCaptureImpl"]; + signalSource?: SandboxExecSignalSource; + spawnRecoveryChild?: OllamaRecoverySpawner; + }, +) => Promise; + +export type OllamaRecoverySpawner = ( + binary: string, + args: readonly string[], + stdio: StdioOptions, + env: NodeJS.ProcessEnv, +) => AgentDispatchChild; + +const defaultOllamaRecoverySpawner: OllamaRecoverySpawner = (binary, args, stdio, env) => + spawn(binary, [...args], { stdio, env }) as unknown as AgentDispatchChild; + +/** Capture one bounded recovery command through the shared signal-aware child supervisor. */ +export async function runOllamaRecoveryCapture( + command: readonly string[], + options: Parameters[1], +): Promise { + const execution = prepareOllamaApiExecution(command, options.host, { + dockerContextIsDefault: options.dockerContextIsDefault, + env: buildSubprocessEnv(), + prepareDockerEnvironment: options.prepareDockerEnvironment, + operation: "Ollama restart recovery", + runCaptureImpl: options.routeProtectionCapture, + }); + const [binary, ...args] = execution.command; + if (!binary) { + execution.cleanup(); + return { + stdout: "", + stderr: "", + exitCode: null, + timedOut: false, + error: new Error("Ollama recovery command is empty"), + }; + } + + let timedOut = false; + let timeout: ReturnType | undefined; + let forceTimeout: ReturnType | undefined; + const spawnRecoveryChild = options.spawnRecoveryChild ?? defaultOllamaRecoverySpawner; + const spawnChild: AgentDispatchSpawner = (runBinary, runArgs, stdio) => { + const child = spawnRecoveryChild(runBinary, runArgs, stdio, execution.env ?? {}); + timeout = setTimeout(() => { + timedOut = true; + if (child.exitCode === null && child.signalCode === null) { + child.kill("SIGTERM"); + forceTimeout = setTimeout(() => { + if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL"); + }, OLLAMA_RESTART_RECOVERY_TERMINATION_GRACE_MILLISECONDS); + forceTimeout.unref?.(); + } + }, options.timeoutMilliseconds); + timeout.unref?.(); + return child; + }; + + try { + const result = await runAgentDispatch( + binary, + args, + { maxBufferBytes: OLLAMA_RESTART_RECOVERY_MAX_BUFFER_BYTES, stdinIsTty: true }, + { signalSource: options.signalSource, spawnChild }, + ); + return { + stdout: result.stdout.trim(), + stderr: result.stderr.trim(), + exitCode: result.status, + timedOut: timedOut || result.status === 28, + signal: result.signal, + ...(result.error ? { error: result.error } : {}), + }; + } finally { + if (timeout) clearTimeout(timeout); + if (forceTimeout) clearTimeout(forceTimeout); + execution.cleanup(); + } +} + +function buildOllamaProbeCommand( + hostname: string, + path: "/api/ps" | "/api/tags", + timeoutMilliseconds: number, +): string[] { + const maxTimeSeconds = timeoutMilliseconds / 1000; + return [ + "curl", + ...buildValidatedCurlCommandArgs([ + "-sf", + "--connect-timeout", + String(Math.min(3, maxTimeSeconds)), + "--max-time", + String(maxTimeSeconds), + `http://${hostname}:${OLLAMA_PORT}${path}`, + ]), + ]; +} + +function buildWarmCommand(model: string, hostname: string, maxTimeSeconds: number): string[] { const body = JSON.stringify({ model, prompt: "Hello, reply in less than 5 words", @@ -156,21 +288,35 @@ function buildWarmCommand(model: string, hostname: string): string[] { keep_alive: "15m", options: { num_predict: 16 }, }); - return getOllamaApiCommand( - buildValidatedCurlCommandArgs([ + return [ + "curl", + ...buildValidatedCurlCommandArgs([ "-sS", "--connect-timeout", "3", "--max-time", - String(OLLAMA_RESTART_RECOVERY_TIMEOUT_SECONDS), + String(maxTimeSeconds), "-H", "Content-Type: application/json", "-d", body, `http://${hostname}:${OLLAMA_PORT}/api/generate`, ]), - hostname, - ); + ]; +} + +function recoveryDeadlineMilliseconds( + timeoutSeconds: number | undefined, + now: () => number, +): number | null { + if (timeoutSeconds === undefined) return null; + return now() + Math.min(timeoutSeconds, OLLAMA_RESTART_RECOVERY_TIMEOUT_SECONDS) * 1000; +} + +function remainingRecoveryMilliseconds(deadline: number | null, now: () => number): number { + return deadline === null + ? OLLAMA_RESTART_RECOVERY_TIMEOUT_SECONDS * 1000 + : Math.max(0, Math.floor(deadline - now())); } function validateWarmResponse(stdout: string): "ok" | "ollama-error" | "invalid-response" { @@ -195,9 +341,11 @@ function validateWarmResponse(stdout: string): "ok" | "ollama-error" | "invalid- } } -function boundedWarmFailureDetail(value: unknown, fallback: string): string { - const detail = String(value ?? "") - .replace(/\s+/g, " ") +export function boundedOllamaRestartRecoveryDetail(value: unknown, fallback: string): string { + const raw = value instanceof Error ? value.message : String(value ?? ""); + const detail = redactFull(redact(raw)) + .replace(/[\u0000-\u001f\u007f-\u009f\u061c\u200e\u200f\u2028-\u202e\u2066-\u2069]/gu, " ") + .replace(/\s+/gu, " ") .trim(); return (detail || fallback).slice(0, 300); } @@ -206,10 +354,10 @@ function boundedWarmFailureDetail(value: unknown, fallback: string): string { * Warm a registered local Ollama model only when `/api/ps` proves that the * daemon is reachable and the selected model is no longer loaded. */ -export function maybeWarmOllamaAfterDaemonRestart( +export async function maybeWarmOllamaAfterDaemonRestart( route: OllamaRestartRecoveryRoute, deps: OllamaRestartRecoveryDeps = {}, -): OllamaRestartRecoveryResult { +): Promise { if (normalizeRouteValue(route.provider) !== OLLAMA_LOCAL_PROVIDER) { return { kind: "skipped", reason: "not-ollama" }; } @@ -227,65 +375,110 @@ export function maybeWarmOllamaAfterDaemonRestart( (() => findReachableOllamaHost(undefined, {}, undefined, { revalidate: true })) )(); if (revalidatedHost !== OLLAMA_HOST_DOCKER_INTERNAL) { - return { kind: "skipped", reason: "unreachable" }; + return { + kind: "skipped", + reason: "unreachable", + endpoint: `http://${rawHost}:${OLLAMA_PORT}`, + }; } } const rawEndpoint = `http://${rawHost}:${OLLAMA_PORT}`; - const probe = deps.probeRuntimeModelStatus ?? probeOllamaRuntimeModelStatus; - const rawCapture = createOllamaApiCapture( - deps.runCaptureImpl, - rawHost, - deps.prepareDockerEnvironment, - ); + if ( + deps.timeoutSeconds !== undefined && + (!Number.isFinite(deps.timeoutSeconds) || deps.timeoutSeconds <= 0) + ) { + return { kind: "skipped", reason: "deadline-exhausted", endpoint: rawEndpoint }; + } + const now = deps.now ?? Date.now; + const recoveryDeadline = recoveryDeadlineMilliseconds(deps.timeoutSeconds, now); + const probeBudgetMilliseconds = remainingRecoveryMilliseconds(recoveryDeadline, now); + if (probeBudgetMilliseconds === 0) { + return { kind: "skipped", reason: "deadline-exhausted", endpoint: rawEndpoint }; + } let status: OllamaRuntimeModelStatus; try { - status = probe(model, () => rawHost, rawCapture); + const statusTimeoutMilliseconds = Math.min( + OLLAMA_RESTART_RECOVERY_PROBE_TIMEOUT_MILLISECONDS, + probeBudgetMilliseconds, + ); + const capture = deps.runRecoveryCaptureImpl ?? runOllamaRecoveryCapture; + const result = await capture( + buildOllamaProbeCommand(rawHost, "/api/ps", statusTimeoutMilliseconds), + { + host: rawHost, + timeoutMilliseconds: statusTimeoutMilliseconds, + dockerContextIsDefault: deps.dockerContextIsDefault, + prepareDockerEnvironment: deps.prepareDockerEnvironment, + routeProtectionCapture: deps.routeProtectionCapture, + signalSource: deps.signalSource, + spawnRecoveryChild: deps.spawnRecoveryChild, + }, + ); + if (result.signal && !result.timedOut) { + return { kind: "cancelled", signal: result.signal }; + } + status = + result.exitCode === 0 && !result.error + ? parseOllamaRuntimeModelStatus(model, result.stdout) + : { probed: false, loaded: false, cpuOnly: false }; } catch { - return { kind: "skipped", reason: "unreachable" }; + return { kind: "skipped", reason: "unreachable", endpoint: rawEndpoint }; } if (!status.probed) { - return { kind: "skipped", reason: "unreachable" }; + return { kind: "skipped", reason: "unreachable", endpoint: rawEndpoint }; } if (status.loaded) { return { kind: "skipped", reason: "already-loaded" }; } - const captureEx = deps.runCaptureExImpl ?? runCaptureEx; + const warmupTimeoutMilliseconds = remainingRecoveryMilliseconds(recoveryDeadline, now); + if (warmupTimeoutMilliseconds === 0) { + return { kind: "skipped", reason: "deadline-exhausted", endpoint: rawEndpoint }; + } + const warmupTimeoutSeconds = warmupTimeoutMilliseconds / 1000; + try { - const execution = (deps.prepareOllamaApiExecution ?? prepareOllamaApiExecution)( - buildWarmCommand(model, rawHost), - rawHost, - { operation: `Ollama restart warm-up for '${model}'` }, - ); - let result; - try { - result = captureEx(execution.command, { - ...(execution.env === undefined ? {} : { env: execution.env }), - }); - } finally { - execution.cleanup(); + const command = buildWarmCommand(model, rawHost, warmupTimeoutSeconds); + const result = await (deps.runRecoveryCaptureImpl ?? runOllamaRecoveryCapture)(command, { + host: rawHost, + timeoutMilliseconds: warmupTimeoutMilliseconds, + dockerContextIsDefault: deps.dockerContextIsDefault, + prepareDockerEnvironment: deps.prepareDockerEnvironment, + routeProtectionCapture: deps.routeProtectionCapture, + signalSource: deps.signalSource, + spawnRecoveryChild: deps.spawnRecoveryChild, + }); + if (result.signal && !result.timedOut) { + return { kind: "cancelled", signal: result.signal }; } if (result.timedOut) { return { kind: "warmed", ok: false, - timedOut: true, reason: "timeout", endpoint: rawEndpoint, - detail: boundedWarmFailureDetail( + detail: boundedOllamaRestartRecoveryDetail( result.stderr, - `warm-up exceeded ${OLLAMA_RESTART_RECOVERY_TIMEOUT_SECONDS} seconds`, + `warm-up exceeded ${String(warmupTimeoutSeconds)} seconds`, ), }; } + if (result.error) { + return { + kind: "warmed", + ok: false, + reason: "spawn-failed", + endpoint: rawEndpoint, + detail: boundedOllamaRestartRecoveryDetail(result.error, "warm-up process could not start"), + }; + } if (result.exitCode !== 0) { return { kind: "warmed", ok: false, - timedOut: false, reason: "command-failed", endpoint: rawEndpoint, - detail: boundedWarmFailureDetail( + detail: boundedOllamaRestartRecoveryDetail( result.stderr || result.stdout, `warm-up exited ${String(result.exitCode)}`, ), @@ -298,36 +491,63 @@ export function maybeWarmOllamaAfterDaemonRestart( // valid (#9455). Ask the same daemon for its inventory to tell them apart; // an unreadable inventory keeps the original warm-failure reason. if (response === "ollama-error") { - const probeInventory = deps.probeModelInventory ?? probeOllamaEndpointInventory; - const inventory = probeInventory(rawHost, rawCapture); - if (inventory && !ollamaInventoryContainsModel(inventory, model)) { - return { - kind: "skipped", - reason: "model-absent", - endpoint: `http://${rawHost}:${OLLAMA_PORT}`, - inventoryLabel: describeModelInventory(inventory), - }; + const inventoryBudgetMilliseconds = remainingRecoveryMilliseconds(recoveryDeadline, now); + if (inventoryBudgetMilliseconds > 0) { + let inventory: string[] | null = null; + try { + const inventoryTimeoutMilliseconds = Math.min( + OLLAMA_RESTART_RECOVERY_PROBE_TIMEOUT_MILLISECONDS, + inventoryBudgetMilliseconds, + ); + const inventoryResult = await (deps.runRecoveryCaptureImpl ?? runOllamaRecoveryCapture)( + buildOllamaProbeCommand(rawHost, "/api/tags", inventoryTimeoutMilliseconds), + { + host: rawHost, + timeoutMilliseconds: inventoryTimeoutMilliseconds, + dockerContextIsDefault: deps.dockerContextIsDefault, + prepareDockerEnvironment: deps.prepareDockerEnvironment, + routeProtectionCapture: deps.routeProtectionCapture, + signalSource: deps.signalSource, + spawnRecoveryChild: deps.spawnRecoveryChild, + }, + ); + if (inventoryResult.signal && !inventoryResult.timedOut) { + return { kind: "cancelled", signal: inventoryResult.signal }; + } + inventory = + inventoryResult.exitCode === 0 && !inventoryResult.error + ? parseOllamaModelInventory(inventoryResult.stdout) + : null; + } catch { + // Inventory only refines the original warm-up error. + } + if (inventory && !ollamaInventoryContainsModel(inventory, model)) { + return { + kind: "skipped", + reason: "model-absent", + endpoint: `http://${rawHost}:${OLLAMA_PORT}`, + inventoryLabel: describeModelInventory(inventory), + }; + } } } if (response !== "ok") { return { kind: "warmed", ok: false, - timedOut: false, reason: response, endpoint: rawEndpoint, - detail: boundedWarmFailureDetail(result.stdout, `Ollama returned ${response}`), + detail: boundedOllamaRestartRecoveryDetail(result.stdout, `Ollama returned ${response}`), }; } - return { kind: "warmed", ok: true, timedOut: false }; + return { kind: "warmed", ok: true }; } catch (error) { return { kind: "warmed", ok: false, - timedOut: false, reason: "spawn-failed", endpoint: rawEndpoint, - detail: boundedWarmFailureDetail( + detail: boundedOllamaRestartRecoveryDetail( error instanceof Error ? error.message : error, "warm-up process could not start", ), diff --git a/src/lib/actions/sandbox/agent/passthrough-dispatch.test.ts b/src/lib/actions/sandbox/agent/passthrough-dispatch.test.ts index f43acf7dfdd..690e4274430 100644 --- a/src/lib/actions/sandbox/agent/passthrough-dispatch.test.ts +++ b/src/lib/actions/sandbox/agent/passthrough-dispatch.test.ts @@ -12,12 +12,13 @@ import { agentDispatchStdio, isSilentAgentDispatch, isTimedOutAgentDispatch, + replaceRequestedAgentTimeoutSeconds, requestedAgentTimeoutSeconds, runAgentDispatch, SILENT_AGENT_DISPATCH_EXIT_CODE, TIMED_OUT_AGENT_TURN_EXIT_CODE, } from "./passthrough-dispatch"; -import { computeExitCode, type SandboxExecSignalSource } from "../exec"; +import { buildOpenshellExecArgs, computeExitCode, type SandboxExecSignalSource } from "../exec"; function dispatchHarness() { const childEvents = new EventEmitter(); @@ -41,7 +42,7 @@ function dispatchHarness() { add: (signal, listener) => signalEvents.on(signal, listener), remove: (signal, listener) => signalEvents.off(signal, listener), }; - return { child, signalEvents, signalSource, stderr, stdout }; + return { child, childEvents, signalEvents, signalSource, stderr, stdout }; } describe("runAgentDispatch", () => { @@ -114,6 +115,60 @@ describe("runAgentDispatch", () => { expect(result.stdout).toBe("1234"); expect(result.stderr).toBe(""); }); + + it("delivers a turn timeout reported 20.8 seconds after its requested deadline (#8723)", async () => { + vi.useFakeTimers(); + const harness = dispatchHarness(); + const requestedDeadlineSeconds = 30; + const delayedFinishMilliseconds = (requestedDeadlineSeconds + 20.8) * 1000; + const timeoutReport = "Request timed out before a response was generated.\n"; + const command = [ + "openclaw", + "agent", + "--timeout", + String(requestedDeadlineSeconds), + "-m", + "ping", + ]; + const args = buildOpenshellExecArgs("alpha", command, { + tty: false, + timeoutSeconds: agentDispatchDeadlineSeconds(command), + }); + + try { + const pending = runAgentDispatch( + "openshell", + args, + { stdinIsTty: true }, + { + signalSource: harness.signalSource, + spawnChild: (_binary, spawnArgs) => { + const hostTimeoutIndex = spawnArgs.indexOf("--timeout"); + const hostTimeoutMilliseconds = Number(spawnArgs[hostTimeoutIndex + 1]) * 1000; + setTimeout(() => harness.child.kill("SIGTERM"), hostTimeoutMilliseconds); + setTimeout(() => { + harness.stdout.emit("data", timeoutReport); + harness.child.exitCode = 0; + harness.childEvents.emit("close", 0, null); + }, delayedFinishMilliseconds); + return harness.child; + }, + }, + ); + + await vi.advanceTimersByTimeAsync(delayedFinishMilliseconds); + expect(await pending).toMatchObject({ + status: 0, + signal: null, + stdout: timeoutReport, + stderr: "", + }); + expect(harness.child.kill).not.toHaveBeenCalled(); + } finally { + vi.clearAllTimers(); + vi.useRealTimers(); + } + }); }); describe("isSilentAgentDispatch", () => { @@ -174,11 +229,19 @@ describe("requestedAgentTimeoutSeconds", () => { }); it("reads a separated --timeout value (#8723)", () => { - expect(requestedAgentTimeoutSeconds(agent("--agent", "main", "--timeout", "30"))).toBe(30); + const command = agent("--agent", "main", "--timeout", "30"); + expect(requestedAgentTimeoutSeconds(command)).toBe(30); + expect(replaceRequestedAgentTimeoutSeconds(command, 20)).toEqual( + agent("--agent", "main", "--timeout", "20"), + ); }); it("reads an equals-form --timeout value (#8723)", () => { - expect(requestedAgentTimeoutSeconds(agent("--timeout=45", "-m", "hi"))).toBe(45); + const command = agent("--timeout=45", "-m", "hi"); + expect(requestedAgentTimeoutSeconds(command)).toBe(45); + expect(replaceRequestedAgentTimeoutSeconds(command, 20)).toEqual( + agent("--timeout=20", "-m", "hi"), + ); }); it("reads a timeout after documented boolean and equals-form options (#8723)", () => { @@ -234,10 +297,6 @@ describe("agentDispatchDeadlineSeconds", () => { expect(agentDispatchDeadlineSeconds(["openclaw", "agent", "-m", "hi"])).toBeUndefined(); }); - it("holds the deadline buffer above the longest aborted-run finish measured (#8723)", () => { - expect(AGENT_DISPATCH_DEADLINE_BUFFER_SECONDS).toBeGreaterThan(20); - }); - it("stays unbounded when the buffered deadline leaves the safe-integer range (#8723)", () => { const ceiling = String(Number.MAX_SAFE_INTEGER); expect(requestedAgentTimeoutSeconds(["openclaw", "agent", "--timeout", ceiling])).toBe( diff --git a/src/lib/actions/sandbox/agent/passthrough-dispatch.ts b/src/lib/actions/sandbox/agent/passthrough-dispatch.ts index 134e58a4de0..346141bd34a 100644 --- a/src/lib/actions/sandbox/agent/passthrough-dispatch.ts +++ b/src/lib/actions/sandbox/agent/passthrough-dispatch.ts @@ -320,23 +320,25 @@ export const OPENCLAW_AGENT_BOOLEAN_FLAGS = new Set(["--deliver"]); */ export const AGENT_DISPATCH_DEADLINE_BUFFER_SECONDS = 30; -/** - * The `--timeout` an `openclaw agent` argv requests, or null when the argv - * requests none. - * - * Mirrors the documented flag grammar only far enough to read one value. - * Anything unrecognized, malformed, or past a `--` terminator returns null so - * the host keeps the wait unbounded rather than shortening a turn without - * evidence. `--timeout 0` disables the deadline upstream and returns null here - * for the same reason. - */ -export function requestedAgentTimeoutSeconds(argv: readonly string[]): number | null { +type RequestedAgentTimeout = { + seconds: number; + argumentIndex: number; + inline: boolean; +}; + +function findRequestedAgentTimeout(argv: readonly string[]): RequestedAgentTimeout | null { if (argv[0] !== "openclaw" || argv[1] !== "agent") return null; for (let index = 2; index < argv.length; index += 1) { const arg = argv[index] as string; if (arg === "--") return null; - if (arg === "--timeout") return parseDeadlineSeconds(argv[index + 1]); - if (arg.startsWith("--timeout=")) return parseDeadlineSeconds(arg.slice("--timeout=".length)); + if (arg === "--timeout") { + const seconds = parseDeadlineSeconds(argv[index + 1]); + return seconds === null ? null : { seconds, argumentIndex: index, inline: false }; + } + if (arg.startsWith("--timeout=")) { + const seconds = parseDeadlineSeconds(arg.slice("--timeout=".length)); + return seconds === null ? null : { seconds, argumentIndex: index, inline: true }; + } if (OPENCLAW_AGENT_VALUE_FLAGS.has(arg)) { index += 1; continue; @@ -349,11 +351,7 @@ export function requestedAgentTimeoutSeconds(argv: readonly string[]): number | ) { continue; } - if ( - arg === "--json" || - arg.startsWith("--json=") || - OPENCLAW_AGENT_BOOLEAN_FLAGS.has(arg) - ) { + if (arg === "--json" || arg.startsWith("--json=") || OPENCLAW_AGENT_BOOLEAN_FLAGS.has(arg)) { continue; } return null; @@ -361,6 +359,37 @@ export function requestedAgentTimeoutSeconds(argv: readonly string[]): number | return null; } +/** + * The `--timeout` an `openclaw agent` argv requests, or null when the argv + * requests none. + * + * Mirrors the documented flag grammar only far enough to read one value. + * Anything unrecognized, malformed, or past a `--` terminator returns null so + * the host keeps the wait unbounded rather than shortening a turn without + * evidence. `--timeout 0` disables the deadline upstream and returns null here + * for the same reason. + */ +export function requestedAgentTimeoutSeconds(argv: readonly string[]): number | null { + return findRequestedAgentTimeout(argv)?.seconds ?? null; +} + +/** Replace a valid agent timeout while preserving its separated or equals form. */ +export function replaceRequestedAgentTimeoutSeconds( + argv: readonly string[], + timeoutSeconds: number, +): readonly string[] { + const requested = findRequestedAgentTimeout(argv); + if (requested === null) return argv; + const replacement = String(Math.max(1, Math.floor(timeoutSeconds))); + const command = [...argv]; + if (requested.inline) { + command[requested.argumentIndex] = `--timeout=${replacement}`; + } else { + command[requested.argumentIndex + 1] = replacement; + } + return command; +} + function parseDeadlineSeconds(raw: string | undefined): number | null { if (raw === undefined || !/^\d+$/.test(raw)) return null; const seconds = Number(raw); diff --git a/src/lib/actions/sandbox/agent/passthrough-ollama-recovery.test.ts b/src/lib/actions/sandbox/agent/passthrough-ollama-recovery.test.ts index 1d784ccf685..a5d9d2d72a9 100644 --- a/src/lib/actions/sandbox/agent/passthrough-ollama-recovery.test.ts +++ b/src/lib/actions/sandbox/agent/passthrough-ollama-recovery.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it, vi } from "vitest"; import { type AgentPassthroughDeps, runAgentPassthrough } from "./passthrough"; +import { requestedAgentTimeoutSeconds } from "./passthrough-dispatch"; import { runOllamaRestartRecovery } from "./passthrough-ollama-recovery"; function makeProcMock() { @@ -17,7 +18,7 @@ describe("runOllamaRestartRecovery", () => { it.each([ ["auth proxy", "http://host.openshell.internal:11435/v1"], ["WSL direct bridge", "http://host.openshell.internal:11434/v1"], - ])("forwards the persisted %s route to recovery", (_name, endpointUrl) => { + ])("forwards the persisted %s route to recovery", async (_name, endpointUrl) => { const recoverOllama = vi.fn(() => ({ kind: "skipped" as const, reason: "already-loaded" as const, @@ -29,35 +30,40 @@ describe("runOllamaRestartRecovery", () => { endpointUrl, }; - runOllamaRestartRecovery(route, proc, recoverOllama); + await runOllamaRestartRecovery(route, proc, {}, recoverOllama); - expect(recoverOllama).toHaveBeenCalledWith(route); + expect(recoverOllama).toHaveBeenCalledWith(route, {}); expect(writes.join("")).toContain("Ollama model 'qwen3.6:35b' is already loaded"); }); - it("reports a successful warm-up", () => { + it("reports a successful warm-up", async () => { const { writes, proc } = makeProcMock(); - runOllamaRestartRecovery({ provider: "ollama-local", model: "qwen3.6:35b" }, proc, () => ({ - kind: "warmed", - ok: true, - timedOut: false, - })); + await runOllamaRestartRecovery( + { provider: "ollama-local", model: "qwen3.6:35b" }, + proc, + {}, + () => ({ kind: "warmed", ok: true }), + ); expect(writes.join("")).toContain("Ollama model 'qwen3.6:35b' is loaded and ready"); }); - it("reports a timeout before continuing to OpenClaw", () => { + it("reports a timeout before continuing to OpenClaw", async () => { const { writes, proc } = makeProcMock(); - runOllamaRestartRecovery({ provider: "ollama-local", model: "qwen3.6:35b" }, proc, () => ({ - kind: "warmed", - ok: false, - timedOut: true, - reason: "timeout", - endpoint: "http://host.docker.internal:11434", - detail: "curl timed out after 300 seconds", - })); + await runOllamaRestartRecovery( + { provider: "ollama-local", model: "qwen3.6:35b" }, + proc, + {}, + () => ({ + kind: "warmed", + ok: false, + reason: "timeout", + endpoint: "http://host.docker.internal:11434", + detail: "curl timed out after 300 seconds", + }), + ); const stderr = writes.join(""); expect(stderr).toContain("Checking whether the Ollama model is loaded"); @@ -66,6 +72,10 @@ describe("runOllamaRestartRecovery", () => { expect(stderr).toContain("at http://host.docker.internal:11434"); expect(stderr).toContain("OpenClaw dispatch will continue"); expect(stderr).toContain("confirm that it serves 'qwen3.6:35b'"); + expect(stderr).toContain( + "NemoClaw will check the model before the next `nemoclaw agent` command and warm it if necessary", + ); + expect(stderr).not.toContain("rerun this command"); }); it.each([ @@ -73,51 +83,108 @@ describe("runOllamaRestartRecovery", () => { ["ollama-error", "Ollama returned an error"], ["invalid-response", "Ollama returned an invalid response"], ["spawn-failed", "the warm-up process could not start"], - ] as const)("reports a %s warm-up failure", (reason, message) => { + ] as const)("reports a %s warm-up failure", async (reason, message) => { const { writes, proc } = makeProcMock(); - runOllamaRestartRecovery({ provider: "ollama-local", model: "qwen3.6:35b" }, proc, () => ({ - kind: "warmed", - ok: false, - timedOut: false, - reason, - endpoint: "http://host.docker.internal:11434", - detail: "bounded failure detail", - })); + await runOllamaRestartRecovery( + { provider: "ollama-local", model: "qwen3.6:35b" }, + proc, + {}, + () => ({ + kind: "warmed", + ok: false, + reason, + endpoint: "http://host.docker.internal:11434", + detail: "bounded failure detail", + }), + ); const stderr = writes.join(""); expect(stderr).toContain(message); expect(stderr).toContain("http://host.docker.internal:11434"); expect(stderr).toContain("OpenClaw dispatch will continue"); expect(stderr).toContain("confirm that it serves 'qwen3.6:35b'"); + expect(stderr).toContain( + "NemoClaw will check the model before the next `nemoclaw agent` command and warm it if necessary", + ); + expect(stderr).not.toContain("rerun this command"); }); it.each([ ["already-loaded", "Ollama model 'qwen3.6:35b' is already loaded"], - ["unreachable", "Ollama was unreachable during the model check"], ["missing-model", "No Ollama model is recorded for this sandbox"], ["not-ollama", "Checking whether the Ollama model is loaded"], - ] as const)("handles the %s skip reason", (reason, message) => { + ] as const)("reports the diagnostic for the %s recovery result", async (reason, message) => { const { writes, proc } = makeProcMock(); - runOllamaRestartRecovery({ provider: "ollama-local", model: "qwen3.6:35b" }, proc, () => ({ - kind: "skipped", - reason, - })); + await runOllamaRestartRecovery( + { provider: "ollama-local", model: "qwen3.6:35b" }, + proc, + {}, + () => ({ kind: "skipped", reason }), + ); expect(writes.join("")).toContain(message); }); - it("names the endpoint and its reported models when the model is absent (#9455)", () => { + it("reports the endpoint, model, and recovery action when Ollama is unreachable", async () => { const { writes, proc } = makeProcMock(); - runOllamaRestartRecovery( + await runOllamaRestartRecovery( + { provider: "ollama-local", model: "qwen3.6:35b" }, + proc, + {}, + () => ({ + kind: "skipped", + reason: "unreachable", + endpoint: "http://host.docker.internal:11434", + }), + ); + + const stderr = writes.join(""); + expect(stderr).toContain("http://host.docker.internal:11434"); + expect(stderr).toContain("qwen3.6:35b"); + expect(stderr).toContain("Restore Ollama access"); + expect(stderr).toContain("confirm that it serves"); + expect(stderr).toContain( + "NemoClaw will check the model before the next `nemoclaw agent` command and warm it if necessary", + ); + expect(stderr).not.toContain("rerun this command"); + }); + + it("reports a warm-up skipped after the command timeout budget is exhausted", async () => { + const { writes, proc } = makeProcMock(); + + await runOllamaRestartRecovery( + { provider: "ollama-local", model: "qwen3.6:35b" }, + proc, + { timeoutSeconds: 1 }, + () => ({ + kind: "skipped", + reason: "deadline-exhausted", + endpoint: "http://host.docker.internal:11434", + }), + ); + + const stderr = writes.join(""); + expect(stderr).toContain("warm-up for 'qwen3.6:35b'"); + expect(stderr).toContain("http://host.docker.internal:11434"); + expect(stderr).toContain("was skipped"); + expect(stderr).toContain("timeout left no recovery budget"); + expect(stderr).toContain("continuing to OpenClaw dispatch"); + }); + + it("names the endpoint and its reported models when the model is absent (#9455)", async () => { + const { writes, proc } = makeProcMock(); + + await runOllamaRestartRecovery( { provider: "ollama-local", model: "gemma4:26b", endpointUrl: "http://host.openshell.internal:11434/v1", }, proc, + {}, () => ({ kind: "skipped", reason: "model-absent", @@ -136,16 +203,46 @@ describe("runOllamaRestartRecovery", () => { expect(stderr).not.toContain("Ollama was unreachable during the restart check"); }); - it("continues OpenClaw dispatch when Ollama recovery throws", () => { + it("redacts and bounds recovery exceptions", async () => { const { writes, proc } = makeProcMock(); + const exposedToken = "sk-proj-NOT-A-REAL-SECRET-1234567890"; + const directionalControls = + "\u061c\u200e\u200f\u2028\u2029\u202a\u202b\u202c\u202d\u202e\u2066\u2067\u2068\u2069"; - expect(() => - runOllamaRestartRecovery({ provider: "ollama-local", model: "qwen3.6:35b" }, proc, () => { - throw new Error("unexpected"); - }), - ).not.toThrow(); - expect(writes.join("")).toContain( - "Ollama restart recovery failed unexpectedly; continuing to OpenClaw dispatch", + await expect( + runOllamaRestartRecovery( + { + provider: "ollama-local", + model: `qwen3.6:35b${directionalControls}\u001b[2J\u0007`, + endpointUrl: `http://host.openshell.internal:11434/v1${directionalControls}\u001b]52;c;clipboard\u0007`, + }, + proc, + {}, + () => { + throw new Error( + `synthetic\u001b[2J Docker transport failure\u0007 OPENAI_API_KEY=${exposedToken} ${"x".repeat(400)} END-OF-DETAIL`, + ); + }, + ), + ).resolves.toBeNull(); + const stderr = writes.join(""); + expect(stderr).toContain("Ollama restart recovery for 'qwen3.6:35b"); + expect(stderr).toContain("at the recorded endpoint http://host.openshell.internal:11434/v1"); + expect(stderr).toContain("synthetic"); + expect(stderr).toContain("Docker transport failure"); + expect(stderr).toContain("OPENAI_API_KEY="); + expect(stderr).toContain("OpenClaw dispatch will continue"); + expect(stderr).toContain("Restore Ollama access to that endpoint"); + expect(stderr).toContain( + "NemoClaw will check the model before the next `nemoclaw agent` command and warm it if necessary", + ); + expect(stderr).not.toContain("rerun this command"); + expect(stderr).not.toContain(exposedToken); + expect(stderr).not.toContain("END-OF-DETAIL"); + expect(stderr).not.toContain("\u001b"); + expect(stderr).not.toContain("\u0007"); + expect(stderr.replace(/\n/gu, "")).not.toMatch( + /[\u0000-\u001f\u007f-\u009f\u061c\u200e\u200f\u2028-\u202e\u2066-\u2069]/u, ); }); }); @@ -202,21 +299,83 @@ describe("agent passthrough Ollama recovery ordering", () => { ), ).rejects.toThrow("__exit:0"); - expect(runRecovery).toHaveBeenCalledWith(expect.objectContaining(route), deps.process); + expect(runRecovery).toHaveBeenCalledWith(expect.objectContaining(route), deps.process, {}); expect(events).toEqual(["recovery", "dispatch"]); }); - it("checks a WSL direct route before non-JSON dispatch", async () => { + it.each([ + ["partial recovery", 5_000, 25], + ["an exhausted recovery budget", 30_000, 1], + ])( + "reduces a 30-second timeout after %s", + async (_name, elapsedMilliseconds, expectedTimeout) => { + const events: string[] = []; + const dispatchedTimeouts: number[] = []; + const route = { + provider: "ollama-local", + model: "qwen3.6:35b", + endpointUrl: "http://host.openshell.internal:11434/v1", + }; + const deps = makePassthroughDeps(route, events); + const now = vi + .fn() + .mockReturnValueOnce(0) + .mockReturnValueOnce(0) + .mockReturnValueOnce(elapsedMilliseconds); + const runRecovery = vi.fn(() => { + events.push("recovery"); + }); + const execNonJson = vi.fn((( + _sandboxName: string, + dispatchedCommand: readonly string[], + ): never => { + dispatchedTimeouts.push(requestedAgentTimeoutSeconds(dispatchedCommand) ?? -1); + events.push("dispatch"); + throw new Error("__exit:0"); + }) as NonNullable); + + await expect( + runAgentPassthrough( + "alpha", + { extraArgs: ["--agent", "main", "--timeout", "30", "-m", "ping"] }, + { ...deps, execNonJson, now, runOllamaRestartRecovery: runRecovery }, + ), + ).rejects.toThrow("__exit:0"); + + expect(runRecovery).toHaveBeenCalledWith(expect.objectContaining(route), deps.process, { + timeoutSeconds: 29, + }); + expect(events).toEqual(["recovery", "dispatch"]); + expect(dispatchedTimeouts).toEqual([expectedTimeout]); + }, + ); + + it("dispatches after reporting an Ollama recovery exception", async () => { const events: string[] = []; + const diagnostics: string[] = []; const route = { provider: "ollama-local", model: "qwen3.6:35b", endpointUrl: "http://host.openshell.internal:11434/v1", }; const deps = makePassthroughDeps(route, events); - const runRecovery = vi.fn(() => { - events.push("recovery"); - }); + deps.process = { + ...deps.process!, + stderr: { + write: (value: string) => { + diagnostics.push(value); + value.includes("failed unexpectedly") && events.push("diagnostic"); + return true; + }, + }, + }; + const runRecovery = ( + registeredRoute: Parameters[0], + proc: Parameters[1], + ) => + runOllamaRestartRecovery(registeredRoute, proc, {}, () => { + throw new Error("synthetic recovery failure"); + }); await expect( runAgentPassthrough( @@ -226,8 +385,32 @@ describe("agent passthrough Ollama recovery ordering", () => { ), ).rejects.toThrow("__exit:0"); - expect(runRecovery).toHaveBeenCalledWith(expect.objectContaining(route), deps.process); - expect(events).toEqual(["recovery", "dispatch"]); + expect(events).toEqual(["diagnostic", "dispatch"]); + expect(diagnostics.join("")).toContain("failed unexpectedly"); + }); + + it("does not dispatch after Ollama recovery receives SIGTERM", async () => { + const events: string[] = []; + const route = { + provider: "ollama-local", + model: "qwen3.6:35b", + endpointUrl: "http://host.openshell.internal:11434/v1", + }; + const deps = makePassthroughDeps(route, events); + const runRecovery = vi.fn(async () => { + events.push("recovery-cancelled"); + return "SIGTERM" as const; + }); + + await expect( + runAgentPassthrough( + "alpha", + { extraArgs: ["--agent", "main", "-m", "ping"] }, + { ...deps, runOllamaRestartRecovery: runRecovery }, + ), + ).rejects.toThrow("__exit:143"); + + expect(events).toEqual(["recovery-cancelled"]); }); it("does not run Ollama recovery for a non-Ollama route", async () => { diff --git a/src/lib/actions/sandbox/agent/passthrough-ollama-recovery.ts b/src/lib/actions/sandbox/agent/passthrough-ollama-recovery.ts index 4687c5ca201..1e8b1a622ff 100644 --- a/src/lib/actions/sandbox/agent/passthrough-ollama-recovery.ts +++ b/src/lib/actions/sandbox/agent/passthrough-ollama-recovery.ts @@ -2,9 +2,11 @@ // SPDX-License-Identifier: Apache-2.0 import { + boundedOllamaRestartRecoveryDetail, maybeWarmOllamaAfterDaemonRestart, OLLAMA_LOCAL_PROVIDER, type OllamaRestartRecoveryFailureReason, + type OllamaRestartRecoveryOptions, type OllamaRestartRecoveryResult, type OllamaRestartRecoveryRoute, } from "./ollama-restart-recovery"; @@ -13,12 +15,24 @@ export { OLLAMA_LOCAL_PROVIDER }; export type OllamaRestartRecoveryFn = ( route: OllamaRestartRecoveryRoute, -) => OllamaRestartRecoveryResult; + options?: OllamaRestartRecoveryOptions, +) => OllamaRestartRecoveryResult | Promise; export interface OllamaRestartRecoveryProcess { stderr: { write(s: string): unknown }; } +function boundedRecoveryEndpoint(value: unknown, fallback: string): string { + const original = String(value ?? "").trim(); + const endpoint = boundedOllamaRestartRecoveryDetail(value, fallback); + return endpoint.endsWith("/") && !original.endsWith("/") ? endpoint.slice(0, -1) : endpoint; +} + +function recordedEndpointLabel(route: OllamaRestartRecoveryRoute): string { + const endpoint = boundedRecoveryEndpoint(route.endpointUrl, ""); + return endpoint ? `at the recorded endpoint ${endpoint}` : "at the saved local Ollama endpoint"; +} + function describeWarmFailure(reason: OllamaRestartRecoveryFailureReason): string { switch (reason) { case "timeout": @@ -36,28 +50,32 @@ function describeWarmFailure(reason: OllamaRestartRecoveryFailureReason): string function reportRecovery( route: OllamaRestartRecoveryRoute, - result: OllamaRestartRecoveryResult, + result: Exclude, proc: OllamaRestartRecoveryProcess, ): void { - const model = String(route.model ?? "").trim() || "the registered model"; + const model = boundedOllamaRestartRecoveryDetail(route.model, "the registered model"); if (result.kind === "warmed") { if (result.ok) { proc.stderr.write(` Ollama model '${model}' is loaded and ready.\n`); return; } + const endpoint = boundedRecoveryEndpoint(result.endpoint, "the saved local Ollama endpoint"); + const detail = boundedOllamaRestartRecoveryDetail(result.detail, "unknown warm-up error"); proc.stderr.write( - ` Ollama warm-up for '${model}' at ${result.endpoint} ${describeWarmFailure(result.reason)} ` + - `(${result.detail}). OpenClaw dispatch will continue. To retry the warm-up, restore ` + - `Ollama access to ${result.endpoint} and confirm that it serves '${model}', then rerun ` + - `this command.\n`, + ` Ollama warm-up for '${model}' at ${endpoint} ${describeWarmFailure(result.reason)} ` + + `(${detail}). OpenClaw dispatch will continue. Restore Ollama access to ${endpoint} ` + + `and confirm that it serves '${model}'. NemoClaw will check the model before the next ` + + `\`nemoclaw agent\` command and warm it if necessary.\n`, ); return; } if (result.reason === "model-absent") { + const endpoint = boundedRecoveryEndpoint(result.endpoint, "the saved local Ollama endpoint"); + const inventoryLabel = boundedOllamaRestartRecoveryDetail(result.inventoryLabel, "none"); proc.stderr.write( - ` Ollama at ${result.endpoint} reports '${model}' as unavailable ` + - `(reported models: ${result.inventoryLabel}); continuing to OpenClaw dispatch.\n`, + ` Ollama at ${endpoint} reports '${model}' as unavailable ` + + `(reported models: ${inventoryLabel}); continuing to OpenClaw dispatch.\n`, ); proc.stderr.write( ` Either the daemon answering that endpoint changed, or the model was removed from ` + @@ -67,16 +85,30 @@ function reportRecovery( return; } + if (result.reason === "deadline-exhausted") { + const endpoint = boundedRecoveryEndpoint(result.endpoint, "the saved local Ollama endpoint"); + proc.stderr.write( + ` Ollama warm-up for '${model}' at ${endpoint} was skipped because the agent command ` + + `timeout left no recovery budget; continuing to OpenClaw dispatch.\n`, + ); + return; + } + const reason = result.reason; switch (reason) { case "already-loaded": proc.stderr.write(` Ollama model '${model}' is already loaded.\n`); break; - case "unreachable": + case "unreachable": { + const endpoint = boundedRecoveryEndpoint(result.endpoint, "the saved local Ollama endpoint"); proc.stderr.write( - " Ollama was unreachable during the model check; continuing to OpenClaw dispatch.\n", + ` Ollama at ${endpoint} was unreachable while checking '${model}'; continuing to ` + + `OpenClaw dispatch. Restore Ollama access to ${endpoint}, confirm that it serves ` + + `'${model}'. NemoClaw will check the model before the next ` + + `\`nemoclaw agent\` command and warm it if necessary.\n`, ); break; + } case "missing-model": proc.stderr.write( " No Ollama model is recorded for this sandbox; continuing to OpenClaw dispatch.\n", @@ -91,18 +123,28 @@ function reportRecovery( } } -/** Run best-effort Ollama recovery without blocking the canonical agent error path. */ -export function runOllamaRestartRecovery( +/** Continue after best-effort recovery failures, but preserve an operator cancellation. */ +export async function runOllamaRestartRecovery( route: OllamaRestartRecoveryRoute, proc: OllamaRestartRecoveryProcess, + options: OllamaRestartRecoveryOptions = {}, recoverOllama: OllamaRestartRecoveryFn = maybeWarmOllamaAfterDaemonRestart, -): void { +): Promise { proc.stderr.write(" Checking whether the Ollama model is loaded...\n"); try { - reportRecovery(route, recoverOllama(route), proc); - } catch { + const result = await recoverOllama(route, options); + if (result.kind === "cancelled") return result.signal; + reportRecovery(route, result, proc); + } catch (error) { + const model = boundedOllamaRestartRecoveryDetail(route.model, "the registered model"); + const endpoint = recordedEndpointLabel(route); + const detail = boundedOllamaRestartRecoveryDetail(error, "unknown recovery error"); proc.stderr.write( - " Ollama restart recovery failed unexpectedly; continuing to OpenClaw dispatch.\n", + ` Ollama restart recovery for '${model}' ${endpoint} failed unexpectedly: ${detail}. ` + + `OpenClaw dispatch will continue. Restore Ollama access to that endpoint, confirm it ` + + `serves '${model}'. NemoClaw will check the model before the next ` + + `\`nemoclaw agent\` command and warm it if necessary.\n`, ); } + return null; } diff --git a/src/lib/actions/sandbox/agent/passthrough.ts b/src/lib/actions/sandbox/agent/passthrough.ts index 563644cc391..37d64c5144a 100644 --- a/src/lib/actions/sandbox/agent/passthrough.ts +++ b/src/lib/actions/sandbox/agent/passthrough.ts @@ -106,6 +106,7 @@ // - Drop Ollama pre-dispatch recovery when supported daemon restarts preserve // loaded runners or NemoClaw manages and warms the daemon lifecycle. +import { performance } from "node:perf_hooks"; import { type AgentDefinition, isTerminalAgent, listAgents, loadAgent } from "../../../agent/defs"; import { CLI_NAME } from "../../../cli/branding"; import { isStdinTty } from "../../../core/stdin"; @@ -126,6 +127,8 @@ import { isTimedOutAgentDispatch, OPENCLAW_AGENT_BOOLEAN_FLAGS, OPENCLAW_AGENT_VALUE_FLAGS, + replaceRequestedAgentTimeoutSeconds, + requestedAgentTimeoutSeconds, runAgentDispatch, SILENT_AGENT_DISPATCH_EXIT_CODE, TIMED_OUT_AGENT_TURN_EXIT_CODE, @@ -232,7 +235,10 @@ export interface AgentPassthroughDeps { exec?: typeof execSandbox; execJson?: typeof runAgentJsonPassthrough; execNonJson?: typeof runAgentNonJsonPassthrough; - runOllamaRestartRecovery?: typeof runOllamaRestartRecovery; + runOllamaRestartRecovery?: ( + ...args: Parameters + ) => ReturnType | NodeJS.Signals | null | void; + now?: () => number; process?: { exit(code: number): never; stdout?: { write(s: string): unknown }; @@ -240,6 +246,41 @@ export interface AgentPassthroughDeps { }; } +const MINIMUM_AGENT_DISPATCH_BUDGET_SECONDS = 1; + +function startAgentCommandDeadline(command: readonly string[], now: () => number): number | null { + const timeoutSeconds = requestedAgentTimeoutSeconds(command); + const timeoutMilliseconds = timeoutSeconds === null ? null : timeoutSeconds * 1000; + return timeoutMilliseconds === null || !Number.isSafeInteger(timeoutMilliseconds) + ? null + : now() + timeoutMilliseconds; +} + +function remainingAgentCommandSeconds(deadline: number | null, now: () => number): number | null { + return deadline === null ? null : Math.max(0, (deadline - now()) / 1000); +} + +function recoveryBudgetSeconds(deadline: number | null, now: () => number): number | null { + const remaining = remainingAgentCommandSeconds(deadline, now); + // Keep one second for the actual turn so best-effort recovery cannot consume + // the entire user-requested deadline before OpenClaw starts. + return remaining === null ? null : Math.max(0, remaining - MINIMUM_AGENT_DISPATCH_BUDGET_SECONDS); +} + +function commandWithRemainingDeadline( + command: readonly string[], + deadline: number | null, + now: () => number, +): readonly string[] { + const remaining = remainingAgentCommandSeconds(deadline, now); + return remaining === null + ? command + : replaceRequestedAgentTimeoutSeconds( + command, + Math.max(MINIMUM_AGENT_DISPATCH_BUDGET_SECONDS, Math.ceil(remaining)), + ); +} + type RegistryReadResult = | { kind: "missing" } | { @@ -540,15 +581,27 @@ export async function runAgentPassthrough( if (isOpenClawPassthroughCommand(command) && !hasTargetSelector(extraArgs)) { rejectNoTargetSelector(proc); } + let dispatchCommand: readonly string[] = command; if (isOpenClawPassthroughCommand(command)) { + const now = deps.now ?? (() => performance.now()); + const commandDeadline = startAgentCommandDeadline(command, now); if (lookup.kind === "agent" && lookup.provider === OLLAMA_LOCAL_PROVIDER) { const recoverOllama = deps.runOllamaRestartRecovery ?? runOllamaRestartRecovery; - recoverOllama(lookup, proc); + const timeoutSeconds = recoveryBudgetSeconds(commandDeadline, now); + const recoverySignal = await recoverOllama( + lookup, + proc, + timeoutSeconds === null ? {} : { timeoutSeconds }, + ); + if (recoverySignal) { + return proc.exit(computeExitCode({ status: null, signal: recoverySignal }).code); + } } + dispatchCommand = commandWithRemainingDeadline(command, commandDeadline, now); } if (isOpenClawPassthroughCommand(command) && requestsOpenClawJsonOutput(extraArgs)) { const execJson = deps.execJson ?? runAgentJsonPassthrough; - await execJson(sandboxName, command, { + await execJson(sandboxName, dispatchCommand, { exit: proc.exit.bind(proc), stdout: proc.stdout ?? process.stdout, stderr: proc.stderr, @@ -557,7 +610,7 @@ export async function runAgentPassthrough( } if (isOpenClawPassthroughCommand(command)) { const execNonJson = deps.execNonJson ?? runAgentNonJsonPassthrough; - await execNonJson(sandboxName, command, proc); + await execNonJson(sandboxName, dispatchCommand, proc); return; } const exec = deps.exec ?? execSandbox; diff --git a/src/lib/actions/sandbox/doctor-flow.test.ts b/src/lib/actions/sandbox/doctor-flow.test.ts index 2b6e73897cb..9c427c022a9 100644 --- a/src/lib/actions/sandbox/doctor-flow.test.ts +++ b/src/lib/actions/sandbox/doctor-flow.test.ts @@ -51,6 +51,7 @@ function createDoctorHarness( getSandboxSpy: MockInstance; getNamedGatewayLifecycleStateSpy: MockInstance; healthProbeSpy: MockInstance; + ollamaInventoryProbeSpy: MockInstance; inspectMutableConfigPermsSpy: MockInstance; loadAgentSpy: MockInstance; probeSandboxInferenceGatewayHealthSpy: MockInstance; @@ -192,6 +193,10 @@ function createDoctorHarness( endpoint: "http://127.0.0.1:11434/v1/chat/completions", detail: "healthy", }); + const ollamaInventoryProbeSpy = vi.spyOn(health, "probeOllamaHostInventory").mockReturnValue({ + endpoint: "http://127.0.0.1:11434/api/tags", + inventory: ["m"], + }); const probeSandboxInferenceGatewayHealthSpy = vi .spyOn(inferenceRouteHealth, "probeSandboxInferenceGatewayHealth") .mockResolvedValue({ @@ -270,6 +275,7 @@ function createDoctorHarness( getSandboxSpy, getNamedGatewayLifecycleStateSpy, healthProbeSpy, + ollamaInventoryProbeSpy, inspectMutableConfigPermsSpy, loadAgentSpy, probeSandboxInferenceGatewayHealthSpy, @@ -488,6 +494,7 @@ describe("runSandboxDoctor flow", () => { ]), ); expect(exitSpy).not.toHaveBeenCalled(); + expect(harness.ollamaInventoryProbeSpy).toHaveBeenCalledOnce(); expect(harness.logSpy).not.toHaveBeenCalled(); }, ); diff --git a/src/lib/actions/sandbox/doctor-system-checks.test.ts b/src/lib/actions/sandbox/doctor-system-checks.test.ts index 704d73adf82..d28b74acb8e 100644 --- a/src/lib/actions/sandbox/doctor-system-checks.test.ts +++ b/src/lib/actions/sandbox/doctor-system-checks.test.ts @@ -1,15 +1,47 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { chmodSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { createRequire } from "node:module"; -import { afterEach, describe, expect, it, vi } from "vitest"; +import { tmpdir } from "node:os"; +import { delimiter, join } from "node:path"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; const requireDist = createRequire(import.meta.url); const modulePath = "./doctor-system-checks.js"; describe("doctor system checks", () => { + const originalPath = process.env.PATH; + let fakeDockerDir: string; + + beforeAll(() => { + fakeDockerDir = mkdtempSync(join(tmpdir(), "nemoclaw-doctor-fake-docker-desktop-")); + const fakeDockerPath = join(fakeDockerDir, "docker"); + writeFileSync( + fakeDockerPath, + [ + "#!/bin/sh", + 'if [ "$1" = "info" ]; then', + " printf '%s\\n' 'Operating System: Docker Desktop'", + " exit 0", + "fi", + "exit 1", + "", + ].join("\n"), + ); + chmodSync(fakeDockerPath, 0o755); + process.env.PATH = `${fakeDockerDir}${delimiter}${originalPath ?? ""}`; + }); + + afterAll(() => { + Reflect.deleteProperty(process.env, "PATH"); + Object.assign(process.env, originalPath === undefined ? {} : { PATH: originalPath }); + rmSync(fakeDockerDir, { recursive: true, force: true }); + }); + afterEach(() => { vi.restoreAllMocks(); + vi.unstubAllEnvs(); delete requireDist.cache[requireDist.resolve(modulePath)]; }); @@ -31,4 +63,98 @@ describe("doctor system checks", () => { hint: "expected host port 19080 for this sandbox gateway", }); }); + + it("probes a loopback Ollama route with direct curl", () => { + const runCaptureImpl = vi.fn((_command: readonly string[]) => + JSON.stringify({ models: [{ name: "qwen3.6:35b" }] }), + ); + const prepareDockerEnvironment = vi.fn(); + const { ollamaDoctorCheck } = requireDist(modulePath); + + expect( + ollamaDoctorCheck("ollama-local", { + getOllamaHost: () => "127.0.0.1", + runCaptureImpl, + prepareDockerEnvironment, + }), + ).toMatchObject({ + status: "ok", + detail: "reachable at http://127.0.0.1:11434/api/tags (1 model(s))", + }); + expect(runCaptureImpl.mock.calls[0]?.[0]?.[0]).toBe("curl"); + expect(runCaptureImpl.mock.calls[0]?.[0]).toContain("http://127.0.0.1:11434/api/tags"); + expect(prepareDockerEnvironment).not.toHaveBeenCalled(); + }); + + it("probes a persisted Windows Ollama route through credential-free Docker", () => { + vi.spyOn(process, "platform", "get").mockReturnValue("linux"); + vi.stubEnv("DOCKER_CONTEXT", "default"); + vi.stubEnv("DOCKER_HOST", ""); + vi.stubEnv("WSL_DISTRO_NAME", "Ubuntu"); + const cleanup = vi.fn(() => ({ ok: true as const })); + const prepareDockerEnvironment = vi.fn(() => ({ + env: { DOCKER_CONFIG: "/tmp/credential-free-docker" }, + isolatedCredentialConfig: true, + cleanup, + })); + const runCaptureImpl = vi.fn( + (command: readonly string[], options?: { env?: NodeJS.ProcessEnv }) => + command.join(" ").includes("Get-NetTCPConnection") + ? "127.0.0.1" + : command[0] === "docker" && + options?.env?.DOCKER_CONFIG === "/tmp/credential-free-docker" + ? command.includes("Host: rebinding.invalid") + ? "403" + : command.some( + (argument) => argument === "http://host.docker.internal:11434/api/tags", + ) + ? JSON.stringify({ models: [] }) + : "" + : "", + ); + const { ollamaDoctorCheck } = requireDist(modulePath); + + expect( + ollamaDoctorCheck("ollama-local", { + getOllamaHost: () => "host.docker.internal", + runCaptureImpl, + prepareDockerEnvironment, + }), + ).toMatchObject({ + status: "ok", + detail: "reachable at http://host.docker.internal:11434/api/tags (0 model(s))", + }); + expect(runCaptureImpl.mock.calls).toHaveLength(4); + expect(runCaptureImpl.mock.calls.slice(1).every(([command]) => command[0] === "docker")).toBe( + true, + ); + expect(runCaptureImpl.mock.calls[1]?.[0]).toEqual( + expect.arrayContaining(["http://host.docker.internal:11434/api/tags"]), + ); + expect(runCaptureImpl.mock.calls[2]?.[0]).toEqual( + expect.arrayContaining(["Host: rebinding.invalid"]), + ); + expect(prepareDockerEnvironment).toHaveBeenCalledTimes(3); + expect(cleanup).toHaveBeenCalledTimes(3); + }); + + it.each([ + ["malformed JSON", "not-json"], + ["JSON without a models array", JSON.stringify({ status: "ok" })], + ])("rejects %s from the selected Ollama endpoint", (_name, output) => { + const { ollamaDoctorCheck } = requireDist(modulePath); + + expect( + ollamaDoctorCheck("ollama-local", { + getOllamaHost: () => "127.0.0.1", + runCaptureImpl: () => output, + }), + ).toEqual({ + group: "Local services", + label: "Ollama", + status: "fail", + detail: "not reachable or invalid response at http://127.0.0.1:11434/api/tags", + hint: "start Ollama or change the sandbox inference provider", + }); + }); }); diff --git a/src/lib/actions/sandbox/doctor-system-checks.ts b/src/lib/actions/sandbox/doctor-system-checks.ts index fa8eeb59d90..ed434c4983d 100644 --- a/src/lib/actions/sandbox/doctor-system-checks.ts +++ b/src/lib/actions/sandbox/doctor-system-checks.ts @@ -2,11 +2,14 @@ // SPDX-License-Identifier: Apache-2.0 import path from "node:path"; -import { buildValidatedCurlCommandArgs } from "../../adapters/http/curl-args"; import { stripAnsi } from "../../adapters/openshell/client"; import { CLI_NAME } from "../../cli/branding"; +import { GATEWAY_PORT } from "../../core/ports"; import { gatewayStartGuidance } from "../../gateway-start-guidance"; -import { GATEWAY_PORT, OLLAMA_PORT } from "../../core/ports"; +import { + type OllamaHostInventoryProbeOptions, + probeOllamaHostInventory, +} from "../../inference/health"; import { CURRENT_RUNTIME_PROVIDER_BUNDLES, resolveCurrentRuntimeProviderBundle, @@ -173,36 +176,29 @@ export function cloudflaredDoctorCheck(sandboxName: string): DoctorCheck { } } -export function ollamaDoctorCheck(currentProvider: string): DoctorCheck { - const endpoint = `http://127.0.0.1:${OLLAMA_PORT}/api/tags`; - const result = captureHostCommand( - "curl", - buildValidatedCurlCommandArgs(["-sS", "--connect-timeout", "2", "--max-time", "4", endpoint]), - 6000, - ); +export type OllamaDoctorCheckDeps = OllamaHostInventoryProbeOptions; + +export function ollamaDoctorCheck( + currentProvider: string, + deps: OllamaDoctorCheckDeps = {}, +): DoctorCheck { + const { endpoint, inventory } = probeOllamaHostInventory(deps); const required = currentProvider === "ollama-local"; - if (result.status !== 0) { + if (inventory === null) { return { group: "Local services", label: "Ollama", status: required ? "fail" : "info", - detail: `not reachable at ${endpoint}`, + detail: `not reachable or invalid response at ${endpoint}`, hint: required ? "start Ollama or change the sandbox inference provider" : undefined, }; } - let modelCount = "unknown model count"; - try { - const parsed = JSON.parse(result.stdout); - if (Array.isArray(parsed.models)) modelCount = `${parsed.models.length} model(s)`; - } catch { - /* keep generic detail */ - } return { group: "Local services", label: "Ollama", status: "ok", - detail: `reachable at ${endpoint} (${modelCount})`, + detail: `reachable at ${endpoint} (${inventory.length} model(s))`, }; } diff --git a/src/lib/inference/context-window.test.ts b/src/lib/inference/context-window.test.ts index 8a1127dd5e1..d130f085a14 100644 --- a/src/lib/inference/context-window.test.ts +++ b/src/lib/inference/context-window.test.ts @@ -160,7 +160,7 @@ describe("resolveContextWindowForModel default dependencies (#8974)", () => { cleanup(); } }); - vi.mocked(getOllamaProbeCommand).mockReturnValue(["docker", "run", "ollama-probe"]); + vi.mocked(getOllamaProbeCommand).mockReturnValue(["curl", "ollama-probe"]); vi.mocked(resolveOllamaRuntimeContextWindow).mockReturnValue(16384); expect(resolveContextWindowForModel("ollama-local", "qwen3.5:9b")).toBe(16384); diff --git a/src/lib/inference/health.ts b/src/lib/inference/health.ts index da41d47c5f0..cf00cf77b60 100644 --- a/src/lib/inference/health.ts +++ b/src/lib/inference/health.ts @@ -14,8 +14,15 @@ import type { CurlProbeOptions, CurlProbeResult } from "../adapters/http/probe"; import { runCurlProbe } from "../adapters/http/probe"; import { normalizeCredentialValue, resolveProviderCredential } from "../credentials/store"; import { getProviderSelectionConfig } from "./config"; -import type { LocalProviderHealthProbeOptions } from "./local"; -import { probeLocalProviderHealth } from "./local"; +import { + getResolvedOllamaHost, + loadPersistedOllamaHost, + type LocalProviderHealthProbeOptions, + OLLAMA_PORT, + probeOllamaEndpointInventory, + probeLocalProviderHealth, + type RunCaptureFn, +} from "./local"; import { MIN_PROBE_REPLY_TOKENS } from "./max-tokens-field"; import { getChatCompletionsProbeCurlArgs } from "./onboard-probes"; import { usesNvidiaEndpointProbePayload } from "./openai-probe-models"; @@ -55,6 +62,30 @@ export interface ProviderHealthProbeOptions { isWsl?: boolean; } +export type OllamaHostInventoryProbeOptions = { + getOllamaHost?: () => string; + runCaptureImpl?: RunCaptureFn; + prepareDockerEnvironment?: Parameters[3]; +}; + +/** Probe the persisted raw Ollama daemon through its platform-specific host transport. */ +export function probeOllamaHostInventory(options: OllamaHostInventoryProbeOptions = {}): { + endpoint: string; + inventory: string[] | null; +} { + const host = options.getOllamaHost + ? options.getOllamaHost() + : (loadPersistedOllamaHost() ?? getResolvedOllamaHost()); + const endpoint = `http://${host}:${OLLAMA_PORT}/api/tags`; + const inventory = probeOllamaEndpointInventory( + host, + options.runCaptureImpl, + 5_000, + options.prepareDockerEnvironment, + ); + return { endpoint, inventory }; +} + const COMPATIBLE_PROVIDERS = new Set(["compatible-endpoint", "compatible-anthropic-endpoint"]); const NVIDIA_HEALTH_CREDENTIAL_ENV = "NVIDIA_INFERENCE_API_KEY"; const HEALTH_PROBE_CONNECT_TIMEOUT_SECONDS = "3"; diff --git a/src/lib/inference/local-windows-ollama-transport.test.ts b/src/lib/inference/local-windows-ollama-transport.test.ts index b99f8bf04e1..6905f456f81 100644 --- a/src/lib/inference/local-windows-ollama-transport.test.ts +++ b/src/lib/inference/local-windows-ollama-transport.test.ts @@ -18,6 +18,7 @@ import { OLLAMA_HOST_DOCKER_INTERNAL, loadPersistedOllamaHost, persistResolvedOllamaHost, + prepareOllamaApiExecution, probeLocalProviderHealth, probeOllamaModelCapabilities, probeWindowsHostOllamaRouteProtection, @@ -31,6 +32,8 @@ import { } from "./local"; import { withOllamaModelOwnershipTransaction } from "./ollama/proxy"; +const WINDOWS_OLLAMA_TAGS_URL = "http://host.docker.internal:11434/api/tags"; + function respondsOnlyThroughDockerDesktop(apiPath: string, response: string) { return vi.fn((command: readonly string[]) => { const rendered = command.join(" "); @@ -126,14 +129,20 @@ describe("Windows-host Ollama transport", () => { ["-sf", "http://host.docker.internal:11434/api/tags"], OLLAMA_HOST_DOCKER_INTERNAL, ), - ).toEqual([ - "docker", - "run", - "--rm", - CONTAINER_REACHABILITY_IMAGE, - "-sf", - "http://host.docker.internal:11434/api/tags", - ]); + ).toEqual( + expect.arrayContaining([ + "docker", + "run", + "--rm", + "HTTP_PROXY=", + "HTTPS_PROXY=", + "ALL_PROXY=", + "NO_PROXY=host.docker.internal", + CONTAINER_REACHABILITY_IMAGE, + "-sf", + "http://host.docker.internal:11434/api/tags", + ]), + ); expect(getOllamaApiCommand(["-sf", "http://127.0.0.1:11434/api/tags"], "127.0.0.1")).toEqual([ "curl", "-sf", @@ -141,13 +150,67 @@ describe("Windows-host Ollama transport", () => { ]); }); + it("clears container proxy variables without changing Docker client authority", () => { + const cleanup = vi.fn(() => ({ ok: true as const })); + const dockerEnv = { + DOCKER_CONFIG: "/tmp/healthy-docker-config", + DOCKER_CONTEXT: "default", + HTTPS_PROXY: "https://operator:private-token@proxy.example", + }; + const execution = prepareOllamaApiExecution( + ["curl", "-sf", WINDOWS_OLLAMA_TAGS_URL], + OLLAMA_HOST_DOCKER_INTERNAL, + { + env: dockerEnv, + runCaptureImpl: (command) => { + const rendered = command.join(" "); + return rendered.includes("Get-NetTCPConnection") + ? "127.0.0.1" + : command.includes("Host: rebinding.invalid") + ? "403" + : command.some((argument) => argument === WINDOWS_OLLAMA_TAGS_URL) + ? JSON.stringify({ models: [] }) + : ""; + }, + prepareDockerEnvironment: () => ({ + env: dockerEnv, + isolatedCredentialConfig: false, + cleanup, + }), + }, + ); + + expect(execution.env).toEqual( + expect.objectContaining({ + DOCKER_CONFIG: dockerEnv.DOCKER_CONFIG, + DOCKER_CONTEXT: "default", + }), + ); + expect(execution.env?.HTTPS_PROXY).toBeUndefined(); + expect(execution.command).toEqual( + expect.arrayContaining([ + "HTTP_PROXY=", + "http_proxy=", + "HTTPS_PROXY=", + "https_proxy=", + "ALL_PROXY=", + "all_proxy=", + "NO_PROXY=host.docker.internal", + "no_proxy=host.docker.internal", + ]), + ); + expect(execution.command.join(" ")).not.toContain("private-token"); + execution.cleanup(); + expect(cleanup).toHaveBeenCalledTimes(3); + }); + it("accepts route protection only when both probes use Docker Desktop", () => { const capture = vi.fn((command) => { const usesDockerDesktop = command[0] === "docker" && command[1] === "run" && command[2] === "--rm" && - command[3] === CONTAINER_REACHABILITY_IMAGE; + command.includes(CONTAINER_REACHABILITY_IMAGE); return usesDockerDesktop ? command.includes("Host: rebinding.invalid") ? "403" @@ -555,7 +618,7 @@ describe("Windows-host Ollama transport", () => { command[0] === "docker" && command[1] === "run" && command[2] === "--rm" && - command[3] === CONTAINER_REACHABILITY_IMAGE && + command.includes(CONTAINER_REACHABILITY_IMAGE) && command.some((argument) => argument === "http://host.docker.internal:11434/api/tags") ? JSON.stringify({ models: [] }) : "", @@ -622,7 +685,7 @@ describe("Windows-host Ollama transport", () => { ), ).toHaveLength(3); expect(sleeps).toEqual([500, 1_000]); - }); + }, 10_000); it("rejects an invalid Windows-host inventory after bounded retries (#10259)", () => { setResolvedOllamaHost(OLLAMA_HOST_DOCKER_INTERNAL); @@ -652,7 +715,7 @@ describe("Windows-host Ollama transport", () => { command[0] === "docker" && command[1] === "run" && command[2] === "--rm" && - command[3] === CONTAINER_REACHABILITY_IMAGE && + command.includes(CONTAINER_REACHABILITY_IMAGE) && command.some((argument) => argument === "http://host.docker.internal:11434/api/generate"); return { stdout: expected ? JSON.stringify({ done: true, response: "ready" }) : "", @@ -938,18 +1001,14 @@ describe("Windows-host Ollama transport", () => { const [command, options] = run.mock.calls[0] ?? []; expect({ cleanupCalls: cleanup.mock.calls.length, - commandPrefix: command?.slice(0, 5), + commandPrefix: command?.slice(0, 4), + image: command?.find((argument: string) => argument === CONTAINER_REACHABILITY_IMAGE), endpoint: command?.find((argument: string) => argument.endsWith("/api/generate")), options, }).toEqual({ cleanupCalls: 3, - commandPrefix: [ - "docker", - "run", - "--rm", - "-d", - CONTAINER_REACHABILITY_IMAGE, - ], + commandPrefix: ["docker", "run", "--rm", "-d"], + image: CONTAINER_REACHABILITY_IMAGE, endpoint: "http://host.docker.internal:11434/api/generate", options: { ignoreError: true, diff --git a/src/lib/inference/local.ts b/src/lib/inference/local.ts index 2637b502c41..6d729c645e3 100644 --- a/src/lib/inference/local.ts +++ b/src/lib/inference/local.ts @@ -38,6 +38,8 @@ import { import { type CaptureResult, run, runCapture, runCaptureEx, shellQuote } from "../runner"; import { buildSubprocessEnv } from "../subprocess-env"; +export { sleepSeconds }; + import { isLocalOllamaRouteOwner, OLLAMA_HOST_DOCKER_INTERNAL, @@ -139,7 +141,10 @@ export { MIN_OLLAMA_VERSION, } from "./ollama-version"; -export type RunCaptureExFn = (cmd: string[], opts?: { env?: NodeJS.ProcessEnv }) => CaptureResult; +export type RunCaptureExFn = ( + cmd: string[], + opts?: { env?: NodeJS.ProcessEnv; timeout?: number }, +) => CaptureResult; // Hosts that local-provider discovery may try when probing Ollama. The Windows // onboarding path separately checks host.docker.internal from Docker Desktop's @@ -344,7 +349,6 @@ export function getWindowsHostOllamaDockerHostValidationArgs(): string[] { ...getWindowsHostOllamaHostValidationCurlArgs(), ]; } - let _resolvedOllamaHost: string | null = null; const OLLAMA_HOST_RECEIPT_NAME = "ollama-host.json"; @@ -427,9 +431,9 @@ export function findReachableOllamaHost( "5", `http://${host}:${OLLAMA_PORT}/api/tags`, ], - { ignoreError: true }, + { ignoreError: true, timeout: 5_000 }, ); - if (result) { + if (isValidOllamaTagsResponseBody(result)) { if (runningOnWsl) { const networkingMode = capture(["wslinfo", "--networking-mode"], { ignoreError: true, @@ -524,6 +528,29 @@ export function clearPersistedOllamaHostIfUnused( } /** Keep Windows-host Ollama requests in Docker Desktop's verified network context. */ +const OLLAMA_DOCKER_PROXY_GUARD_ARGS = [ + "--env", + "HTTP_PROXY=", + "--env", + "http_proxy=", + "--env", + "HTTPS_PROXY=", + "--env", + "https_proxy=", + "--env", + "ALL_PROXY=", + "--env", + "all_proxy=", + "--env", + "FTP_PROXY=", + "--env", + "ftp_proxy=", + "--env", + `NO_PROXY=${OLLAMA_HOST_DOCKER_INTERNAL}`, + "--env", + `no_proxy=${OLLAMA_HOST_DOCKER_INTERNAL}`, +] as const; + export function getOllamaApiCommand( curlArgs: readonly string[], host: string = getResolvedOllamaHost(), @@ -535,6 +562,7 @@ export function getOllamaApiCommand( "run", "--rm", ...(options.dockerDetached ? ["-d"] : []), + ...OLLAMA_DOCKER_PROXY_GUARD_ARGS, CONTAINER_REACHABILITY_IMAGE, ...curlArgs, ] @@ -580,6 +608,7 @@ export function prepareOllamaApiExecution( host === OLLAMA_HOST_DOCKER_INTERNAL && windowsHostOllamaRouteProtectionProbeDepth === 0 && !probeWindowsHostOllamaRouteProtection(options.runCaptureImpl ?? runCapture, { + dockerContextIsDefault: options.dockerContextIsDefault, env: sourceEnv, prepareDockerEnvironment: options.prepareDockerEnvironment, }).protected @@ -937,7 +966,10 @@ export function ollamaInventoryContainsModel(inventory: string[], model: string) } function sanitizeModelNameForDisplay(value: string): string { - const sanitized = value.replace(/[\u0000-\u001f\u007f-\u009f]/g, ""); + const sanitized = value.replace( + /[\u0000-\u001f\u007f-\u009f\u061c\u200e\u200f\u2028-\u202e\u2066-\u2069]/gu, + "", + ); return sanitized.length > 120 ? `${sanitized.slice(0, 117)}...` : sanitized; } @@ -1679,21 +1711,28 @@ export function getLocalProviderContainerReachabilityCheck( export function probeOllamaEndpointInventory( host: string, runCaptureImpl?: RunCaptureFn, + timeoutMilliseconds = 5_000, + prepareDockerEnvironment: PrepareDockerEnvironmentFn = prepareIsolatedDockerEnvironment, ): string[] | null { - const capture = createOllamaApiCapture(runCaptureImpl, host); + const capture = createOllamaApiCapture(runCaptureImpl, host, prepareDockerEnvironment); + const normalizedTimeoutMilliseconds = + Number.isFinite(timeoutMilliseconds) && timeoutMilliseconds > 0 + ? Math.floor(timeoutMilliseconds) + : 5_000; + const boundedTimeoutMilliseconds = Math.max(1, Math.min(5_000, normalizedTimeoutMilliseconds)); const body = capture( [ "curl", ...buildValidatedCurlCommandArgs([ "-sf", "--connect-timeout", - "3", + String(Math.min(3, boundedTimeoutMilliseconds / 1000)), "--max-time", - "5", + String(boundedTimeoutMilliseconds / 1000), `http://${host}:${OLLAMA_PORT}/api/tags`, ]), ], - { ignoreError: true }, + { ignoreError: true, timeout: boundedTimeoutMilliseconds }, ); return parseOllamaModelInventory(body); } @@ -2349,7 +2388,8 @@ export function getOllamaProbeCommand( keep_alive: keepAlive, options: { num_predict: 16 }, }); - const endpoint = `http://${getResolvedOllamaHost()}:${OLLAMA_PORT}/api/generate`; + const host = getResolvedOllamaHost(); + const endpoint = `http://${host}:${OLLAMA_PORT}/api/generate`; return [ "curl", ...buildValidatedCurlCommandArgs([ @@ -2370,13 +2410,16 @@ export function validateOllamaModel( runCaptureImpl?: RunCaptureFn, isSparkImpl?: () => boolean, runCaptureExImpl?: RunCaptureExFn, - options: { allowToolsIncompatible?: boolean } = {}, + options: { + allowToolsIncompatible?: boolean; + prepareDockerEnvironment?: PrepareDockerEnvironmentFn; + } = {}, ): ValidationResult { const capture = runCaptureImpl ?? runCapture; const captureEx = createOllamaApiCaptureEx( runCaptureExImpl ?? runCaptureEx, getResolvedOllamaHost(), - prepareIsolatedDockerEnvironment, + options.prepareDockerEnvironment, capture, ); const isSpark = isSparkImpl ?? (() => detectNvidiaPlatform() === "spark"); diff --git a/src/lib/inference/ollama-runtime-context.test.ts b/src/lib/inference/ollama-runtime-context.test.ts index 37005554dcf..7aa131c51ec 100644 --- a/src/lib/inference/ollama-runtime-context.test.ts +++ b/src/lib/inference/ollama-runtime-context.test.ts @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { afterEach, describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; import { OLLAMA_PORT } from "../core/ports"; import { @@ -66,6 +66,31 @@ describe("Ollama runtime context helpers", () => { ).toBeNull(); }); + it("bounds the daemon status probe process and curl request", () => { + const capture = vi.fn( + (_command: readonly string[], _options?: { ignoreError?: boolean; timeout?: number }) => + JSON.stringify({ models: [] }), + ); + + probeOllamaRuntimeModelStatus("qwen3.6:35b", getOllamaHost, capture, 1_200); + + expect(capture.mock.calls[0][0]).toEqual( + expect.arrayContaining(["--connect-timeout", "1.2", "--max-time", "1.2"]), + ); + expect(capture.mock.calls[0][1]).toMatchObject({ ignoreError: true, timeout: 1_200 }); + }); + + it.each(["{}", "null", '{"models":{}}'])( + "treats an invalid daemon status shape as a failed probe: %s", + (response) => { + expect(probeOllamaRuntimeModelStatus("qwen3.6:35b", getOllamaHost, () => response)).toEqual({ + probed: false, + loaded: false, + cpuOnly: false, + }); + }, + ); + it.each(["bogus", "1.5", 0, -1])( "warns and ignores malformed Ollama /api/ps context length %#", (value) => { diff --git a/src/lib/inference/ollama-runtime-context.ts b/src/lib/inference/ollama-runtime-context.ts index 39c22515e58..fa11c2512af 100644 --- a/src/lib/inference/ollama-runtime-context.ts +++ b/src/lib/inference/ollama-runtime-context.ts @@ -16,7 +16,7 @@ import { runCapture } from "../runner"; export type OllamaRuntimeRunCaptureFn = ( cmd: readonly string[], - opts?: { ignoreError?: boolean }, + opts?: { ignoreError?: boolean; timeout?: number }, ) => string; export interface OllamaRuntimeModelStatus { @@ -140,28 +140,45 @@ export function probeOllamaRuntimeModelStatus( model: string, getOllamaHost: () => string, runCaptureImpl?: OllamaRuntimeRunCaptureFn, + timeoutMilliseconds = 5_000, ): OllamaRuntimeModelStatus { const capture = runCaptureImpl ?? runCapture; const host = getOllamaHost(); + const normalizedTimeoutMilliseconds = + Number.isFinite(timeoutMilliseconds) && timeoutMilliseconds > 0 + ? Math.floor(timeoutMilliseconds) + : 5_000; + const boundedTimeoutMilliseconds = Math.max(1, Math.min(5_000, normalizedTimeoutMilliseconds)); const output = capture( [ "curl", ...buildValidatedCurlCommandArgs([ "-sf", "--connect-timeout", - "3", + String(Math.min(3, boundedTimeoutMilliseconds / 1000)), "--max-time", - "5", + String(boundedTimeoutMilliseconds / 1000), `http://${host}:${OLLAMA_PORT}/api/ps`, ]), ], - { ignoreError: true }, + { ignoreError: true, timeout: boundedTimeoutMilliseconds }, ); + return parseOllamaRuntimeModelStatus(model, output); +} + +/** Parse one completed Ollama `/api/ps` response without performing I/O. */ +export function parseOllamaRuntimeModelStatus( + model: string, + output: string, +): OllamaRuntimeModelStatus { if (!output) return { probed: false, loaded: false, cpuOnly: false }; try { - const parsed = JSON.parse(String(output || "")); - const models = Array.isArray(parsed?.models) ? parsed.models : []; + const parsed = JSON.parse(output) as { models?: unknown } | null; + if (!parsed || !Array.isArray(parsed.models)) { + return { probed: false, loaded: false, cpuOnly: false }; + } + const models = parsed.models; const target = normalizeOllamaModelName(model); const loaded = models.find((entry: { name?: unknown; model?: unknown }) => { return ( @@ -193,7 +210,7 @@ export function probeOllamaRuntimeModelStatus( ...(hasSizeVram ? { sizeVram: rawSizeVram } : {}), }; } catch { - return { probed: true, loaded: false, cpuOnly: false }; + return { probed: false, loaded: false, cpuOnly: false }; } } diff --git a/src/lib/inference/ollama/windows.test.ts b/src/lib/inference/ollama/windows.test.ts index e87469d6844..6691cba5adf 100644 --- a/src/lib/inference/ollama/windows.test.ts +++ b/src/lib/inference/ollama/windows.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { EventEmitter } from "node:events"; import { createRequire } from "node:module"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -17,6 +18,151 @@ function commandText(command: string | string[]): string { return Array.isArray(command) ? command.join(" ") : String(command); } +function isWindowsOllamaTagsUrl(value: string): boolean { + try { + const url = new URL(value); + return ( + url.protocol === "http:" && + url.username === "" && + url.password === "" && + url.hostname === "host.docker.internal" && + url.port === "11434" && + url.pathname === "/api/tags" && + url.search === "" && + url.hash === "" + ); + } catch { + return false; + } +} + +function isDockerTagsRequest(command: string | string[]): boolean { + return Array.isArray(command) && command[0] === "docker" && command.some(isWindowsOllamaTagsUrl); +} + +function successfulRun(stdout = "") { + return { status: 0, stdout, stderr: "" }; +} + +function hostSnapshotRun( + userHost: string | null, + watcherPath: string | null, + daemonPath: string | null, +) { + return successfulRun(JSON.stringify({ userHost, watcherPath, daemonPath })); +} + +type WindowsSetupState = { + userHost: string | null; + watcherRunning: boolean; + daemonRunning: boolean; + events: string[]; +}; + +class PreservedWindowsInterrupt extends Error { + constructor(readonly signal: NodeJS.Signals) { + super(`preserved ${signal}`); + } +} + +function createWindowsSetupBoundary(options: { + userHost: string | null; + watcherPath: string | null; + daemonPath: string | null; + persistBindingResult?: boolean; + launchStatuses?: number[]; + readinessResults?: boolean[]; + rollbackStatus?: number; + interruptSignal?: "SIGINT" | "SIGTERM"; +}) { + const snapshot = { + userHost: options.userHost, + watcherPath: options.watcherPath, + daemonPath: options.daemonPath, + }; + const state: WindowsSetupState = { + userHost: snapshot.userHost, + watcherRunning: Boolean(snapshot.watcherPath), + daemonRunning: Boolean(snapshot.daemonPath), + events: [], + }; + const launchStatuses = options.launchStatuses ?? [1, 1, 1]; + const readinessResults = options.readinessResults ?? []; + const rollbackStatus = options.rollbackStatus ?? 0; + let launchIndex = 0; + let readinessIndex = 0; + let interruptHandler = (_signal: "SIGINT" | "SIGTERM"): void | Promise => {}; + const operations = { + captureSnapshot: vi.fn(() => { + state.events.push("snapshot"); + return snapshot; + }), + persistBinding: vi.fn(() => { + state.events.push("persist"); + state.userHost = "127.0.0.1:11434"; + return options.persistBindingResult ?? true; + }), + stopProcesses: vi.fn(() => { + state.events.push("stop-existing"); + state.watcherRunning = false; + state.daemonRunning = false; + }), + wait: vi.fn(), + launchOperations: { + runAttempt: vi.fn((attempt: { kind: "watcher" | "installed" | "path" }) => { + const status = launchStatuses[launchIndex] ?? 1; + launchIndex += 1; + state.events.push(`launch:${attempt.kind}`); + state.daemonRunning = status === 0; + options.interruptSignal ? interruptHandler(options.interruptSignal) : undefined; + return status === 0 + ? successfulRun() + : { status, stderr: `${attempt.kind} launch unavailable` }; + }), + awaitReady: vi.fn(() => { + const ready = readinessResults[readinessIndex] ?? false; + readinessIndex += 1; + state.events.push("readiness"); + ready + ? require(LOCAL_INFERENCE_PATH).setResolvedOllamaHost("host.docker.internal") + : undefined; + return ready; + }), + stopProcesses: vi.fn(() => { + state.events.push("stop-launch-attempt"); + state.watcherRunning = false; + state.daemonRunning = false; + }), + wait: vi.fn(), + }, + rollbackSnapshot: vi.fn(() => { + state.events.push("stop-replacement", "restore"); + const restored = rollbackStatus === 0; + state.userHost = restored ? snapshot.userHost : state.userHost; + state.watcherRunning = restored ? Boolean(snapshot.watcherPath) : false; + state.daemonRunning = restored + ? Boolean(snapshot.daemonPath) && !snapshot.watcherPath + : false; + return restored; + }), + registerInterruptHandler: vi.fn( + (handler: (signal: "SIGINT" | "SIGTERM") => void | Promise) => { + interruptHandler = handler; + return vi.fn(); + }, + ), + preserveInterrupt: vi.fn((signal: "SIGINT" | "SIGTERM") => { + state.events.push(`signal:${signal}`); + throw new PreservedWindowsInterrupt(signal); + }), + }; + return { + operations, + state, + triggerInterrupt: (signal: "SIGINT" | "SIGTERM") => interruptHandler(signal), + }; +} + function loadWindowsOllamaWithMocks( run: ReturnType, runCapture: ReturnType, @@ -33,9 +179,9 @@ function loadWindowsOllamaWithMocks( const originalDetectContainerRuntimeFromDockerInfo = dockerAdapter.detectContainerRuntimeFromDockerInfo; const originalIsWsl = platform.isWsl; + const originalSpawn = childProcess.spawn; const originalRun = runner.run; const originalRunCapture = runner.runCapture; - const originalSpawn = childProcess.spawn; // Stub the blocking wait so this test does not spend time on retry delays. const atomicsWaitStub = vi.spyOn(Atomics, "wait").mockReturnValue("timed-out"); @@ -62,6 +208,96 @@ function loadWindowsOllamaWithMocks( }; } +function createMockChildProcess() { + return Object.assign(new EventEmitter(), { + stdout: new EventEmitter(), + stderr: new EventEmitter(), + kill: vi.fn(() => true), + }); +} + +function captureDefaultWindowsRollbackInvocation(userHost: string | null) { + const watcherPath = "C:\\Users\\tester\\Ollama\\ollama app.exe"; + const daemonPath = "C:\\Users\\tester\\Ollama\\ollama.exe"; + const launchFailure = { status: 1, stdout: "", stderr: "launch unavailable" }; + const runResults = [ + hostSnapshotRun(userHost, watcherPath, daemonPath), + launchFailure, + launchFailure, + launchFailure, + successfulRun(), + ]; + const run = vi.fn( + (_command: string | string[], _options?: { env?: NodeJS.ProcessEnv }) => + runResults.shift() ?? successfulRun(), + ); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks( + run, + vi.fn((command: string | string[]) => + commandText(command).includes("SetEnvironmentVariable('OLLAMA_HOST'") + ? "127.0.0.1:11434" + : "", + ), + ); + + try { + expect(windows.setupWindowsOllamaLoopbackBinding({ installedPath: daemonPath })).toEqual({ + ok: false, + reason: "readiness", + }); + } finally { + restore(); + logSpy.mockRestore(); + errorSpy.mockRestore(); + } + + return { daemonPath, run, watcherPath }; +} + +function createWindowsInstallBoundary(options: { + userHost: string | null; + watcherPath: string | null; + daemonPath: string | null; + persistBindingResult?: boolean; + installedPath?: string; + initialReady?: boolean; + launchStatuses?: number[]; + readinessResults?: boolean[]; + rollbackStatus?: number; + installerInterrupt?: "SIGINT" | "SIGTERM"; + cancelInstaller?: () => Promise; +}) { + const boundary = createWindowsSetupBoundary(options); + const cancelInstaller = vi.fn(async () => { + boundary.state.events.push("cancel-installer"); + await options.cancelInstaller?.(); + }); + const completion = options.installerInterrupt + ? Promise.resolve().then(() => boundary.triggerInterrupt(options.installerInterrupt!)) + : Promise.resolve(); + const operations = { + ...boundary.operations, + startInstaller: vi.fn(() => { + boundary.state.events.push("install"); + boundary.state.userHost = "127.0.0.1:11434"; + boundary.state.watcherRunning = true; + boundary.state.daemonRunning = true; + return { completion, cancelAndWait: cancelInstaller }; + }), + resolveInstalledPath: vi.fn(() => { + boundary.state.events.push("resolve-path"); + return options.installedPath ?? "C:\\Users\\tester\\Ollama\\ollama.exe"; + }), + awaitReady: vi.fn(() => { + boundary.state.events.push("installer-readiness"); + return options.initialReady ?? false; + }), + }; + return { ...boundary, cancelInstaller, operations }; +} + describe("Windows Ollama helper", () => { beforeEach(() => { vi.stubEnv("DOCKER_CONTEXT", "default"); @@ -72,28 +308,148 @@ describe("Windows Ollama helper", () => { vi.unstubAllEnvs(); }); - it("rejects a nonempty invalid Docker readiness response (#10100)", () => { + it("leaves the persistent installer binding under the mutation transaction", () => { + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + const installerCommand = windows.buildWindowsOllamaInstallerCommand(); + expect(installerCommand).toContain("$env:OLLAMA_HOST='127.0.0.1:11434'"); + expect(installerCommand).not.toContain("SetEnvironmentVariable('OLLAMA_HOST'"); + } finally { + restore(); + } + }); + + it("terminates the PowerShell wrapper when cancellation precedes the PID sentinel", async () => { + const child = createMockChildProcess(); + const spawnProcess = vi.fn(() => child); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn(), spawnProcess); + + try { + const installer = windows.startWindowsOllamaInstaller(); + const cancellation = installer.cancelAndWait(); + expect(child.kill).toHaveBeenCalledWith("SIGTERM"); + + child.emit("close", null); + await expect(cancellation).resolves.toBeUndefined(); + } finally { + restore(); + } + }); + + it("terminates the reported Windows installer process tree", async () => { + const installerChild = createMockChildProcess(); + const taskkillChild = createMockChildProcess(); + const spawnProcess = vi.fn((command: string) => + command === "taskkill.exe" ? taskkillChild : installerChild, + ); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn(), spawnProcess); + + try { + const installer = windows.startWindowsOllamaInstaller(); + installerChild.stdout.emit("data", Buffer.from("__NEMOCLAW_WINDOWS_INSTALLER_PID__:4321\n")); + const cancellation = installer.cancelAndWait(); + expect(installerChild.kill).not.toHaveBeenCalled(); + await vi.waitFor(() => + expect(spawnProcess).toHaveBeenLastCalledWith( + "taskkill.exe", + ["/PID", "4321", "/T", "/F"], + { stdio: "ignore" }, + ), + ); + + taskkillChild.emit("close", 0); + installerChild.emit("close", null); + await expect(cancellation).resolves.toBeUndefined(); + } finally { + restore(); + } + }); + + it("streams an oversized unterminated PID prefix and cancels the wrapper", async () => { + const child = createMockChildProcess(); + const spawnProcess = vi.fn(() => child); + const stdoutWrite = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn(), spawnProcess); + const output = Buffer.alloc(256, "x"); + + try { + const installer = windows.startWindowsOllamaInstaller(); + child.stdout.emit("data", output); + expect(stdoutWrite).toHaveBeenCalledWith(output); + child.stdout.emit("data", Buffer.from("\n__NEMOCLAW_WINDOWS_INSTALLER_PID__:4321\n")); + + const cancellation = installer.cancelAndWait(); + expect(child.kill).toHaveBeenCalledWith("SIGTERM"); + child.emit("close", null); + await expect(cancellation).resolves.toBeUndefined(); + expect(spawnProcess).toHaveBeenCalledTimes(1); + } finally { + restore(); + stdoutWrite.mockRestore(); + } + }); + + it("describes Docker-client isolation in Windows Ollama timeout diagnostics", () => { + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + windows.printWindowsOllamaTimeoutDiagnostics(); + } finally { + restore(); + } + + const diagnostic = errorSpy.mock.calls.map(([message]) => String(message)).join("\n"); + errorSpy.mockRestore(); + expect(diagnostic).toContain("nemoclaw onboard"); + expect(diagnostic).toContain("isolated Docker client configuration"); + expect(diagnostic).toContain("removes that temporary configuration afterward"); + expect(diagnostic).toContain("docker run"); + expect(diagnostic).toContain(REBINDING_PROBE_HOST_HEADER); + }); + + it("describes a snapshot inspection failure without claiming a startup timeout", () => { + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + windows.printWindowsOllamaSnapshotDiagnostics(); + } finally { + restore(); + } + + const diagnostic = errorSpy.mock.calls.map(([message]) => String(message)).join("\n"); + errorSpy.mockRestore(); + expect(diagnostic).toContain("User-scope OLLAMA_HOST"); + expect(diagnostic).toContain("existing Ollama processes"); + expect(diagnostic).toContain("nemoclaw onboard"); + expect(diagnostic).not.toContain("Timed out waiting for Ollama to start"); + }); + + it("continues probing after a nonempty invalid Docker readiness response (#10100)", () => { const run = vi.fn(); const localInference = require(LOCAL_INFERENCE_PATH); - const runCapture = vi.fn((command: string | string[]) => { - if (commandText(command).includes("Get-NetTCPConnection")) return "127.0.0.1"; - expect(command).toEqual( - expect.arrayContaining([ - "docker", - "run", - "--rm", - localInference.CONTAINER_REACHABILITY_IMAGE, - WINDOWS_OLLAMA_TAGS_URL, - ]), - ); - expect(command.slice(0, 4)).toEqual([ - "docker", - "run", - "--rm", - localInference.CONTAINER_REACHABILITY_IMAGE, - ]); - expect(command.at(-1)).toBe(WINDOWS_OLLAMA_TAGS_URL); + let invalidResponseServed = false; + let validResponseServed = false; + const serveInvalidResponse = () => { + invalidResponseServed = true; return "proxy response"; + }; + const serveValidResponse = () => { + validResponseServed = true; + return JSON.stringify({ models: [] }); + }; + const runCapture = vi.fn((command: string | string[]) => { + return commandText(command).includes("Get-NetTCPConnection") + ? "127.0.0.1" + : Array.isArray(command) && command.includes(REBINDING_PROBE_HOST_HEADER) + ? "403" + : isDockerTagsRequest(command) + ? invalidResponseServed + ? serveValidResponse() + : serveInvalidResponse() + : ""; }); localInference.resetOllamaHostCache(); const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); @@ -102,16 +458,16 @@ describe("Windows Ollama helper", () => { try { expect( windows.awaitWindowsOllamaReady({ - delay: vi.fn(), prepareDockerEnvironment: () => ({ env: {}, isolatedCredentialConfig: false, cleanup: () => ({ ok: true }), }), }), - ).toBe(false); - expect(runCapture.mock.calls.length).toBeGreaterThan(0); - expect(localInference.getResolvedOllamaHost()).toBe("127.0.0.1"); + ).toBe(true); + expect(invalidResponseServed).toBe(true); + expect(validResponseServed).toBe(true); + expect(localInference.getResolvedOllamaHost()).toBe("host.docker.internal"); } finally { localInference.resetOllamaHostCache(); restore(); @@ -120,18 +476,25 @@ describe("Windows Ollama helper", () => { }); it("rejects readiness when the active runtime changes away from Docker Desktop", () => { - const run = vi.fn(); - const runCapture = vi.fn(); const detectContainerRuntimeFromDockerInfo = vi.fn(() => "docker"); const currentIsWsl = vi.fn(() => true); const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); - const { windows, restore } = loadWindowsOllamaWithMocks(run, runCapture, undefined, { + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn(), undefined, { detectContainerRuntimeFromDockerInfo, isWsl: currentIsWsl, }); try { - expect(windows.awaitWindowsOllamaReady({ delay: vi.fn() })).toBe(false); + expect( + windows.awaitWindowsOllamaReady({ + delay: vi.fn(), + prepareDockerEnvironment: () => ({ + env: {}, + isolatedCredentialConfig: false, + cleanup: () => ({ ok: true }), + }), + }), + ).toBe(false); } finally { restore(); logSpy.mockRestore(); @@ -139,23 +502,22 @@ describe("Windows Ollama helper", () => { expect(detectContainerRuntimeFromDockerInfo).toHaveBeenCalledTimes(15); expect(currentIsWsl).toHaveBeenCalledTimes(15); - expect(runCapture).not.toHaveBeenCalled(); }); it("rejects a reachable daemon that accepts a rebinding Host header", () => { - const run = vi.fn(); - const localInference = require(LOCAL_INFERENCE_PATH); const runCapture = vi.fn((command: string | string[]) => { - if (commandText(command).includes("Get-NetTCPConnection")) return "127.0.0.1"; - return Array.isArray(command) && command.includes(REBINDING_PROBE_HOST_HEADER) - ? "200" - : Array.isArray(command) && command.at(-1) === WINDOWS_OLLAMA_TAGS_URL - ? JSON.stringify({ models: [] }) - : ""; + return commandText(command).includes("Get-NetTCPConnection") + ? "127.0.0.1" + : Array.isArray(command) && command.includes(REBINDING_PROBE_HOST_HEADER) + ? "200" + : isDockerTagsRequest(command) + ? JSON.stringify({ models: [] }) + : ""; }); + const localInference = require(LOCAL_INFERENCE_PATH); localInference.resetOllamaHostCache(); const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); - const { windows, restore } = loadWindowsOllamaWithMocks(run, runCapture); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), runCapture); try { expect( @@ -176,186 +538,522 @@ describe("Windows Ollama helper", () => { } }); - it("falls back from a stale watcher path and checks readiness from Docker Desktop (#8127)", () => { + it("falls back from a stale watcher to the verified executable and selects Windows route", () => { const watcherPath = "C:\\Users\\tester\\AppData\\Local\\Programs\\Ollama\\ollama app.exe"; const installedPath = "C:\\Users\\tester\\AppData\\Local\\Programs\\Ollama\\ollama.exe"; - const launchScripts: string[] = []; - const stopCommands: string[] = []; - const persistedHostCommands: string[] = []; - - const run = vi.fn((command: string[]) => { - const script = command[2] || ""; - launchScripts.push(script); - if (script.includes(watcherPath)) { - return { status: 1, stderr: "stale watcher path" }; - } - return { status: 0, stderr: "" }; + const boundary = createWindowsSetupBoundary({ + userHost: "127.0.0.1:11434", + watcherPath, + daemonPath: installedPath, + launchStatuses: [1, 0], + readinessResults: [true], }); - const runCapture = vi.fn((command: string | string[]) => { - const cmd = commandText(command); - switch (true) { - case cmd.includes("Get-Process 'ollama app'") && cmd.includes("ExpandProperty Path"): - return watcherPath; - case cmd.includes("Stop-Process"): - stopCommands.push(cmd); - return ""; - case cmd.includes("SetEnvironmentVariable('OLLAMA_HOST'"): - persistedHostCommands.push(cmd); - return "127.0.0.1:11434"; - case cmd.includes("Get-NetTCPConnection"): - return "127.0.0.1"; - case Array.isArray(command) && command.includes(REBINDING_PROBE_HOST_HEADER): - return "403"; - case Array.isArray(command) && command.at(-1) === WINDOWS_OLLAMA_TAGS_URL: - return command[0] === "docker" && - launchScripts.some((script) => script.includes(installedPath)) - ? JSON.stringify({ models: [] }) - : ""; - default: - return ""; - } + const localInference = require(LOCAL_INFERENCE_PATH); + localInference.resetOllamaHostCache(); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + const result = windows.setupWindowsOllamaLoopbackBinding( + { installedPath }, + boundary.operations, + ); + expect(result.ok).toBe(true); + result.commit(); + expect(boundary.state.events).toEqual([ + "snapshot", + "persist", + "stop-existing", + "launch:watcher", + "stop-launch-attempt", + "launch:installed", + "readiness", + ]); + expect(localInference.getResolvedOllamaHost()).toBe("host.docker.internal"); + } finally { + localInference.resetOllamaHostCache(); + restore(); + logSpy.mockRestore(); + errorSpy.mockRestore(); + } + }); + + it("restores the prior binding and watcher when every rebound launch fails", () => { + const priorHost = "127.0.0.1:11434"; + const watcherPath = "C:\\Users\\tester\\Ollama\\ollama app.exe"; + const daemonPath = "C:\\Users\\tester\\Ollama\\ollama.exe"; + const boundary = createWindowsSetupBoundary({ + userHost: priorHost, + watcherPath, + daemonPath, }); const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); - const delay = vi.fn(); - const { windows, restore } = loadWindowsOllamaWithMocks(run, runCapture); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); try { - expect(windows.setupWindowsOllamaLoopbackBinding({ installedPath, delay })).toBe(true); + expect( + windows.setupWindowsOllamaLoopbackBinding( + { installedPath: daemonPath }, + boundary.operations, + ), + ).toEqual({ ok: false, reason: "readiness" }); } finally { restore(); logSpy.mockRestore(); errorSpy.mockRestore(); } - expect(run).toHaveBeenCalledTimes(2); - expect(launchScripts[0]).toContain(watcherPath); - expect(launchScripts[1]).toContain(installedPath); - expect(launchScripts[1]).toContain("-ArgumentList 'serve'"); - expect(launchScripts.every((script) => !script.includes("0.0.0.0:11434"))).toBe(true); - expect(launchScripts.some((script) => script.includes("127.0.0.1:11434"))).toBe(true); - expect( - launchScripts.some((script) => script.includes("Start-Process -FilePath ollama.exe")), - ).toBe(false); - expect(stopCommands[0]).toContain("Get-Process 'ollama app'"); - expect(stopCommands[1]).toContain("Get-Process ollama"); - expect(persistedHostCommands).toEqual([expect.stringContaining("'127.0.0.1:11434'")]); - expect(persistedHostCommands[0]).toContain("GetEnvironmentVariable('OLLAMA_HOST','User')"); - expect(persistedHostCommands[0]).not.toContain("0.0.0.0:11434"); - expect(runCapture).toHaveBeenCalledWith( - [ - "docker", - "run", - "--rm", - "docker.io/curlimages/curl@sha256:d9b4541e214bcd85196d6e92e2753ac6d0ea699f0af5741f8c6cccbfcf00ef4b", - "-sf", - "--connect-timeout", - "2", - "--max-time", - "5", - "http://host.docker.internal:11434/api/tags", - ], - expect.objectContaining({ ignoreError: true }), - ); - expect(delay).toHaveBeenCalled(); - expect(delay.mock.calls.every(([seconds]) => seconds > 0 && seconds <= 2)).toBe(true); + expect(boundary.state.userHost).toBe(priorHost); + expect(boundary.state.watcherRunning).toBe(true); + expect(boundary.state.daemonRunning).toBe(false); + expect(boundary.state.events.at(-1)).toBe("restore"); }); - it("explains recovery when every launch fails after persisting the loopback binding", () => { - const watcherPath = "C:\\Users\\tester\\AppData\\Local\\Programs\\Ollama\\ollama app.exe"; - const installedPath = "C:\\Users\\tester\\AppData\\Local\\Programs\\Ollama\\ollama.exe"; - const run = vi.fn(() => ({ status: 1, stderr: "launch failed" })); - const runCapture = vi.fn((command: string | string[]) => { - const rendered = commandText(command); - return rendered.includes("Get-Process 'ollama app'") && - rendered.includes("ExpandProperty Path") - ? watcherPath - : rendered.includes("SetEnvironmentVariable('OLLAMA_HOST'") - ? "127.0.0.1:11434" - : ""; + it("restores the prior state when existing-install persistence is uncertain (#10855)", () => { + const priorHost = "127.0.0.1:11434"; + const watcherPath = "C:\\Users\\tester\\Ollama\\ollama app.exe"; + const boundary = createWindowsSetupBoundary({ + userHost: priorHost, + watcherPath, + daemonPath: null, + persistBindingResult: false, }); const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); - const { windows, restore } = loadWindowsOllamaWithMocks(run, runCapture); - let errors: string[] = []; + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); try { - expect( - windows.setupWindowsOllamaLoopbackBinding({ - installedPath, - delay: vi.fn(), - }), - ).toBe(false); - errors = errorSpy.mock.calls.map(([message]) => String(message)); + expect(windows.setupWindowsOllamaLoopbackBinding({}, boundary.operations)).toEqual({ + ok: false, + reason: "binding", + }); } finally { restore(); errorSpy.mockRestore(); } - expect(run).toHaveBeenCalledTimes(3); - expect(errors).toContainEqual( - expect.stringContaining("OLLAMA_HOST=127.0.0.1:11434 setting was persisted"), - ); - expect(errors).toContainEqual(expect.stringContaining("Ollama may now be stopped")); - expect(errors).toContainEqual(expect.stringContaining("rerun `nemoclaw onboard`")); + expect(boundary.state.userHost).toBe(priorHost); + expect(boundary.state.watcherRunning).toBe(true); + expect(boundary.state.events).toEqual(["snapshot", "persist", "stop-replacement", "restore"]); }); - it("fails repair before stopping Ollama when the persistent loopback setting is rejected", () => { - const run = vi.fn(); - const runCapture = vi.fn((command: string | string[]) => - commandText(command).includes("SetEnvironmentVariable('OLLAMA_HOST'") ? "0.0.0.0:11434" : "", - ); + it("redacts the prior binding from a failed Windows rollback diagnostic", () => { + const priorHost = "https://operator:private-token@ollama.example:11434"; + const daemonPath = "C:\\Users\\tester\\Ollama\\ollama.exe"; + const boundary = createWindowsSetupBoundary({ + userHost: priorHost, + watcherPath: null, + daemonPath, + rollbackStatus: 1, + }); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); - const { windows, restore } = loadWindowsOllamaWithMocks(run, runCapture); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + expect(windows.setupWindowsOllamaLoopbackBinding({}, boundary.operations)).toEqual({ + ok: false, + reason: "readiness", + }); + } finally { + restore(); + logSpy.mockRestore(); + } + + const diagnostic = errorSpy.mock.calls.map(([message]) => String(message)).join("\n"); + errorSpy.mockRestore(); + expect(diagnostic).toContain("Failed to restore the previous Windows Ollama state"); + expect(diagnostic).toContain("restore your previous User-scope OLLAMA_HOST value"); + expect(diagnostic).toContain("relaunch the previous Ollama app or daemon"); + expect(diagnostic).not.toContain(priorHost); + expect(diagnostic).not.toContain("private-token"); + expect(diagnostic).not.toContain(Buffer.from(priorHost, "utf8").toString("base64")); + }); + + it("keeps a credential-bearing prior binding out of Windows rollback argv", () => { + const priorHost = "https://operator:private-token@ollama.example:11434"; + const { daemonPath, run, watcherPath } = captureDefaultWindowsRollbackInvocation(priorHost); + expect(run).toHaveBeenCalledTimes(5); + const rollbackCall = run.mock.calls.at(-1); + expect(rollbackCall).toBeDefined(); + const [rollbackCommand, rollbackOptions] = rollbackCall!; + const argv = commandText(rollbackCommand); + + expect(argv).toContain("$env:NEMOCLAW_OLLAMA_RESTORE_HOST"); + expect(argv).toContain("Remove-Item Env:NEMOCLAW_OLLAMA_RESTORE_HOST"); + expect(argv).not.toContain(priorHost); + expect(argv).not.toContain(Buffer.from(priorHost, "utf8").toString("base64")); + expect(argv).not.toContain(watcherPath); + expect(argv).not.toContain(daemonPath); + expect(rollbackOptions?.env).toMatchObject({ + NEMOCLAW_OLLAMA_RESTORE_HOST: priorHost, + NEMOCLAW_OLLAMA_RESTORE_HOST_PRESENT: "1", + NEMOCLAW_OLLAMA_RESTORE_WATCHER: watcherPath, + NEMOCLAW_OLLAMA_RESTORE_DAEMON: daemonPath, + }); + }); + + it("preserves a missing prior binding distinctly during Windows rollback", () => { + const { run } = captureDefaultWindowsRollbackInvocation(null); + const rollbackCall = run.mock.calls.at(-1); + expect(rollbackCall).toBeDefined(); + const [, rollbackOptions] = rollbackCall!; + + expect(rollbackOptions?.env).toMatchObject({ + NEMOCLAW_OLLAMA_RESTORE_HOST: "", + NEMOCLAW_OLLAMA_RESTORE_HOST_PRESENT: "0", + }); + }); + + it("restores the prior Windows state when install cannot resolve ollama.exe", async () => { + const priorHost = "127.0.0.1:11434"; + const watcherPath = "C:\\Users\\tester\\Ollama\\ollama app.exe"; + const boundary = createWindowsInstallBoundary({ + userHost: priorHost, + watcherPath, + daemonPath: null, + installedPath: "", + }); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); try { - expect(windows.setupWindowsOllamaLoopbackBinding()).toBe(false); + await expect(windows.installOllamaOnWindowsHost({}, boundary.operations)).resolves.toEqual({ + ok: false, + path: "", + reason: "install", + }); } finally { restore(); + logSpy.mockRestore(); errorSpy.mockRestore(); } - expect(run).not.toHaveBeenCalled(); - expect( - runCapture.mock.calls.some(([command]) => commandText(command).includes("Stop-Process")), - ).toBe(false); + expect(boundary.state.userHost).toBe(priorHost); + expect(boundary.state.watcherRunning).toBe(true); + expect(boundary.state.daemonRunning).toBe(false); + expect(boundary.state.events).toEqual([ + "snapshot", + "persist", + "install", + "resolve-path", + "stop-replacement", + "restore", + ]); }); - it("fails fresh installation before spawning when the persistent loopback setting is rejected", async () => { - const run = vi.fn(); - const runCapture = vi.fn((command: string | string[]) => - commandText(command).includes("SetEnvironmentVariable('OLLAMA_HOST'") ? "0.0.0.0:11434" : "", - ); - const spawn = vi.fn(); + it("restores the prior state when installer persistence is uncertain (#10855)", async () => { + const priorHost = "127.0.0.1:11434"; + const daemonPath = "C:\\Users\\tester\\Ollama\\ollama.exe"; + const boundary = createWindowsInstallBoundary({ + userHost: priorHost, + watcherPath: null, + daemonPath, + persistBindingResult: false, + }); const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); - const { windows, restore } = loadWindowsOllamaWithMocks(run, runCapture, spawn); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); try { - await expect(windows.installOllamaOnWindowsHost()).resolves.toEqual({ ok: false, path: "" }); + await expect(windows.installOllamaOnWindowsHost({}, boundary.operations)).resolves.toEqual({ + ok: false, + path: "", + reason: "binding", + }); } finally { restore(); logSpy.mockRestore(); errorSpy.mockRestore(); } - expect(spawn).not.toHaveBeenCalled(); + expect(boundary.state.userHost).toBe(priorHost); + expect(boundary.state.daemonRunning).toBe(true); + expect(boundary.state.events).toEqual(["snapshot", "persist", "stop-replacement", "restore"]); }); + it("restores the prior Windows state when installed Ollama remains unreachable", async () => { + const priorHost = "127.0.0.1:11434"; + const daemonPath = "C:\\Users\\tester\\Ollama\\ollama.exe"; + const boundary = createWindowsInstallBoundary({ + userHost: priorHost, + watcherPath: null, + daemonPath, + launchStatuses: [1, 1], + }); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + await expect(windows.installOllamaOnWindowsHost({}, boundary.operations)).resolves.toEqual({ + ok: false, + path: daemonPath, + reason: "readiness", + }); + } finally { + restore(); + logSpy.mockRestore(); + errorSpy.mockRestore(); + } + + expect(boundary.state.userHost).toBe(priorHost); + expect(boundary.state.watcherRunning).toBe(false); + expect(boundary.state.daemonRunning).toBe(true); + expect(boundary.state.events.at(-2)).toBe("stop-replacement"); + expect(boundary.state.events.at(-1)).toBe("restore"); + }); + + it("cancels install and restores prior Windows state before preserving SIGTERM", async () => { + const priorHost = "127.0.0.1:11434"; + const watcherPath = "C:\\Users\\tester\\Ollama\\ollama app.exe"; + let finishProcessTreeCancellation = () => {}; + const processTreeDrained = new Promise((resolve) => { + finishProcessTreeCancellation = resolve; + }); + const boundary = createWindowsInstallBoundary({ + userHost: priorHost, + watcherPath, + daemonPath: null, + installerInterrupt: "SIGTERM", + cancelInstaller: () => processTreeDrained, + }); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + const installation = windows.installOllamaOnWindowsHost({}, boundary.operations); + await vi.waitFor(() => expect(boundary.cancelInstaller).toHaveBeenCalledOnce()); + expect(boundary.state.events).toEqual(["snapshot", "persist", "install", "cancel-installer"]); + expect(boundary.operations.rollbackSnapshot).not.toHaveBeenCalled(); + expect(boundary.operations.preserveInterrupt).not.toHaveBeenCalled(); + + finishProcessTreeCancellation(); + await expect(installation).rejects.toThrow(PreservedWindowsInterrupt); + } finally { + restore(); + logSpy.mockRestore(); + errorSpy.mockRestore(); + } + + expect(boundary.state.events).toEqual([ + "snapshot", + "persist", + "install", + "cancel-installer", + "stop-replacement", + "restore", + "signal:SIGTERM", + ]); + expect(boundary.state.userHost).toBe(priorHost); + expect(boundary.state.watcherRunning).toBe(true); + expect(boundary.cancelInstaller).toHaveBeenCalledOnce(); + }); + + it("does not restore mutable Windows state when installer tree cancellation fails", async () => { + const boundary = createWindowsInstallBoundary({ + userHost: "127.0.0.1:11434", + watcherPath: "C:\\Users\\tester\\Ollama\\ollama app.exe", + daemonPath: null, + installerInterrupt: "SIGINT", + cancelInstaller: async () => { + throw new Error("taskkill tree termination failed"); + }, + }); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + await expect(windows.installOllamaOnWindowsHost({}, boundary.operations)).rejects.toThrow( + "taskkill tree termination failed", + ); + } finally { + restore(); + logSpy.mockRestore(); + errorSpy.mockRestore(); + } + + expect(boundary.cancelInstaller).toHaveBeenCalledOnce(); + expect(boundary.operations.rollbackSnapshot).not.toHaveBeenCalled(); + expect(boundary.operations.preserveInterrupt).not.toHaveBeenCalled(); + expect(boundary.state.events).toEqual(["snapshot", "persist", "install", "cancel-installer"]); + }); + + it("stops waiting without rollback when cancellation has no PID or child close (#10855)", async () => { + vi.useFakeTimers(); + const child = createMockChildProcess(); + const spawnProcess = vi.fn(() => child); + const boundary = createWindowsInstallBoundary({ + userHost: "127.0.0.1:11434", + watcherPath: "C:\\Users\\tester\\Ollama\\ollama app.exe", + daemonPath: null, + }); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn(), spawnProcess); + boundary.operations.startInstaller = vi.fn(() => { + boundary.state.events.push("install"); + const installer = windows.startWindowsOllamaInstaller(); + queueMicrotask(() => { + void Promise.resolve(boundary.triggerInterrupt("SIGTERM")).catch(() => {}); + }); + return installer; + }); + + try { + const installation = windows.installOllamaOnWindowsHost({}, boundary.operations); + const rejection = expect(installation).rejects.toThrow( + "Timed out while confirming Windows Ollama installer cancellation", + ); + await Promise.resolve(); + expect(child.kill).toHaveBeenCalledWith("SIGTERM"); + + await vi.advanceTimersByTimeAsync(5_000); + await rejection; + expect(spawnProcess).toHaveBeenCalledTimes(1); + expect(boundary.operations.rollbackSnapshot).not.toHaveBeenCalled(); + expect(boundary.operations.preserveInterrupt).not.toHaveBeenCalled(); + const diagnostic = errorSpy.mock.calls.map(([message]) => String(message)).join("\n"); + expect(diagnostic).toContain("Could not confirm that the Windows Ollama installer stopped"); + expect(diagnostic).toContain("did not restore the previous Windows Ollama state"); + expect(diagnostic).toContain("stop the installer and Ollama processes"); + expect(diagnostic).toContain("restore the previous User-scope OLLAMA_HOST value"); + expect(diagnostic).toContain("nemoclaw onboard"); + } finally { + restore(); + logSpy.mockRestore(); + errorSpy.mockRestore(); + vi.useRealTimers(); + } + }); + + it("reports manual recovery when install rollback fails", async () => { + const boundary = createWindowsInstallBoundary({ + userHost: "127.0.0.1:11434", + watcherPath: null, + daemonPath: null, + installedPath: "", + rollbackStatus: 1, + }); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + await windows.installOllamaOnWindowsHost({}, boundary.operations); + } finally { + restore(); + logSpy.mockRestore(); + } + + const diagnostic = errorSpy.mock.calls.map(([message]) => String(message)).join("\n"); + errorSpy.mockRestore(); + expect(diagnostic).toContain("Failed to restore the previous Windows Ollama state"); + expect(diagnostic).toContain("restore your previous User-scope OLLAMA_HOST value"); + }); + + it("stops a final unready replacement before restoring the prior Windows state", () => { + const priorHost = "127.0.0.1:11434"; + const watcherPath = "C:\\Users\\tester\\Ollama\\ollama app.exe"; + const daemonPath = "C:\\Users\\tester\\Ollama\\ollama.exe"; + const boundary = createWindowsSetupBoundary({ + userHost: priorHost, + watcherPath, + daemonPath, + launchStatuses: [1, 1, 0], + readinessResults: [false], + }); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + expect( + windows.setupWindowsOllamaLoopbackBinding( + { installedPath: daemonPath }, + boundary.operations, + ), + ).toEqual({ ok: false, reason: "readiness" }); + } finally { + restore(); + logSpy.mockRestore(); + errorSpy.mockRestore(); + } + + const finalLaunch = boundary.state.events.lastIndexOf("launch:path"); + const finalStop = boundary.state.events.lastIndexOf("stop-replacement"); + const rollback = boundary.state.events.lastIndexOf("restore"); + expect(finalLaunch).toBeLessThan(finalStop); + expect(finalStop).toBeLessThan(rollback); + expect(boundary.state.userHost).toBe(priorHost); + expect(boundary.state.watcherRunning).toBe(true); + expect(boundary.state.daemonRunning).toBe(false); + }); + + it.each(["SIGINT", "SIGTERM"] as const)( + "restores the prior Windows state before preserving %s", + (signal) => { + const priorHost = "127.0.0.1:11434"; + const watcherPath = "C:\\Users\\tester\\Ollama\\ollama app.exe"; + const daemonPath = "C:\\Users\\tester\\Ollama\\ollama.exe"; + const boundary = createWindowsSetupBoundary({ + userHost: priorHost, + watcherPath, + daemonPath, + launchStatuses: [0], + interruptSignal: signal, + }); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + const { windows, restore } = loadWindowsOllamaWithMocks(vi.fn(), vi.fn()); + + try { + expect(() => + windows.setupWindowsOllamaLoopbackBinding( + { installedPath: daemonPath }, + boundary.operations, + ), + ).toThrow(PreservedWindowsInterrupt); + } finally { + restore(); + logSpy.mockRestore(); + errorSpy.mockRestore(); + } + + expect(boundary.state.events).toEqual([ + "snapshot", + "persist", + "stop-existing", + "launch:watcher", + "stop-replacement", + "restore", + `signal:${signal}`, + ]); + expect(boundary.state.userHost).toBe(priorHost); + expect(boundary.state.watcherRunning).toBe(true); + expect(boundary.state.daemonRunning).toBe(false); + expect(boundary.operations.preserveInterrupt).toHaveBeenCalledWith(signal); + }, + ); + it("isolates Docker credentials while waiting for the Windows-host daemon", () => { const run = vi.fn(); const cleanup = vi.fn(() => ({ ok: true as const })); const runCapture = vi.fn( (command: string | string[], options?: { env?: NodeJS.ProcessEnv }) => { - if (commandText(command).includes("Get-NetTCPConnection")) return "127.0.0.1"; - return Array.isArray(command) && - command[0] === "docker" && - options?.env?.DOCKER_CONFIG === "/tmp/credential-free-docker" - ? command.includes(REBINDING_PROBE_HOST_HEADER) - ? "403" - : command.at(-1) === WINDOWS_OLLAMA_TAGS_URL - ? JSON.stringify({ models: [] }) - : "" - : ""; + return commandText(command).includes("Get-NetTCPConnection") + ? "127.0.0.1" + : !Array.isArray(command) || options?.env?.DOCKER_CONFIG !== "/tmp/credential-free-docker" + ? "" + : command.includes(REBINDING_PROBE_HOST_HEADER) + ? "403" + : isDockerTagsRequest(command) + ? JSON.stringify({ models: [] }) + : ""; }, ); const localInference = require(LOCAL_INFERENCE_PATH); @@ -377,7 +1075,6 @@ describe("Windows Ollama helper", () => { expect(runCapture).toHaveBeenCalledWith( expect.arrayContaining(["docker", "run", "--rm", WINDOWS_OLLAMA_TAGS_URL]), expect.objectContaining({ - ignoreError: true, env: expect.objectContaining({ DOCKER_CONFIG: "/tmp/credential-free-docker", DOCKER_CONTEXT: "default", @@ -391,33 +1088,4 @@ describe("Windows Ollama helper", () => { logSpy.mockRestore(); } }); - - it("prints both Docker reachability and Host-validation timeout diagnostics", () => { - const run = vi.fn(); - const runCapture = vi.fn(); - const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {}); - const { windows, restore } = loadWindowsOllamaWithMocks(run, runCapture); - let diagnostics: string[] = []; - - try { - windows.printWindowsOllamaTimeoutDiagnostics(); - diagnostics = errorSpy.mock.calls.map(([message]) => String(message)); - } finally { - restore(); - errorSpy.mockRestore(); - } - - expect(diagnostics).toContainEqual( - expect.stringContaining( - `docker run --rm ${require(LOCAL_INFERENCE_PATH).CONTAINER_REACHABILITY_IMAGE} -sf`, - ), - ); - expect(diagnostics).toContainEqual( - expect.stringContaining( - `docker run --rm ${require(LOCAL_INFERENCE_PATH).CONTAINER_REACHABILITY_IMAGE} -sS --output /dev/null --write-out %{http_code}`, - ), - ); - expect(diagnostics).toContainEqual(expect.stringContaining(REBINDING_PROBE_HOST_HEADER)); - expect(diagnostics).toContainEqual(expect.stringContaining("Expected output: 403")); - }); }); diff --git a/src/lib/inference/ollama/windows.ts b/src/lib/inference/ollama/windows.ts index 026582ed499..fd6429ac19b 100644 --- a/src/lib/inference/ollama/windows.ts +++ b/src/lib/inference/ollama/windows.ts @@ -1,4 +1,3 @@ -// @ts-nocheck // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 // @@ -32,41 +31,184 @@ function psSingleQuote(value: string): string { return `'${String(value).replace(/'/g, "''")}'`; } -// Pre-set OLLAMA_HOST in both User scope (persists across logins) and the -// current PowerShell session (inherited by the installer's auto-spawned -// ollama_app + daemon) so the new daemon stays on Windows loopback. Ollama -// enables its Host-header validation only for loopback listeners, which is -// required to reject same-host DNS-rebinding requests. +type WindowsOllamaInstallerProcess = { + completion: Promise; + cancelAndWait: () => Promise; +}; + +const WINDOWS_INSTALLER_PID_SENTINEL = "__NEMOCLAW_WINDOWS_INSTALLER_PID__:"; +// A Windows DWORD PID has at most 10 digits; allow CRLF after it. +const WINDOWS_INSTALLER_PID_LINE_MAX_BYTES = Buffer.byteLength(WINDOWS_INSTALLER_PID_SENTINEL) + 12; +const WINDOWS_INSTALLER_CANCELLATION_TIMEOUT_MS = 5_000; + +function reportUnconfirmedWindowsInstallerCancellation(): void { + console.error(" Could not confirm that the Windows Ollama installer stopped."); + console.error( + " NemoClaw did not restore the previous Windows Ollama state because the installer may still change it.", + ); + console.error( + " In Windows PowerShell, stop the installer and Ollama processes, restore the previous User-scope OLLAMA_HOST value, then retry:", + ); + console.error(" nemoclaw onboard"); +} + +function waitForWindowsInstallerCancellation(operation: Promise): Promise { + let timer: NodeJS.Timeout | undefined; + const deadline = new Promise((_resolve, reject) => { + timer = setTimeout(() => { + reject(new Error("Timed out while confirming Windows Ollama installer cancellation")); + }, WINDOWS_INSTALLER_CANCELLATION_TIMEOUT_MS); + timer.unref(); + }); + return Promise.race([operation, deadline]).finally(() => { + if (timer) clearTimeout(timer); + }); +} + +function parseWindowsProcessId(value: string): number | null { + if (!/^[1-9][0-9]*$/.test(value)) return null; + const pid = Number(value); + return Number.isSafeInteger(pid) && pid <= 0xffff_ffff ? pid : null; +} + +function terminateWindowsProcessTree(pid: number): Promise { + return new Promise((resolve, reject) => { + const taskkill = spawn("taskkill.exe", ["/PID", String(pid), "/T", "/F"], { + stdio: "ignore", + }); + taskkill.once("error", (error: NodeJS.ErrnoException) => { + reject( + new Error( + `Failed to start taskkill.exe for Windows installer PID ${pid}: ${error.message}`, + ), + ); + }); + taskkill.once("close", (code: number | null) => { + if (code === 0) resolve(); + else + reject( + new Error(`taskkill.exe failed for Windows installer PID ${pid} (exit ${String(code)})`), + ); + }); + }); +} + +// Pre-set OLLAMA_HOST in the current PowerShell session so the installer's +// auto-spawned ollama_app + daemon inherit the loopback binding. The enclosing +// mutation transaction owns the separate User-scope write and its rollback. +// Ollama enables its Host-header validation only for loopback listeners, which +// is required to reject same-host DNS-rebinding requests. // Don't use stdio:inherit here. When powershell.exe is spawned through // WSL interop, its stdout looks like a pipe (not a console), so PowerShell // holds output in an internal buffer and the user sees long silent gaps. // Reading the pipe from Node and re-writing to our own TTY shows progress // as soon as PowerShell flushes a chunk. -async function installOllamaOnWindowsHost(): Promise<{ ok: boolean; path: string }> { - console.log(" Installing Ollama on Windows host..."); - console.log(" This can take several minutes. Output may pause silently"); - if (!persistOllamaLoopbackHostEnvVar()) { - console.error(" Failed to persist the Windows Ollama loopback binding."); - return { ok: false, path: "" }; - } - await new Promise((resolve) => { - const child = spawn( - "powershell.exe", - [ - "-Command", - `$env:OLLAMA_HOST='${OLLAMA_LOOPBACK_HOST}'; irm https://ollama.com/install.ps1 | iex`, - ], - { stdio: ["ignore", "pipe", "pipe"] }, - ); - child.stdout?.on("data", (chunk: Buffer) => process.stdout.write(chunk)); +function buildWindowsOllamaInstallerCommand(): string { + return ( + `[Console]::OutputEncoding = [System.Text.Encoding]::UTF8; [Console]::Out.WriteLine('${WINDOWS_INSTALLER_PID_SENTINEL}' + $PID); [Console]::Out.Flush(); ` + + `$env:OLLAMA_HOST='${OLLAMA_LOOPBACK_HOST}'; irm https://ollama.com/install.ps1 | iex` + ); +} + +function startWindowsOllamaInstaller(): WindowsOllamaInstallerProcess { + const child = spawn("powershell.exe", ["-Command", buildWindowsOllamaInstallerCommand()], { + stdio: ["ignore", "pipe", "pipe"], + }); + let windowsPid: number | null = null; + let stdoutPrefix = Buffer.alloc(0); + let awaitingPidLine = true; + let resolveWindowsPid: (pid: number | null) => void = () => {}; + let rejectCompletion: (error: unknown) => void = () => {}; + const windowsPidReady = new Promise((resolve) => { + resolveWindowsPid = resolve; + }); + const settlePidLine = (line: Buffer): boolean => { + awaitingPidLine = false; + const text = line.toString("utf8").replace(/\r?\n$/, ""); + const candidate = text.slice(WINDOWS_INSTALLER_PID_SENTINEL.length); + const parsedPid = text.startsWith(WINDOWS_INSTALLER_PID_SENTINEL) + ? parseWindowsProcessId(candidate) + : null; + windowsPid = parsedPid; + resolveWindowsPid(parsedPid); + return parsedPid !== null; + }; + const streamInstallerStdout = (chunk: Buffer) => { + if (!awaitingPidLine) { + process.stdout.write(chunk); + return; + } + const remainingBytes = WINDOWS_INSTALLER_PID_LINE_MAX_BYTES - stdoutPrefix.length; + const inspected = chunk.subarray(0, remainingBytes); + const newlineIndex = inspected.indexOf(0x0a); + if (newlineIndex >= 0) { + const line = Buffer.concat([stdoutPrefix, inspected.subarray(0, newlineIndex + 1)]); + if (!settlePidLine(line)) process.stdout.write(line); + process.stdout.write(chunk.subarray(newlineIndex + 1)); + stdoutPrefix = Buffer.alloc(0); + return; + } + if (chunk.length >= remainingBytes) { + awaitingPidLine = false; + resolveWindowsPid(null); + process.stdout.write(stdoutPrefix); + process.stdout.write(chunk); + stdoutPrefix = Buffer.alloc(0); + return; + } + stdoutPrefix = Buffer.concat([stdoutPrefix, chunk]); + }; + const completion = new Promise((resolve, reject) => { + rejectCompletion = reject; + child.stdout?.on("data", streamInstallerStdout); child.stderr?.on("data", (chunk: Buffer) => process.stderr.write(chunk)); - child.on("close", () => resolve()); + child.on("close", () => { + if (awaitingPidLine) { + if (!settlePidLine(stdoutPrefix)) process.stdout.write(stdoutPrefix); + stdoutPrefix = Buffer.alloc(0); + } + resolve(); + }); child.on("error", (err: NodeJS.ErrnoException) => { console.error(` Failed to spawn powershell.exe: ${err.message}`); + resolveWindowsPid(null); resolve(); }); }); - const installedPath = runCapture( + let cancellation: Promise | null = null; + return { + completion, + cancelAndWait: () => { + cancellation ??= waitForWindowsInstallerCancellation( + (async () => { + if (windowsPid === null) { + try { + child.kill("SIGTERM"); + } catch { + // The wrapper has already exited, so its close event remains authoritative. + } + } + const pid = await windowsPidReady; + if (pid !== null) { + await terminateWindowsProcessTree(pid); + } + await completion; + })(), + ).catch((error: unknown) => { + reportUnconfirmedWindowsInstallerCancellation(); + // Unblock the install path without treating the process as stopped. + // The rejected completion reaches the mutation owner, which leaves the + // snapshot untouched because cancellation was not confirmed. + rejectCompletion(error); + throw error; + }); + return cancellation; + }, + }; +} + +function resolveWindowsOllamaInstalledPath(): string { + return runCapture( [ "powershell.exe", "-Command", @@ -74,24 +216,96 @@ async function installOllamaOnWindowsHost(): Promise<{ ok: boolean; path: string ], { ignoreError: true }, ).trim(); - if (!installedPath) { - return { ok: false, path: "" }; - } - console.log(` ✓ Installed: ${installedPath}`); - return { ok: true, path: installedPath }; } -// Capture the watcher path so we can relaunch from the same exe after kill, -// preserving the tray icon and the watcher's auto-restart behavior. -function captureWindowsOllamaWatcherPath(): string { - return runCapture( +type WindowsOllamaHostSnapshot = { + userHost: string | null; + watcherPath: string | null; + daemonPath: string | null; +}; + +function optionalSnapshotPath(value: unknown): string | null | undefined { + if (value === null) return null; + if (typeof value !== "string") return undefined; + return value.trim() || null; +} + +// Capture every state item that setup mutates before changing the User-scope +// binding or stopping processes. A failed snapshot leaves the host untouched. +function captureWindowsOllamaHostSnapshot(): WindowsOllamaHostSnapshot | null { + const result = run( [ "powershell.exe", "-Command", - "Get-Process 'ollama app' -EA SilentlyContinue | Select-Object -First 1 -ExpandProperty Path", + "$userHost = [Environment]::GetEnvironmentVariable('OLLAMA_HOST','User'); " + + "$watcherPath = Get-Process 'ollama app' -EA SilentlyContinue | Select-Object -First 1 -ExpandProperty Path; " + + "$daemonPath = Get-Process ollama -EA SilentlyContinue | Select-Object -First 1 -ExpandProperty Path; " + + "[PSCustomObject]@{userHost=$userHost;watcherPath=$watcherPath;daemonPath=$daemonPath} | ConvertTo-Json -Compress", ], - { ignoreError: true }, - ).trim(); + { ignoreError: true, suppressOutput: true }, + ); + if (result.error || result.status !== 0) return null; + try { + const parsed = JSON.parse(String(result.stdout || "")) as Record | null; + if (!parsed || (parsed.userHost !== null && typeof parsed.userHost !== "string")) return null; + const watcherPath = optionalSnapshotPath(parsed.watcherPath); + const daemonPath = optionalSnapshotPath(parsed.daemonPath); + if (watcherPath === undefined || daemonPath === undefined) return null; + return { userHost: parsed.userHost, watcherPath, daemonPath }; + } catch { + return null; + } +} + +const WINDOWS_OLLAMA_RESTORE_HOST_ENV = "NEMOCLAW_OLLAMA_RESTORE_HOST"; +const WINDOWS_OLLAMA_RESTORE_HOST_PRESENT_ENV = "NEMOCLAW_OLLAMA_RESTORE_HOST_PRESENT"; +const WINDOWS_OLLAMA_RESTORE_WATCHER_ENV = "NEMOCLAW_OLLAMA_RESTORE_WATCHER"; +const WINDOWS_OLLAMA_RESTORE_DAEMON_ENV = "NEMOCLAW_OLLAMA_RESTORE_DAEMON"; + +function runWindowsOllamaStateScript(script: string, env?: NodeJS.ProcessEnv): boolean { + const result = run(["powershell.exe", "-Command", `$ErrorActionPreference='Stop'; ${script}`], { + env, + ignoreError: true, + suppressOutput: true, + }); + return !result.error && result.status === 0; +} + +function buildWindowsOllamaRestoreScript(): string { + return [ + `$previousHostPresent = $env:${WINDOWS_OLLAMA_RESTORE_HOST_PRESENT_ENV}`, + `$previousHost = $env:${WINDOWS_OLLAMA_RESTORE_HOST_ENV}`, + `$previousWatcher = $env:${WINDOWS_OLLAMA_RESTORE_WATCHER_ENV}`, + `$previousDaemon = $env:${WINDOWS_OLLAMA_RESTORE_DAEMON_ENV}`, + `Remove-Item Env:${WINDOWS_OLLAMA_RESTORE_HOST_PRESENT_ENV} -EA SilentlyContinue`, + `Remove-Item Env:${WINDOWS_OLLAMA_RESTORE_HOST_ENV} -EA SilentlyContinue`, + `Remove-Item Env:${WINDOWS_OLLAMA_RESTORE_WATCHER_ENV} -EA SilentlyContinue`, + `Remove-Item Env:${WINDOWS_OLLAMA_RESTORE_DAEMON_ENV} -EA SilentlyContinue`, + "if ($previousHostPresent -ne '1') { $previousHost = $null }", + "Get-Process 'ollama app' -EA SilentlyContinue | Stop-Process -Force", + "Get-Process ollama -EA SilentlyContinue | Stop-Process -Force", + "[Environment]::SetEnvironmentVariable('OLLAMA_HOST',$previousHost,'User')", + "$env:OLLAMA_HOST = $previousHost", + "if ($previousWatcher) { Start-Process -FilePath $previousWatcher -WindowStyle Hidden -ErrorAction Stop } " + + "elseif ($previousDaemon) { Start-Process -FilePath $previousDaemon -ArgumentList 'serve' -WindowStyle Hidden -ErrorAction Stop }", + ].join("; "); +} + +function rollbackWindowsOllamaHostSnapshot(snapshot: WindowsOllamaHostSnapshot): boolean { + return runWindowsOllamaStateScript(buildWindowsOllamaRestoreScript(), { + [WINDOWS_OLLAMA_RESTORE_HOST_PRESENT_ENV]: snapshot.userHost === null ? "0" : "1", + [WINDOWS_OLLAMA_RESTORE_HOST_ENV]: snapshot.userHost ?? "", + [WINDOWS_OLLAMA_RESTORE_WATCHER_ENV]: snapshot.watcherPath ?? "", + [WINDOWS_OLLAMA_RESTORE_DAEMON_ENV]: snapshot.daemonPath ?? "", + }); +} + +function reportWindowsOllamaRollbackFailure(): void { + console.error(" Failed to restore the previous Windows Ollama state."); + console.error( + " In Windows PowerShell, stop Ollama, restore your previous User-scope OLLAMA_HOST " + + "value, and relaunch the previous Ollama app or daemon.", + ); } // User-scope so the next login-time tray launch remains loopback-only without @@ -149,19 +363,48 @@ function awaitWindowsOllamaReady( return false; } +type WindowsOllamaLaunchAttempt = { + kind: "watcher" | "installed" | "path"; + label: string; + script: string; +}; + +type WindowsOllamaLaunchOperations = { + runAttempt: (attempt: WindowsOllamaLaunchAttempt) => { + status: number | null; + stderr?: string; + error?: Error; + }; + awaitReady: () => boolean; + stopProcesses: () => void; + wait: (seconds: number) => void; +}; + +const WINDOWS_OLLAMA_LAUNCH_OPERATIONS: WindowsOllamaLaunchOperations = { + runAttempt: (attempt) => + run(["powershell.exe", "-Command", attempt.script], { + ignoreError: true, + suppressOutput: true, + }), + awaitReady: awaitWindowsOllamaReady, + stopProcesses: killWindowsOllamaProcesses, + wait: sleep, +}; + // Relaunch via the watcher path when available so the tray icon and the // watcher's auto-restart survive; fall back through the verified installed // path and finally refreshed PATH because stale watcher paths are possible. function launchAndAwaitWindowsOllama( - opts: { watcherPath?: string; installedPath?: string; delay?: (seconds: number) => void } = {}, + opts: { watcherPath?: string; installedPath?: string } = {}, + operations: WindowsOllamaLaunchOperations = WINDOWS_OLLAMA_LAUNCH_OPERATIONS, ): boolean { console.log(" Starting Ollama on Windows host via WSL interop..."); - const delay = opts.delay ?? sleep; const watcherPath = typeof opts.watcherPath === "string" ? opts.watcherPath.trim() : ""; const installedPath = typeof opts.installedPath === "string" ? opts.installedPath.trim() : ""; - const launchAttempts: Array<{ label: string; script: string }> = []; + const launchAttempts: WindowsOllamaLaunchAttempt[] = []; if (watcherPath) { launchAttempts.push({ + kind: "watcher", label: "Ollama tray app", script: `$env:OLLAMA_HOST='127.0.0.1:11434'; Start-Process -FilePath ${psSingleQuote(watcherPath)} ` + @@ -170,6 +413,7 @@ function launchAndAwaitWindowsOllama( } if (installedPath) { launchAttempts.push({ + kind: "installed", label: "verified ollama.exe", script: `$env:OLLAMA_HOST='127.0.0.1:11434'; Start-Process -FilePath ${psSingleQuote(installedPath)} ` + @@ -177,6 +421,7 @@ function launchAndAwaitWindowsOllama( }); } launchAttempts.push({ + kind: "path", label: "refreshed Windows PATH", script: "$env:PATH = [Environment]::GetEnvironmentVariable('PATH','Machine') + ';' + [Environment]::GetEnvironmentVariable('PATH','User'); " + @@ -185,11 +430,8 @@ function launchAndAwaitWindowsOllama( for (let i = 0; i < launchAttempts.length; i++) { const attempt = launchAttempts[i]; - const result = run(["powershell.exe", "-Command", attempt.script], { - ignoreError: true, - suppressOutput: true, - }); - if (result.status === 0 && awaitWindowsOllamaReady({ delay })) { + const result = operations.runAttempt(attempt); + if (result.status === 0 && operations.awaitReady()) { return true; } @@ -201,13 +443,233 @@ function launchAndAwaitWindowsOllama( : error || `exit ${result.status}${stderr ? `: ${stderr}` : ""}`; console.error(` PowerShell launch via ${attempt.label} failed: ${detail}`); if (i < launchAttempts.length - 1) { - killWindowsOllamaProcesses(); - delay(1); + operations.stopProcesses(); + operations.wait(1); } } return false; } +type WindowsOllamaInterruptSignal = "SIGINT" | "SIGTERM"; + +type WindowsOllamaSetupOperations = { + captureSnapshot: () => WindowsOllamaHostSnapshot | null; + persistBinding: () => boolean; + stopProcesses: () => void; + wait: (seconds: number) => void; + launchOperations: WindowsOllamaLaunchOperations; + rollbackSnapshot: (snapshot: WindowsOllamaHostSnapshot) => boolean; + registerInterruptHandler: ( + handler: (signal: WindowsOllamaInterruptSignal) => void | Promise, + ) => () => void; + preserveInterrupt: (signal: WindowsOllamaInterruptSignal) => void; +}; + +type WindowsOllamaInstallOperations = WindowsOllamaSetupOperations & { + startInstaller: () => WindowsOllamaInstallerProcess; + resolveInstalledPath: () => string; + awaitReady: () => boolean; +}; + +export type WindowsOllamaMutationSession = { + commit: () => void; + rollback: () => void | Promise; +}; + +export type WindowsOllamaFailureReason = "binding" | "install" | "readiness" | "snapshot"; + +export type WindowsOllamaInstallResult = + | { ok: false; path: string; reason: WindowsOllamaFailureReason } + | ({ ok: true; path: string } & WindowsOllamaMutationSession); + +export type WindowsOllamaSetupResult = + | { ok: false; reason: Exclude } + | ({ ok: true } & WindowsOllamaMutationSession); + +function registerWindowsOllamaInterruptHandler( + handler: (signal: WindowsOllamaInterruptSignal) => void | Promise, +): () => void { + const reportFailure = (error: unknown) => { + console.error(` Windows Ollama interrupt rollback failed: ${String(error)}`); + process.exitCode = 1; + }; + const runHandler = (signal: WindowsOllamaInterruptSignal) => { + try { + Promise.resolve(handler(signal)).catch(reportFailure); + } catch (error) { + reportFailure(error); + } + }; + const onSigint = () => runHandler("SIGINT"); + const onSigterm = () => runHandler("SIGTERM"); + process.once("SIGINT", onSigint); + process.once("SIGTERM", onSigterm); + return () => { + process.off("SIGINT", onSigint); + process.off("SIGTERM", onSigterm); + }; +} + +const WINDOWS_OLLAMA_SETUP_OPERATIONS: WindowsOllamaSetupOperations = { + captureSnapshot: captureWindowsOllamaHostSnapshot, + persistBinding: persistOllamaLoopbackHostEnvVar, + stopProcesses: killWindowsOllamaProcesses, + wait: sleep, + launchOperations: WINDOWS_OLLAMA_LAUNCH_OPERATIONS, + rollbackSnapshot: rollbackWindowsOllamaHostSnapshot, + registerInterruptHandler: registerWindowsOllamaInterruptHandler, + preserveInterrupt: (signal) => { + process.kill(process.pid, signal); + }, +}; + +const WINDOWS_OLLAMA_INSTALL_OPERATIONS: WindowsOllamaInstallOperations = { + ...WINDOWS_OLLAMA_SETUP_OPERATIONS, + startInstaller: startWindowsOllamaInstaller, + resolveInstalledPath: resolveWindowsOllamaInstalledPath, + awaitReady: awaitWindowsOllamaReady, +}; + +function rollbackWindowsOllamaSetup( + snapshot: WindowsOllamaHostSnapshot, + operations: WindowsOllamaSetupOperations, +): void { + if (!operations.rollbackSnapshot(snapshot)) reportWindowsOllamaRollbackFailure(); +} + +function beginWindowsOllamaMutation( + snapshot: WindowsOllamaHostSnapshot, + operations: WindowsOllamaSetupOperations, +) { + let active = true; + let mutated = false; + let cancelActiveOperation: () => void | Promise = () => {}; + let removeInterruptHandler = () => {}; + let rollbackPromise: Promise | null = null; + let interruptPromise: Promise | null = null; + const commit = () => { + if (!active) return; + active = false; + cancelActiveOperation = () => {}; + removeInterruptHandler(); + }; + const restore = () => { + if (mutated) rollbackWindowsOllamaSetup(snapshot, operations); + }; + const rollback = (): void | Promise => { + if (rollbackPromise) return rollbackPromise; + if (!active) return; + active = false; + removeInterruptHandler(); + const cancel = cancelActiveOperation; + cancelActiveOperation = () => {}; + const cancellation = cancel(); + if (cancellation && typeof cancellation.then === "function") { + rollbackPromise = Promise.resolve(cancellation).then(restore); + return rollbackPromise; + } + restore(); + }; + removeInterruptHandler = operations.registerInterruptHandler((signal) => { + const result = rollback(); + if (result && typeof result.then === "function") { + interruptPromise = result.then(() => operations.preserveInterrupt(signal)); + return interruptPromise; + } + operations.preserveInterrupt(signal); + }); + return { + commit, + markMutated: () => { + mutated = true; + }, + rollback, + setInterruptCancellation: (cancel: () => void | Promise) => { + if (active) cancelActiveOperation = cancel; + }, + waitForInterrupt: async () => { + if (interruptPromise) await interruptPromise; + }, + }; +} + +function applyWindowsOllamaBinding( + opts: { announceStop?: boolean; installedPath?: string } = {}, + snapshot: WindowsOllamaHostSnapshot, + operations: WindowsOllamaSetupOperations, + markMutated: () => void, +): { ok: true } | { ok: false; reason: "binding" | "readiness" } { + markMutated(); + if (!operations.persistBinding()) { + console.error(" Could not persist the Windows Ollama host binding."); + return { ok: false, reason: "binding" }; + } + if (opts.announceStop) { + console.log(" Stopping existing Ollama on Windows host..."); + } + operations.stopProcesses(); + operations.wait(1); + return launchAndAwaitWindowsOllama( + { + watcherPath: snapshot.watcherPath || undefined, + installedPath: opts.installedPath, + }, + operations.launchOperations, + ) + ? { ok: true } + : { ok: false, reason: "readiness" }; +} + +async function installOllamaOnWindowsHost( + opts: { beforeRestart?: () => void } = {}, + operations: WindowsOllamaInstallOperations = WINDOWS_OLLAMA_INSTALL_OPERATIONS, +): Promise { + const snapshot = operations.captureSnapshot(); + if (!snapshot) { + console.error(" Could not capture the existing Windows Ollama state; leaving it unchanged."); + return { ok: false, path: "", reason: "snapshot" }; + } + const mutation = beginWindowsOllamaMutation(snapshot, operations); + console.log(" Installing Ollama on Windows host..."); + console.log(" This can take several minutes. Output may pause silently"); + try { + mutation.markMutated(); + if (!operations.persistBinding()) { + await mutation.rollback(); + return { ok: false, path: "", reason: "binding" }; + } + const installer = operations.startInstaller(); + mutation.setInterruptCancellation(installer.cancelAndWait); + await installer.completion; + mutation.setInterruptCancellation(() => {}); + await mutation.waitForInterrupt(); + const installedPath = operations.resolveInstalledPath(); + if (!installedPath) { + await mutation.rollback(); + return { ok: false, path: "", reason: "install" }; + } + console.log(` ✓ Installed: ${installedPath}`); + if (!operations.awaitReady()) { + console.log(" Installer did not leave a reachable Ollama daemon; restarting it..."); + opts.beforeRestart?.(); + const setupResult = applyWindowsOllamaBinding( + { installedPath }, + snapshot, + operations, + mutation.markMutated, + ); + if (!setupResult.ok) { + await mutation.rollback(); + return { ok: false, path: installedPath, reason: setupResult.reason }; + } + } + return { ok: true, path: installedPath, commit: mutation.commit, rollback: mutation.rollback }; + } catch (error) { + await mutation.rollback(); + throw error; + } +} + // Used by start and restart paths to force a loopback-only binding on an // already installed Ollama. Fresh install fallback passes installedPath to // avoid relying on a newly-mutated Windows PATH from this process. @@ -217,32 +679,42 @@ function setupWindowsOllamaLoopbackBinding( installedPath?: string; delay?: (seconds: number) => void; } = {}, -): boolean { - const delay = opts.delay ?? sleep; - const watcherPath = captureWindowsOllamaWatcherPath(); - if (!persistOllamaLoopbackHostEnvVar()) { - console.error(" Failed to persist the Windows Ollama loopback binding."); - return false; - } - if (opts.announceStop) { - console.log(" Stopping existing Ollama on Windows host..."); + operations: WindowsOllamaSetupOperations = WINDOWS_OLLAMA_SETUP_OPERATIONS, +): WindowsOllamaSetupResult { + const delay = opts.delay; + const effectiveOperations = delay + ? { + ...operations, + wait: delay, + launchOperations: { + ...operations.launchOperations, + awaitReady: () => awaitWindowsOllamaReady({ delay }), + wait: delay, + }, + } + : operations; + const snapshot = effectiveOperations.captureSnapshot(); + if (!snapshot) { + console.error(" Could not capture the existing Windows Ollama state; leaving it unchanged."); + return { ok: false, reason: "snapshot" }; } - killWindowsOllamaProcesses(); - delay(1); - const launched = launchAndAwaitWindowsOllama({ - watcherPath: watcherPath || undefined, - installedPath: opts.installedPath, - delay, - }); - if (!launched) { - console.error( - ` The Windows user OLLAMA_HOST=${OLLAMA_LOOPBACK_HOST} setting was persisted, and Ollama may now be stopped.`, - ); - console.error( - " Resolve the launch or probe failure below, then rerun `nemoclaw onboard` to retry the bounded restart.", + const mutation = beginWindowsOllamaMutation(snapshot, effectiveOperations); + try { + const setupResult = applyWindowsOllamaBinding( + opts, + snapshot, + effectiveOperations, + mutation.markMutated, ); + if (!setupResult.ok) { + mutation.rollback(); + return setupResult; + } + return { ok: true, commit: mutation.commit, rollback: mutation.rollback }; + } catch (error) { + mutation.rollback(); + throw error; } - return launched; } function printWindowsOllamaTimeoutDiagnostics(): void { @@ -257,12 +729,28 @@ function printWindowsOllamaTimeoutDiagnostics(): void { console.error(` docker ${getWindowsHostOllamaDockerReachabilityArgs().join(" ")}`); console.error(` docker ${getWindowsHostOllamaDockerHostValidationArgs().join(" ")}`); console.error(" Expected output: 403 (other values mean Host validation is disabled)."); + console.error(" After correcting the Windows process or listener, retry:"); + console.error(" nemoclaw onboard"); + console.error( + " NemoClaw repeats the reachability check with an isolated Docker client configuration and removes that temporary configuration afterward.", + ); +} + +function printWindowsOllamaSnapshotDiagnostics(): void { + console.error(" NemoClaw could not inspect the existing Windows Ollama state."); + console.error( + " In Windows PowerShell, verify that the current user can query the User-scope OLLAMA_HOST value and existing Ollama processes, then retry:", + ); + console.error(" nemoclaw onboard"); } module.exports = { installOllamaOnWindowsHost, awaitWindowsOllamaReady, + buildWindowsOllamaInstallerCommand, + startWindowsOllamaInstaller, setupWindowsOllamaLoopbackBinding, sleep, + printWindowsOllamaSnapshotDiagnostics, printWindowsOllamaTimeoutDiagnostics, }; diff --git a/src/lib/inference/sandbox-facing-ollama-model.test.ts b/src/lib/inference/sandbox-facing-ollama-model.test.ts index fbb4c1530c0..cf50f1b58f8 100644 --- a/src/lib/inference/sandbox-facing-ollama-model.test.ts +++ b/src/lib/inference/sandbox-facing-ollama-model.test.ts @@ -4,6 +4,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { OLLAMA_PORT, OLLAMA_PROXY_PORT } from "../core/ports"; import { + describeModelInventory, getOllamaContainerPort, getLocalProviderContainerReachabilityCheck, ollamaInventoryContainsModel, @@ -100,11 +101,20 @@ describe("sandbox-facing Ollama model validation", () => { describe("Ollama model inventory", () => { it("queries the given daemon for its inventory", () => { - const capture = vi.fn((_command: readonly string[]) => tagsBody("llama3.2:1b")); + const capture = vi.fn( + ( + _command: readonly string[], + _options?: { ignoreError?: boolean; env?: NodeJS.ProcessEnv; timeout?: number }, + ) => tagsBody("llama3.2:1b"), + ); - const inventory = probeOllamaEndpointInventory("127.0.0.1", capture); + const inventory = probeOllamaEndpointInventory("127.0.0.1", capture, 1_200); expect(commandUrl(capture.mock.calls[0][0])).toBe(`http://127.0.0.1:${OLLAMA_PORT}/api/tags`); + expect(capture.mock.calls[0][0]).toEqual( + expect.arrayContaining(["--connect-timeout", "1.2", "--max-time", "1.2"]), + ); + expect(capture.mock.calls[0][1]).toMatchObject({ ignoreError: true, timeout: 1_200 }); expect(inventory).toEqual(["llama3.2:1b"]); expect(ollamaInventoryContainsModel(inventory ?? [], "gemma4:26b")).toBe(false); }); @@ -121,4 +131,18 @@ describe("Ollama model inventory", () => { it("keeps a valid empty inventory authoritative", () => { expect(probeOllamaEndpointInventory("127.0.0.1", () => tagsBody())).toEqual([]); }); + + it("removes directional controls from inventory labels and validation messages", () => { + const controls = + "\u061c\u200e\u200f\u2028\u2029\u202a\u202b\u202c\u202d\u202e\u2066\u2067\u2068\u2069"; + const model = `qwen3.6${controls}:35b`; + + expect(describeModelInventory([model])).toBe("qwen3.6:35b"); + + setResolvedOllamaHost(OLLAMA_HOST_DOCKER_INTERNAL); + const result = validateSandboxFacingOllamaModel("llama3.2:1b", () => tagsBody(model)); + expect(result.ok).toBe(false); + expect(result.message).toContain("reported models: qwen3.6:35b"); + expect(result.message).not.toMatch(/[\u061c\u200e\u200f\u2028-\u202e\u2066-\u2069]/u); + }); }); diff --git a/src/lib/onboard.ts b/src/lib/onboard.ts index 2d19488a13e..e07c1338632 100644 --- a/src/lib/onboard.ts +++ b/src/lib/onboard.ts @@ -227,7 +227,7 @@ const { } = require("./inference/ollama/proxy"); const { installOllamaOnWindowsHost, - awaitWindowsOllamaReady, + printWindowsOllamaSnapshotDiagnostics, setupWindowsOllamaLoopbackBinding, printWindowsOllamaTimeoutDiagnostics, } = require("./inference/ollama/windows"); @@ -987,7 +987,7 @@ const { getLocalProviderBaseUrl, selectAndValidateOllamaModel, installOllamaOnWindowsHost, - awaitWindowsOllamaReady, + printWindowsOllamaSnapshotDiagnostics, setupWindowsOllamaLoopbackBinding, printWindowsOllamaTimeoutDiagnostics, resetOllamaHostCache, @@ -1673,7 +1673,7 @@ async function selectAndValidateOllamaModel( "non-interactive mode cannot prompt for confirmation. " + "Re-run with --yes / -y (or NEMOCLAW_YES=1) to authorise the download.", ); - process.exit(1); + ollamaFlow.deferOllamaProcessExit(); } else { const proceed = await promptYesNoOrDefault( ` Download Ollama model '${selectedModel}' (${sizeLabel})?`, @@ -1704,7 +1704,7 @@ async function selectAndValidateOllamaModel( const allowToolsIncompatible = probe.allowToolsIncompatible === true; const validationBaseUrl = getLocalProviderValidationBaseUrl(provider); if (!validationBaseUrl) - abortNonInteractive("Local Ollama validation URL could not be determined."); + ollamaFlow.deferAbort("Local Ollama validation URL could not be determined."); const validation = await validateOpenAiLikeSelection( "Local Ollama", validationBaseUrl!, @@ -1716,7 +1716,7 @@ async function selectAndValidateOllamaModel( ); if (validation.retry === "selection") return { outcome: "back-to-selection" }; if (!validation.ok) { - if (isNonInteractive()) abortNonInteractive(`model '${selectedModel}' failed validation.`); + if (isNonInteractive()) ollamaFlow.deferAbort(`model '${selectedModel}' failed validation.`); continue; } // Ollama's /v1/responses endpoint does not produce correctly formatted diff --git a/src/lib/onboard/ollama-probe-failure.test.ts b/src/lib/onboard/ollama-probe-failure.test.ts index e4c3225da82..a5eef14e844 100644 --- a/src/lib/onboard/ollama-probe-failure.test.ts +++ b/src/lib/onboard/ollama-probe-failure.test.ts @@ -11,8 +11,19 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import { completeOllamaRuntimeContextSelection, handleOllamaProbeFailure, + OllamaSelectionFatalError, } from "./ollama-probe-failure"; +function captureFatalSelection(run: () => unknown): OllamaSelectionFatalError { + try { + run(); + } catch (error) { + expect(error).toBeInstanceOf(OllamaSelectionFatalError); + return error as OllamaSelectionFatalError; + } + throw new Error("expected a fatal Ollama selection outcome"); +} + describe("handleOllamaProbeFailure (#4365)", () => { let originalProvider: string | undefined; let originalNonInteractive: string | undefined; @@ -29,22 +40,23 @@ describe("handleOllamaProbeFailure (#4365)", () => { else process.env.NEMOCLAW_NON_INTERACTIVE = originalNonInteractive; } - it("exits when a pinned Ollama provider hits a daemon failure", () => { + it("defers pinned-provider termination when Ollama hits a daemon failure", () => { process.env.NEMOCLAW_PROVIDER = "ollama"; const errSpy = vi.spyOn(console, "error").mockImplementation(() => {}); const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); - const exitSpy = vi.spyOn(process, "exit").mockImplementation(((code?: number) => { - throw new Error(`process.exit:${code ?? 0}`); - }) as never); try { - expect(() => + const fatal = captureFatalSelection(() => handleOllamaProbeFailure( { ok: false, message: "runner crashed", daemonFailure: true }, "nemotron-3-nano:30b", () => false, ), - ).toThrow(/process\.exit:1/); + ); + expect(fatal).toMatchObject({ + termination: "process", + message: "Ollama daemon is unhealthy for model 'nemotron-3-nano:30b'.", + }); const errLines = errSpy.mock.calls.map((c) => String(c[0])); expect( errLines.some((l) => @@ -56,33 +68,31 @@ describe("handleOllamaProbeFailure (#4365)", () => { } finally { errSpy.mockRestore(); logSpy.mockRestore(); - exitSpy.mockRestore(); restore(); } }); - it("aborts non-interactive runs on a daemon failure", () => { + it("defers non-interactive aborts on a daemon failure", () => { delete process.env.NEMOCLAW_PROVIDER; const errSpy = vi.spyOn(console, "error").mockImplementation(() => {}); const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); - const exitSpy = vi.spyOn(process, "exit").mockImplementation(((code?: number) => { - throw new Error(`process.exit:${code ?? 0}`); - }) as never); try { - expect(() => + const fatal = captureFatalSelection(() => handleOllamaProbeFailure( { ok: false, message: "runner died", daemonFailure: true }, "nemotron-3-nano:30b", () => true, ), - ).toThrow(/process\.exit:1/); - const errLines = errSpy.mock.calls.map((c) => String(c[0])); - expect(errLines.some((l) => l.includes("Aborting: Ollama daemon is unhealthy"))).toBe(true); + ); + expect(fatal).toMatchObject({ + termination: "non-interactive", + message: "Ollama daemon is unhealthy for model 'nemotron-3-nano:30b'.", + hint: expect.stringContaining("Pick a non-Ollama provider"), + }); } finally { errSpy.mockRestore(); logSpy.mockRestore(); - exitSpy.mockRestore(); restore(); } }); @@ -136,30 +146,26 @@ describe("handleOllamaProbeFailure (#4365)", () => { } }); - it("aborts non-interactive model-level failures via the legacy message", () => { + it("defers non-interactive model-level termination with the legacy message", () => { delete process.env.NEMOCLAW_PROVIDER; const errSpy = vi.spyOn(console, "error").mockImplementation(() => {}); const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); - const exitSpy = vi.spyOn(process, "exit").mockImplementation(((code?: number) => { - throw new Error(`process.exit:${code ?? 0}`); - }) as never); try { - expect(() => + const fatal = captureFatalSelection(() => handleOllamaProbeFailure( { ok: false, message: "model requires more system memory" }, "qwen3.5:9b", () => true, ), - ).toThrow(/process\.exit:1/); - const errLines = errSpy.mock.calls.map((c) => String(c[0])); - expect( - errLines.some((l) => l.includes("Aborting: Ollama model 'qwen3.5:9b' unavailable")), - ).toBe(true); + ); + expect(fatal).toMatchObject({ + termination: "non-interactive", + message: "Ollama model 'qwen3.5:9b' unavailable.", + }); } finally { errSpy.mockRestore(); logSpy.mockRestore(); - exitSpy.mockRestore(); restore(); } }); @@ -198,51 +204,49 @@ describe("completeOllamaRuntimeContextSelection (#6760)", () => { } }); - it("aborts non-interactive runs after a runtime context failure", () => { + it("defers non-interactive termination after a runtime context failure", () => { const errSpy = vi.spyOn(console, "error").mockImplementation(() => {}); - const exitSpy = vi.spyOn(process, "exit").mockImplementation(((code?: number) => { - throw new Error(`process.exit:${code ?? 0}`); - }) as never); try { - expect(() => + const fatal = captureFatalSelection(() => completeOllamaRuntimeContextSelection( { ok: false, message: "restart Ollama with OLLAMA_CONTEXT_LENGTH=64000" }, selected, () => true, ), - ).toThrow(/process\.exit:1/); - expect(errSpy).toHaveBeenCalledWith( - " [non-interactive] Aborting: restart Ollama with OLLAMA_CONTEXT_LENGTH=64000", ); + expect(fatal).toMatchObject({ + termination: "non-interactive", + message: "restart Ollama with OLLAMA_CONTEXT_LENGTH=64000", + }); + expect(errSpy).not.toHaveBeenCalled(); } finally { errSpy.mockRestore(); - exitSpy.mockRestore(); } }); - it("exits pinned interactive runs after a runtime context failure", () => { + it("defers pinned interactive termination after a runtime context failure", () => { vi.stubEnv("NEMOCLAW_PROVIDER", "ollama"); const errSpy = vi.spyOn(console, "error").mockImplementation(() => {}); const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); - const exitSpy = vi.spyOn(process, "exit").mockImplementation(((code?: number) => { - throw new Error(`process.exit:${code ?? 0}`); - }) as never); try { - expect(() => + const fatal = captureFatalSelection(() => completeOllamaRuntimeContextSelection( { ok: false, message: "restart Ollama with OLLAMA_CONTEXT_LENGTH=64000" }, selected, () => false, ), - ).toThrow(/process\.exit:1/); + ); + expect(fatal).toMatchObject({ + termination: "process", + message: "restart Ollama with OLLAMA_CONTEXT_LENGTH=64000", + }); expect(errSpy).toHaveBeenCalledWith(" restart Ollama with OLLAMA_CONTEXT_LENGTH=64000"); expect(logSpy).not.toHaveBeenCalledWith(" Returning to provider selection."); } finally { errSpy.mockRestore(); logSpy.mockRestore(); - exitSpy.mockRestore(); vi.unstubAllEnvs(); } }); diff --git a/src/lib/onboard/ollama-probe-failure.ts b/src/lib/onboard/ollama-probe-failure.ts index 1c7b9d5d3a5..377d1a20b18 100644 --- a/src/lib/onboard/ollama-probe-failure.ts +++ b/src/lib/onboard/ollama-probe-failure.ts @@ -2,7 +2,6 @@ // SPDX-License-Identifier: Apache-2.0 import type { ApplyOllamaRuntimeContextWindowResult } from "../inference/ollama-runtime-context"; -import { abortNonInteractive } from "./non-interactive-abort"; import { isOllamaProviderPinned } from "./ollama-startup"; export interface OllamaProbeFailureInput { @@ -21,6 +20,28 @@ type SelectedOllamaModel = { export type OllamaModelSelectionOutcome = SelectedOllamaModel | { outcome: "back-to-selection" }; +/** A fatal selection outcome whose process termination must happen after transactional rollback. */ +export class OllamaSelectionFatalError extends Error { + constructor( + readonly termination: "process" | "non-interactive", + message: string, + readonly hint?: string, + ) { + super(message); + this.name = "OllamaSelectionFatalError"; + } +} + +/** Defer a fatal process exit to the owner of any active Ollama mutation. */ +export function deferOllamaProcessExit(): never { + throw new OllamaSelectionFatalError("process", "Fatal Ollama model selection failure."); +} + +/** Defer a non-interactive abort to the owner of any active Ollama mutation. */ +export function deferAbort(message: string, hint?: string): never { + throw new OllamaSelectionFatalError("non-interactive", message, hint); +} + /** Finish Ollama selection, returning to provider selection on an interactive context failure. */ export function completeOllamaRuntimeContextSelection( result: ApplyOllamaRuntimeContextWindowResult, @@ -28,9 +49,13 @@ export function completeOllamaRuntimeContextSelection( isNonInteractive: () => boolean, ): OllamaModelSelectionOutcome { if (result.ok) return selected; - if (isNonInteractive()) abortNonInteractive(result.message); + if (isNonInteractive()) { + throw new OllamaSelectionFatalError("non-interactive", result.message); + } console.error(` ${result.message}`); - if (isOllamaProviderPinned()) process.exit(1); + if (isOllamaProviderPinned()) { + throw new OllamaSelectionFatalError("process", result.message); + } console.log(" Returning to provider selection."); console.log(""); return { outcome: "back-to-selection" }; @@ -60,10 +85,14 @@ export function handleOllamaProbeFailure( console.error( " NEMOCLAW_PROVIDER pins onboarding to Ollama but the Ollama model runner is unhealthy; refusing to loop on Ollama model selection.", ); - process.exit(1); + throw new OllamaSelectionFatalError( + "process", + `Ollama daemon is unhealthy for model '${selectedModel}'.`, + ); } if (isNonInteractive()) { - abortNonInteractive( + throw new OllamaSelectionFatalError( + "non-interactive", `Ollama daemon is unhealthy for model '${selectedModel}'.`, "Pick a non-Ollama provider, restart Ollama, or rerun with NEMOCLAW_PROVIDER set explicitly.", ); @@ -77,7 +106,12 @@ export function handleOllamaProbeFailure( console.log(""); return "back-to-selection"; } - if (isNonInteractive()) abortNonInteractive(`Ollama model '${selectedModel}' unavailable.`); + if (isNonInteractive()) { + throw new OllamaSelectionFatalError( + "non-interactive", + `Ollama model '${selectedModel}' unavailable.`, + ); + } console.log(" Choose a different Ollama model or select Other."); console.log(""); return "continue"; diff --git a/src/lib/onboard/provider-host-state.test.ts b/src/lib/onboard/provider-host-state.test.ts index 6f08450b2e4..f6cb0b44d8f 100644 --- a/src/lib/onboard/provider-host-state.test.ts +++ b/src/lib/onboard/provider-host-state.test.ts @@ -564,6 +564,7 @@ describe("detectInferenceProviderHostState", () => { wslDetection: { isWsl: true }, env: {}, loopbackOnly: true, + prepareDockerEnvironment: undefined, }); }); diff --git a/src/lib/onboard/provider-host-state.ts b/src/lib/onboard/provider-host-state.ts index f9270c8ca04..11ad8184f1d 100644 --- a/src/lib/onboard/provider-host-state.ts +++ b/src/lib/onboard/provider-host-state.ts @@ -3,6 +3,7 @@ import { dockerCapture as defaultDockerCapture } from "../adapters/docker"; import { + createOllamaApiCapture, detectLocalTcpListener, findReachableOllamaHost, getLocalProviderAvailabilityEndpoint, @@ -32,10 +33,7 @@ import { type OllamaInstallMenuResult, resolveOllamaInstallMenuEntry } from "./o import { buildVllmMenuEntries, type VllmMenuEntry } from "./vllm-menu"; import { detectWindowsHostOllama, type WindowsHostOllamaState } from "./windows-host-ollama"; -type DockerCapture = ( - args: string[], - options?: { env?: NodeJS.ProcessEnv; ignoreError?: boolean; timeout?: number }, -) => string; +type DockerCapture = RunCaptureFn; export interface InferenceProviderHostGpu { nimCapable?: boolean; @@ -90,6 +88,7 @@ export interface DetectInferenceProviderHostStateDeps { detectVllmProfile: (gpu: InferenceProviderHostGpu | null | undefined) => VllmProfile | null; getLocalProviderAvailabilityEndpoint: (provider: string) => string | null; detectLocalTcpListener: (port: number) => boolean | null; + prepareDockerEnvironment?: Parameters[2]; probeWindowsHostOllamaRouteProtection: typeof probeWindowsHostOllamaRouteProtection; resetOllamaHostCache: () => void; } @@ -123,6 +122,7 @@ function buildDeps( getLocalProviderAvailabilityEndpoint: overrides.getLocalProviderAvailabilityEndpoint ?? getLocalProviderAvailabilityEndpoint, detectLocalTcpListener: overrides.detectLocalTcpListener ?? detectLocalTcpListener, + prepareDockerEnvironment: overrides.prepareDockerEnvironment, probeWindowsHostOllamaRouteProtection: overrides.probeWindowsHostOllamaRouteProtection ?? probeWindowsHostOllamaRouteProtection, resetOllamaHostCache: overrides.resetOllamaHostCache ?? defaultResetOllamaHostCache, @@ -212,6 +212,7 @@ export function detectInferenceProviderHostState( wslDetection: { isWsl }, env: input.env, loopbackOnly: hasWindowsOllama ? winOllamaState.loopbackOnly : undefined, + prepareDockerEnvironment: deps.prepareDockerEnvironment, }); const windowsOllamaReachable = windowsOllamaProtection.reachable; const windowsOllamaRouteProtected = windowsOllamaProtection.protected; diff --git a/src/lib/onboard/setup-nim-ollama.test.ts b/src/lib/onboard/setup-nim-ollama.test.ts index 75d41e4d5b5..a45841dd2c3 100644 --- a/src/lib/onboard/setup-nim-ollama.test.ts +++ b/src/lib/onboard/setup-nim-ollama.test.ts @@ -6,6 +6,7 @@ import assert from "node:assert/strict"; import { afterEach, describe, expect, it, vi } from "vitest"; import { MIN_HERMES_OLLAMA_CONTEXT_WINDOW } from "../inference/ollama-runtime-context"; +import { OllamaSelectionFatalError } from "./ollama-probe-failure"; import { createSetupNimOllamaHandlers } from "./setup-nim-ollama"; import type { SetupNimSelectionState } from "./setup-nim-selection"; @@ -47,9 +48,18 @@ function makeDeps(overrides: Partial = {}): Deps { model: "llama3.1:8b", allowToolsIncompatible: true, }), - installOllamaOnWindowsHost: async () => ({ ok: true, path: "C:/Ollama/ollama.exe" }), - awaitWindowsOllamaReady: () => true, - setupWindowsOllamaLoopbackBinding: () => true, + installOllamaOnWindowsHost: async () => ({ + ok: true, + path: "C:/Ollama/ollama.exe", + commit: () => {}, + rollback: () => {}, + }), + setupWindowsOllamaLoopbackBinding: () => ({ + ok: true, + commit: () => {}, + rollback: () => {}, + }), + printWindowsOllamaSnapshotDiagnostics: () => {}, printWindowsOllamaTimeoutDiagnostics: () => {}, resetOllamaHostCache: () => {}, installOllamaOnMacOS: () => ({ ok: true }), @@ -256,13 +266,23 @@ describe("createSetupNimOllamaHandlers", () => { ); }); - it("does not install or restart Windows Ollama when preflight rejects", async () => { + it("rolls back a Windows Ollama restart when route preflight rejects", async () => { const state = makeState(); state.assertRouteCompatible = () => { throw new Error("route conflict"); }; - const install = vi.fn(async () => ({ ok: true })); - const restart = vi.fn(() => true); + const install = vi.fn(async () => ({ + ok: true as const, + path: "C:/Ollama/ollama.exe", + commit: () => {}, + rollback: () => {}, + })); + const rollback = vi.fn(); + const restart = vi.fn(() => ({ + ok: true as const, + commit: () => {}, + rollback, + })); const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( makeDeps({ installOllamaOnWindowsHost: install, @@ -281,7 +301,8 @@ describe("createSetupNimOllamaHandlers", () => { ), ).rejects.toThrow("route conflict"); expect(install).not.toHaveBeenCalled(); - expect(restart).not.toHaveBeenCalled(); + expect(restart).toHaveBeenCalledOnce(); + expect(rollback).toHaveBeenCalledOnce(); }); it("stops before Windows Ollama install effects when sandbox identity changes (#9833)", async () => { @@ -289,8 +310,17 @@ describe("createSetupNimOllamaHandlers", () => { selection.revalidateSandboxIdentity = () => { throw new Error("Sandbox identity changed before local inference"); }; - const install = vi.fn(async () => ({ ok: true, path: "C:/Ollama/ollama.exe" })); - const start = vi.fn(() => true); + const install = vi.fn(async () => ({ + ok: true as const, + path: "C:/Ollama/ollama.exe", + commit: () => {}, + rollback: () => {}, + })); + const start = vi.fn(() => ({ + ok: true as const, + commit: () => {}, + rollback: () => {}, + })); const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( makeDeps({ installOllamaOnWindowsHost: install, @@ -313,6 +343,203 @@ describe("createSetupNimOllamaHandlers", () => { expect(start).not.toHaveBeenCalled(); }); + it("reports a Windows snapshot failure without claiming a startup timeout", async () => { + const snapshotDiagnostic = vi.fn(); + const timeoutDiagnostic = vi.fn(); + const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( + makeDeps({ + isNonInteractive: () => false, + printWindowsOllamaSnapshotDiagnostics: snapshotDiagnostic, + printWindowsOllamaTimeoutDiagnostics: timeoutDiagnostic, + setupWindowsOllamaLoopbackBinding: () => ({ ok: false, reason: "snapshot" }), + }), + ); + + await expect( + handleWindowsHostOllamaSelection( + null, + "start-windows-ollama", + "qwen3:8b", + false, + "C:/Ollama/ollama.exe", + makeState(), + ), + ).resolves.toBe("retry-selection"); + + expect(snapshotDiagnostic).toHaveBeenCalledOnce(); + expect(timeoutDiagnostic).not.toHaveBeenCalled(); + }); + + it("commits a new Windows Ollama install after model selection", async () => { + const commit = vi.fn(); + const rollback = vi.fn(); + const revalidate = vi.fn(); + const install = vi.fn(async (args: { beforeRestart: () => void }) => { + args.beforeRestart(); + return { + ok: true as const, + path: "C:/Ollama/ollama.exe", + commit, + rollback, + }; + }); + const state = makeState(); + state.revalidateSandboxIdentity = revalidate; + const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( + makeDeps({ installOllamaOnWindowsHost: install }), + ); + + await expect( + handleWindowsHostOllamaSelection( + null, + "install-windows-ollama", + "qwen3:8b", + false, + null, + state, + ), + ).resolves.toBe("selected"); + + expect(revalidate.mock.calls.map(([operation]) => operation)).toEqual([ + "install the Windows Ollama runtime", + "start the Windows Ollama runtime", + ]); + expect(commit).toHaveBeenCalledOnce(); + expect(rollback).not.toHaveBeenCalled(); + }); + + it("rolls back a new Windows Ollama install when model selection returns", async () => { + const commit = vi.fn(); + const rollback = vi.fn(); + const install = vi.fn(async () => ({ + ok: true as const, + path: "C:/Ollama/ollama.exe", + commit, + rollback, + })); + const resetHost = vi.fn(); + const state = makeState(); + const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( + makeDeps({ + isNonInteractive: () => false, + installOllamaOnWindowsHost: install, + resetOllamaHostCache: resetHost, + selectAndValidateOllamaModel: async () => ({ outcome: "back-to-selection" }), + }), + ); + + await expect( + handleWindowsHostOllamaSelection( + null, + "install-windows-ollama", + "qwen3:8b", + false, + null, + state, + ), + ).resolves.toBe("retry-selection"); + + expect(commit).not.toHaveBeenCalled(); + expect(rollback).toHaveBeenCalledOnce(); + expect(resetHost).toHaveBeenCalledOnce(); + }); + + it("rolls back a Windows Ollama restart when model selection returns", async () => { + const commit = vi.fn(); + const rollback = vi.fn(); + const restart = vi.fn(() => ({ ok: true as const, commit, rollback })); + const resetHost = vi.fn(); + const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( + makeDeps({ + isNonInteractive: () => false, + resetOllamaHostCache: resetHost, + selectAndValidateOllamaModel: async () => ({ outcome: "back-to-selection" }), + setupWindowsOllamaLoopbackBinding: restart, + }), + ); + + await expect( + handleWindowsHostOllamaSelection( + null, + "start-windows-ollama", + "qwen3:8b", + false, + "C:/Ollama/ollama.exe", + makeState(), + ), + ).resolves.toBe("retry-selection"); + + expect(restart).toHaveBeenCalledOnce(); + expect(commit).not.toHaveBeenCalled(); + expect(rollback).toHaveBeenCalledOnce(); + expect(resetHost).toHaveBeenCalledOnce(); + }); + + it("rolls back a Windows Ollama restart when model selection throws", async () => { + const rollback = vi.fn(); + const restart = vi.fn(() => ({ ok: true as const, commit: vi.fn(), rollback })); + const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( + makeDeps({ + selectAndValidateOllamaModel: async () => { + throw new Error("model selection failed"); + }, + setupWindowsOllamaLoopbackBinding: restart, + }), + ); + + await expect( + handleWindowsHostOllamaSelection( + null, + "start-windows-ollama", + "qwen3:8b", + false, + "C:/Ollama/ollama.exe", + makeState(), + ), + ).rejects.toThrow("model selection failed"); + + expect(restart).toHaveBeenCalledOnce(); + expect(rollback).toHaveBeenCalledOnce(); + }); + + it("awaits Windows rollback before terminating a fatal model selection", async () => { + const events: string[] = []; + const rollback = vi.fn(async () => { + events.push("rollback:start"); + await Promise.resolve(); + events.push("rollback:done"); + }); + const exit = vi.fn((code?: number): never => { + events.push(`exit:${String(code)}`); + throw new Error(`process.exit:${String(code)}`); + }); + const restart = vi.fn(() => ({ ok: true as const, commit: vi.fn(), rollback })); + const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( + makeDeps({ + process: { ...process, exit } as unknown as NodeJS.Process, + selectAndValidateOllamaModel: async () => { + throw new OllamaSelectionFatalError("process", "fatal model selection"); + }, + setupWindowsOllamaLoopbackBinding: restart, + }), + ); + + await expect( + handleWindowsHostOllamaSelection( + null, + "start-windows-ollama", + "qwen3:8b", + false, + "C:/Ollama/ollama.exe", + makeState(), + ), + ).rejects.toThrow("process.exit:1"); + + expect(events).toEqual(["rollback:start", "rollback:done", "exit:1"]); + expect(rollback).toHaveBeenCalledOnce(); + expect(exit).toHaveBeenCalledWith(1); + }); + it("preserves accepted tools-incompatible state for running Ollama", async () => { const state = makeState(); const { handleRunningOllamaSelection } = createSetupNimOllamaHandlers(makeDeps()); @@ -436,7 +663,7 @@ describe("createSetupNimOllamaHandlers", () => { getLocalProviderBaseUrl: () => endpointUrl, setupWindowsOllamaLoopbackBinding: () => { endpointUrl = "http://host.docker.internal:11434/v1"; - return true; + return { ok: true, commit: () => {}, rollback: () => {} }; }, selectAndValidateOllamaModel: selectModel, }), @@ -453,10 +680,7 @@ describe("createSetupNimOllamaHandlers", () => { expect(result).toBe("selected"); expect(state.endpointUrl).toBe("http://host.docker.internal:11434/v1"); - expect(compatibilityEndpoints).toEqual([ - "http://host.openshell.internal:11435/v1", - "http://host.docker.internal:11434/v1", - ]); + expect(compatibilityEndpoints).toEqual(["http://host.docker.internal:11434/v1"]); expect(selectModel).toHaveBeenCalledTimes(1); }); diff --git a/src/lib/onboard/setup-nim-ollama.ts b/src/lib/onboard/setup-nim-ollama.ts index 17179498349..bbc67197950 100644 --- a/src/lib/onboard/setup-nim-ollama.ts +++ b/src/lib/onboard/setup-nim-ollama.ts @@ -3,6 +3,11 @@ import type { OllamaStartupOutcome } from "./ollama-startup"; import type { SetupNimSelectionState } from "./setup-nim-selection"; +import type { + WindowsOllamaInstallResult, + WindowsOllamaSetupResult, +} from "../inference/ollama/windows"; +import { OllamaSelectionFatalError } from "./ollama-probe-failure"; const { getRequestedModelFromEnv, @@ -47,12 +52,14 @@ type SetupNimOllamaDeps = { | { outcome: "back-to-selection" } | { outcome: "selected"; model: string; allowToolsIncompatible: boolean } >; - installOllamaOnWindowsHost: () => Promise<{ ok: boolean; path?: string | null }>; - awaitWindowsOllamaReady: () => boolean; + installOllamaOnWindowsHost: (args: { + beforeRestart: () => void; + }) => Promise; setupWindowsOllamaLoopbackBinding: (args: { announceStop?: boolean; installedPath?: string | null; - }) => boolean; + }) => WindowsOllamaSetupResult; + printWindowsOllamaSnapshotDiagnostics?: () => void; printWindowsOllamaTimeoutDiagnostics: () => void; resetOllamaHostCache: () => void; installOllamaOnMacOS: (args: { @@ -66,7 +73,7 @@ type SetupNimOllamaDeps = { restartOnly?: boolean; contextWindowFloor?: number; }) => { ok: boolean }; - abortNonInteractive: (message: string) => never; + abortNonInteractive: (message: string, hint?: string) => never; assertOllamaUpgradeApplied: (menu: { hasUpgradableOllama: boolean; }) => { ok: true } | { ok: false; message: string }; @@ -141,6 +148,13 @@ export function createSetupNimOllamaHandlers(deps: SetupNimOllamaDeps): { } } + function terminateFatalSelection(error: OllamaSelectionFatalError): never { + if (error.termination === "non-interactive") { + deps.abortNonInteractive(error.message, error.hint); + } + deps.process.exit(1); + } + function configureOllamaState(state: SetupNimSelectionState): void { state.provider = "ollama-local"; state.credentialEnv = null; @@ -195,7 +209,6 @@ export function createSetupNimOllamaHandlers(deps: SetupNimOllamaDeps): { if (!deps.checkOllamaPortsOrWarn({ isNonInteractive: deps.isNonInteractive })) { return "retry-selection"; } - preflightOllamaRoute(state, requestedModel, null); const isInstall = selectedKey === "install-windows-ollama"; const isRestart = !isInstall; const promptMsg = isInstall @@ -208,44 +221,62 @@ export function createSetupNimOllamaHandlers(deps: SetupNimOllamaDeps): { : !(await deps.prompt(promptMsg)).trim().toLowerCase().startsWith("n"); if (!proceed) return "retry-selection"; - if (isInstall) { - state.revalidateSandboxIdentity?.("install the Windows Ollama runtime"); - const installResult = await deps.installOllamaOnWindowsHost(); - if (!installResult.ok) { - console.error( - " Install did not produce ollama.exe on PATH. Check the installer output above.", - ); - if (deps.isNonInteractive()) deps.process.exit(1); - return "retry-selection"; - } - if (!deps.awaitWindowsOllamaReady()) { - console.log(" Installer did not leave a reachable Ollama daemon; restarting it..."); - state.revalidateSandboxIdentity?.("start the Windows Ollama runtime"); - if (!deps.setupWindowsOllamaLoopbackBinding({ installedPath: installResult.path })) { - deps.printWindowsOllamaTimeoutDiagnostics(); + let mutationSession: { commit: () => void; rollback: () => void | Promise } | null = null; + try { + if (isInstall) { + state.revalidateSandboxIdentity?.("install the Windows Ollama runtime"); + const installResult = await deps.installOllamaOnWindowsHost({ + beforeRestart: () => + state.revalidateSandboxIdentity?.("start the Windows Ollama runtime"), + }); + if (!installResult.ok) { + if (installResult.reason === "snapshot") { + deps.printWindowsOllamaSnapshotDiagnostics?.(); + } else if (installResult.reason === "readiness") { + deps.printWindowsOllamaTimeoutDiagnostics(); + } else if (installResult.reason === "install") { + console.error( + " Install did not produce ollama.exe on PATH. Check the installer output above.", + ); + } if (deps.isNonInteractive()) deps.process.exit(1); return "retry-selection"; } - } - console.log(` ✓ Using Ollama on host.docker.internal:${deps.OLLAMA_PORT}`); - } else { - state.revalidateSandboxIdentity?.("start the Windows Ollama runtime"); - if ( - !deps.setupWindowsOllamaLoopbackBinding({ + mutationSession = installResult; + console.log(` ✓ Using Ollama on host.docker.internal:${deps.OLLAMA_PORT}`); + } else { + state.revalidateSandboxIdentity?.("start the Windows Ollama runtime"); + const setupResult = deps.setupWindowsOllamaLoopbackBinding({ announceStop: isRestart, installedPath: winOllamaInstalledPath || undefined, - }) - ) { - deps.printWindowsOllamaTimeoutDiagnostics(); - if (deps.isNonInteractive()) deps.process.exit(1); - return "retry-selection"; + }); + if (!setupResult.ok) { + if (setupResult.reason === "snapshot") { + deps.printWindowsOllamaSnapshotDiagnostics?.(); + } else if (setupResult.reason === "readiness") { + deps.printWindowsOllamaTimeoutDiagnostics(); + } + if (deps.isNonInteractive()) deps.process.exit(1); + return "retry-selection"; + } + mutationSession = setupResult; + console.log(` ✓ Using Ollama on host.docker.internal:${deps.OLLAMA_PORT}`); } - console.log(` ✓ Using Ollama on host.docker.internal:${deps.OLLAMA_PORT}`); + + const lockedModel = preflightOllamaRoute(state, requestedModel, null); + const result = await selectModel(gpu, state, requestedModel, null, lockedModel); + if (result === "retry-selection") { + await mutationSession?.rollback(); + deps.resetOllamaHostCache(); + } else { + mutationSession?.commit(); + } + return result; + } catch (error) { + await mutationSession?.rollback(); + if (error instanceof OllamaSelectionFatalError) terminateFatalSelection(error); + throw error; } - const lockedModel = preflightOllamaRoute(state, requestedModel, null); - const result = await selectModel(gpu, state, requestedModel, null, lockedModel); - if (result === "retry-selection") deps.resetOllamaHostCache(); - return result; } async function handleRunningOllamaSelection( @@ -302,7 +333,12 @@ export function createSetupNimOllamaHandlers(deps: SetupNimOllamaDeps): { return "selected"; case "ready": announceOllamaRoute(); - return selectModel(gpu, state, requestedModel, recoveredModel, lockedModel); + try { + return await selectModel(gpu, state, requestedModel, recoveredModel, lockedModel); + } catch (error) { + if (error instanceof OllamaSelectionFatalError) terminateFatalSelection(error); + throw error; + } default: { const kind = (startup as { kind?: unknown }).kind; Object.assign(state, initialState); @@ -350,7 +386,12 @@ export function createSetupNimOllamaHandlers(deps: SetupNimOllamaDeps): { return "retry-selection"; } announceOllamaRoute(); - return selectModel(gpu, state, requestedModel, recoveredModel, lockedModel); + try { + return await selectModel(gpu, state, requestedModel, recoveredModel, lockedModel); + } catch (error) { + if (error instanceof OllamaSelectionFatalError) terminateFatalSelection(error); + throw error; + } } return { diff --git a/src/lib/runner.ts b/src/lib/runner.ts index 49680e1f406..cb415fc0807 100644 --- a/src/lib/runner.ts +++ b/src/lib/runner.ts @@ -451,6 +451,7 @@ function validateName(name: string, label = "name"): string { export { ROOT, + buildSubprocessEnv, redact, redactFull, redactFullWithUrls, diff --git a/test/onboarding/onboard-inference-reconciliation.test.ts b/test/onboarding/onboard-inference-reconciliation.test.ts index 9d25955a331..8adc8f63837 100644 --- a/test/onboarding/onboard-inference-reconciliation.test.ts +++ b/test/onboarding/onboard-inference-reconciliation.test.ts @@ -8,6 +8,11 @@ import os from "node:os"; import path from "node:path"; import { describe, expect, it, vi } from "vitest"; +import { + clearPersistedOllamaHostIfUnused, + loadPersistedOllamaHost, + persistResolvedOllamaHost, +} from "../../src/lib/inference/local.js"; import { createLocalInferenceRouteApplier } from "../../src/lib/onboard/local-inference-route.js"; import type { SetupInference, SetupInferenceDeps } from "../../src/lib/onboard/setup-inference.js"; import { writeOkOpenshell } from "../helpers/onboard-openshell-fixture"; @@ -1087,22 +1092,31 @@ describe("re-onboard Ollama GPU release (#9110)", () => { expect(unloadOllamaModels).toHaveBeenCalledWith(["llama3"]); }); - it("retires the final Ollama route receipt after switching providers", async () => { - const clearPersistedOllamaHostIfUnused = vi.fn(() => true); - const harness = releaseHarness({ - getSandbox: () => priorEntry, - sandboxes: [{ ...priorEntry, provider: "vllm-local", model: "vllm-model" }], - unloadOllamaModels: vi.fn(), - clearPersistedOllamaHostIfUnused, - }); + it("retires the final Windows-host Ollama route receipt after switching providers", async () => { + const stateRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-provider-switch-ollama-")); + const finalRoutes = [{ ...priorEntry, provider: "vllm-local", model: "vllm-model" }]; + const clearReceipt = vi.fn((routes: readonly ReleaseEntry[]) => + clearPersistedOllamaHostIfUnused(routes, stateRoot), + ); + try { + persistResolvedOllamaHost("host.docker.internal", stateRoot); + const harness = releaseHarness({ + getSandbox: () => priorEntry, + sandboxes: finalRoutes, + unloadOllamaModels: vi.fn(), + loadPersistedOllamaHost: () => loadPersistedOllamaHost(stateRoot), + clearPersistedOllamaHostIfUnused: clearReceipt, + }); - await expect(harness.setupInference("test-box", "vllm-model", "vllm-local")).resolves.toEqual({ - ok: true, - }); + await expect(harness.setupInference("test-box", "vllm-model", "vllm-local")).resolves.toEqual( + { ok: true }, + ); - expect(clearPersistedOllamaHostIfUnused).toHaveBeenCalledWith([ - { ...priorEntry, provider: "vllm-local", model: "vllm-model" }, - ]); + expect(clearReceipt).toHaveBeenCalledWith(finalRoutes); + expect(loadPersistedOllamaHost(stateRoot)).toBeNull(); + } finally { + fs.rmSync(stateRoot, { recursive: true, force: true }); + } }); it("keeps the successful route when the superseded model unload fails (#9110)", async () => { @@ -1277,29 +1291,38 @@ describe("re-onboard Ollama GPU release (#9110)", () => { expect(unloadOllamaModels).not.toHaveBeenCalled(); }); - it("keeps the route and shared model for a compatible local Ollama peer", async () => { + it("keeps the Windows-host route and shared model for a compatible local Ollama peer", async () => { + const stateRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-provider-switch-peer-")); const unloadOllamaModels = vi.fn<(onlyModels: readonly string[]) => void>(); - const clearPersistedOllamaHostIfUnused = vi.fn(() => true); + const clearReceipt = vi.fn((routes: readonly ReleaseEntry[]) => + clearPersistedOllamaHostIfUnused(routes, stateRoot), + ); const peer: ReleaseEntry = { name: "peer", provider: "compatible-endpoint", model: "llama3:latest", - endpointUrl: "http://127.0.0.1:11434/v1", + endpointUrl: "http://host.docker.internal:11434/v1", }; - const harness = releaseHarness({ - getSandbox: () => priorEntry, - sandboxes: [{ ...priorEntry, provider: "vllm-local", model: "vllm-model" }, peer], - unloadOllamaModels, - loadPersistedOllamaHost: () => "127.0.0.1", - clearPersistedOllamaHostIfUnused, - }); + try { + persistResolvedOllamaHost("host.docker.internal", stateRoot); + const harness = releaseHarness({ + getSandbox: () => priorEntry, + sandboxes: [{ ...priorEntry, provider: "vllm-local", model: "vllm-model" }, peer], + unloadOllamaModels, + loadPersistedOllamaHost: () => loadPersistedOllamaHost(stateRoot), + clearPersistedOllamaHostIfUnused: clearReceipt, + }); - await expect(harness.setupInference("test-box", "vllm-model", "vllm-local")).resolves.toEqual({ - ok: true, - }); + await expect(harness.setupInference("test-box", "vllm-model", "vllm-local")).resolves.toEqual( + { ok: true }, + ); - expect(unloadOllamaModels).not.toHaveBeenCalled(); - expect(clearPersistedOllamaHostIfUnused).not.toHaveBeenCalled(); + expect(unloadOllamaModels).not.toHaveBeenCalled(); + expect(clearReceipt).not.toHaveBeenCalled(); + expect(loadPersistedOllamaHost(stateRoot)).toBe("host.docker.internal"); + } finally { + fs.rmSync(stateRoot, { recursive: true, force: true }); + } }); it("reads the prior route and releases the model inside the sandbox mutation lock (#9110)", async () => { diff --git a/test/onboarding/onboard-ollama-upgrade-version-floor.test.ts b/test/onboarding/onboard-ollama-upgrade-version-floor.test.ts index 982e076d460..b531afc6a07 100644 --- a/test/onboarding/onboard-ollama-upgrade-version-floor.test.ts +++ b/test/onboarding/onboard-ollama-upgrade-version-floor.test.ts @@ -100,9 +100,17 @@ function makeOllamaDeps(overrides: Partial = {}): SetupNimOl model: "qwen3:8b", allowToolsIncompatible: false, }), - installOllamaOnWindowsHost: async () => ({ ok: true }), - awaitWindowsOllamaReady: () => true, - setupWindowsOllamaLoopbackBinding: () => true, + installOllamaOnWindowsHost: async () => ({ + ok: true, + path: "C:/Ollama/ollama.exe", + commit: () => {}, + rollback: () => {}, + }), + setupWindowsOllamaLoopbackBinding: () => ({ + ok: true, + commit: () => {}, + rollback: () => {}, + }), printWindowsOllamaTimeoutDiagnostics: () => {}, resetOllamaHostCache: () => {}, installOllamaOnMacOS: () => ({ ok: true }), diff --git a/test/onboarding/onboard-selection.test.ts b/test/onboarding/onboard-selection.test.ts index 089f1f51831..dde543b0dbb 100644 --- a/test/onboarding/onboard-selection.test.ts +++ b/test/onboarding/onboard-selection.test.ts @@ -75,6 +75,7 @@ const CREDENTIAL_RETRY_PROMPT_RE = const OLLAMA_CHAT_COMPLETIONS_TOOL_CALL_RESPONSE = '{"choices":[{"message":{"role":"assistant","content":"","tool_calls":[{"type":"function","function":{"name":"emit_ok","arguments":"{\\"ok\\":true}"}}]}}]}'; const PROVIDER_SELECTION_TEST_TIMEOUT_MS = testTimeout(60_000); +const WINDOWS_SETUP_SUCCESS = { ok: true as const, commit: () => {}, rollback: () => {} }; const repoRoot = path.join(import.meta.dirname, "../.."); const onboardPath = JSON.stringify(path.join(repoRoot, "src", "lib", "onboard.ts")); const credentialsPath = JSON.stringify( @@ -482,9 +483,11 @@ function makeSetupNimOllamaDeps(overrides: Partial = {}): Se model: "qwen3:8b", allowToolsIncompatible: false, }), - installOllamaOnWindowsHost: async () => ({ ok: true }), - awaitWindowsOllamaReady: () => true, - setupWindowsOllamaLoopbackBinding: () => true, + installOllamaOnWindowsHost: async () => ({ + ...WINDOWS_SETUP_SUCCESS, + path: "C:/Ollama/ollama.exe", + }), + setupWindowsOllamaLoopbackBinding: () => WINDOWS_SETUP_SUCCESS, printWindowsOllamaTimeoutDiagnostics: () => {}, resetOllamaHostCache: () => {}, installOllamaOnMacOS: () => ({ ok: true }), @@ -1183,7 +1186,11 @@ describe("onboard provider selection UX", { timeout: PROVIDER_SELECTION_TEST_TIM }); it("offers Gemini 3.6 Flash instead of 2.5 Flash and supports Other (#9298)", async () => { - const acceptedDefault = await promptRemoteModel("Google Gemini", "gemini", "gemini-3.6-flash", null, + const acceptedDefault = await promptRemoteModel( + "Google Gemini", + "gemini", + "gemini-3.6-flash", + null, { promptFn: async () => "", writeLine: () => {} }, ); assert.equal(acceptedDefault, "gemini-3.6-flash"); @@ -1482,9 +1489,12 @@ reportChildScenario(async () => { ); }); - it("treats an implicit latest Ollama model as installed during systemd repair", { - timeout: PROVIDER_SELECTION_TEST_TIMEOUT_MS, - }, () => { + it( + "treats an implicit latest Ollama model as installed during systemd repair", + { + timeout: PROVIDER_SELECTION_TEST_TIMEOUT_MS, + }, + () => { const workspace = onboardProcessWorkspace("nemoclaw-onboard-ollama-systemd-"); const { root: tmpDir } = workspace; const fakeBin = workspace.binDir; @@ -1576,11 +1586,15 @@ reportChildScenario(async () => { ), "should install and wait for the Ollama systemd drop-in restart", ); - }); + }, + ); - it("preserves existing Ollama systemd override settings while repairing loopback", { - timeout: 10_000, - }, () => { + it( + "preserves existing Ollama systemd override settings while repairing loopback", + { + timeout: 10_000, + }, + () => { const workspace = onboardProcessWorkspace("nemoclaw-onboard-ollama-systemd-merge-"); const { root: tmpDir } = workspace; const fakeBin = workspace.binDir; @@ -1712,11 +1726,15 @@ reportChildScenario(async () => { payload.installedBody.includes('Environment="HTTPS_PROXY=http://proxy.internal:8080"'), "other Environment= settings should be preserved", ); - }); + }, + ); - it("adds Spark CUDA v13 and enables the Ollama systemd service on managed install", { - timeout: 10_000, - }, () => { + it( + "adds Spark CUDA v13 and enables the Ollama systemd service on managed install", + { + timeout: 10_000, + }, + () => { const workspace = onboardProcessWorkspace("nemoclaw-ollama-systemd-spark-"); const { root: tmpDir } = workspace; @@ -1789,11 +1807,15 @@ reportChildScenario(() => { ), "managed Ollama installs should enable the service for reboot survival", ); - }); + }, + ); - it("allows prompt-capable sudo in non-interactive Ollama systemd setup", { - timeout: 10_000, - }, () => { + it( + "allows prompt-capable sudo in non-interactive Ollama systemd setup", + { + timeout: 10_000, + }, + () => { const workspace = onboardProcessWorkspace("nemoclaw-ollama-systemd-sudo-mode-"); const { root: tmpDir } = workspace; @@ -1849,7 +1871,8 @@ reportChildScenario(() => { ), "prompt sudo mode should not use sudo -n", ); - }); + }, + ); it("rejects unsupported non-interactive sudo mode values", () => { const previousMode = process.env.NEMOCLAW_NON_INTERACTIVE_SUDO_MODE; @@ -1882,9 +1905,12 @@ reportChildScenario(() => { } }); - it("repairs already-loopback systemd Ollama without starting a duplicate daemon", { - timeout: 10_000, - }, () => { + it( + "repairs already-loopback systemd Ollama without starting a duplicate daemon", + { + timeout: 10_000, + }, + () => { const workspace = onboardProcessWorkspace("nemoclaw-onboard-ollama-systemd-loopback-"); const { root: tmpDir } = workspace; const fakeBin = workspace.binDir; @@ -1982,11 +2008,15 @@ reportChildScenario(async () => { payload.events.slice(restartIndex + 1).includes("tags"), "should re-probe after the systemd restart instead of trusting a stale loopback cache", ); - }); + }, + ); - it("fails closed instead of starting unmanaged Ollama when systemd restart stays unreachable", { - timeout: 15_000, - }, () => { + it( + "fails closed instead of starting unmanaged Ollama when systemd restart stays unreachable", + { + timeout: 15_000, + }, + () => { const workspace = onboardProcessWorkspace("nemoclaw-onboard-existing-systemd-restart-fail-"); const { root: tmpDir } = workspace; @@ -2047,7 +2077,8 @@ const { setupNim } = require(${onboardPath}); assert.equal(result.status, 1); assert.match(result.stderr, /Ollama systemd restart did not recover/); assert.doesNotMatch(result.stderr, /manual-start/); - }); + }, + ); it("fails closed when an existing Ollama systemd override cannot be applied", () => { const workspace = onboardProcessWorkspace("nemoclaw-onboard-existing-systemd-fail-"); @@ -3539,9 +3570,7 @@ reportChildScenario(async () => { assert.ok(zstdWarningIndex >= 0 && zstdWarningIndex < zstdCommandIndex); assert.ok(installerWarningIndex >= 0 && installerWarningIndex < installerCommandIndex); assert.equal(events[installerCommandIndex]?.stdio, "inherit"); - assert.ok( - commands.some((command) => command.includes("/install.sh'")), - ); + assert.ok(commands.some((command) => command.includes("/install.sh'"))); assert.ok(!commands.some((command) => command.includes("brew install"))); assert.ok( commands.some((command) => command.includes("OLLAMA_HOST=127.0.0.1:11434 ollama serve")), @@ -3849,62 +3878,12 @@ const { setupNim } = require(${onboardPath}); assert.equal(prompt.mock.calls.length, 0); assert.equal(result.provider, "ollama-local"); assert.ok(notes.some((line) => line.includes("[non-interactive] Provider: ollama"))); - assert.ok( - commands.some((command) => command.includes("/install.sh'")), - ); + assert.ok(commands.some((command) => command.includes("/install.sh'"))); } finally { resetOllamaHostCache(); } }); - it("restarts Windows-host Ollama after install when installer auto-start is not reachable", async () => { - const installedPath = "C:\\Users\\tester\\AppData\\Local\\Programs\\Ollama\\ollama.exe"; - const install = vi.fn(async () => ({ ok: true, path: installedPath })); - const awaitReady = vi.fn(() => false); - const setup = vi.fn(() => true); - const lines: string[] = []; - const log = vi.spyOn(console, "log").mockImplementation((...args) => { - lines.push(args.join(" ")); - }); - const state = makeOllamaSelectionState(); - const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( - makeSetupNimOllamaDeps({ - installOllamaOnWindowsHost: install, - awaitWindowsOllamaReady: awaitReady, - setupWindowsOllamaLoopbackBinding: setup, - }), - ); - - try { - const result = await handleWindowsHostOllamaSelection( - null, - "install-windows-ollama", - "qwen3:8b", - false, - null, - state, - ); - - assert.equal(result, "selected"); - assert.equal(state.provider, "ollama-local"); - assert.equal(state.model, "qwen3:8b"); - assert.equal(install.mock.calls.length, 1); - assert.equal(awaitReady.mock.calls.length, 1); - assert.deepEqual( - setup.mock.calls.map(([options]) => options), - [{ installedPath }], - ); - assert.ok( - lines.some((line) => - line.includes("Installer did not leave a reachable Ollama daemon; restarting it"), - ), - ); - assert.ok(lines.some((line) => line.includes("Using Ollama on host.docker.internal:11434"))); - } finally { - log.mockRestore(); - } - }); - it("shows Windows-host Ollama in the menu with a Docker Desktop requirement on native Docker WSL", () => { const requirement = getWindowsHostOllamaDockerRequirement("docker"); const { options } = buildWindowsProviderMenu(requirement, { @@ -3919,6 +3898,15 @@ const { setupNim } = require(${onboardPath}); assert.doesNotMatch(menuOutput, /Start Ollama on Windows host \(suggested\)/); }); + it("reports native Docker as unsupported for Windows-host Ollama", () => { + const requirement = getWindowsHostOllamaDockerRequirement("docker"); + + assert.equal(requirement.supported, false); + assert.match(requirement.detectedRuntime, /Docker/); + assert.match(requirement.installLabel, /requires Docker Desktop WSL integration/); + assert.match(requirement.reason, /requires Docker Desktop WSL integration/); + }); + it("uses the Windows-host start path when install-windows-ollama is requested but Ollama is already installed", async () => { const requirement = getWindowsHostOllamaDockerRequirement("docker-desktop"); const installedPath = "C:\\Users\\tester\\AppData\\Local\\Programs\\Ollama\\ollama.exe"; @@ -3930,8 +3918,12 @@ const { setupNim } = require(${onboardPath}); const selectedResolution = requireSelectedProviderResolution(resolution); assert.equal(selectedResolution.selected.key, "start-windows-ollama"); - const install = vi.fn(async () => ({ ok: false, path: "" })); - const setup = vi.fn(() => true); + const install = vi.fn(async () => ({ + ok: false as const, + path: "", + reason: "install" as const, + })); + const setup = vi.fn((_args?: unknown) => WINDOWS_SETUP_SUCCESS); const lines: string[] = []; const log = vi.spyOn(console, "log").mockImplementation((...args) => { lines.push(args.join(" ")); @@ -3982,8 +3974,12 @@ const { setupNim } = require(${onboardPath}); loopbackOnly: true, }); - const install = vi.fn(async () => ({ ok: false, path: "" })); - const setup = vi.fn(() => true); + const install = vi.fn(async () => ({ + ok: false as const, + path: "", + reason: "install" as const, + })); + const setup = vi.fn((_args?: unknown) => WINDOWS_SETUP_SUCCESS); const log = vi.spyOn(console, "log").mockImplementation(() => {}); const state = makeOllamaSelectionState(); const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( @@ -4028,8 +4024,12 @@ const { setupNim } = require(${onboardPath}); loopbackOnly: true, }); - const install = vi.fn(async () => ({ ok: false, path: "" })); - const setup = vi.fn(() => true); + const install = vi.fn(async () => ({ + ok: false as const, + path: "", + reason: "install" as const, + })); + const setup = vi.fn((_args?: unknown) => WINDOWS_SETUP_SUCCESS); const log = vi.spyOn(console, "log").mockImplementation(() => {}); const state = makeOllamaSelectionState(); const { handleWindowsHostOllamaSelection } = createSetupNimOllamaHandlers( diff --git a/test/support/onboard-selection-test-helpers.ts b/test/support/onboard-selection-test-helpers.ts index 74ad97d76ce..102fd3c1d0e 100644 --- a/test/support/onboard-selection-test-helpers.ts +++ b/test/support/onboard-selection-test-helpers.ts @@ -265,11 +265,16 @@ local.getOllamaModelOptions = () => { }; windows.installOllamaOnWindowsHost = async () => { console.error("WINDOWS_INSTALL_CALLED"); - return { ok: true, path: "C:\\Users\\tester\\AppData\\Local\\Programs\\Ollama\\ollama.exe" }; + return { + ok: true, + path: "C:\\Users\\tester\\AppData\\Local\\Programs\\Ollama\\ollama.exe", + commit: () => {}, + rollback: () => {}, + }; }; windows.setupWindowsOllamaLoopbackBinding = () => { console.error("WINDOWS_SETUP_CALLED"); - return true; + return { ok: true, commit: () => {}, rollback: () => {} }; }; const { setupNim } = require(${onboardPath});