Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion docs/inference/set-up-ollama.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,20 @@ Set `NEMOCLAW_OLLAMA_INSTALL_MODE=system` or `NEMOCLAW_OLLAMA_INSTALL_MODE=user`

The user-local mode downloads and extracts the Ollama release to `${HOME}/.local`, then starts `${HOME}/.local/bin/ollama serve` once.
It does not configure a systemd service, create the `ollama` system user, or install CUDA drivers.
Restart the daemon manually after a reboot.
For ordinary user-local installs, restart the daemon manually after a reboot.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Scope the automatic-start statement to the applicable flow.

Line 54 says that an ordinary user-local daemon needs a manual restart after reboot. The earlier unqualified statement says NemoClaw starts any installed stopped daemon. Scope that statement to installation or onboarding, or document the portable receipt-owned recovery exception there.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/inference/set-up-ollama.mdx` at line 54, Update the earlier unqualified
automatic-start statement to scope it to the installation or onboarding flow, or
explicitly document the portable receipt-owned recovery exception. Keep the
guidance for ordinary user-local installs in line 54 stating that the daemon
requires a manual restart after reboot.


<AgentOnly variant="openclaw">
For a sandbox created with the portable experimental profile, `$$nemoclaw <name> connect --probe-only` and `$$nemoclaw <name> recover` can restart the user-local daemon that NemoClaw installed.
Before it decides whether to start Ollama, recovery probes `http://127.0.0.1:11434/api/tags`.
If the API is unhealthy, recovery starts Ollama only when the sandbox records `ollama-local` and a valid ownership receipt names the fixed regular executable.
It refuses to launch a duplicate when another `ollama` process exists but the API is unhealthy.
After it launches the daemon, it waits up to 30 seconds for `/api/tags` to return valid JSON with a `models` array.
Recovery then requires a trusted reachable result from the sandbox `inference.local` route.
When route repair needs the local Ollama dependency check, it verifies or recovers the authenticated proxy before it retries `inference.local`.
If Ollama does not become healthy within 30 seconds, the command identifies the exact receipt-bound executable and its `serve` argument, then tells you to retry recovery.
The command exits non-zero with recovery guidance when startup or route validation fails.
It does not start, stop, or replace a system service or another user-managed Ollama daemon.
</AgentOnly>

NemoClaw prints a `PATH` hint when `${HOME}/.local/bin` is missing from your `PATH`.
Add `export PATH="${HOME}/.local/bin:$PATH"` to your shell profile if you want to invoke `ollama` directly.
Expand Down
17 changes: 17 additions & 0 deletions docs/manage-sandboxes/recover-rebuild-sandboxes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,23 @@ Other workflows, including onboarding, rebuild, and `doctor --fix`, can explicit
`$$nemoclaw <name> gateway restart` instead restarts only the supported agent gateway inside the named sandbox.
</AgentOnly>

<AgentOnly variant="openclaw">
### Recover Portable Local Ollama

For a portable experimental-profile sandbox with the recorded `ollama-local` provider, `connect --probe-only` and `recover` also verify the host-side inference chain.
Before it decides whether to start Ollama, the command probes `http://127.0.0.1:11434/api/tags` and leaves a healthy daemon unchanged.
When that API is unhealthy, it starts the fixed user-local executable only if NemoClaw has a valid ownership receipt.
It refuses to launch a duplicate when another `ollama` process exists but the API remains unhealthy.
After a launch, recovery waits up to 30 seconds for `/api/tags` to return valid JSON with a `models` array.
It does not take over a system service or an unrelated user-managed daemon.
It refuses a symbolic link, non-regular file, or non-executable file at the receipt path.

The command also requires a trusted reachable route probe at `https://inference.local/v1/models` before it reports success.
When route repair needs the local Ollama dependency check, it verifies or recovers the authenticated proxy on port `11435` before it retries the route.
If Ollama does not become healthy within 30 seconds, the command identifies the exact receipt-bound executable and its `serve` argument, then tells you to retry recovery.
An Ollama startup or route failure exits non-zero and prints the available recovery guidance.
</AgentOnly>

## Rebuild While Preserving State

<AgentOnly variant="openclaw">
Expand Down
19 changes: 18 additions & 1 deletion docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1020,10 +1020,21 @@ Set `NEMOCLAW_NO_CONNECT_HINT=1` to suppress the hint in scripted workflows.
If the sandbox is running an outdated agent version, a non-blocking warning prints before connecting with a `$$nemoclaw <name> rebuild` hint.
If another terminal is already connected to the sandbox, `connect` prints a note with the number of existing sessions before proceeding. Multiple concurrent sessions are allowed.

`connect` does not pull or serve a model itself, but it does inspect managed-vLLM install variables such as `NEMOCLAW_VLLM_MODEL` and `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` if you exported them in the same shell.
`connect` does not pull a model itself, but it does inspect managed-vLLM install variables such as `NEMOCLAW_VLLM_MODEL` and `NEMOCLAW_VLLM_EXTRA_ARGS_JSON` if you exported them in the same shell.
An unknown model slug, malformed extra-args JSON, or a gated model (for example `deepseek-r1-distill-70b`) with no `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` exits non-zero with the same error the installer would emit, before any sandbox readiness probe or SSH attach.
Unset the managed-vLLM variable, or fix the value, before retrying.
Comment on lines +1023 to 1025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Document the --probe-only preflight exception.

These lines state that every connect inspects managed-vLLM variables. runConnectEntryPreflight skips that preflight when probeOnly is true. Qualify this text as applying to connect without --probe-only.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/reference/commands.mdx` around lines 1023 - 1025, Qualify the
managed-vLLM preflight behavior described in the connect documentation to apply
only when running connect without --probe-only. Update the related error and
retry guidance consistently, preserving the existing behavior for regular
connect while noting that runConnectEntryPreflight is skipped in probe-only
mode.


<AgentOnly variant="openclaw">
For a portable experimental-profile sandbox with the recorded `ollama-local` provider, `connect --probe-only` probes `http://127.0.0.1:11434/api/tags` before it decides whether to start Ollama.
If the API is unhealthy and the ownership receipt is valid, it starts the fixed user-local Ollama executable with `serve`.
It refuses to launch a duplicate when another `ollama` process exists but the API remains unhealthy.
After a launch, it waits up to 30 seconds for valid `/api/tags` JSON with a `models` array.
It then requires the existing sandbox `inference.local` route probe to return a trusted reachable result before it exits successfully.
When route repair needs the local Ollama dependency check, it verifies or recovers the authenticated proxy before it retries `inference.local`.
If Ollama does not become healthy within 30 seconds, the command identifies the exact receipt-bound executable and its `serve` argument, then tells you to retry recovery.
The command does not take over a system service or an unrelated user-managed Ollama daemon.
</AgentOnly>

Before reading or changing the live OpenShell gateway inference route, `connect` verifies the shared provider and sandbox metadata.
When the live route differs and the metadata is compatible, `connect` warns and re-points the route to the target sandbox's recorded provider and model.
Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for provider-global identity, route drift, and hard-error recovery.
Expand Down Expand Up @@ -1378,6 +1389,12 @@ $$nemoclaw my-assistant recover

</AgentOnly>

<AgentOnly variant="openclaw">

For a portable experimental-profile sandbox with the recorded `ollama-local` provider, `recover` also runs the ownership-bound Ollama and `inference.local` route reachability checks described for [`connect --probe-only`](#$$nemoclaw-name-connect).

</AgentOnly>

<AgentOnly variant="hermes">

`recover` re-evaluates the documented Hermes secret boundary against `/sandbox/.hermes/.env` and the supervisor runtime environment on every run, including when the gateway is already healthy.
Expand Down
1 change: 1 addition & 0 deletions docs/reference/host-files-and-state.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Share redacted diagnostics only.
| `~/.nemoclaw/onboard-session.json` | Resume marker for an onboarding attempt that failed before completion. | Yes, when you intentionally want to discard the failed session and start over. Prefer `$$nemoclaw onboard --fresh` when available. |
| `~/.nemoclaw/usage-notice.json` | Records the third-party software notice version in `acceptedVersion` and the acceptance time in `acceptedAt`. Install, onboarding, and rebuild flows consult this file and prompt again when its recorded version differs from the current notice or the file is absent. | Yes; deleting it makes the next applicable install, onboarding, or rebuild flow prompt for acceptance again. |
| `~/.nemoclaw/ollama-proxy-token` | Local auth token used by the host-side Ollama auth proxy. | Yes, but re-run onboarding afterward so NemoClaw recreates and registers the proxy token. |
| `~/.nemoclaw/ollama/user-local-ownership.json` | Mode-`0600`, credential-free receipt containing a schema version and the fixed `~/.local/bin/ollama` path. Portable OpenClaw recovery requires this receipt before it starts the binary. A successful system Ollama install removes the receipt. | No while a portable OpenClaw sandbox uses the user-local daemon. Deleting it disables automatic daemon restart until you reinstall Ollama through onboarding. |
| `~/.nemoclaw/dual-station-vllm-api-key` | Owner-only host-global bearer API key shared by authenticated NemoClaw-managed vLLM profiles. | No while any managed vLLM runtime uses it. Full uninstall removes it only after matching runtime cleanup succeeds and no sibling gateway remains. |
| `~/.nemoclaw/host-local-vllm-runtime.json` | Owner-only, credential-free receipt for a catalog-selected single-host vLLM runtime. It binds the immutable catalog, preset, and recipe digests to the exact container ID and API-key fingerprint. | No while that runtime exists. Recovery and full uninstall fail closed if its profile labels do not match this receipt, and uninstall removes it only after exact container cleanup succeeds. |
| `~/.nemoclaw/managed-llama-cpp/api-key` | Owner-only bearer API key mounted read-only into the managed llama.cpp container. | No while the managed llama.cpp runtime exists. A retry reuses the same key. Full uninstall removes it with the managed state only after runtime cleanup succeeds. |
Expand Down
21 changes: 21 additions & 0 deletions src/lib/actions/sandbox/connect-flow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,27 @@ describe("connectSandbox flow", () => {
);
});

it("probe-only mode exits before reporting success when inference.local returns no trusted result (#8502)", async () => {
const harness = createConnectHarness({
registryEntry: {
provider: "nvidia-prod",
model: "nvidia/nemotron-3-super-120b-a12b",
},
inferenceGetOutput: "Provider: nvidia-prod\nModel: nvidia/nemotron-3-super-120b-a12b\n",
inferenceProbeResponses: ["route probe unavailable"],
});

await expect(harness.connectSandbox("alpha", { probeOnly: true })).rejects.toThrow(
"process.exit(1)",
);

expect(harness.logSpy.mock.calls.flat().join("\n")).not.toContain("Probe complete");
expect(harness.errorSpy.mock.calls.flat().join("\n")).toContain(
"inference route is not known healthy",
);
expect(exitSpy).toHaveBeenCalledWith(1);
});

it("probe-only mode reports an ordinary running gateway for an already-running completion (#7919)", async () => {
const harness = createConnectHarness({
processCheck: {
Expand Down
13 changes: 7 additions & 6 deletions src/lib/actions/sandbox/connect.ts
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@ async function runSandboxConnectProbe(sandboxName: string): Promise<void> {
);
}
if (processCheck.wasRunning) {
await ensureSandboxInferenceRoute(sandboxName, agent, { quiet: true });
await ensureSandboxInferenceRouteOrExit(sandboxName, agent);
// Defense-in-depth scope-upgrade approval on the probe-only / `recover`
// path (#4504): the gateway is up, so deterministically clear any pending
// allowlisted CLI/webchat scope upgrade. Best-effort; never throws.
Expand All @@ -299,7 +299,7 @@ async function runSandboxConnectProbe(sandboxName: string): Promise<void> {
return;
}
if (processCheck.recovered) {
await ensureSandboxInferenceRoute(sandboxName, agent, { quiet: true });
await ensureSandboxInferenceRouteOrExit(sandboxName, agent);
// Same defense-in-depth approval after a recovery (#4504); best-effort.
runConnectAutoPairApprovalPass(sandboxName);
const managedControlCompletion =
Expand All @@ -313,7 +313,7 @@ async function runSandboxConnectProbe(sandboxName: string): Promise<void> {
}
return;
}
await ensureSandboxInferenceRoute(sandboxName, agent, { quiet: true });
await ensureSandboxInferenceRouteOrExit(sandboxName, agent);
console.error(
` Probe failed: ${agentName} gateway is not running in '${sandboxName}' and automatic recovery failed.`,
);
Expand Down Expand Up @@ -1087,9 +1087,10 @@ async function runConnectEntryPreflight(
console.error(` Error: ${error instanceof Error ? error.message : String(error)}`);
process.exit(1);
}
// probe-only / recover never install or serve a model, so skip the
// express-vLLM model preflight for them (it only steers the install path
// and would otherwise hard-exit a recovery on a stale NEMOCLAW_VLLM_MODEL).
// probe-only / recover can restart receipt-owned local inference, but they
// never select, install, or pull a model. Skip the express-vLLM model
// preflight because it only steers installation and can reject recovery on
// a stale NEMOCLAW_VLLM_MODEL.
if (!probeOnly) preflightVllmModelEnvOrExit();
const live = await ensureLiveSandboxOrExit(sandboxName, {
allowNonReadyPhase: true,
Expand Down
4 changes: 2 additions & 2 deletions src/lib/actions/sandbox/gateway-state.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,13 +91,13 @@ function gatewayScopedArgs(args: string[], gatewayName?: string): string[] {
/** Recover a receipt-bound portable sandbox before the live lookup rejects a stopped container. */
export function recoverPortableDemoSandboxLifecycleForConnect(
sandboxName: string,
sandbox: Pick<SandboxEntry, "agent"> | null,
sandbox: Pick<SandboxEntry, "agent" | "provider"> | null,
gatewayName: string,
): PortableDemoLifecycleRecoveryResult {
if (!sandbox) return { kind: "not-installed" };
return recoverPortableDemoSandboxLifecycle(
sandboxName,
{ agent: sandbox.agent, gatewayName },
{ agent: sandbox.agent, gatewayName, provider: sandbox.provider },
{
openshellBinary: getOpenshellBinary(),
captureOpenshell: (args, timeoutMs) => {
Expand Down
75 changes: 75 additions & 0 deletions src/lib/onboard/experimental/ollama-user-local-runtime.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import fs from "node:fs";
import os from "node:os";
import path from "node:path";

import { afterEach, describe, expect, it } from "vitest";

import {
loadUserLocalOllamaOwnership,
recordUserLocalOllamaOwnership,
removeUserLocalOllamaOwnership,
userLocalOllamaOwnershipInternals,
} from "./ollama-user-local-runtime";

const temporaryDirectories: string[] = [];

function createFixture(): { homeDir: string; stateDir: string; binPath: string } {
const homeDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-ollama-owner-"));
temporaryDirectories.push(homeDir);
return {
homeDir,
stateDir: path.join(homeDir, ".nemoclaw"),
binPath: path.join(homeDir, ".local", "bin", "ollama"),
};
}

afterEach(() => {
for (const directory of temporaryDirectories.splice(0)) {
fs.rmSync(directory, { force: true, recursive: true });
}
});

describe("user-local Ollama ownership receipt", () => {
it("records and reloads only the fixed NemoClaw user-local path (#8502)", () => {
const fixture = createFixture();

recordUserLocalOllamaOwnership(fixture.binPath, fixture);

expect(loadUserLocalOllamaOwnership(fixture)).toBe(fixture.binPath);
const receipt = userLocalOllamaOwnershipInternals.receiptPath(fixture);
expect(fs.statSync(receipt).mode & 0o777).toBe(0o600);
});

it("refuses to record an Ollama path outside the fixed user-local install (#8502)", () => {
const fixture = createFixture();

expect(() => recordUserLocalOllamaOwnership("/usr/local/bin/ollama", fixture)).toThrow(
"unexpected user-local Ollama path",
);
});

it("rejects a receipt that redirects recovery to another executable (#8502)", () => {
const fixture = createFixture();
const receipt = userLocalOllamaOwnershipInternals.receiptPath(fixture);
fs.mkdirSync(path.dirname(receipt), { recursive: true });
fs.writeFileSync(
receipt,
`${JSON.stringify({ schemaVersion: 1, binPath: "/tmp/unrelated" })}\n`,
{ mode: 0o600 },
);

expect(() => loadUserLocalOllamaOwnership(fixture)).toThrow("ownership receipt is invalid");
});

it("removes obsolete ownership after a system installation (#8502)", () => {
const fixture = createFixture();
recordUserLocalOllamaOwnership(fixture.binPath, fixture);

removeUserLocalOllamaOwnership(fixture);

expect(loadUserLocalOllamaOwnership(fixture)).toBeNull();
});
});
120 changes: 120 additions & 0 deletions src/lib/onboard/experimental/ollama-user-local-runtime.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import fs from "node:fs";
import os from "node:os";
import path from "node:path";

import { openRegularFileNoFollow } from "../../adapters/fs/regular-file";
import { OLLAMA_PORT } from "../../core/ports";
import { ensureConfigDir } from "../../state/config-io";

export { OLLAMA_PORT };

const RECEIPT_DIRECTORY = "ollama";
const RECEIPT_FILE = "user-local-ownership.json";
const MAX_RECEIPT_BYTES = 1024;

interface UserLocalOllamaOwnershipReceipt {
schemaVersion: 1;
binPath: string;
}

export interface UserLocalOllamaOwnershipDeps {
homeDir?: string;
stateDir?: string;
}

function resolveHomeDir(deps: UserLocalOllamaOwnershipDeps): string {
return deps.homeDir ?? os.homedir();
}

function resolveStateDir(deps: UserLocalOllamaOwnershipDeps): string {
return deps.stateDir ?? path.join(resolveHomeDir(deps), ".nemoclaw");
}

function receiptPath(deps: UserLocalOllamaOwnershipDeps): string {
return path.join(resolveStateDir(deps), RECEIPT_DIRECTORY, RECEIPT_FILE);
}

function expectedBinPath(deps: UserLocalOllamaOwnershipDeps): string {
return path.join(resolveHomeDir(deps), ".local", "bin", "ollama");
}

function parseReceipt(
value: unknown,
deps: UserLocalOllamaOwnershipDeps,
): UserLocalOllamaOwnershipReceipt {
if (typeof value !== "object" || value === null || Array.isArray(value)) {
throw new Error("NemoClaw-managed user-local Ollama ownership receipt is malformed");
}
const receipt = value as Record<string, unknown>;
if (
Object.keys(receipt).sort().join(",") !== "binPath,schemaVersion" ||
receipt.schemaVersion !== 1 ||
receipt.binPath !== expectedBinPath(deps)
) {
throw new Error("NemoClaw-managed user-local Ollama ownership receipt is invalid");
}
return receipt as unknown as UserLocalOllamaOwnershipReceipt;
}

/** Record the fixed user-local binary only after NemoClaw starts it successfully. */
export function recordUserLocalOllamaOwnership(
binPath: string,
deps: UserLocalOllamaOwnershipDeps = {},
): void {
if (binPath !== expectedBinPath(deps)) {
throw new Error("NemoClaw refused to record an unexpected user-local Ollama path");
}
const target = receiptPath(deps);
ensureConfigDir(path.dirname(target));
let file;
try {
file = openRegularFileNoFollow(target, { writable: true });
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
file = openRegularFileNoFollow(target, { create: true, mode: 0o600, writable: true });
}
try {
const receipt: UserLocalOllamaOwnershipReceipt = { schemaVersion: 1, binPath };
file.replaceUtf8(`${JSON.stringify(receipt, null, 2)}\n`, 0o600);
} finally {
file.close();
}
}

/** Load the exact receipt-bound user-local path, or null when no receipt exists. */
export function loadUserLocalOllamaOwnership(
deps: UserLocalOllamaOwnershipDeps = {},
): string | null {
let file;
try {
file = openRegularFileNoFollow(receiptPath(deps));
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
throw error;
}
try {
const receipt = parseReceipt(JSON.parse(file.readUtf8(MAX_RECEIPT_BYTES)), deps);
return receipt.binPath;
} catch (error) {
if (error instanceof SyntaxError) {
throw new Error("NemoClaw-managed user-local Ollama ownership receipt is malformed");
}
throw error;
} finally {
file.close();
}
}

/** Remove stale user-local ownership after a successful system installation. */
export function removeUserLocalOllamaOwnership(deps: UserLocalOllamaOwnershipDeps = {}): void {
try {
fs.unlinkSync(receiptPath(deps));
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
}
}

export const userLocalOllamaOwnershipInternals = { receiptPath };
Loading
Loading