Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1181,7 +1181,7 @@ Use this form when you care about a specific sandbox's live OpenShell state, age
Do not pass a sandbox name to `$$nemoclaw status`; that command is the global all-sandbox/service overview.

Pass `--json` to emit a structured per-sandbox report instead of the text renderer.
The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `failureLayer`, `terminalRuntimeHealth`, and `dockerPaused`.
The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`.
The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable.
Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route.
When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`.
Expand All @@ -1193,6 +1193,10 @@ Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-ga
In that case, text output keeps OpenShell's authoritative phase but prints a `docker unpause <container>` recovery hint instead of sending you directly to rebuild.
For terminal runtime sandboxes, the command also checks cgroup OOM kill counters.
If the counter records an OOM kill, text output prints `Runtime health: degraded (... OOM kill recorded)` and points you to `$$nemoclaw <name> rebuild`; JSON output reports `terminalRuntimeHealth.kind: "degraded"` with the OOM kill count and source counter path.
For a present gateway runtime, text output prints `Serving process (<agent> gateway): not checked`, and JSON output reports `servingProcessHealth: { "checked": false }`.
The existing inference probes run in a fresh sandbox command, so they do not attest that the long-running gateway process has equivalent inference access.
NemoClaw does not probe the serving process yet.
For terminal runtimes, `servingProcessHealth` is `null` and the text output omits this line because there is no long-running gateway process.
The command exits non-zero when the sandbox is missing locally, the gateway state is not `present`, the gateway reports a schema/protobuf mismatch (mirrored as `rpcIssue`), `failureLayer` is non-null, the authoritative in-sandbox inference route fails or cannot be probed, or a terminal runtime sandbox reports a recorded OOM kill.
The alias form `$$nemoclaw <name> status --json` requires the sandbox to be registered locally; the canonical form `$$nemoclaw sandbox status <name> --json` is the one to use from automation that may run against an unknown sandbox name, since it still emits a JSON document with `found: false` instead of a text error.

Expand Down Expand Up @@ -1355,6 +1359,9 @@ For inference health, `doctor` treats the probe to `https://inference.local/v1/m
HTTP responses from `200` through `499`, including `401` and `403`, pass this check.
HTTP `500` through `599`, interim `100` through `199`, transport failures with status `000`, invalid status values, and an unavailable authoritative probe fail the check.
Direct provider and upstream probes use the same authenticated model-invocation checks as status and remain diagnostic only, so their failure does not fail `doctor` when the authoritative in-sandbox route is reachable.
For gateway runtimes, `doctor` also reports an informational `Serving process: not checked` result because its fresh sandbox probes do not attest the long-running gateway process.
This result does not fail the readiness check.
Terminal runtimes omit it because they have no long-running gateway process.

Warnings do not make the command fail.
Failed checks, including a failed or unavailable authoritative inference route, exit non-zero so scripts can use `doctor` as a readiness gate.
Expand Down
55 changes: 52 additions & 3 deletions src/lib/actions/sandbox/doctor-flow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,43 @@ describe("runSandboxDoctor flow", () => {
},
);

it.each([
"openclaw",
"hermes",
] as const)("keeps serving-process health explicitly unchecked for the %s gateway (#7003)", async (agent) => {
const harness = createDoctorHarness();
harness.loadAgentSpy.mockReturnValue({
name: agent,
runtime: { kind: "gateway" },
configPaths: {
dir: "/sandbox/.agent",
configFile: "config.json",
format: "json",
},
});
harness.getSandboxSpy.mockReturnValue({
name: "alpha",
agent,
model: "registry-model",
provider: "ollama-local",
openshellDriver: "docker",
gatewayName: "nemoclaw-19080",
gatewayPort: 19080,
});

const report = await harness.runSandboxDoctor("alpha", ["--json"], { quietJson: true });

expect(harness.loadAgentSpy).toHaveBeenCalledWith(agent);
expect(report?.checks).toContainEqual(
expect.objectContaining({
group: "Inference",
label: "Serving process",
status: "info",
detail: "not checked — serving-process probing is not implemented",
}),
);
});

it("rejects mutating --fix when JSON output was requested", async () => {
const harness = createDoctorHarness();

Expand Down Expand Up @@ -412,21 +449,33 @@ describe("runSandboxDoctor flow", () => {
]);
});

it("skips OpenClaw tool-scope checks for other agents", async () => {
it("skips gateway-specific and OpenClaw checks for terminal agents", async () => {
const harness = createDoctorHarness();
harness.getSandboxSpy.mockReturnValue({
name: "alpha",
agent: "hermes",
agent: "langchain-deepagents-code",
model: "registry-model",
provider: "ollama-local",
openshellDriver: "docker",
gatewayName: "nemoclaw-19080",
gatewayPort: 19080,
});
harness.loadAgentSpy.mockReturnValue({
name: "langchain-deepagents-code",
runtime: { kind: "terminal", interactive_command: "deepagents" },
configPaths: {
dir: "/sandbox/.deepagents",
configFile: "config.json",
format: "json",
},
});

await harness.runSandboxDoctor("alpha", ["--json"], { quietJson: true });
const report = await harness.runSandboxDoctor("alpha", ["--json"], { quietJson: true });

expect(harness.buildToolScopeChecksSpy).not.toHaveBeenCalled();
expect(report?.checks).not.toContainEqual(
expect.objectContaining({ group: "Inference", label: "Serving process" }),
);
});

it("appends the local gateway result without mutating provider health", async () => {
Expand Down
35 changes: 35 additions & 0 deletions src/lib/actions/sandbox/doctor-inference.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,41 @@ describe("doctor inference checks", () => {
);
});

it("keeps serving-process health explicitly unchecked until a probe contract exists (#7003)", async () => {
const checks = await collectInferenceChecks(
"alpha",
{ provider: "nvidia-prod", model: "model" },
true,
{
probeProviderHealthImpl: () => upstream(),
probeSandboxInferenceGatewayHealthImpl: async () => gateway(true),
},
);

expect(checks).toContainEqual(
expect.objectContaining({
label: "Serving process",
status: "info",
detail: "not checked — serving-process probing is not implemented",
}),
);
});

it("omits serving-process health for terminal agents without a gateway process (#7003)", async () => {
const checks = await collectInferenceChecks(
"alpha",
{ provider: "nvidia-prod", model: "model" },
true,
{
probeProviderHealthImpl: () => upstream(),
probeSandboxInferenceGatewayHealthImpl: async () => gateway(true),
includeServingProcessCheck: false,
},
);

expect(checks).not.toContainEqual(expect.objectContaining({ label: "Serving process" }));
});

it("does not mutate direct provider health while adding route evidence", async () => {
const providerHealth = upstream();

Expand Down
14 changes: 14 additions & 0 deletions src/lib/actions/sandbox/doctor-inference.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ export type DoctorInferenceRoute = {
type DoctorInferenceDeps = {
probeProviderHealthImpl?: typeof probeProviderHealth;
probeSandboxInferenceGatewayHealthImpl?: typeof probeSandboxInferenceGatewayHealth;
/** False for terminal agents that do not have a long-running gateway serving process. */
includeServingProcessCheck?: boolean;
};

function pushInferenceHealthCheck(
Expand Down Expand Up @@ -156,5 +158,17 @@ export async function collectInferenceChecks(
)) {
pushInferenceHealthCheck(checks, diagnostic, { authoritative: false });
}
// Serving-process leg: the above probes run in a fresh exec with OpenShell's
// injected env, so they cannot attest what the long-running gateway process
// can reach. Until NemoClaw defines and implements a process-owned probe
// contract, keep this honest result explicit (#7003).
if (deps.includeServingProcessCheck !== false) {
checks.push({
group: "Inference",
label: "Serving process",
status: "info",
detail: "not checked — serving-process probing is not implemented",
});
}
return checks;
}
16 changes: 15 additions & 1 deletion src/lib/actions/sandbox/doctor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { stripAnsi } from "../../adapters/openshell/client";
import { resolveOpenshell } from "../../adapters/openshell/resolve";
import { captureOpenshell } from "../../adapters/openshell/runtime";
import { OPENSHELL_PROBE_TIMEOUT_MS } from "../../adapters/openshell/timeouts";
import { getAgentRuntimeKind, loadAgent } from "../../agent/defs";
import * as agentRuntime from "../../agent/runtime";
import { CLI_NAME } from "../../cli/branding";
import { GATEWAY_PORT } from "../../core/ports";
Expand Down Expand Up @@ -386,6 +387,17 @@ function collectToolScopeChecks(
});
}

function shouldReportServingProcessHealth(agentName: string | null | undefined): boolean {
const resolvedName = agentName || "openclaw";
try {
return getAgentRuntimeKind(loadAgent(resolvedName)) === "gateway";
} catch {
// Status preserves OpenClaw's gateway default if its manifest cannot be
// loaded, while unknown non-default agents are classified as unknown.
return resolvedName === "openclaw";
}
}

async function collectDoctorChecks(
sandboxName: string,
sb: SandboxEntry | null | undefined,
Expand All @@ -400,7 +412,9 @@ async function collectDoctorChecks(
...host.checks,
...gateway.checks,
...sandbox.checks,
...(await collectInferenceChecks(sandboxName, route, sandbox.reachable)),
...(await collectInferenceChecks(sandboxName, route, sandbox.reachable, {
includeServingProcessCheck: shouldReportServingProcessHealth(sb?.agent),
})),
...collectRegisteredSandboxChecks(sandboxName, sb, intent.wantsFix, sandbox.reachable),
...collectToolScopeChecks(sandboxName, sb, sandbox.reachable, intent.wantsFix),
ollamaDoctorCheck(route.provider),
Expand Down
15 changes: 15 additions & 0 deletions src/lib/actions/sandbox/status-flow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@ describe("showSandboxStatus flow", () => {
expect(output).toContain("Model: nvidia/nemotron");
expect(output).toContain("Inference: reachable");
expect(output).toContain("Inference (ollama backend):");
expect(output).toContain("Serving process (openclaw gateway):");
expect(output).toContain("not checked");
expect(output).toContain("Host GPU: yes");
expect(output).toContain("last CUDA proof failed: cuInit");
expect(output).toContain("CUDA initialization failed");
Expand All @@ -128,6 +130,19 @@ describe("showSandboxStatus flow", () => {
expect(exitSpy).not.toHaveBeenCalled();
});

it("omits serving-process status when the gateway is unavailable (#7003)", async () => {
const harness = createStatusFlowHarness({
lookupState: "missing",
servingProcessHealth: null,
});

await expect(harness.showSandboxStatus("alpha")).rejects.toThrow("process.exit(1)");

const output = harness.logSpy.mock.calls.map((call) => String(call[0])).join("\n");
expect(output).not.toContain("Serving process");
expect(exitSpy).toHaveBeenCalledWith(1);
});

it.each([
{ label: "unreachable" as const, detail: "inference.local is unreachable" },
{ label: "unhealthy" as const, detail: "inference.local returned HTTP 503" },
Expand Down
58 changes: 52 additions & 6 deletions src/lib/actions/sandbox/status-inference.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,16 @@
// SPDX-License-Identifier: Apache-2.0

import { describe, expect, it, vi } from "vitest";
import { collectSandboxStatusSnapshot, getSandboxStatusInferenceHealth } from "./status";
import {
collectSandboxStatusSnapshot,
getSandboxStatusInferenceHealth,
getSandboxStatusReport,
} from "./status";

describe("sandbox status inference.local route health (#6192)", () => {
function snapshotDeps(options: {
agent?: string;
lookupState?: "present" | "missing";
provider?: string;
liveProvider?: string;
liveModel?: string;
Expand All @@ -23,17 +29,17 @@ describe("sandbox status inference.local route health (#6192)", () => {
const reportInferenceProbeError = vi.fn();
const sandbox = {
name: "alpha",
agent: "openclaw",
agent: options.agent ?? "openclaw",
model: "nvidia/nemotron",
provider,
};
return {
getSandbox: () => sandbox,
listSandboxes: () => ({ sandboxes: [sandbox], defaultSandbox: "alpha" }),
reconcile: async () => ({
state: "present" as const,
output: "Name: alpha\nPhase: Ready\n",
}),
reconcile: async () =>
options.lookupState === "missing"
? { state: "missing" as const, output: "sandbox alpha not found" }
: { state: "present" as const, output: "Name: alpha\nPhase: Ready\n" },
captureOpenshellForStatusImpl: async () =>
({
status: 0,
Expand All @@ -51,6 +57,7 @@ describe("sandbox status inference.local route health (#6192)", () => {
? async () => Promise.reject(new Error("openshell unavailable TOKEN=super-secret"))
: async () => options.routeHealth,
),
probeTerminalRuntimeHealth: vi.fn(() => ({ kind: "ok" as const, oomKillCount: 0 as const })),
reportInferenceProbeError,
};
}
Expand Down Expand Up @@ -83,6 +90,45 @@ describe("sandbox status inference.local route health (#6192)", () => {
expect(snapshot.inferenceHealth?.subprobes).toEqual([
expect.objectContaining({ ok: true, probeLabel: "upstream" }),
]);
expect(snapshot.servingProcessHealth).toEqual({ checked: false });

const report = await getSandboxStatusReport("alpha", deps);
expect(report.servingProcessHealth).toEqual({ checked: false });
});

it("does not invent serving-process health for terminal agents (#7003)", async () => {
const deps = snapshotDeps({
agent: "langchain-deepagents-code",
routeHealth: {
ok: true,
endpoint: "https://inference.local/v1/models",
httpStatus: 200,
detail: "route reachable",
},
});

const snapshot = await collectSandboxStatusSnapshot("alpha", { deps });

expect(snapshot.servingProcessHealth).toBeNull();
expect(deps.probeTerminalRuntimeHealth).toHaveBeenCalledWith("alpha");

const report = await getSandboxStatusReport("alpha", deps);
expect(report.servingProcessHealth).toBeNull();
});

it("does not invent serving-process health when the gateway is unavailable (#7003)", async () => {
const deps = snapshotDeps({
lookupState: "missing",
routeHealth: null,
});

const snapshot = await collectSandboxStatusSnapshot("alpha", { deps });

expect(snapshot.servingProcessHealth).toBeNull();
expect(deps.probeSandboxInferenceGatewayHealthImpl).not.toHaveBeenCalled();

const report = await getSandboxStatusReport("alpha", deps);
expect(report.servingProcessHealth).toBeNull();
});

it.each([
Expand Down
Loading
Loading