From 657f105603d3983d09c67f68379e8f48d0d35cbf Mon Sep 17 00:00:00 2001 From: Hai Nguyen Date: Thu, 3 Sep 2026 10:27:54 +0000 Subject: [PATCH 01/12] fix(status): retry a transient inference request refusal `nemoclaw status` exited nonzero for a Phase Ready sandbox when the one in-sandbox inference request it sends came back HTTP 503, while the same output still reported route reachability as reachable, the upstream provider as healthy, and the phase as Ready. `collectSandboxStatusSnapshot` already wrapped the route and invocation probes in `retryUntilAsync`, but derived the attempt count from `recoveredManagedGateway`, which requires this run to have restarted a dead gateway. A Ready sandbox whose gateway is already up therefore got exactly one attempt, so a single transient gateway or availability answer became `failureLabel: "unhealthy"` and exit 1. Move the retry policy out of the attempt count and into the `accept` predicate: retry only when the inference request itself was refused with HTTP 429, 502, 503, or 504, the same signature the onboarding probes already treat as transient. A route that never serves the request still reports unhealthy and exits nonzero after three bounded attempts, and HTTP 401, 403, 404, and 500, an invalid 2xx body, a statusless request, and a failing /v1/models route probe all stay final on the first attempt with no added delay. Fixes #10709 Signed-off-by: Hai Nguyen --- docs/reference/commands.mdx | 2 +- .../status-snapshot-inference-health.test.ts | 214 ++++++++++++++++++ src/lib/actions/sandbox/status-snapshot.ts | 51 ++++- 3 files changed, 259 insertions(+), 8 deletions(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index b559e619624..8f0204d13db 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -1397,7 +1397,7 @@ For Portable Hermes, `status` reports `Portable lifecycle phase: pending`, `conf For a `compatible-endpoint` route that uses `openai-completions`, the text output prints `Reasoning effort` as `low`, `medium`, `high`, or `endpoint-default`. The line is omitted for another provider or API family. -Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request through the stored provider credential, with a 30-second timeout, and consumes provider tokens on a hosted route. When the same `status` run recovers a managed gateway, it retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts. Each attempt can consume another 16 tokens on a hosted route. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed: +Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request through the stored provider credential, with a 30-second timeout, and consumes provider tokens on a hosted route. When the inference request returns HTTP `429`, `502`, `503`, or `504`, `status` retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts, because those statuses are transient gateway and availability answers rather than evidence that the route is broken. It retries the same way after the run recovers a managed gateway. Every other failure is final on the first attempt with no delay: HTTP `401`, `403`, `404`, and `500`, an invalid 2xx response body, a request that returned no HTTP status, and a failing `/v1/models` route probe. Each attempt can consume another 16 tokens on a hosted route. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed: - `unauthorized` when the route rejected it with HTTP `401` or `403`. - `unhealthy` when the route returned another failing HTTP status or an invalid 2xx response body. diff --git a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts index 953a831ba14..b02e78333bf 100644 --- a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts +++ b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts @@ -730,4 +730,218 @@ describe("collectSandboxStatusSnapshot inference route health", () => { expect(snapshot.inferenceHealth).toMatchObject({ ok: false }); expect(snapshot.inferenceHealth?.okLabel).toBeUndefined(); }); + + it("serves a Ready managed sandbox after one transient HTTP 503 inference request (#10709)", async () => { + const healthy: SandboxInferenceRouteHealth = { + ok: true, + endpoint: "https://inference.local/v1/models", + httpStatus: 200, + detail: "reachable", + }; + const options = snapshotDeps(healthy, null, { ok: true }, { openshellDriver: "docker" }); + options.deps.reconcile = async () => ({ + state: "present", + phase: "Ready", + output: "Phase: Ready", + }); + const probeSandboxInferenceInvocationImpl = vi + .fn() + .mockReturnValueOnce({ + ok: false, + detail: "sandbox inference invocation probe returned HTTP 503", + httpStatus: 503, + }) + .mockReturnValueOnce({ ok: true }); + const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const recoverSandboxProcesses = vi.fn(() => ({ + checked: true, + wasRunning: true, + recovered: false, + })) as never; + + const snapshot = await collectSandboxStatusSnapshot("alpha", { + ...options, + deps: { + ...options.deps, + delayInferenceRecoveryProbe, + probeSandboxInferenceInvocationImpl, + recoverSandboxProcesses, + }, + }); + + expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes(2); + expect(delayInferenceRecoveryProbe).toHaveBeenCalledOnce(); + expect(delayInferenceRecoveryProbe).toHaveBeenCalledWith(2_000); + expect(snapshot.inferenceHealth).toMatchObject({ ok: true, probed: true }); + }); + + it("reports unhealthy after three transient HTTP 503 inference requests (#10709)", async () => { + const healthy: SandboxInferenceRouteHealth = { + ok: true, + endpoint: "https://inference.local/v1/models", + httpStatus: 200, + detail: "reachable", + }; + const refused: SandboxInferenceInvocationResult = { + ok: false, + detail: "sandbox inference invocation probe returned HTTP 503", + httpStatus: 503, + }; + const options = snapshotDeps(healthy, null, { ok: true }, { openshellDriver: "docker" }); + options.deps.reconcile = async () => ({ + state: "present", + phase: "Ready", + output: "Phase: Ready", + }); + const probeSandboxInferenceInvocationImpl = vi.fn(() => refused); + const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const recoverSandboxProcesses = vi.fn(() => ({ + checked: true, + wasRunning: true, + recovered: false, + })) as never; + + const snapshot = await collectSandboxStatusSnapshot("alpha", { + ...options, + deps: { + ...options.deps, + delayInferenceRecoveryProbe, + probeSandboxInferenceInvocationImpl, + recoverSandboxProcesses, + }, + }); + + expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes(3); + expect(delayInferenceRecoveryProbe).toHaveBeenCalledTimes(2); + expect(delayInferenceRecoveryProbe).toHaveBeenCalledWith(2_000); + expect(snapshot.inferenceHealth).toMatchObject({ ok: false, failureLabel: "unhealthy" }); + expect(snapshot.inferenceHealth?.subprobes).toContainEqual( + expect.objectContaining({ probeLabel: "route reachability", okLabel: "reachable" }), + ); + }); + + it.each([ + { + label: "unauthorized", + invocation: { + ok: false as const, + detail: "sandbox inference invocation probe returned HTTP 401", + httpStatus: 401, + }, + failureLabel: "unauthorized", + }, + { + label: "not found", + invocation: { + ok: false as const, + detail: "sandbox inference invocation probe returned HTTP 404", + httpStatus: 404, + }, + failureLabel: "unhealthy", + }, + { + label: "internal error", + invocation: { + ok: false as const, + detail: "sandbox inference invocation probe returned HTTP 500", + httpStatus: 500, + }, + failureLabel: "unhealthy", + }, + { + label: "invalid response body", + invocation: { + ok: false as const, + detail: "sandbox inference invocation probe returned an invalid response body", + httpStatus: 200, + }, + failureLabel: "unhealthy", + }, + { + label: "statusless", + invocation: { + ok: false as const, + detail: "sandbox inference invocation probe was unavailable", + httpStatus: null, + }, + failureLabel: "unreachable", + }, + ])("fails a $label inference request on the first attempt (#10709)", async (testCase) => { + const healthy: SandboxInferenceRouteHealth = { + ok: true, + endpoint: "https://inference.local/v1/models", + httpStatus: 200, + detail: "reachable", + }; + const options = snapshotDeps(healthy, null, { ok: true }, { openshellDriver: "docker" }); + options.deps.reconcile = async () => ({ + state: "present", + phase: "Ready", + output: "Phase: Ready", + }); + const probeSandboxInferenceInvocationImpl = vi.fn(() => testCase.invocation); + const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const recoverSandboxProcesses = vi.fn(() => ({ + checked: true, + wasRunning: true, + recovered: false, + })) as never; + + const snapshot = await collectSandboxStatusSnapshot("alpha", { + ...options, + deps: { + ...options.deps, + delayInferenceRecoveryProbe, + probeSandboxInferenceInvocationImpl, + recoverSandboxProcesses, + }, + }); + + expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledOnce(); + expect(delayInferenceRecoveryProbe).not.toHaveBeenCalled(); + expect(snapshot.inferenceHealth).toMatchObject({ + ok: false, + failureLabel: testCase.failureLabel, + }); + }); + + it("fails a 5xx models route on the first attempt without sending an inference request (#10709)", async () => { + const brokenRoute: SandboxInferenceRouteHealth = { + ok: false, + endpoint: "https://inference.local/v1/models", + httpStatus: 503, + detail: + "Inference gateway returned HTTP 503 on https://inference.local/v1/models; the route is reachable but unhealthy.", + }; + const options = snapshotDeps(brokenRoute, null, { ok: true }, { openshellDriver: "docker" }); + options.deps.reconcile = async () => ({ + state: "present", + phase: "Ready", + output: "Phase: Ready", + }); + const probeSandboxInferenceGatewayHealthImpl = vi.fn(async () => brokenRoute); + const probeSandboxInferenceInvocationImpl = vi.fn(() => ({ ok: true }) as const); + const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const recoverSandboxProcesses = vi.fn(() => ({ + checked: true, + wasRunning: true, + recovered: false, + })) as never; + + const snapshot = await collectSandboxStatusSnapshot("alpha", { + ...options, + deps: { + ...options.deps, + delayInferenceRecoveryProbe, + probeSandboxInferenceGatewayHealthImpl, + probeSandboxInferenceInvocationImpl, + recoverSandboxProcesses, + }, + }); + + expect(probeSandboxInferenceGatewayHealthImpl).toHaveBeenCalledOnce(); + expect(probeSandboxInferenceInvocationImpl).not.toHaveBeenCalled(); + expect(delayInferenceRecoveryProbe).not.toHaveBeenCalled(); + expect(snapshot.inferenceHealth).toMatchObject({ ok: false, failureLabel: "unhealthy" }); + }); }); diff --git a/src/lib/actions/sandbox/status-snapshot.ts b/src/lib/actions/sandbox/status-snapshot.ts index 98b8438ecb5..3321d614079 100644 --- a/src/lib/actions/sandbox/status-snapshot.ts +++ b/src/lib/actions/sandbox/status-snapshot.ts @@ -63,8 +63,38 @@ type ProbeProviderHealth = ( type ProbeSandboxInferenceGatewayHealth = typeof probeSandboxInferenceGatewayHealth; type DelayInferenceRecoveryProbe = (delayMs: number) => Promise; -const RECOVERED_INFERENCE_PROBE_ATTEMPTS = 3; -const RECOVERED_INFERENCE_PROBE_DELAY_MS = 2_000; +const INFERENCE_PROBE_ATTEMPTS = 3; +const INFERENCE_PROBE_RETRY_DELAY_MS = 2_000; + +// 429 = Too Many Requests; 502/503/504 = gateway and availability answers from +// the proxy in front of `inference.local` and from hosted providers. This is +// the same signature `src/lib/inference/probe-retry.ts` already retries for the +// onboarding probes (#2980, #3033); that module is CommonJS and `@ts-nocheck`, +// so it cannot export the set to a typed module. A Ready sandbox must not +// report a failing route because one of these landed in the single request +// `status` sends (#10709). +const TRANSIENT_INFERENCE_INVOCATION_STATUSES: ReadonlySet = new Set([429, 502, 503, 504]); + +/** + * True only when the inference request itself was refused with a transient + * gateway or availability status, so one more 16-token request is worth + * sending. + * + * HTTP 401, 403, 404, and 500, an invalid 2xx response body, and a request that + * never reached an HTTP status all return false. Those stay final on the first + * attempt and add no delay to the interactive `status` path. A failing + * `/v1/models` route probe is not retried here either: it reports a different + * hop, and status renders the route probe result for that hop. + */ +function inferenceInvocationFailureIsTransient( + invocation: ReturnType | null, +): boolean { + if (invocation === null || invocation.ok) return false; + return ( + invocation.httpStatus !== null && + TRANSIENT_INFERENCE_INVOCATION_STATUSES.has(invocation.httpStatus) + ); +} /** * Honest serving-process state while the self-report response and probe @@ -597,7 +627,6 @@ export async function collectSandboxStatusSnapshot( try { const probe = opts.deps?.probeSandboxInferenceGatewayHealthImpl ?? probeSandboxInferenceGatewayHealth; - const attempts = recoveredManagedGateway ? RECOVERED_INFERENCE_PROBE_ATTEMPTS : 1; await retryUntilAsync( async () => { gatewayChain = gatewayName ? await probe(sandboxName, { gatewayName }) : null; @@ -623,11 +652,19 @@ export async function collectSandboxStatusSnapshot( return { gatewayChain, invocation }; }, { - accept: ({ gatewayChain: chain, invocation: result }) => - Boolean(chain?.ok && (!canProbeInvocation || result?.ok)), + accept: ({ gatewayChain: chain, invocation: result }) => { + if (chain?.ok && (!canProbeInvocation || result?.ok)) return true; + // After this run recovered a managed gateway, keep waiting for the + // restarted chain to settle whatever the failure shape (#8572). + if (recoveredManagedGateway) return false; + // Otherwise retry only a transient gateway or availability status on + // the inference request. Every other request failure, and every + // route probe failure, is final on the first attempt (#10709). + return !inferenceInvocationFailureIsTransient(result); + }, retryDelaysMs: Array.from( - { length: attempts - 1 }, - () => RECOVERED_INFERENCE_PROBE_DELAY_MS, + { length: INFERENCE_PROBE_ATTEMPTS - 1 }, + () => INFERENCE_PROBE_RETRY_DELAY_MS, ), sleep: opts.deps?.delayInferenceRecoveryProbe ?? sleep, }, From f80c299cb8a372654e76c65e1a384199cb758c7e Mon Sep 17 00:00:00 2001 From: Hai Nguyen Date: Thu, 3 Sep 2026 10:39:24 +0000 Subject: [PATCH 02/12] docs(status): scope the first-attempt rule to an ordinary run The retry sentence read as if every non-transient failure were final on the first attempt. That is true only for an ordinary run: after the same run recovers a managed gateway, `status` still retries any failed route or inference probe while the restarted delivery chain settles. Name both paths so the timing is unambiguous. Signed-off-by: Hai Nguyen --- docs/reference/commands.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 8f0204d13db..1f649438630 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -1397,7 +1397,7 @@ For Portable Hermes, `status` reports `Portable lifecycle phase: pending`, `conf For a `compatible-endpoint` route that uses `openai-completions`, the text output prints `Reasoning effort` as `low`, `medium`, `high`, or `endpoint-default`. The line is omitted for another provider or API family. -Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request through the stored provider credential, with a 30-second timeout, and consumes provider tokens on a hosted route. When the inference request returns HTTP `429`, `502`, `503`, or `504`, `status` retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts, because those statuses are transient gateway and availability answers rather than evidence that the route is broken. It retries the same way after the run recovers a managed gateway. Every other failure is final on the first attempt with no delay: HTTP `401`, `403`, `404`, and `500`, an invalid 2xx response body, a request that returned no HTTP status, and a failing `/v1/models` route probe. Each attempt can consume another 16 tokens on a hosted route. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed: +Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request through the stored provider credential, with a 30-second timeout, and consumes provider tokens on a hosted route. When the inference request returns HTTP `429`, `502`, `503`, or `504`, `status` retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts, because those statuses are transient gateway and availability answers rather than evidence that the route is broken. On an ordinary run, every other failure is final on the first attempt with no delay: HTTP `401`, `403`, `404`, and `500`, an invalid 2xx response body, a request that returned no HTTP status, and a failing `/v1/models` route probe. When the same run recovers a managed gateway, `status` retries any failed route or inference probe on that schedule instead, while the restarted delivery chain settles. Each attempt can consume another 16 tokens on a hosted route. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed: - `unauthorized` when the route rejected it with HTTP `401` or `403`. - `unhealthy` when the route returned another failing HTTP status or an invalid 2xx response body. From e9214cc574a6777e06dfebca570af825b91a5453 Mon Sep 17 00:00:00 2001 From: Hai Nguyen Date: Thu, 3 Sep 2026 10:53:50 +0000 Subject: [PATCH 03/12] refactor(inference): give the transient probe statuses one owner The status retry added a second copy of the HTTP 429/502/503/504 set that `probe-retry.ts` already owned for the onboarding probes, so a later change to one retry policy could leave the other behind. Move the set to `src/lib/inference/probe/transient-http-policy.ts`, a typed ESM module that `probe-retry.ts` requires the same way it already requires `core/retry`, and that sandbox code imports directly. Put the invocation-result predicate in `inference-route-health.ts` next to `classifyInferenceInvocationFailureLabel`, which already owns how an invocation result is classified; `status-snapshot.ts` reads it through the import it already had, so its fan-out is unchanged. No behavior change. Signed-off-by: Hai Nguyen --- .../sandbox/inference-route-health.test.ts | 57 +++++++++++++++++++ .../actions/sandbox/inference-route-health.ts | 17 ++++++ src/lib/actions/sandbox/status-snapshot.ts | 33 +---------- src/lib/inference/probe-retry.ts | 10 ++-- .../inference/probe/transient-http-policy.ts | 18 ++++++ 5 files changed, 99 insertions(+), 36 deletions(-) create mode 100644 src/lib/inference/probe/transient-http-policy.ts diff --git a/src/lib/actions/sandbox/inference-route-health.test.ts b/src/lib/actions/sandbox/inference-route-health.test.ts index 0fa665fd55c..0e5ab77c7db 100644 --- a/src/lib/actions/sandbox/inference-route-health.test.ts +++ b/src/lib/actions/sandbox/inference-route-health.test.ts @@ -8,6 +8,7 @@ import { } from "./connect-inference-route-probe"; import { buildSandboxInferenceRouteHealth, + isTransientInferenceInvocationFailure, probeSandboxInferenceGatewayHealth, type SandboxInferenceRouteHealth, } from "./inference-route-health"; @@ -278,3 +279,59 @@ describe("buildSandboxInferenceRouteHealth (#10080)", () => { expect(result.ok).toBe(true); }); }); + +describe("transient inference invocation failures", () => { + it.each([429, 502, 503, 504])( + "treats HTTP %i as a transient inference request failure (#10709)", + (httpStatus) => { + expect( + isTransientInferenceInvocationFailure({ + ok: false, + detail: `sandbox inference invocation probe returned HTTP ${httpStatus}`, + httpStatus, + }), + ).toBe(true); + }, + ); + + it.each([400, 401, 403, 404, 405, 500, 501])( + "treats HTTP %i as a settled inference request failure (#10709)", + (httpStatus) => { + expect( + isTransientInferenceInvocationFailure({ + ok: false, + detail: `sandbox inference invocation probe returned HTTP ${httpStatus}`, + httpStatus, + }), + ).toBe(false); + }, + ); + + it("treats a served request as no failure at all (#10709)", () => { + expect(isTransientInferenceInvocationFailure({ ok: true })).toBe(false); + }); + + it("treats an invalid 2xx response body as a settled failure (#10709)", () => { + expect( + isTransientInferenceInvocationFailure({ + ok: false, + detail: "sandbox inference invocation probe returned an invalid response body", + httpStatus: 200, + }), + ).toBe(false); + }); + + it("treats a request that reached no HTTP status as a settled failure (#10709)", () => { + expect( + isTransientInferenceInvocationFailure({ + ok: false, + detail: "sandbox inference invocation probe was unavailable", + httpStatus: null, + }), + ).toBe(false); + }); + + it("reports no failure when no inference request was sent (#10709)", () => { + expect(isTransientInferenceInvocationFailure(null)).toBe(false); + }); +}); diff --git a/src/lib/actions/sandbox/inference-route-health.ts b/src/lib/actions/sandbox/inference-route-health.ts index ac59437c078..5c6846b3b7c 100644 --- a/src/lib/actions/sandbox/inference-route-health.ts +++ b/src/lib/actions/sandbox/inference-route-health.ts @@ -5,6 +5,7 @@ import { captureOpenshellForStatus, isCommandTimeout } from "../../adapters/open import { OPENSHELL_INFERENCE_ROUTE_PROBE_TIMEOUT_MS } from "../../adapters/openshell/timeouts"; import * as agentRuntime from "../../agent/runtime"; import type { ProviderHealthStatus } from "../../inference/health"; +import { RETRIABLE_HTTP_PROBE_STATUSES } from "../../inference/probe/transient-http-policy"; import { buildSandboxInferenceRouteProbeArgs, classifyInferenceRouteFailureLabel, @@ -154,6 +155,22 @@ function classifyInferenceInvocationFailureLabel( return "unhealthy"; } +/** + * True only when the inference request itself was declined with a transient + * gateway or availability status, so sending it again is worthwhile. + * + * HTTP 401, 403, 404, and 500, an invalid 2xx response body, and a request + * that never reached an HTTP status all return false: those describe the route + * as it is, so a caller must report them without retrying. A null invocation + * also returns false, because no inference request was sent. + */ +export function isTransientInferenceInvocationFailure( + invocation: SandboxInferenceInvocationResult | null, +): boolean { + if (invocation === null || invocation.ok) return false; + return invocation.httpStatus !== null && RETRIABLE_HTTP_PROBE_STATUSES.has(invocation.httpStatus); +} + /** * Report the reachable route as its own hop so an operator can tell a broken * route from a reachable route that will not serve an inference request. diff --git a/src/lib/actions/sandbox/status-snapshot.ts b/src/lib/actions/sandbox/status-snapshot.ts index 3321d614079..189e53f2c18 100644 --- a/src/lib/actions/sandbox/status-snapshot.ts +++ b/src/lib/actions/sandbox/status-snapshot.ts @@ -39,6 +39,7 @@ import type { SandboxGatewayState } from "./gateway-state"; import { getReconciledSandboxGatewayState, getSandboxGatewayStateForStatus } from "./gateway-state"; import { buildSandboxInferenceRouteHealth, + isTransientInferenceInvocationFailure, type ProbeSandboxInferenceInvocation, probeSandboxInferenceGatewayHealth, runSandboxInferenceInvocationProbe, @@ -66,36 +67,6 @@ type DelayInferenceRecoveryProbe = (delayMs: number) => Promise; const INFERENCE_PROBE_ATTEMPTS = 3; const INFERENCE_PROBE_RETRY_DELAY_MS = 2_000; -// 429 = Too Many Requests; 502/503/504 = gateway and availability answers from -// the proxy in front of `inference.local` and from hosted providers. This is -// the same signature `src/lib/inference/probe-retry.ts` already retries for the -// onboarding probes (#2980, #3033); that module is CommonJS and `@ts-nocheck`, -// so it cannot export the set to a typed module. A Ready sandbox must not -// report a failing route because one of these landed in the single request -// `status` sends (#10709). -const TRANSIENT_INFERENCE_INVOCATION_STATUSES: ReadonlySet = new Set([429, 502, 503, 504]); - -/** - * True only when the inference request itself was refused with a transient - * gateway or availability status, so one more 16-token request is worth - * sending. - * - * HTTP 401, 403, 404, and 500, an invalid 2xx response body, and a request that - * never reached an HTTP status all return false. Those stay final on the first - * attempt and add no delay to the interactive `status` path. A failing - * `/v1/models` route probe is not retried here either: it reports a different - * hop, and status renders the route probe result for that hop. - */ -function inferenceInvocationFailureIsTransient( - invocation: ReturnType | null, -): boolean { - if (invocation === null || invocation.ok) return false; - return ( - invocation.httpStatus !== null && - TRANSIENT_INFERENCE_INVOCATION_STATUSES.has(invocation.httpStatus) - ); -} - /** * Honest serving-process state while the self-report response and probe * contracts remain undefined. Do not add a checked result until both contracts @@ -660,7 +631,7 @@ export async function collectSandboxStatusSnapshot( // Otherwise retry only a transient gateway or availability status on // the inference request. Every other request failure, and every // route probe failure, is final on the first attempt (#10709). - return !inferenceInvocationFailureIsTransient(result); + return !isTransientInferenceInvocationFailure(result); }, retryDelaysMs: Array.from( { length: INFERENCE_PROBE_ATTEMPTS - 1 }, diff --git a/src/lib/inference/probe-retry.ts b/src/lib/inference/probe-retry.ts index cc1e153a301..fd935b09ff1 100644 --- a/src/lib/inference/probe-retry.ts +++ b/src/lib/inference/probe-retry.ts @@ -18,11 +18,11 @@ const { retryUntil } = require("../core/retry"); const CURL_TIMEOUT_STATUS = 28; const NODE_SPAWN_TIMEOUT_STATUS = -110; -// 429 = Too Many Requests; 502/503/504 = upstream gateway/availability flakes -// (NVIDIA Endpoints and other hosted providers periodically emit these for -// minutes at a time). All four are transient — retry with backoff before -// surfacing a hard failure to the wizard. See issues #2980 and #3033. -const RETRIABLE_HTTP_PROBE_STATUSES = new Set([429, 502, 503, 504]); +// Which statuses are transient is shared policy, not a wizard-local decision: +// the sandbox status probe classifies the same four (#10709). The definition +// and its rationale live in ./probe/transient-http-policy; this module keeps +// re-exporting the set so existing consumers are unaffected. +const { RETRIABLE_HTTP_PROBE_STATUSES } = require("./probe/transient-http-policy"); const HTTP_PROBE_RETRY_DELAYS_MS = [5_000, 15_000, 30_000]; function sleepSync(ms) { diff --git a/src/lib/inference/probe/transient-http-policy.ts b/src/lib/inference/probe/transient-http-policy.ts new file mode 100644 index 00000000000..b500d23b7cd --- /dev/null +++ b/src/lib/inference/probe/transient-http-policy.ts @@ -0,0 +1,18 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +/** + * The one statement of which HTTP statuses an inference probe may retry. + * + * 429 = Too Many Requests; 502/503/504 = upstream gateway and availability + * flakes (NVIDIA Endpoints and other hosted providers periodically emit these + * for minutes at a time). All four mean the route answered and declined to + * serve the request, which is not evidence that the route is broken, so a + * probe retries them with backoff before reporting a hard failure. See issues + * #2980, #3033, and #10709. + * + * This module is plain typed ESM so the require()-based onboarding probe loop + * in `./probe-retry` and the typed sandbox status probe can read one + * definition instead of keeping their own copies. + */ +export const RETRIABLE_HTTP_PROBE_STATUSES: ReadonlySet = new Set([429, 502, 503, 504]); From 8f82b24ef445b343a2f8aa9a403102d3bbb4676f Mon Sep 17 00:00:00 2001 From: Hai Nguyen Date: Thu, 3 Sep 2026 10:53:50 +0000 Subject: [PATCH 04/12] test(status): cover every transient status and the repeated route probe The retry tests proved only HTTP 503 and only that the inference request ran again. Dropping 429, 502, or 504 from the transient set, or moving the route probe out of the retried operation, would have left them green. Parameterize the recovery test over all four transient statuses, assert the `/v1/models` probe runs once per attempt on both the recovery and the exhaustion path, and add the HTTP 403 case so an authorization denial is pinned as final rather than retried with the stored provider credential. Signed-off-by: Hai Nguyen --- .../status-snapshot-inference-health.test.ts | 105 +++++++++++------- 1 file changed, 63 insertions(+), 42 deletions(-) diff --git a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts index b02e78333bf..51aad01f71b 100644 --- a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts +++ b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts @@ -731,49 +731,58 @@ describe("collectSandboxStatusSnapshot inference route health", () => { expect(snapshot.inferenceHealth?.okLabel).toBeUndefined(); }); - it("serves a Ready managed sandbox after one transient HTTP 503 inference request (#10709)", async () => { - const healthy: SandboxInferenceRouteHealth = { - ok: true, - endpoint: "https://inference.local/v1/models", - httpStatus: 200, - detail: "reachable", - }; - const options = snapshotDeps(healthy, null, { ok: true }, { openshellDriver: "docker" }); - options.deps.reconcile = async () => ({ - state: "present", - phase: "Ready", - output: "Phase: Ready", - }); - const probeSandboxInferenceInvocationImpl = vi - .fn() - .mockReturnValueOnce({ - ok: false, - detail: "sandbox inference invocation probe returned HTTP 503", - httpStatus: 503, - }) - .mockReturnValueOnce({ ok: true }); - const delayInferenceRecoveryProbe = vi.fn(async () => undefined); - const recoverSandboxProcesses = vi.fn(() => ({ - checked: true, - wasRunning: true, - recovered: false, - })) as never; - - const snapshot = await collectSandboxStatusSnapshot("alpha", { - ...options, - deps: { - ...options.deps, - delayInferenceRecoveryProbe, - probeSandboxInferenceInvocationImpl, - recoverSandboxProcesses, - }, - }); + // Every transient status must repeat the whole probe pair, not just the + // inference request: a retry that skipped the route probe could report a + // served request against a route it never rechecked. + it.each([429, 502, 503, 504])( + "serves a Ready managed sandbox after one transient HTTP %i inference request (#10709)", + async (httpStatus) => { + const healthy: SandboxInferenceRouteHealth = { + ok: true, + endpoint: "https://inference.local/v1/models", + httpStatus: 200, + detail: "reachable", + }; + const options = snapshotDeps(healthy, null, { ok: true }, { openshellDriver: "docker" }); + options.deps.reconcile = async () => ({ + state: "present", + phase: "Ready", + output: "Phase: Ready", + }); + const probeSandboxInferenceGatewayHealthImpl = vi.fn(async () => healthy); + const probeSandboxInferenceInvocationImpl = vi + .fn() + .mockReturnValueOnce({ + ok: false, + detail: `sandbox inference invocation probe returned HTTP ${httpStatus}`, + httpStatus, + }) + .mockReturnValueOnce({ ok: true }); + const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const recoverSandboxProcesses = vi.fn(() => ({ + checked: true, + wasRunning: true, + recovered: false, + })) as never; + + const snapshot = await collectSandboxStatusSnapshot("alpha", { + ...options, + deps: { + ...options.deps, + delayInferenceRecoveryProbe, + probeSandboxInferenceGatewayHealthImpl, + probeSandboxInferenceInvocationImpl, + recoverSandboxProcesses, + }, + }); - expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes(2); - expect(delayInferenceRecoveryProbe).toHaveBeenCalledOnce(); - expect(delayInferenceRecoveryProbe).toHaveBeenCalledWith(2_000); - expect(snapshot.inferenceHealth).toMatchObject({ ok: true, probed: true }); - }); + expect(probeSandboxInferenceGatewayHealthImpl).toHaveBeenCalledTimes(2); + expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes(2); + expect(delayInferenceRecoveryProbe).toHaveBeenCalledOnce(); + expect(delayInferenceRecoveryProbe).toHaveBeenCalledWith(2_000); + expect(snapshot.inferenceHealth).toMatchObject({ ok: true, probed: true }); + }, + ); it("reports unhealthy after three transient HTTP 503 inference requests (#10709)", async () => { const healthy: SandboxInferenceRouteHealth = { @@ -793,6 +802,7 @@ describe("collectSandboxStatusSnapshot inference route health", () => { phase: "Ready", output: "Phase: Ready", }); + const probeSandboxInferenceGatewayHealthImpl = vi.fn(async () => healthy); const probeSandboxInferenceInvocationImpl = vi.fn(() => refused); const delayInferenceRecoveryProbe = vi.fn(async () => undefined); const recoverSandboxProcesses = vi.fn(() => ({ @@ -806,11 +816,13 @@ describe("collectSandboxStatusSnapshot inference route health", () => { deps: { ...options.deps, delayInferenceRecoveryProbe, + probeSandboxInferenceGatewayHealthImpl, probeSandboxInferenceInvocationImpl, recoverSandboxProcesses, }, }); + expect(probeSandboxInferenceGatewayHealthImpl).toHaveBeenCalledTimes(3); expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes(3); expect(delayInferenceRecoveryProbe).toHaveBeenCalledTimes(2); expect(delayInferenceRecoveryProbe).toHaveBeenCalledWith(2_000); @@ -830,6 +842,15 @@ describe("collectSandboxStatusSnapshot inference route health", () => { }, failureLabel: "unauthorized", }, + { + label: "forbidden", + invocation: { + ok: false as const, + detail: "sandbox inference invocation probe returned HTTP 403", + httpStatus: 403, + }, + failureLabel: "unauthorized", + }, { label: "not found", invocation: { From b37ec6ff95e89c57ca4bbfb4eeed81de44559c39 Mon Sep 17 00:00:00 2001 From: Hai Nguyen Date: Thu, 3 Sep 2026 11:24:49 +0000 Subject: [PATCH 05/12] refactor(inference): finish the transient status consolidation The extraction left `openai-validation-session.ts` on its own copy of the same four statuses, so the module that claims to own the policy did not yet own it and a later change could move the probe paths apart. Read the shared set there too, and cover the native retry path from the settled side: an HTTP 500 reaches the curl fallback after one request, so widening the shared set fails a test instead of silently spending retries. Each caller keeps its own delay schedule, which is genuinely local. No behavior change. Signed-off-by: Hai Nguyen --- ...openai-validation-session-fallback.test.ts | 32 +++++++++++++++++++ .../inference/openai-validation-session.ts | 4 +-- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/src/lib/inference/openai-validation-session-fallback.test.ts b/src/lib/inference/openai-validation-session-fallback.test.ts index 96e19a54d6a..7b24b69157b 100644 --- a/src/lib/inference/openai-validation-session-fallback.test.ts +++ b/src/lib/inference/openai-validation-session-fallback.test.ts @@ -308,4 +308,36 @@ describe("OpenAI validation curl fallback", () => { expect(legacyProbe).toHaveBeenCalledTimes(1); expect(harness.sessionOptions!.lookup).not.toHaveBeenCalled(); }); + + // Pins the shared transient policy from the native retry side: a status the + // policy does not list must reach the fallback without spending retries + // (#10709). + it("does not retry a settled HTTP failure before falling back", async () => { + vi.stubEnv("NEMOCLAW_TEST_NO_SLEEP", "1"); + let requests = 0; + const server = http.createServer((request, response) => { + request.resume(); + requests += 1; + response.statusCode = 500; + response.end('{"error":{"message":"internal"}}'); + }); + const port = await listen(server); + const legacyProbe: OpenAiValidationSessionDeps["legacyProbe"] = vi.fn(() => ({ + ok: false, + message: "curl settled diagnostic", + })); + const harness = createOpenAiValidationTestDeps(legacyProbe); + + const result = await probeOpenAiLikeEndpointWithValidationSession( + `http://provider.example.test:${port}/v1`, + "test-model", + "test-key", + { skipResponsesProbe: true }, + harness, + ); + + expect(result).toEqual({ ok: false, message: "curl settled diagnostic" }); + expect(requests).toBe(1); + expect(legacyProbe).toHaveBeenCalledTimes(1); + }); }); diff --git a/src/lib/inference/openai-validation-session.ts b/src/lib/inference/openai-validation-session.ts index 00ac699e4aa..c42dd063533 100644 --- a/src/lib/inference/openai-validation-session.ts +++ b/src/lib/inference/openai-validation-session.ts @@ -19,8 +19,8 @@ import { strictToolProbeReasoningRetryMessage, } from "./openai-probe-models"; import { STREAMING_EVENT_PROBE_MAX_SECONDS } from "./probe-http-helpers"; +import { RETRIABLE_HTTP_PROBE_STATUSES } from "./probe/transient-http-policy"; -const RETRIABLE_HTTP_STATUSES = new Set([429, 502, 503, 504]); const RETRY_DELAYS_MS = [5_000, 15_000, 30_000]; export interface OpenAiValidationOptions { @@ -208,7 +208,7 @@ async function requestWithHttpRetry( accept: (result) => !retryTransientHttp || result.curlStatus !== 0 || - !RETRIABLE_HTTP_STATUSES.has(result.httpStatus), + !RETRIABLE_HTTP_PROBE_STATUSES.has(result.httpStatus), retryDelaysMs: RETRY_DELAYS_MS, onRetry: (result, delayMs) => { console.log( From 82e140b25a3f6c379563e13c1e71679efb8ef5e8 Mon Sep 17 00:00:00 2001 From: Hai Nguyen Date: Thu, 3 Sep 2026 12:33:47 +0000 Subject: [PATCH 06/12] test(status): prove the retry believes the rechecked route The retry repeats the route probe and the inference request as a pair, but the existing tests returned the same healthy route on every attempt. A regression that re-probed the route and then ignored the answer would have passed their call counts while reporting a healthy sandbox against a route that had just failed. Add a case where the route answers 200, the inference request returns a transient 503, and the second route probe comes back unreachable. Assert that no second inference request is sent and that the reported health is the second route result. Signed-off-by: Hai Nguyen --- .../status-snapshot-inference-health.test.ts | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts index 51aad01f71b..a01f70c178f 100644 --- a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts +++ b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts @@ -784,6 +784,65 @@ describe("collectSandboxStatusSnapshot inference route health", () => { }, ); + // Repeating the probe pair is only worth anything if the retry believes the + // second route result. A retry that re-probed and then ignored the answer + // would still pass a call-count assertion (#10709). + it("stops at a route that fails between attempts without sending a second inference request (#10709)", async () => { + const healthy: SandboxInferenceRouteHealth = { + ok: true, + endpoint: "https://inference.local/v1/models", + httpStatus: 200, + detail: "reachable", + }; + const brokenRoute: SandboxInferenceRouteHealth = { + ok: false, + endpoint: "https://inference.local/v1/models", + httpStatus: 0, + detail: "Inference gateway unreachable on https://inference.local/v1/models.", + }; + const options = snapshotDeps(healthy, null, { ok: true }, { openshellDriver: "docker" }); + options.deps.reconcile = async () => ({ + state: "present", + phase: "Ready", + output: "Phase: Ready", + }); + const probeSandboxInferenceGatewayHealthImpl = vi + .fn() + .mockResolvedValueOnce(healthy) + .mockResolvedValueOnce(brokenRoute); + const probeSandboxInferenceInvocationImpl = vi.fn(() => ({ + ok: false as const, + detail: "sandbox inference invocation probe returned HTTP 503", + httpStatus: 503, + })); + const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const recoverSandboxProcesses = vi.fn(() => ({ + checked: true, + wasRunning: true, + recovered: false, + })) as never; + + const snapshot = await collectSandboxStatusSnapshot("alpha", { + ...options, + deps: { + ...options.deps, + delayInferenceRecoveryProbe, + probeSandboxInferenceGatewayHealthImpl, + probeSandboxInferenceInvocationImpl, + recoverSandboxProcesses, + }, + }); + + expect(probeSandboxInferenceGatewayHealthImpl).toHaveBeenCalledTimes(2); + expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledOnce(); + expect(delayInferenceRecoveryProbe).toHaveBeenCalledOnce(); + expect(snapshot.inferenceHealth).toMatchObject({ + ok: false, + detail: brokenRoute.detail, + failureLabel: "unreachable", + }); + }); + it("reports unhealthy after three transient HTTP 503 inference requests (#10709)", async () => { const healthy: SandboxInferenceRouteHealth = { ok: true, From 7a635229474410ae5d0e375a3f62fdb4490d97c6 Mon Sep 17 00:00:00 2001 From: Hai Nguyen Date: Thu, 3 Sep 2026 12:33:47 +0000 Subject: [PATCH 07/12] docs(status): correct the inference request count outside the command reference Five pages, and one paragraph inside the `status` section itself, still said `status` sends one inference request. That stopped being true for a transient gateway status, and inside the command reference it contradicted the retry paragraph three screens above it. Say "an inference request" where the count is the only claim, and name the bounded retry on the two operational pages whose readers feel the added wait: troubleshooting and the headless-server deployment guide. The command reference keeps sole ownership of the retried statuses, the attempt count, and the token cost. The `start` sentence is unchanged because `start` still sends exactly one request, and the credential rotation page now says outright that a rejection is not retried. Signed-off-by: Hai Nguyen --- docs/deployment/deploy-to-headless-server.mdx | 2 +- docs/inference/set-up-ollama.mdx | 2 +- docs/inference/verify-inference-route.mdx | 2 +- docs/reference/commands.mdx | 2 +- docs/reference/troubleshooting.mdx | 2 +- docs/security/credential-rotation.mdx | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/deployment/deploy-to-headless-server.mdx b/docs/deployment/deploy-to-headless-server.mdx index 372efe56c6d..bdabe68c56d 100644 --- a/docs/deployment/deploy-to-headless-server.mdx +++ b/docs/deployment/deploy-to-headless-server.mdx @@ -206,7 +206,7 @@ $$nemoclaw headless-agent status $$nemoclaw headless-agent connect --probe-only ``` -`$$nemoclaw headless-agent status` exits nonzero when the sandbox, gateway, local container, or authoritative inference route is not verified. Its main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox, then sends one inference request over the same route when that probe reports the route reachable. The line reports `healthy` when the route served the request, `unauthorized` when the route rejected it with HTTP `401` or `403`, and `unhealthy` when the route returned HTTP `500` through `599`. +`$$nemoclaw headless-agent status` exits nonzero when the sandbox, gateway, local container, or authoritative inference route is not verified. Its main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox, then sends an inference request over the same route when that probe reports the route reachable. It repeats both probes up to three total attempts when that request returns a transient gateway status; refer to the [CLI commands reference](../reference/commands) for the retried statuses and their token cost. The line reports `healthy` when the route served the request, `unauthorized` when the route rejected it with HTTP `401` or `403`, and `unhealthy` when the route returned HTTP `500` through `599`. diff --git a/docs/inference/set-up-ollama.mdx b/docs/inference/set-up-ollama.mdx index 942cf47ffca..211f9b4680f 100644 --- a/docs/inference/set-up-ollama.mdx +++ b/docs/inference/set-up-ollama.mdx @@ -266,7 +266,7 @@ Scoped uninstall preserves the shared proxy state while a sibling gateway remain All proxy endpoints require the token, including `GET /api/tags`. The host-side proxy liveness check treats any HTTP response, including `401`, as evidence that the proxy answered. The `status` and `doctor` route probe runs inside the sandbox and reports HTTP `200` through `499` as reachable, HTTP `500` through `599` as `unhealthy`, and transport failures or unavailable probes as `unreachable` or `not probed`. -`status` then sends one inference request over the reachable route and reports `healthy` when the route served it, or `unauthorized` when the proxy token is no longer accepted. +`status` then sends an inference request over the reachable route and reports `healthy` when the route served it, or `unauthorized` when the proxy token is no longer accepted. If Ollama is already bound to a non-loopback address, onboarding restarts it on `127.0.0.1:11434` so the proxy becomes the only network path. diff --git a/docs/inference/verify-inference-route.mdx b/docs/inference/verify-inference-route.mdx index 52e1bd3f157..ac82a978109 100644 --- a/docs/inference/verify-inference-route.mdx +++ b/docs/inference/verify-inference-route.mdx @@ -31,7 +31,7 @@ $$nemoclaw status ``` The `Inference` row first checks the sandbox's `inference.local` path. -When that route responds, `status` sends one inference request through the same path. +When that route responds, `status` sends an inference request through the same path. When the live provider matches the recorded provider, `status` validates the result against the recorded API family, even when only the model differs. When the live provider differs, `status` does not carry the recorded API family to the live provider. The row reports `healthy` only when the route returns a structurally valid result for the selected API family. diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 1f649438630..6b148ae741e 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -1430,7 +1430,7 @@ $$nemoclaw my-assistant status --json $$nemoclaw sandbox status my-assistant --json ``` -The command probes `https://inference.local/v1/models` from inside the sandbox, and when that probe reports the route reachable it sends one inference request over the same route. That inference request is the authoritative inference health check, and both checks exercise the route that agent traffic uses. The main `Inference` line reports one of these states: +The command probes `https://inference.local/v1/models` from inside the sandbox, and when that probe reports the route reachable it sends an inference request over the same route. That inference request is the authoritative inference health check, and both checks exercise the route that agent traffic uses. The main `Inference` line reports one of these states: | State | Meaning | | --- | --- | diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index 28ae642a694..4b0bb7760d1 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -1885,7 +1885,7 @@ Verify that the inference provider endpoint is reachable from the host. Check th $$nemoclaw status ``` -The main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox and then sends one inference request over the same route, so it reflects the route the agent uses. If that line shows `unauthorized`, `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop. An `unauthorized` line means the route answered but rejected the request, so refresh the provider credential rather than the route. +The main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox and then sends an inference request over the same route, so it reflects the route the agent uses. When that request returns a transient gateway status, `status` repeats both probes up to three total attempts before it reports a failure, so a failing line can take a few seconds longer than a healthy one. If that line shows `unauthorized`, `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop. An `unauthorized` line means the route answered but rejected the request, so refresh the provider credential rather than the route. For local Ollama and local vLLM, `Inference (ollama backend)` or the corresponding local-backend line reports the host-side service separately. If a local-backend diagnostic fails, start the backend. For Local Ollama, current releases can also print `Inference (auth proxy)` when a proxy token is available. Docker Desktop on Windows Subsystem for Linux (WSL) reaches host loopback directly, so its auth-proxy diagnostic is non-authoritative. Native Docker Engine inside WSL is unqualified; enable Docker Desktop WSL integration, then rerun onboarding. diff --git a/docs/security/credential-rotation.mdx b/docs/security/credential-rotation.mdx index 8a7f3d988bd..ea755657246 100644 --- a/docs/security/credential-rotation.mdx +++ b/docs/security/credential-rotation.mdx @@ -229,7 +229,7 @@ Plan for recreation downtime when automating messaging or web search rotation. `$$nemoclaw credentials list` confirms that a provider exists, but it does not reveal or validate the stored value. `$$nemoclaw status` performs host-side reachability checks without sending cloud API keys, so even an HTTP `401` or `403` can count as reachable there. -`$$nemoclaw status` also sends one inference request through the stored provider credential and reports `unauthorized` when the route rejects it with HTTP `401` or `403`. +`$$nemoclaw status` also sends an inference request through the stored provider credential and reports `unauthorized` when the route rejects it with HTTP `401` or `403`. It does not retry that rejection. `$$nemoclaw inference get` reports the active route and does not authenticate a model request. Complete a real request through the rotated integration before declaring the rotation successful. From ff1042a10245a6e10291b370a302733421c1325a Mon Sep 17 00:00:00 2001 From: Prekshi Vyas Date: Thu, 3 Sep 2026 12:56:17 -0700 Subject: [PATCH 08/12] docs(status): clarify final inference failure Signed-off-by: Prekshi Vyas --- docs/deployment/deploy-to-headless-server.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployment/deploy-to-headless-server.mdx b/docs/deployment/deploy-to-headless-server.mdx index bdabe68c56d..0b3ae97a6dc 100644 --- a/docs/deployment/deploy-to-headless-server.mdx +++ b/docs/deployment/deploy-to-headless-server.mdx @@ -206,7 +206,7 @@ $$nemoclaw headless-agent status $$nemoclaw headless-agent connect --probe-only ``` -`$$nemoclaw headless-agent status` exits nonzero when the sandbox, gateway, local container, or authoritative inference route is not verified. Its main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox, then sends an inference request over the same route when that probe reports the route reachable. It repeats both probes up to three total attempts when that request returns a transient gateway status; refer to the [CLI commands reference](../reference/commands) for the retried statuses and their token cost. The line reports `healthy` when the route served the request, `unauthorized` when the route rejected it with HTTP `401` or `403`, and `unhealthy` when the route returned HTTP `500` through `599`. +`$$nemoclaw headless-agent status` exits nonzero when the sandbox, gateway, local container, or authoritative inference route is not verified. Its main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox, then sends an inference request over the same route when that probe reports the route reachable. It repeats both probes up to three total attempts when that request returns a transient gateway status; refer to the [CLI commands reference](../reference/commands) for the retried statuses and their token cost. The line reports `healthy` when the route served the request, `unauthorized` when the route rejected it with HTTP `401` or `403`, and `unhealthy` after a final failure or after all transient attempts fail. From 3571b140a5013f5afe28d73538f141ebcb18dc5c Mon Sep 17 00:00:00 2001 From: Prekshi Vyas Date: Thu, 3 Sep 2026 14:04:31 -0700 Subject: [PATCH 09/12] fix(status): report inference probe retries Signed-off-by: Prekshi Vyas --- docs/deployment/deploy-to-headless-server.mdx | 2 +- docs/reference/commands.mdx | 2 +- docs/reference/troubleshooting.mdx | 2 +- .../status-snapshot-inference-health.test.ts | 19 ++++++++++ src/lib/actions/sandbox/status-snapshot.ts | 36 +++++++++++++++++++ 5 files changed, 58 insertions(+), 3 deletions(-) diff --git a/docs/deployment/deploy-to-headless-server.mdx b/docs/deployment/deploy-to-headless-server.mdx index 0b3ae97a6dc..f058fe25dad 100644 --- a/docs/deployment/deploy-to-headless-server.mdx +++ b/docs/deployment/deploy-to-headless-server.mdx @@ -441,7 +441,7 @@ Run `openshell sandbox list` and inspect the row for `headless-agent`. `NotReady ### Inference Returns HTTP 5xx -An HTTP status from `500` through `599` makes the authoritative `inference.local` route unhealthy. Check the configured provider and host egress, then run: +A final HTTP status from `500` through `599` makes the authoritative `inference.local` route unhealthy. HTTP `502`, `503`, and `504` report unhealthy only after all transient attempts fail. Check the configured provider and host egress, then run: ```bash $$nemoclaw headless-agent doctor diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 19673cb0681..0ca9c950025 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -1399,7 +1399,7 @@ For Portable Hermes, `status` reports `Portable lifecycle phase: pending`, `conf For a `compatible-endpoint` route that uses `openai-completions`, the text output prints `Reasoning effort` as `low`, `medium`, `high`, or `endpoint-default`. The line is omitted for another provider or API family. -Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request through the stored provider credential, with a 30-second timeout, and consumes provider tokens on a hosted route. When the inference request returns HTTP `429`, `502`, `503`, or `504`, `status` retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts, because those statuses are transient gateway and availability answers rather than evidence that the route is broken. On an ordinary run, every other failure is final on the first attempt with no delay: HTTP `401`, `403`, `404`, and `500`, an invalid 2xx response body, a request that returned no HTTP status, and a failing `/v1/models` route probe. When the same run recovers a managed gateway, `status` retries any failed route or inference probe on that schedule instead, while the restarted delivery chain settles. Each attempt can consume another 16 tokens on a hosted route. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed: +Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request through the stored provider credential, with a 30-second timeout, and consumes provider tokens on a hosted route. When the inference request returns HTTP `429`, `502`, `503`, or `504`, `status` retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts, because those statuses are transient gateway and availability answers rather than evidence that the route is broken. Before each retry, it writes the failed HTTP status, next attempt, total attempts, and delay to stderr; `--json` keeps stdout machine-readable. On an ordinary run, every other failure is final on the first attempt with no delay: HTTP `401`, `403`, `404`, and `500`, an invalid 2xx response body, a request that returned no HTTP status, and a failing `/v1/models` route probe. When the same run recovers a managed gateway, `status` retries any failed route or inference probe on that schedule instead, while the restarted delivery chain settles. Each attempt can consume another 16 tokens on a hosted route. Each route probe has a 10-second timeout and each inference request has a 30-second timeout, so three complete probe pairs and two delays can take about 124 seconds. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed: - `unauthorized` when the route rejected it with HTTP `401` or `403`. - `unhealthy` when the route returned another failing HTTP status or an invalid 2xx response body. diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index b6e8120b5c9..446651eef21 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -1887,7 +1887,7 @@ Verify that the inference provider endpoint is reachable from the host. Check th $$nemoclaw status ``` -The main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox and then sends an inference request over the same route, so it reflects the route the agent uses. When that request returns a transient gateway status, `status` repeats both probes up to three total attempts before it reports a failure, so a failing line can take a few seconds longer than a healthy one. If that line shows `unauthorized`, `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop. An `unauthorized` line means the route answered but rejected the request, so refresh the provider credential rather than the route. +The main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox and then sends an inference request over the same route, so it reflects the route the agent uses. When that request returns a transient gateway status, `status` repeats both probes up to three total attempts before it reports a failure. It prints the HTTP status, next attempt, and two-second delay to stderr before each retry; the configured timeout envelope for three complete probe pairs and two delays is about 124 seconds. If that line shows `unauthorized`, `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop. An `unauthorized` line means the route answered but rejected the request, so refresh the provider credential rather than the route. For local Ollama and local vLLM, `Inference (ollama backend)` or the corresponding local-backend line reports the host-side service separately. If a local-backend diagnostic fails, start the backend. For Local Ollama, current releases can also print `Inference (auth proxy)` when a proxy token is available. Docker Desktop on Windows Subsystem for Linux (WSL) reaches host loopback directly, so its auth-proxy diagnostic is non-authoritative. Native Docker Engine inside WSL is unqualified; enable Docker Desktop WSL integration, then rerun onboarding. diff --git a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts index a01f70c178f..a3cbc246e3f 100644 --- a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts +++ b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts @@ -42,6 +42,7 @@ function snapshotDeps( probeProviderHealthImpl: () => providerHealth, probeSandboxInferenceGatewayHealthImpl: async () => gateway, probeSandboxInferenceInvocationImpl: () => invocation, + reportInferenceProbeRetry: vi.fn(), }, }; } @@ -759,6 +760,7 @@ describe("collectSandboxStatusSnapshot inference route health", () => { }) .mockReturnValueOnce({ ok: true }); const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const reportInferenceProbeRetry = vi.fn(); const recoverSandboxProcesses = vi.fn(() => ({ checked: true, wasRunning: true, @@ -773,6 +775,7 @@ describe("collectSandboxStatusSnapshot inference route health", () => { probeSandboxInferenceGatewayHealthImpl, probeSandboxInferenceInvocationImpl, recoverSandboxProcesses, + reportInferenceProbeRetry, }, }); @@ -780,6 +783,11 @@ describe("collectSandboxStatusSnapshot inference route health", () => { expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes(2); expect(delayInferenceRecoveryProbe).toHaveBeenCalledOnce(); expect(delayInferenceRecoveryProbe).toHaveBeenCalledWith(2_000); + expect(reportInferenceProbeRetry).toHaveBeenCalledOnce(); + expect(reportInferenceProbeRetry).toHaveBeenCalledWith( + ` Inference request returned HTTP ${httpStatus}; retrying route and request in 2s ` + + "(attempt 2/3)...", + ); expect(snapshot.inferenceHealth).toMatchObject({ ok: true, probed: true }); }, ); @@ -816,6 +824,7 @@ describe("collectSandboxStatusSnapshot inference route health", () => { httpStatus: 503, })); const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const reportInferenceProbeRetry = vi.fn(); const recoverSandboxProcesses = vi.fn(() => ({ checked: true, wasRunning: true, @@ -830,6 +839,7 @@ describe("collectSandboxStatusSnapshot inference route health", () => { probeSandboxInferenceGatewayHealthImpl, probeSandboxInferenceInvocationImpl, recoverSandboxProcesses, + reportInferenceProbeRetry, }, }); @@ -864,6 +874,7 @@ describe("collectSandboxStatusSnapshot inference route health", () => { const probeSandboxInferenceGatewayHealthImpl = vi.fn(async () => healthy); const probeSandboxInferenceInvocationImpl = vi.fn(() => refused); const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const reportInferenceProbeRetry = vi.fn(); const recoverSandboxProcesses = vi.fn(() => ({ checked: true, wasRunning: true, @@ -878,6 +889,7 @@ describe("collectSandboxStatusSnapshot inference route health", () => { probeSandboxInferenceGatewayHealthImpl, probeSandboxInferenceInvocationImpl, recoverSandboxProcesses, + reportInferenceProbeRetry, }, }); @@ -885,6 +897,10 @@ describe("collectSandboxStatusSnapshot inference route health", () => { expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes(3); expect(delayInferenceRecoveryProbe).toHaveBeenCalledTimes(2); expect(delayInferenceRecoveryProbe).toHaveBeenCalledWith(2_000); + expect(reportInferenceProbeRetry.mock.calls).toEqual([ + [" Inference request returned HTTP 503; retrying route and request in 2s (attempt 2/3)..."], + [" Inference request returned HTTP 503; retrying route and request in 2s (attempt 3/3)..."], + ]); expect(snapshot.inferenceHealth).toMatchObject({ ok: false, failureLabel: "unhealthy" }); expect(snapshot.inferenceHealth?.subprobes).toContainEqual( expect.objectContaining({ probeLabel: "route reachability", okLabel: "reachable" }), @@ -961,6 +977,7 @@ describe("collectSandboxStatusSnapshot inference route health", () => { }); const probeSandboxInferenceInvocationImpl = vi.fn(() => testCase.invocation); const delayInferenceRecoveryProbe = vi.fn(async () => undefined); + const reportInferenceProbeRetry = vi.fn(); const recoverSandboxProcesses = vi.fn(() => ({ checked: true, wasRunning: true, @@ -974,11 +991,13 @@ describe("collectSandboxStatusSnapshot inference route health", () => { delayInferenceRecoveryProbe, probeSandboxInferenceInvocationImpl, recoverSandboxProcesses, + reportInferenceProbeRetry, }, }); expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledOnce(); expect(delayInferenceRecoveryProbe).not.toHaveBeenCalled(); + expect(reportInferenceProbeRetry).not.toHaveBeenCalled(); expect(snapshot.inferenceHealth).toMatchObject({ ok: false, failureLabel: testCase.failureLabel, diff --git a/src/lib/actions/sandbox/status-snapshot.ts b/src/lib/actions/sandbox/status-snapshot.ts index 189e53f2c18..b47becf14c9 100644 --- a/src/lib/actions/sandbox/status-snapshot.ts +++ b/src/lib/actions/sandbox/status-snapshot.ts @@ -292,6 +292,7 @@ interface CollectSandboxStatusSnapshotDeps { probeSandboxInferenceInvocationImpl?: ProbeSandboxInferenceInvocation; delayInferenceRecoveryProbe?: DelayInferenceRecoveryProbe; reportInferenceProbeError?: (message: string) => void; + reportInferenceProbeRetry?: (message: string) => void; probeTerminalRuntimeHealth?: ProbeTerminalRuntimeHealth; recoverSandboxProcesses?: RecoverSandboxProcesses; reconcile?: ReconcileSandboxGatewayState; @@ -391,6 +392,33 @@ function reportInferenceProbeError(error: unknown, writer: (message: string) => ); } +function reportInferenceProbeRetry( + gatewayChain: Awaited>, + invocation: ReturnType | null, + delayMs: number, + attempt: number, + writer: (message: string) => void, +): void { + let reason = "route probe did not pass"; + if (gatewayChain?.ok) { + reason = "request probe did not pass"; + if ( + invocation && + !invocation.ok && + invocation.httpStatus !== null && + (invocation.httpStatus < 200 || invocation.httpStatus >= 300) + ) { + reason = `request returned HTTP ${invocation.httpStatus}`; + } + } else if (gatewayChain && gatewayChain.httpStatus > 0) { + reason = `route probe returned HTTP ${gatewayChain.httpStatus}`; + } + writer( + ` Inference ${reason}; retrying route and request in ${delayMs / 1_000}s ` + + `(attempt ${attempt + 1}/${INFERENCE_PROBE_ATTEMPTS})...`, + ); +} + export async function collectSandboxStatusSnapshot( sandboxName: string, opts: { @@ -637,6 +665,14 @@ export async function collectSandboxStatusSnapshot( { length: INFERENCE_PROBE_ATTEMPTS - 1 }, () => INFERENCE_PROBE_RETRY_DELAY_MS, ), + onRetry: ({ gatewayChain: chain, invocation: result }, delayMs, attempt) => + reportInferenceProbeRetry( + chain, + result, + delayMs, + attempt, + opts.deps?.reportInferenceProbeRetry ?? console.error, + ), sleep: opts.deps?.delayInferenceRecoveryProbe ?? sleep, }, ); From 151b69a1176184b0632d8e5bfe8adb2dcb041b9c Mon Sep 17 00:00:00 2001 From: Prekshi Vyas Date: Thu, 3 Sep 2026 15:01:02 -0700 Subject: [PATCH 10/12] docs(status): clarify retry request accounting Signed-off-by: Prekshi Vyas --- docs/reference/commands.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 0ca9c950025..c215e4ee99e 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -1399,7 +1399,7 @@ For Portable Hermes, `status` reports `Portable lifecycle phase: pending`, `conf For a `compatible-endpoint` route that uses `openai-completions`, the text output prints `Reasoning effort` as `low`, `medium`, `high`, or `endpoint-default`. The line is omitted for another provider or API family. -Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request through the stored provider credential, with a 30-second timeout, and consumes provider tokens on a hosted route. When the inference request returns HTTP `429`, `502`, `503`, or `504`, `status` retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts, because those statuses are transient gateway and availability answers rather than evidence that the route is broken. Before each retry, it writes the failed HTTP status, next attempt, total attempts, and delay to stderr; `--json` keeps stdout machine-readable. On an ordinary run, every other failure is final on the first attempt with no delay: HTTP `401`, `403`, `404`, and `500`, an invalid 2xx response body, a request that returned no HTTP status, and a failing `/v1/models` route probe. When the same run recovers a managed gateway, `status` retries any failed route or inference probe on that schedule instead, while the restarted delivery chain settles. Each attempt can consume another 16 tokens on a hosted route. Each route probe has a 10-second timeout and each inference request has a 30-second timeout, so three complete probe pairs and two delays can take about 124 seconds. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed: +Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request per attempt through the stored provider credential, with a 30-second timeout for each request, and consumes provider tokens on a hosted route. When the inference request returns HTTP `429`, `502`, `503`, or `504`, `status` retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts, because those statuses are transient gateway and availability answers rather than evidence that the route is broken. Before each retry, it writes the failed probe boundary, an HTTP status when one is available, the next attempt, total attempts, and delay to stderr; `--json` keeps stdout machine-readable. On an ordinary run, every other failure is final on the first attempt with no delay: HTTP `401`, `403`, `404`, and `500`, an invalid 2xx response body, a request that returned no HTTP status, and a failing `/v1/models` route probe. When the same run recovers a managed gateway, `status` retries any failed route or inference probe on that schedule instead, while the restarted delivery chain settles. Each inference request can consume another 16 tokens on a hosted route. Each route probe has a 10-second timeout and each inference request has a 30-second timeout, so three complete probe pairs and two delays can take about 124 seconds. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed: - `unauthorized` when the route rejected it with HTTP `401` or `403`. - `unhealthy` when the route returned another failing HTTP status or an invalid 2xx response body. From 66749610ea87303ca5017d878ccd36f137a80f94 Mon Sep 17 00:00:00 2001 From: Aaron Erickson Date: Fri, 4 Sep 2026 07:01:29 -0700 Subject: [PATCH 11/12] fix(ci): restore reviewed audit cleanup permissions Signed-off-by: Aaron Erickson --- scripts/audit-reviewed-npm-graph.mts | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/audit-reviewed-npm-graph.mts b/scripts/audit-reviewed-npm-graph.mts index eb7e8ed67c3..b651f8e4ff9 100755 --- a/scripts/audit-reviewed-npm-graph.mts +++ b/scripts/audit-reviewed-npm-graph.mts @@ -1000,6 +1000,7 @@ function main(): void { }); }); } finally { + makeTreeOwnerWritable(tempRoot); fs.rmSync(tempRoot, { recursive: true, force: true }); } } From 98151ddae9cc37ebb79fe560f04d50ee12fab95d Mon Sep 17 00:00:00 2001 From: Aaron Erickson Date: Fri, 4 Sep 2026 07:19:43 -0700 Subject: [PATCH 12/12] chore(ci): use trusted base audit cleanup Signed-off-by: Aaron Erickson --- scripts/audit-reviewed-npm-graph.mts | 1 - 1 file changed, 1 deletion(-) diff --git a/scripts/audit-reviewed-npm-graph.mts b/scripts/audit-reviewed-npm-graph.mts index 64de3764d72..f22ef430d7f 100755 --- a/scripts/audit-reviewed-npm-graph.mts +++ b/scripts/audit-reviewed-npm-graph.mts @@ -1004,7 +1004,6 @@ function main(): void { }); }); } finally { - makeTreeOwnerWritable(tempRoot); fs.rmSync(tempRoot, { recursive: true, force: true }); } }