Skip to content
Merged
2 changes: 2 additions & 0 deletions docs/manage-sandboxes/run-sandboxes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ When another sandbox or a host listener already holds the default API port `8642
<AgentOnly variant="openclaw,hermes">
If you intentionally run separate OpenShell gateways on the same host, set a different `NEMOCLAW_GATEWAY_PORT` before each onboarding run.
NemoClaw isolates the gateway name and local state by port so one port-specific gateway does not replace another.
Sandbox-scoped commands that read, exec into, or inspect a sandbox, such as `$$nemoclaw <name> exec` and `$$nemoclaw <name> status`, address each sandbox through its recorded gateway binding, so the current shell's `NEMOCLAW_GATEWAY_PORT` does not redirect them. Lifecycle commands such as `$$nemoclaw <name> stop` still act on the gateway that `NEMOCLAW_GATEWAY_PORT` selects.
A non-default `NEMOCLAW_GATEWAY_PORT` also gets its own host state root at `~/.nemoclaw/gateways/<port>/`, with a separate sandbox registry, snapshots, and legacy credential-migration files, so gateway-scoped state stays segregated while shared host-level files remain under `~/.nemoclaw/`.
On first use after upgrading, NemoClaw moves legacy rows and related state only when their recorded gateway identity matches the selected port; ambiguous state is left untouched with remediation.
Provider credentials remain in the OpenShell gateway store.
Expand All @@ -73,6 +74,7 @@ If ownership cannot be proved, onboarding fails closed and reports the listener
<AgentOnly variant="deepagents">
If you intentionally run separate OpenShell gateways on the same host, set a different `NEMOCLAW_GATEWAY_PORT` before each onboarding run.
NemoClaw isolates the gateway name and local state by port so one port-specific gateway does not replace another.
Sandbox-scoped commands that read, exec into, or inspect a sandbox, such as `nemo-deepagents <name> exec` and `nemo-deepagents <name> status`, address each sandbox through its recorded gateway binding, so the current shell's `NEMOCLAW_GATEWAY_PORT` does not redirect them. Lifecycle commands such as `nemo-deepagents <name> stop` still act on the gateway that `NEMOCLAW_GATEWAY_PORT` selects.
A non-default `NEMOCLAW_GATEWAY_PORT` also gets its own host state root at `~/.nemoclaw/gateways/<port>/`, with a separate sandbox registry, snapshots, and legacy credential-migration files, so gateway-scoped state stays segregated while shared host-level files remain under `~/.nemoclaw/`.
On first use after upgrading, NemoClaw moves legacy rows and related state only when their recorded gateway identity matches the selected port; ambiguous state is left untouched with remediation.
Provider credentials remain in the OpenShell gateway store.
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -424,7 +424,7 @@ $$nemoclaw list

A gateway port that keeps listening after you uninstall, or that keeps serving after you re-onboard without `NEMOCLAW_GATEWAY_PORT` set, can belong to a second environment.

Onboarding under a non-default `NEMOCLAW_GATEWAY_PORT` registers the sandbox on gateway `nemoclaw-<port>` and stores its registry and state under `~/.nemoclaw/gateways/<port>/`. Every later command reads the state root that its own `NEMOCLAW_GATEWAY_PORT` selects. A command without that variable uses port `8080` unless the installer recorded an automatic alternate-port selection; in that case, the CLI restores the recorded port. Clearing an explicitly supplied variable does not move an existing sandbox back to the default port.
Onboarding under a non-default `NEMOCLAW_GATEWAY_PORT` registers the sandbox on gateway `nemoclaw-<port>` and stores its registry and state under `~/.nemoclaw/gateways/<port>/`. Gateway-lifecycle commands such as `$$nemoclaw onboard`, `$$nemoclaw uninstall`, and the deprecated `$$nemoclaw stop` act on the state root that their own `NEMOCLAW_GATEWAY_PORT` selects, while sandbox-scoped commands that read, exec into, or inspect a sandbox, such as `$$nemoclaw <name> exec` and `$$nemoclaw <name> status`, resolve the named sandbox across every per-port registry root and address its recorded gateway binding. A gateway-lifecycle command without that variable uses port `8080` unless the installer recorded an automatic alternate-port selection; in that case, the CLI restores the recorded port. Clearing an explicitly supplied variable does not move an existing sandbox back to the default port.

List the gateways the host still has:

Expand Down
9 changes: 9 additions & 0 deletions src/lib/actions/sandbox/gateway-failure-classifier.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,15 @@ vi.mock("../../state/registry", () => ({
listSandboxes: vi.fn(() => ({ sandboxes: [] })),
}));

vi.mock("../../state/registry/cross-port", () => ({
findSandboxAcrossGatewayRoots: (...args: unknown[]) => {
const entry = getSandboxMock(...args);
return entry
? { entry, gatewayPort: entry.gatewayPort ?? null, registryFile: "/test/sandboxes.json" }
: null;
},
}));

import {
classifyGatewayFailure,
classifyObservedSandboxContainerFailure,
Expand Down
59 changes: 52 additions & 7 deletions src/lib/actions/sandbox/gateway-select.test.ts
Original file line number Diff line number Diff line change
@@ -1,18 +1,30 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import fs from "node:fs";
import os from "node:os";
import path from "node:path";

import { afterEach, describe, expect, it, vi } from "vitest";

import * as registry from "../../state/registry";
import * as crossPort from "../../state/registry/cross-port";
import { selectSandboxOwningGateway } from "./gateway-select";

function mockSandboxGatewayPort(gatewayPort: number): void {
vi.spyOn(crossPort, "findSandboxAcrossGatewayRoots").mockReturnValue({
entry: { name: "sandbox", gatewayPort },
gatewayPort,
registryFile: "/test/sandboxes.json",
});
}

describe("selectSandboxOwningGateway", () => {
afterEach(() => {
vi.restoreAllMocks();
});

it("selects the owning non-default gateway for a registered sandbox", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
mockSandboxGatewayPort(8091);
const run = vi.fn(() => ({ status: 0 }) as never);

const selected = selectSandboxOwningGateway("beta", run);
Expand All @@ -28,7 +40,7 @@ describe("selectSandboxOwningGateway", () => {
});

it("replays selection output through the stdio adapter", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8080 } as never);
mockSandboxGatewayPort(8080);
const output = "\u001b[32m✓ Active gateway set to 'nemoclaw'\u001b[0m\n";
const run = vi.fn(() => ({ status: 0, stdout: output }) as never);
const write = vi.fn();
Expand All @@ -41,7 +53,7 @@ describe("selectSandboxOwningGateway", () => {
});

it("keeps the bare default gateway name for a default-port sandbox", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8080 } as never);
mockSandboxGatewayPort(8080);
const run = vi.fn(() => ({ status: 0 }) as never);

expect(selectSandboxOwningGateway("alpha", run)).toEqual({
Expand All @@ -52,7 +64,7 @@ describe("selectSandboxOwningGateway", () => {
});

it("does not touch the active gateway for an unregistered sandbox", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue(null);
vi.spyOn(crossPort, "findSandboxAcrossGatewayRoots").mockReturnValue(null);
const run = vi.fn(() => ({ status: 0 }) as never);

expect(selectSandboxOwningGateway("ghost", run)).toEqual({
Expand All @@ -63,7 +75,7 @@ describe("selectSandboxOwningGateway", () => {
});

it("reports failure when the gateway select command exits nonzero", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
mockSandboxGatewayPort(8091);
const run = vi.fn(() => ({ status: 1 }) as never);

expect(selectSandboxOwningGateway("beta", run)).toEqual({
Expand All @@ -73,12 +85,45 @@ describe("selectSandboxOwningGateway", () => {
});

it("reports failure when the gateway select command errors on spawn", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
mockSandboxGatewayPort(8091);
const run = vi.fn(() => ({ status: null, error: new Error("spawn failed") }) as never);

expect(selectSandboxOwningGateway("beta", run)).toEqual({
outcome: "failed",
gatewayName: "nemoclaw-8091",
});
});

it("resolves the owning gateway from a sibling gateway-port registry root", () => {
// Regression: with two gateways on one host, the sandbox's recorded
// binding lives under ~/.nemoclaw/gateways/<its port>/ even when the
// process points at another gateway port.
const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-select-cross-"));
try {
vi.stubEnv("HOME", home);
const siblingDir = path.join(home, ".nemoclaw", "gateways", "8245");
fs.mkdirSync(siblingDir, { recursive: true });
fs.writeFileSync(
path.join(siblingDir, "sandboxes.json"),
JSON.stringify({
defaultSandbox: null,
defaultSelectionRevision: 1,
sandboxes: { "owner-a": { name: "owner-a", gatewayPort: 8245 } },
}),
);
const run = vi.fn(() => ({ status: 0 }) as never);

expect(selectSandboxOwningGateway("owner-a", run)).toEqual({
outcome: "selected",
gatewayName: "nemoclaw-8245",
});
expect(run).toHaveBeenCalledWith(
["gateway", "select", "nemoclaw-8245"],
expect.objectContaining({ ignoreError: true }),
);
} finally {
vi.unstubAllEnvs();
fs.rmSync(home, { recursive: true, force: true });
}
});
});
11 changes: 11 additions & 0 deletions src/lib/actions/sandbox/gateway-state-drift.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ const openshellRuntime = requireDist("../../adapters/openshell/runtime.js");
const gatewayRuntime = requireDist("../../gateway-runtime-action.js");
const dockerDriverRecovery = requireDist("../../onboard/docker-driver-sandbox-recovery.js");
const registry = requireDist("../../state/registry.js");
const crossPortRegistry = requireDist("../../state/registry/cross-port.js");
const gatewaySelect = requireDist("./gateway-select.js");
const gatewayState: GatewayStateModule = requireDist("./gateway-state.js");

Expand Down Expand Up @@ -43,6 +44,7 @@ describe("sandbox gateway state drift guard", () => {
let detectPreflightIssueSpy: MockInstance;
let getNamedGatewayLifecycleStateSpy: MockInstance;
let getSandboxSpy: MockInstance;
let findSandboxAcrossGatewayRootsSpy: MockInstance;
let gatewaySelectSpy: MockInstance;
let recoverNamedGatewayRuntimeSpy: MockInstance;
let runOpenshellSpy: MockInstance;
Expand All @@ -54,6 +56,14 @@ describe("sandbox gateway state drift guard", () => {
errorSpy = vi.spyOn(console, "error").mockImplementation(() => undefined);

getSandboxSpy = vi.spyOn(registry, "getSandbox").mockReturnValue(null);
findSandboxAcrossGatewayRootsSpy = vi
.spyOn(crossPortRegistry, "findSandboxAcrossGatewayRoots")
.mockImplementation((name: unknown) => {
const entry = registry.getSandbox(String(name));
return entry
? { entry, gatewayPort: entry.gatewayPort ?? null, registryFile: "/test/sandboxes.json" }
: null;
});
gatewaySelectSpy = vi
.spyOn(gatewaySelect, "selectSandboxOwningGateway")
.mockReturnValue({ outcome: "selected", gatewayName: "nemoclaw" });
Expand Down Expand Up @@ -100,6 +110,7 @@ describe("sandbox gateway state drift guard", () => {
vi.spyOn(openshellRuntime, "isCommandTimeout").mockReturnValue(false),
getNamedGatewayLifecycleStateSpy,
getSandboxSpy,
findSandboxAcrossGatewayRootsSpy,
gatewaySelectSpy,
recoverNamedGatewayRuntimeSpy,
vi
Expand Down
11 changes: 11 additions & 0 deletions src/lib/actions/sandbox/gateway-state-hints.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ describe("printGatewayLifecycleHint multi-instance hints", () => {
let getSandboxDockerRuntimeSpy: MockInstance;
let getNamedGatewayLifecycleStateSpy: MockInstance;
let getSandboxSpy: MockInstance;
let findSandboxAcrossGatewayRootsSpy: MockInstance;
let recoverNamedGatewayRuntimeSpy: MockInstance;

function mockSandboxPhase(phase: string): void {
Expand All @@ -35,6 +36,7 @@ describe("printGatewayLifecycleHint multi-instance hints", () => {
const openshellRuntime = requireDist("../../adapters/openshell/runtime.js");
const gatewayRuntime = requireDist("../../gateway-runtime-action.js");
const registry = requireDist("../../state/registry.js");
const crossPortRegistry = requireDist("../../state/registry/cross-port.js");
const dockerHealth = requireDist("./docker-health.js");
const gatewaySelect = requireDist("./gateway-select.js");
vi.spyOn(gatewayDrift, "detectOpenShellStateRpcPreflightIssue").mockReturnValue(null);
Expand All @@ -52,6 +54,14 @@ describe("printGatewayLifecycleHint multi-instance hints", () => {
gatewayName: "nemoclaw",
gatewayPort: 8080,
});
findSandboxAcrossGatewayRootsSpy = vi
.spyOn(crossPortRegistry, "findSandboxAcrossGatewayRoots")
.mockImplementation((name: unknown) => {
const entry = registry.getSandbox(String(name));
return entry
? { entry, gatewayPort: entry.gatewayPort ?? null, registryFile: "/test/sandboxes.json" }
: null;
});
getSandboxDockerRuntimeSpy = vi.spyOn(dockerHealth, "getSandboxDockerRuntime").mockReturnValue({
health: "none",
paused: false,
Expand Down Expand Up @@ -83,6 +93,7 @@ describe("printGatewayLifecycleHint multi-instance hints", () => {
expect(combined).toContain("nemoclaw");
expect(combined).toContain("openshell gateway select");
expect(getSandboxSpy).toHaveBeenCalledWith("instance-a");
expect(findSandboxAcrossGatewayRootsSpy).toHaveBeenCalledWith("instance-a");
});

it("uses the sandbox's per-port gateway name in the hint for a non-default `NEMOCLAW_GATEWAY_PORT`", () => {
Expand Down
6 changes: 6 additions & 0 deletions src/lib/actions/sandbox/gateway-state-observe-mode.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import * as gatewayRuntime from "../../gateway-runtime-action";
import * as openshellRuntime from "../../adapters/openshell/runtime";
import * as portableAgentLifecycle from "../../onboard/experimental/portable-agent-lifecycle";
import * as registry from "../../state/registry";
import * as crossPortRegistry from "../../state/registry/cross-port";
import * as gatewaySelect from "./gateway-select";
import {
captureHermesPortableInferenceRecoveryGateway,
Expand All @@ -26,6 +27,11 @@ describe("getReconciledSandboxGatewayState observe mode", () => {
status: "Gateway: nemoclaw-8091\nStatus: Connected",
} as never);
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
vi.spyOn(crossPortRegistry, "findSandboxAcrossGatewayRoots").mockReturnValue({
entry: { name: "beta", gatewayPort: 8091 } as never,
gatewayPort: 8091,
registryFile: "/test/sandboxes.json",
});
});

afterEach(() => {
Expand Down
3 changes: 2 additions & 1 deletion src/lib/actions/sandbox/gateway-target.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@ import {
type SandboxGatewayBinding,
} from "../../onboard/gateway-binding";
import * as registry from "../../state/registry";
import { findSandboxAcrossGatewayRoots } from "../../state/registry/cross-port";

export function getKnownSandboxTarget(sandboxName: string): registry.SandboxEntry | null {
return registry.getSandbox(sandboxName);
return findSandboxAcrossGatewayRoots(sandboxName)?.entry ?? null;
}

export function listPersistedSandboxTargets(): registry.SandboxEntry[] {
Expand Down
9 changes: 9 additions & 0 deletions src/lib/actions/sandbox/policy-channel-conflict.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import { hashCredential } from "../../security/credential-hash";
import * as onboardSession from "../../state/onboard-session";
import type { SandboxEntry } from "../../state/registry";
import * as registry from "../../state/registry";
import * as crossPortRegistry from "../../state/registry/cross-port";
import * as messagingHostForwardLifecycle from "./messaging-host-forward-lifecycle";
import { addSandboxChannel, startSandboxChannel } from "./policy-channel";
import { policyChannelDependencies } from "./policy-channel-dependencies";
Expand Down Expand Up @@ -344,6 +345,14 @@ beforeEach(() => {

// Registry seam.
getSandboxMock = vi.spyOn(registry, "getSandbox").mockReturnValue(null);
vi.spyOn(crossPortRegistry, "findSandboxAcrossGatewayRoots").mockImplementation(
(name: string) => {
const entry = registry.getSandbox(name);
return entry
? { entry, gatewayPort: entry.gatewayPort ?? null, registryFile: "/test/sandboxes.json" }
: null;
},
);
getDisabledChannelsMock = vi.spyOn(registry, "getDisabledChannels").mockReturnValue([]);
listSandboxesMock = vi
.spyOn(registry, "listSandboxes")
Expand Down
9 changes: 9 additions & 0 deletions src/lib/actions/sandbox/rebuild-gateway-drift.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import * as openshellRuntime from "../../adapters/openshell/runtime";
import * as gatewayRuntime from "../../gateway-runtime-action";
import * as dockerDriverRecovery from "../../onboard/docker-driver-sandbox-recovery";
import * as registry from "../../state/registry";
import * as crossPortRegistry from "../../state/registry/cross-port";
import * as registryPersistence from "../../state/registry/persistence";
import { type RebuildSandboxEntry, resolveRebuildLiveState } from "./rebuild-flow-helpers";
import {
Expand Down Expand Up @@ -84,6 +85,14 @@ describe("rebuild gateway drift preflight", () => {
.spyOn(dockerDriverRecovery, "recoverDockerDriverSandbox")
.mockReturnValue({ recovered: false, via: null });
vi.spyOn(registry, "getSandbox").mockReturnValue(makeSandboxEntry() as never);
vi.spyOn(crossPortRegistry, "findSandboxAcrossGatewayRoots").mockImplementation(
(name: string) => {
const entry = registry.getSandbox(name);
return entry
? { entry, gatewayPort: entry.gatewayPort ?? null, registryFile: "/test/sandboxes.json" }
: null;
},
);
vi.spyOn(registryPersistence, "load").mockReturnValue({
sandboxes: { alpha: makeSandboxEntry() },
} as never);
Expand Down
3 changes: 2 additions & 1 deletion src/lib/actions/sandbox/status.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { inspectManagedLlamaCppStatus } from "../../inference/llama-cpp/managed-
import { getGatewayPresets } from "../../policy";
import { withMcpLifecycleLock } from "../../state/mcp-lifecycle-lock-acquisition";
import * as registry from "../../state/registry";
import { findSandboxAcrossGatewayRoots } from "../../state/registry/cross-port";
import { getSandboxDockerRuntime } from "./docker-health";
import {
qualifyPortableAgentLifecycleAuthority,
Expand Down Expand Up @@ -68,7 +69,7 @@ function inspectHermesPortableStatus(
}

function getPublishedSandbox(sandboxName: string): registry.SandboxEntry | null {
const entry = registry.getSandbox(sandboxName);
const entry = findSandboxAcrossGatewayRoots(sandboxName)?.entry ?? null;
return entry && registry.isPublishedSandboxRegistration(entry) ? entry : null;
}

Expand Down
Loading
Loading