Skip to content
Merged
2 changes: 2 additions & 0 deletions docs/manage-sandboxes/run-sandboxes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ When creating a Hermes sandbox, if another sandbox or a host listener already ho
<AgentOnly variant="openclaw,hermes">
If you intentionally run separate OpenShell gateways on the same host, set a different `NEMOCLAW_GATEWAY_PORT` before each onboarding run.
NemoClaw isolates the gateway name and local state by port so one port-specific gateway does not replace another.
Sandbox-scoped commands that read, exec into, or inspect a sandbox, such as `$$nemoclaw <name> exec` and `$$nemoclaw <name> status`, address each sandbox through its recorded gateway binding, so the current shell's `NEMOCLAW_GATEWAY_PORT` does not redirect them. Lifecycle commands such as `$$nemoclaw <name> stop` still act on the gateway that `NEMOCLAW_GATEWAY_PORT` selects.
A non-default `NEMOCLAW_GATEWAY_PORT` also gets its own host state root at `~/.nemoclaw/gateways/<port>/`, with a separate sandbox registry, snapshots, and legacy credential-migration files, so gateway-scoped state stays segregated while shared host-level files remain under `~/.nemoclaw/`.
On first use after upgrading, NemoClaw moves legacy rows and related state only when their recorded gateway identity matches the selected port; ambiguous state is left untouched with remediation.
Provider credentials remain in the OpenShell gateway store.
Expand All @@ -78,6 +79,7 @@ Hermes also retains or restores its registered OpenAI-compatible API forward usi
<AgentOnly variant="deepagents">
If you intentionally run separate OpenShell gateways on the same host, set a different `NEMOCLAW_GATEWAY_PORT` before each onboarding run.
NemoClaw isolates the gateway name and local state by port so one port-specific gateway does not replace another.
Sandbox-scoped commands that read, exec into, or inspect a sandbox, such as `nemo-deepagents <name> exec` and `nemo-deepagents <name> status`, address each sandbox through its recorded gateway binding, so the current shell's `NEMOCLAW_GATEWAY_PORT` does not redirect them. Lifecycle commands such as `nemo-deepagents <name> stop` still act on the gateway that `NEMOCLAW_GATEWAY_PORT` selects.
A non-default `NEMOCLAW_GATEWAY_PORT` also gets its own host state root at `~/.nemoclaw/gateways/<port>/`, with a separate sandbox registry, snapshots, and legacy credential-migration files, so gateway-scoped state stays segregated while shared host-level files remain under `~/.nemoclaw/`.
On first use after upgrading, NemoClaw moves legacy rows and related state only when their recorded gateway identity matches the selected port; ambiguous state is left untouched with remediation.
Provider credentials remain in the OpenShell gateway store.
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -424,7 +424,7 @@ $$nemoclaw list

A gateway port that keeps listening after you uninstall, or that keeps serving after you re-onboard without `NEMOCLAW_GATEWAY_PORT` set, can belong to a second environment.

Onboarding under a non-default `NEMOCLAW_GATEWAY_PORT` registers the sandbox on gateway `nemoclaw-<port>` and stores its registry and state under `~/.nemoclaw/gateways/<port>/`. Every later command reads the state root that its own `NEMOCLAW_GATEWAY_PORT` selects. A command without that variable uses port `8080` unless the installer recorded an automatic alternate-port selection; in that case, the CLI restores the recorded port. Clearing an explicitly supplied variable does not move an existing sandbox back to the default port.
Onboarding under a non-default `NEMOCLAW_GATEWAY_PORT` registers the sandbox on gateway `nemoclaw-<port>` and stores its registry and state under `~/.nemoclaw/gateways/<port>/`. Gateway-lifecycle commands such as `$$nemoclaw onboard`, `$$nemoclaw uninstall`, and the deprecated `$$nemoclaw stop` act on the state root that their own `NEMOCLAW_GATEWAY_PORT` selects, while sandbox-scoped commands that read, exec into, or inspect a sandbox, such as `$$nemoclaw <name> exec` and `$$nemoclaw <name> status`, resolve the named sandbox across every per-port registry root and address its recorded gateway binding. A gateway-lifecycle command without that variable uses port `8080` unless the installer recorded an automatic alternate-port selection; in that case, the CLI restores the recorded port. Clearing an explicitly supplied variable does not move an existing sandbox back to the default port.

List the gateways the host still has:

Expand Down
9 changes: 9 additions & 0 deletions src/lib/actions/sandbox/gateway-failure-classifier.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,15 @@ vi.mock("../../state/registry", () => ({
listSandboxes: vi.fn(() => ({ sandboxes: [] })),
}));

vi.mock("../../state/registry/cross-port", () => ({
findSandboxAcrossGatewayRoots: (...args: unknown[]) => {
const entry = getSandboxMock(...args);
return entry
? { entry, gatewayPort: entry.gatewayPort ?? null, registryFile: "/test/sandboxes.json" }
: null;
},
}));

import {
classifyGatewayFailure,
classifyObservedSandboxContainerFailure,
Expand Down
59 changes: 52 additions & 7 deletions src/lib/actions/sandbox/gateway-select.test.ts
Original file line number Diff line number Diff line change
@@ -1,18 +1,30 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import fs from "node:fs";
import os from "node:os";
import path from "node:path";

import { afterEach, describe, expect, it, vi } from "vitest";

import * as registry from "../../state/registry";
import * as crossPort from "../../state/registry/cross-port";
import { selectSandboxOwningGateway } from "./gateway-select";

function mockSandboxGatewayPort(gatewayPort: number): void {
vi.spyOn(crossPort, "findSandboxAcrossGatewayRoots").mockReturnValue({
entry: { name: "sandbox", gatewayPort },
gatewayPort,
registryFile: "/test/sandboxes.json",
});
}

describe("selectSandboxOwningGateway", () => {
afterEach(() => {
vi.restoreAllMocks();
});

it("selects the owning non-default gateway for a registered sandbox", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
mockSandboxGatewayPort(8091);
const run = vi.fn(() => ({ status: 0 }) as never);

const selected = selectSandboxOwningGateway("beta", run);
Expand All @@ -28,7 +40,7 @@ describe("selectSandboxOwningGateway", () => {
});

it("replays selection output through the stdio adapter", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8080 } as never);
mockSandboxGatewayPort(8080);
const output = "\u001b[32m✓ Active gateway set to 'nemoclaw'\u001b[0m\n";
const run = vi.fn(() => ({ status: 0, stdout: output }) as never);
const write = vi.fn();
Expand All @@ -41,7 +53,7 @@ describe("selectSandboxOwningGateway", () => {
});

it("keeps the bare default gateway name for a default-port sandbox", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8080 } as never);
mockSandboxGatewayPort(8080);
const run = vi.fn(() => ({ status: 0 }) as never);

expect(selectSandboxOwningGateway("alpha", run)).toEqual({
Expand All @@ -52,7 +64,7 @@ describe("selectSandboxOwningGateway", () => {
});

it("does not touch the active gateway for an unregistered sandbox", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue(null);
vi.spyOn(crossPort, "findSandboxAcrossGatewayRoots").mockReturnValue(null);
const run = vi.fn(() => ({ status: 0 }) as never);

expect(selectSandboxOwningGateway("ghost", run)).toEqual({
Expand All @@ -63,7 +75,7 @@ describe("selectSandboxOwningGateway", () => {
});

it("reports failure when the gateway select command exits nonzero", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
mockSandboxGatewayPort(8091);
const run = vi.fn(() => ({ status: 1 }) as never);

expect(selectSandboxOwningGateway("beta", run)).toEqual({
Expand All @@ -73,12 +85,45 @@ describe("selectSandboxOwningGateway", () => {
});

it("reports failure when the gateway select command errors on spawn", () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
mockSandboxGatewayPort(8091);
const run = vi.fn(() => ({ status: null, error: new Error("spawn failed") }) as never);

expect(selectSandboxOwningGateway("beta", run)).toEqual({
outcome: "failed",
gatewayName: "nemoclaw-8091",
});
});

it("resolves the owning gateway from a sibling gateway-port registry root", () => {
// Regression: with two gateways on one host, the sandbox's recorded
// binding lives under ~/.nemoclaw/gateways/<its port>/ even when the
// process points at another gateway port.
const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-select-cross-"));
try {
vi.stubEnv("HOME", home);
const siblingDir = path.join(home, ".nemoclaw", "gateways", "8245");
fs.mkdirSync(siblingDir, { recursive: true });
fs.writeFileSync(
path.join(siblingDir, "sandboxes.json"),
JSON.stringify({
defaultSandbox: null,
defaultSelectionRevision: 1,
sandboxes: { "owner-a": { name: "owner-a", gatewayPort: 8245 } },
}),
);
const run = vi.fn(() => ({ status: 0 }) as never);

expect(selectSandboxOwningGateway("owner-a", run)).toEqual({
outcome: "selected",
gatewayName: "nemoclaw-8245",
});
expect(run).toHaveBeenCalledWith(
["gateway", "select", "nemoclaw-8245"],
expect.objectContaining({ ignoreError: true }),
);
} finally {
vi.unstubAllEnvs();
fs.rmSync(home, { recursive: true, force: true });
}
});
});
11 changes: 11 additions & 0 deletions src/lib/actions/sandbox/gateway-state-drift.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ const openshellRuntime = requireDist("../../adapters/openshell/runtime.js");
const gatewayRuntime = requireDist("../../gateway-runtime-action.js");
const dockerDriverRecovery = requireDist("../../onboard/docker-driver-sandbox-recovery.js");
const registry = requireDist("../../state/registry.js");
const crossPortRegistry = requireDist("../../state/registry/cross-port.js");
const gatewaySelect = requireDist("./gateway-select.js");
const gatewayState: GatewayStateModule = requireDist("./gateway-state.js");

Expand Down Expand Up @@ -43,6 +44,7 @@ describe("sandbox gateway state drift guard", () => {
let detectPreflightIssueSpy: MockInstance;
let getNamedGatewayLifecycleStateSpy: MockInstance;
let getSandboxSpy: MockInstance;
let findSandboxAcrossGatewayRootsSpy: MockInstance;
let gatewaySelectSpy: MockInstance;
let recoverNamedGatewayRuntimeSpy: MockInstance;
let runOpenshellSpy: MockInstance;
Expand All @@ -54,6 +56,14 @@ describe("sandbox gateway state drift guard", () => {
errorSpy = vi.spyOn(console, "error").mockImplementation(() => undefined);

getSandboxSpy = vi.spyOn(registry, "getSandbox").mockReturnValue(null);
findSandboxAcrossGatewayRootsSpy = vi
.spyOn(crossPortRegistry, "findSandboxAcrossGatewayRoots")
.mockImplementation((name: unknown) => {
const entry = registry.getSandbox(String(name));
return entry
? { entry, gatewayPort: entry.gatewayPort ?? null, registryFile: "/test/sandboxes.json" }
: null;
});
gatewaySelectSpy = vi
.spyOn(gatewaySelect, "selectSandboxOwningGateway")
.mockReturnValue({ outcome: "selected", gatewayName: "nemoclaw" });
Expand Down Expand Up @@ -100,6 +110,7 @@ describe("sandbox gateway state drift guard", () => {
vi.spyOn(openshellRuntime, "isCommandTimeout").mockReturnValue(false),
getNamedGatewayLifecycleStateSpy,
getSandboxSpy,
findSandboxAcrossGatewayRootsSpy,
gatewaySelectSpy,
recoverNamedGatewayRuntimeSpy,
vi
Expand Down
11 changes: 11 additions & 0 deletions src/lib/actions/sandbox/gateway-state-hints.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ describe("printGatewayLifecycleHint multi-instance hints", () => {
let getSandboxDockerRuntimeSpy: MockInstance;
let getNamedGatewayLifecycleStateSpy: MockInstance;
let getSandboxSpy: MockInstance;
let findSandboxAcrossGatewayRootsSpy: MockInstance;
let recoverNamedGatewayRuntimeSpy: MockInstance;

function mockSandboxPhase(phase: string): void {
Expand All @@ -35,6 +36,7 @@ describe("printGatewayLifecycleHint multi-instance hints", () => {
const openshellRuntime = requireDist("../../adapters/openshell/runtime.js");
const gatewayRuntime = requireDist("../../gateway-runtime-action.js");
const registry = requireDist("../../state/registry.js");
const crossPortRegistry = requireDist("../../state/registry/cross-port.js");
const dockerHealth = requireDist("./docker-health.js");
const gatewaySelect = requireDist("./gateway-select.js");
vi.spyOn(gatewayDrift, "detectOpenShellStateRpcPreflightIssue").mockReturnValue(null);
Expand All @@ -52,6 +54,14 @@ describe("printGatewayLifecycleHint multi-instance hints", () => {
gatewayName: "nemoclaw",
gatewayPort: 8080,
});
findSandboxAcrossGatewayRootsSpy = vi
.spyOn(crossPortRegistry, "findSandboxAcrossGatewayRoots")
.mockImplementation((name: unknown) => {
const entry = registry.getSandbox(String(name));
return entry
? { entry, gatewayPort: entry.gatewayPort ?? null, registryFile: "/test/sandboxes.json" }
: null;
});
getSandboxDockerRuntimeSpy = vi.spyOn(dockerHealth, "getSandboxDockerRuntime").mockReturnValue({
health: "none",
paused: false,
Expand Down Expand Up @@ -83,6 +93,7 @@ describe("printGatewayLifecycleHint multi-instance hints", () => {
expect(combined).toContain("nemoclaw");
expect(combined).toContain("openshell gateway select");
expect(getSandboxSpy).toHaveBeenCalledWith("instance-a");
expect(findSandboxAcrossGatewayRootsSpy).toHaveBeenCalledWith("instance-a");
});

it("uses the sandbox's per-port gateway name in the hint for a non-default `NEMOCLAW_GATEWAY_PORT`", () => {
Expand Down
6 changes: 6 additions & 0 deletions src/lib/actions/sandbox/gateway-state-observe-mode.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import * as openshellRuntime from "../../adapters/openshell/runtime";
import * as dockerDriverRecovery from "../../onboard/docker-driver-sandbox-recovery";
import * as portableAgentLifecycle from "../../onboard/experimental/portable-agent-lifecycle";
import * as registry from "../../state/registry";
import * as crossPortRegistry from "../../state/registry/cross-port";
import * as gatewaySelect from "./gateway-select";
import {
captureHermesPortableInferenceRecoveryGateway,
Expand All @@ -27,6 +28,11 @@ describe("getReconciledSandboxGatewayState observe mode", () => {
status: "Gateway: nemoclaw-8091\nStatus: Connected",
} as never);
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
vi.spyOn(crossPortRegistry, "findSandboxAcrossGatewayRoots").mockReturnValue({
entry: { name: "beta", gatewayPort: 8091 } as never,
gatewayPort: 8091,
registryFile: "/test/sandboxes.json",
});
});

afterEach(() => {
Expand Down
15 changes: 7 additions & 8 deletions src/lib/actions/sandbox/gateway-state-owning-gateway.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import * as gatewayDrift from "../../adapters/openshell/gateway-drift";
import * as openshellRuntime from "../../adapters/openshell/runtime";
import * as gatewayRuntime from "../../gateway-runtime-action";
import * as registry from "../../state/registry";
import * as crossPortRegistry from "../../state/registry/cross-port";
import * as gatewaySelect from "./gateway-select";
import {
getReconciledSandboxGatewayState,
Expand All @@ -16,6 +16,11 @@ import {

describe("getReconciledSandboxGatewayState owning-gateway guard", () => {
beforeEach(() => {
vi.spyOn(crossPortRegistry, "findSandboxAcrossGatewayRoots").mockReturnValue({
entry: { name: "beta", gatewayPort: 8091 } as never,
gatewayPort: 8091,
registryFile: "/test/sandboxes.json",
});
vi.spyOn(gatewaySelect, "selectSandboxOwningGateway").mockReturnValue({
outcome: "selected",
gatewayName: "nemoclaw-8091",
Expand Down Expand Up @@ -114,7 +119,6 @@ describe("getReconciledSandboxGatewayState owning-gateway guard", () => {
});

it("targets the registered owner on the first lookup instead of accepting ambient sibling state", async () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
const getState = vi.fn((_name: string, gatewayName?: string) =>
gatewayName === "nemoclaw-8091"
? { state: "present", output: "Phase: Ready" }
Expand All @@ -133,7 +137,6 @@ describe("getReconciledSandboxGatewayState owning-gateway guard", () => {
});

it("queries the gateway returned by selection when registry ownership changes between snapshots", async () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
vi.mocked(gatewaySelect.selectSandboxOwningGateway).mockReturnValue({
outcome: "selected",
gatewayName: "nemoclaw-8092",
Expand All @@ -148,7 +151,6 @@ describe("getReconciledSandboxGatewayState owning-gateway guard", () => {
});

it("keeps recovery pinned to the gateway returned by selection after ownership changes", async () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
vi.mocked(gatewaySelect.selectSandboxOwningGateway).mockReturnValue({
outcome: "selected",
gatewayName: "nemoclaw-8092",
Expand Down Expand Up @@ -178,7 +180,6 @@ describe("getReconciledSandboxGatewayState owning-gateway guard", () => {
});

it("does not return an ambient sibling timeout without first querying the healthy owner", async () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
const getState = vi.fn((_name: string, gatewayName?: string) =>
gatewayName === "nemoclaw-8091"
? { state: "present", output: "Phase: Ready" }
Expand All @@ -193,7 +194,6 @@ describe("getReconciledSandboxGatewayState owning-gateway guard", () => {
});

it("does not return an ambient sibling unknown error without first querying the healthy owner", async () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
const getState = vi.fn((_name: string, gatewayName?: string) =>
gatewayName === "nemoclaw-8091"
? { state: "present", output: "Phase: Ready" }
Expand All @@ -208,7 +208,6 @@ describe("getReconciledSandboxGatewayState owning-gateway guard", () => {
});

it("fails closed before lookup when the owning gateway cannot be selected", async () => {
vi.spyOn(registry, "getSandbox").mockReturnValue({ gatewayPort: 8091 } as never);
vi.mocked(gatewaySelect.selectSandboxOwningGateway).mockReturnValue({
outcome: "failed",
gatewayName: "nemoclaw-8091",
Expand All @@ -223,7 +222,7 @@ describe("getReconciledSandboxGatewayState owning-gateway guard", () => {
});

it("leaves an unregistered sandbox on the existing default lookup path", async () => {
vi.spyOn(registry, "getSandbox").mockReturnValue(null);
vi.mocked(crossPortRegistry.findSandboxAcrossGatewayRoots).mockReturnValue(null);
const getState = vi.fn().mockResolvedValue({ state: "present", output: "Phase: Ready" });

const result = await getReconciledSandboxGatewayState("ghost", { getState });
Expand Down
3 changes: 2 additions & 1 deletion src/lib/actions/sandbox/gateway-target.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@ import {
type SandboxGatewayBinding,
} from "../../onboard/gateway-binding";
import * as registry from "../../state/registry";
import { findSandboxAcrossGatewayRoots } from "../../state/registry/cross-port";

export function getKnownSandboxTarget(sandboxName: string): registry.SandboxEntry | null {
return registry.getSandbox(sandboxName);
return findSandboxAcrossGatewayRoots(sandboxName)?.entry ?? null;
}

export function listPersistedSandboxTargets(): registry.SandboxEntry[] {
Expand Down
Loading
Loading