fix(inference): recover dual-Station runtime ownership

docs/inference/set-up-vllm-on-two-dgx-stations.mdx

@@ -142,8 +142,8 @@ Follow [Set Up vLLM](set-up-vllm) for that workflow.
 ## Preserve Cleanup Ownership
 
 After readiness and container validation pass, NemoClaw writes an owner-only cleanup receipt.
-It also copies the SSH binding under the selected gateway state root.
-A later onboarding run that reuses the validated pair recreates this cleanup ownership before it accepts the endpoint.
+It copies the SSH binding beside the receipt under the host-global `~/.nemoclaw/` state root so every gateway port uses the same ownership state.
+A later onboarding run recovers and revalidates this cleanup ownership before it accepts the existing endpoint.
 
 The receipt contains no serving API key.
 It records the peer, cluster, and GPU identities needed to revalidate and remove both managed containers.

docs/reference/host-files-and-state.mdx

@@ -18,14 +18,12 @@ Deep Agents-specific runtime state lives inside the sandbox under `/sandbox/.dee
 </AgentOnly>
 
 <Warning>
-Do not paste a legacy `credentials.json`, provider tokens, bot tokens, proxy tokens, or debug archives containing them into chat or issue comments.
+Do not paste a legacy `credentials.json`, provider tokens, bot tokens, proxy tokens, `dual-station-vllm-api-key`, or debug archives containing them into chat or issue comments.
 Share redacted diagnostics only.
 </Warning>
 
 ## Files
 
-In the table, `<selected-state-root>` is `~/.nemoclaw/` for the default gateway or `~/.nemoclaw/gateways/<port>/` for a non-default gateway.
-
 | Path | Purpose | Safe to delete |
 |---|---|---|
 | `~/.nemoclaw/config.json` | Host-level CLI configuration and defaults created by onboarding or config commands. | Only if you want NemoClaw to forget host defaults and rebuild them on the next setup. |
@@ -34,8 +32,14 @@ In the table, `<selected-state-root>` is `~/.nemoclaw/` for the default gateway
 | `~/.nemoclaw/onboard-session.json` | Resume marker for an onboarding attempt that failed before completion. | Yes, when you intentionally want to discard the failed session and start over. Prefer `$$nemoclaw onboard --fresh` when available. |
 | `~/.nemoclaw/usage-notice.json` | Records the third-party software notice version in `acceptedVersion` and the acceptance time in `acceptedAt`. Install, onboarding, and rebuild flows consult this file and prompt again when its recorded version differs from the current notice or the file is absent. | Yes; deleting it makes the next applicable install, onboarding, or rebuild flow prompt for acceptance again. |
 | `~/.nemoclaw/ollama-proxy-token` | Local auth token used by the host-side Ollama auth proxy. | Yes, but re-run onboarding afterward so NemoClaw recreates and registers the proxy token. |
-| `<selected-state-root>/dual-station-vllm-runtime.json` | Owner-only managed dual-Station cleanup receipt. It contains no serving API key and binds the peer, cluster, and GPU identities used to revalidate and remove both managed vLLM containers during full uninstall. | No while the managed pair exists. A full `$$nemoclaw uninstall` removes the receipt after both exact containers are removed; gateway-scoped uninstall preserves it. |
-| `<selected-state-root>/dual-station-vllm-runtime.json.ssh-binding/` | Owner-only copied SSH host-key and Docker-command binding needed to reach the recorded worker during full uninstall. | No while the managed pair exists. Full uninstall removes it with the cleanup receipt after pair cleanup succeeds; gateway-scoped uninstall preserves it. |
+| `~/.nemoclaw/dual-station-vllm-api-key` | Host-global, owner-only bearer key shared by gateways that route to the managed dual-Station vLLM endpoint. The head requires this key for inference requests, and the worker does not receive it. | No while the managed pair exists. Full uninstall removes it after pair cleanup succeeds; gateway-scoped uninstall preserves it while sibling gateways remain. |
+| `~/.nemoclaw/dual-station-vllm-runtime.json` | Host-global, owner-only managed dual-Station cleanup receipt. It contains no serving API key and binds the peer, cluster, and GPU identities used to revalidate and remove both managed vLLM containers during full uninstall. | No while the managed pair exists. A full `$$nemoclaw uninstall` removes the receipt after both exact containers are removed; gateway-scoped uninstall preserves it. |
+| `~/.nemoclaw/dual-station-vllm-runtime.json.ssh-binding/` | Host-global, owner-only copied SSH host-key and Docker-command binding needed to reach the recorded worker during full uninstall. | No while the managed pair exists. Full uninstall removes it with the cleanup receipt after pair cleanup succeeds; gateway-scoped uninstall preserves it. |
+
+Earlier releases can store the runtime receipt and its SSH-binding directory under `~/.nemoclaw/gateways/<port>/` when the managed pair was created from a non-default gateway port.
+The API key remains host-global at `~/.nemoclaw/dual-station-vllm-api-key`.
+Current releases discover and validate the legacy receipt location without moving or duplicating its ownership state.
+Do not move these files or directories manually.
 
 `sandboxes.json` is the current registry file name.
 If you see `registry.json` in older tests, notes, or discussions, treat it as legacy wording for the sandbox registry unless a specific release note says otherwise.

src/lib/actions/uninstall/run-plan-dual-station.test.ts

@@ -48,6 +48,12 @@ describe("dual-Station runtime uninstall", () => {
     fs.writeFileSync(path.join(stateDir, "dual-station-vllm-runtime.json"), "{}\n", {
       mode: 0o600,
     });
+    fs.writeFileSync(path.join(stateDir, "dual-station-vllm-api-key"), "ab".repeat(32), {
+      mode: 0o600,
+    });
+    fs.mkdirSync(path.join(stateDir, "dual-station-vllm-runtime.json.ssh-binding"), {
+      mode: 0o700,
+    });
     const runDualStationRuntimeCleanup = vi.fn(() => ok());
     const rmSync = vi.fn();
     const runDocker = vi.fn(() => ok());
@@ -79,6 +85,111 @@ describe("dual-Station runtime uninstall", () => {
     }
   });
 
+  it("finds host-global pair ownership when the final gateway uses a non-default port", async () => {
+    const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-uninstall-dual-port-"));
+    const port = 9123;
+    const stateDir = path.join(home, ".nemoclaw");
+    const legacyStateDir = path.join(stateDir, "gateways", String(port));
+    fs.mkdirSync(legacyStateDir, { recursive: true, mode: 0o700 });
+    fs.writeFileSync(path.join(legacyStateDir, "dual-station-vllm-runtime.json"), "{}\n", {
+      mode: 0o600,
+    });
+    fs.writeFileSync(path.join(stateDir, "dual-station-vllm-api-key"), "ab".repeat(32), {
+      mode: 0o600,
+    });
+    fs.mkdirSync(path.join(legacyStateDir, "dual-station-vllm-runtime.json.ssh-binding"), {
+      mode: 0o700,
+    });
+    const runDualStationRuntimeCleanup = vi.fn(() => ok());
+
+    try {
+      vi.stubEnv("NEMOCLAW_GATEWAY_PORT", String(port));
+      vi.resetModules();
+      const { runUninstallPlan: runPortUninstallBase } = await import("./run-plan");
+      const result = runPortUninstallBase(
+        { assumeYes: true, deleteModels: false, keepOpenShell: true },
+        {
+          commandExists: () => true,
+          env: {
+            HOME: home,
+            NEMOCLAW_GATEWAY_PORT: String(port),
+            TMPDIR: home,
+          } as NodeJS.ProcessEnv,
+          existsSync: () => false,
+          isTty: false,
+          log: vi.fn(),
+          resolveGatewayTeardownAuthority: ({ gatewayName, gatewayPort }) => ({
+            gatewayName,
+            gatewayPort,
+            mode: "nemoclaw-managed",
+            source: "standalone",
+            endpoint: null,
+            stateDir: null,
+            supervisor: null,
+            requiredCapabilities: [],
+          }),
+          rmSync: vi.fn(),
+          run: (command, args) =>
+            command === "openshell" && args[0] === "gateway" && args[1] === "list"
+              ? ok(JSON.stringify([{ name: `nemoclaw-${String(port)}` }]))
+              : ok(),
+          runDocker: () => ok(),
+          runDualStationRuntimeCleanup,
+        },
+      );
+
+      expect(result.exitCode).toBe(0);
+      expect(runDualStationRuntimeCleanup).toHaveBeenCalledOnce();
+    } finally {
+      vi.unstubAllEnvs();
+      vi.resetModules();
+      fs.rmSync(home, { recursive: true, force: true });
+    }
+  });
+
+  it("preserves host-global pair ownership while sibling gateways remain", () => {
+    const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-uninstall-dual-scoped-"));
+    const stateDir = path.join(home, ".nemoclaw");
+    const apiKeyPath = path.join(stateDir, "dual-station-vllm-api-key");
+    const receiptPath = path.join(stateDir, "dual-station-vllm-runtime.json");
+    const bindingPath = `${receiptPath}.ssh-binding`;
+    const selectedStatePath = path.join(stateDir, "selected-only");
+    fs.mkdirSync(bindingPath, { recursive: true, mode: 0o700 });
+    fs.writeFileSync(apiKeyPath, "ab".repeat(32), { mode: 0o600 });
+    fs.writeFileSync(receiptPath, "{}\n", { mode: 0o600 });
+    fs.writeFileSync(selectedStatePath, "remove me\n");
+    const runDualStationRuntimeCleanup = vi.fn(() => ok());
+
+    try {
+      const result = runUninstallPlan(
+        { assumeYes: true, deleteModels: false, destroyUserData: true, keepOpenShell: true },
+        {
+          commandExists: (command) => command === "openshell",
+          env: { HOME: home, TMPDIR: home } as NodeJS.ProcessEnv,
+          existsSync: fs.existsSync,
+          isTty: false,
+          log: vi.fn(),
+          rmSync: fs.rmSync,
+          run: (command, args) =>
+            command === "openshell" && args[0] === "gateway" && args[1] === "list"
+              ? ok(JSON.stringify([{ name: "nemoclaw" }, { name: "sibling" }]))
+              : ok(),
+          runDocker: () => ok(),
+          runDualStationRuntimeCleanup,
+        },
+      );
+
+      expect(result.exitCode).toBe(0);
+      expect(fs.existsSync(apiKeyPath)).toBe(true);
+      expect(fs.existsSync(receiptPath)).toBe(true);
+      expect(fs.existsSync(bindingPath)).toBe(true);
+      expect(fs.existsSync(selectedStatePath)).toBe(false);
+      expect(runDualStationRuntimeCleanup).not.toHaveBeenCalled();
+    } finally {
+      fs.rmSync(home, { recursive: true, force: true });
+    }
+  });
+
   it("does not start the remaining uninstall steps when managed pair cleanup fails", () => {
     const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-uninstall-dual-fail-"));
     const stateDir = path.join(home, ".nemoclaw");

src/lib/actions/uninstall/run-plan.ts

@@ -27,7 +27,11 @@ import {
 } from "../../domain/uninstall/paths";
 import { buildUninstallPlan, type UninstallPlan } from "../../domain/uninstall/plan";
 import { isOllamaAuthProxyCommandLine } from "../../inference/ollama/process";
-import { DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE } from "../../inference/vllm-station-runtime-receipt-path";
+import {
+  DUAL_STATION_VLLM_API_KEY_FILE,
+  DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE,
+  discoverDualStationVllmRuntimeReceiptStateDirs,
+} from "../../inference/vllm-station-runtime-receipt-path";
 import { buildDockerGatewayDebEnvFile } from "../../onboard/docker-driver-gateway-env";
 import {
   getNemoclawOpenShellGatewayUserServicePath,
@@ -251,6 +255,9 @@ const SHARED_HOST_STATE_ENTRIES = new Set([
   "source",
   GATEWAYS_SUBDIR,
   "managed_swap",
+  DUAL_STATION_VLLM_API_KEY_FILE,
+  DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE,
+  `${DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE}.ssh-binding`,
   ...HTTPS_PIN_RUNTIME_ADAPTER_STATE_ENTRIES,
 ]);
 
@@ -1209,14 +1216,32 @@ function removeManagedDualStationRuntime(
   paths: UninstallPaths,
   runtime: UninstallRuntime,
 ): boolean {
-  const receiptPath = path.join(paths.nemoclawStateDir, DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE);
+  const sharedStateDir = path.dirname(paths.managedSwapMarkerPath);
+  let receiptPaths: string[];
   try {
-    fs.lstatSync(receiptPath);
+    receiptPaths = discoverDualStationVllmRuntimeReceiptStateDirs(
+      sharedStateDir,
+      GATEWAYS_SUBDIR,
+    ).map((stateDir) => path.join(stateDir, DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE));
   } catch (error) {
-    if ((error as NodeJS.ErrnoException).code === "ENOENT") return true;
     runtime.error(`Could not inspect managed dual-Station rollback state: ${formatError(error)}`);
     return false;
   }
+  let receiptFound = false;
+  for (const receiptPath of receiptPaths) {
+    try {
+      fs.lstatSync(receiptPath);
+      receiptFound = true;
+    } catch (error) {
+      if ((error as NodeJS.ErrnoException).code !== "ENOENT") {
+        runtime.error(
+          `Could not inspect managed dual-Station rollback state: ${formatError(error)}`,
+        );
+        return false;
+      }
+    }
+  }
+  if (!receiptFound) return true;
   const result = runtime.runDualStationRuntimeCleanup({
     env: runtime.env,
     stdio: "inherit",
@@ -1753,6 +1778,7 @@ function executePlan(
             ...(scopedToSelectedGateway
               ? [
                   ...HTTPS_PIN_RUNTIME_ADAPTER_STATE_ENTRIES,
+                  DUAL_STATION_VLLM_API_KEY_FILE,
                   DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE,
                   `${DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE}.ssh-binding`,
                 ]

src/lib/inference/vllm-api-key.ts

@@ -9,8 +9,8 @@ import path from "node:path";
 import { DEFAULT_GATEWAY_PORT } from "../core/ports";
 import { nemoclawStateRoot } from "../state/state-root";
 import { ensureLocalAdapterStateDir } from "./local-adapter-lifecycle";
+import { DUAL_STATION_VLLM_API_KEY_FILE } from "./vllm-station-runtime-receipt-path";
 
-export const DUAL_STATION_VLLM_API_KEY_FILE = "dual-station-vllm-api-key";
 export const DUAL_STATION_VLLM_API_KEY_PATTERN = /^[a-f0-9]{64}$/;
 
 export interface DualStationVllmApiKeyOptions {

src/lib/inference/vllm-dual-station.test.ts

@@ -29,6 +29,7 @@ const mocks = vi.hoisted(() => ({
   preflightOwnership: vi.fn(),
   persistRuntimeReceipt: vi.fn(),
   probeCapability: vi.fn(),
+  recoverRuntime: vi.fn(),
   probeDockerStorage: vi.fn(),
   probeHostStorage: vi.fn(),
   runCapture: vi.fn(),
@@ -93,6 +94,7 @@ vi.mock("./vllm-station-cluster-lifecycle", () => ({
 
 vi.mock("./vllm-station-runtime-receipt", () => ({
   persistDualStationVllmRuntimeReceipt: mocks.persistRuntimeReceipt,
+  recoverInstalledDualStationVllmRuntime: mocks.recoverRuntime,
 }));
 
 vi.mock("./vllm-api-key", () => ({
@@ -244,6 +246,7 @@ beforeEach(() => {
   mocks.cleanup.mockReturnValue({ ok: true, removedContainerIds: [] });
   mocks.commitLegacyMigration.mockResolvedValue({ ok: true, cleanupWarnings: [] });
   mocks.persistRuntimeReceipt.mockImplementation(() => {});
+  mocks.recoverRuntime.mockReturnValue({ kind: "not-installed" });
   mocks.rollbackLegacyMigration.mockResolvedValue({ ok: true });
   mocks.measureDirectorySizeBytes.mockReturnValue(0n);
   mocks.probeDockerStorage.mockReturnValue({
@@ -310,6 +313,7 @@ describe("dual DGX Station running-runtime receipt adoption", () => {
     });
 
     expect(mocks.probeCapability).toHaveBeenCalledOnce();
+    expect(mocks.recoverRuntime).not.toHaveBeenCalled();
     expect(mocks.preflightOwnership).toHaveBeenCalledWith(plan());
     expect(mocks.areContainersRunning).toHaveBeenCalledWith(plan());
     expect(mocks.persistRuntimeReceipt).toHaveBeenCalledWith(plan());
@@ -335,9 +339,106 @@ describe("dual DGX Station running-runtime receipt adoption", () => {
       ok: false,
       reason: "the managed dual-Station peer configuration is missing",
     });
+    expect(mocks.recoverRuntime).toHaveBeenCalledOnce();
     expect(mocks.probeCapability).not.toHaveBeenCalled();
     expect(mocks.persistRuntimeReceipt).not.toHaveBeenCalled();
   });
+
+  it("recovers and revalidates persisted pair ownership for a later onboarding process", async () => {
+    delete process.env.NEMOCLAW_DGX_STATION_PEER;
+    const recoveredPlan = plan();
+    let lifecycleActive = false;
+    mocks.recoverRuntime.mockReturnValue({ kind: "ready", plan: recoveredPlan });
+    mocks.withLifecycle.mockImplementation(async (operation) => {
+      lifecycleActive = true;
+      try {
+        return await operation();
+      } finally {
+        lifecycleActive = false;
+      }
+    });
+    mocks.recoverRuntime.mockImplementation(() => {
+      expect(lifecycleActive).toBe(true);
+      return { kind: "ready", plan: recoveredPlan };
+    });
+    mocks.preflightOwnership.mockImplementation(() => {
+      expect(lifecycleActive).toBe(true);
+      return { ok: true };
+    });
+    mocks.areContainersRunning.mockImplementation(() => {
+      expect(lifecycleActive).toBe(true);
+      return true;
+    });
+
+    await expect(persistConfiguredDualStationVllmRuntimeReceipt()).resolves.toEqual({
+      ok: true,
+      persisted: true,
+    });
+
+    expect(mocks.recoverRuntime).toHaveBeenCalledOnce();
+    expect(mocks.probeCapability).not.toHaveBeenCalled();
+    expect(mocks.preflightOwnership).toHaveBeenCalledWith(recoveredPlan);
+    expect(mocks.areContainersRunning).toHaveBeenCalledWith(recoveredPlan);
+    expect(mocks.persistRuntimeReceipt).not.toHaveBeenCalled();
+    expect(lifecycleActive).toBe(false);
+  });
+
+  it("fails closed when the receipt disappears before locked recovery", async () => {
+    delete process.env.NEMOCLAW_DGX_STATION_PEER;
+    let lifecycleActive = false;
+    mocks.withLifecycle.mockImplementation(async (operation) => {
+      lifecycleActive = true;
+      try {
+        mocks.recoverRuntime.mockImplementation(() => {
+          expect(lifecycleActive).toBe(true);
+          return { kind: "not-installed" };
+        });
+        return await operation();
+      } finally {
+        lifecycleActive = false;
+      }
+    });
+
+    await expect(persistConfiguredDualStationVllmRuntimeReceipt()).resolves.toEqual({
+      ok: false,
+      reason: "the managed dual-Station peer configuration is missing",
+    });
+    expect(mocks.recoverRuntime).toHaveBeenCalledOnce();
+    expect(mocks.preflightOwnership).not.toHaveBeenCalled();
+    expect(mocks.persistRuntimeReceipt).not.toHaveBeenCalled();
+    expect(lifecycleActive).toBe(false);
+  });
+
+  it("fails closed when persisted pair ownership is unsafe", async () => {
+    delete process.env.NEMOCLAW_DGX_STATION_PEER;
+    mocks.recoverRuntime.mockReturnValue({
+      kind: "unsafe",
+      reason: "could not revalidate the managed pair: managed runtime identity changed",
+    });
+
+    await expect(persistConfiguredDualStationVllmRuntimeReceipt()).resolves.toEqual({
+      ok: false,
+      reason:
+        "the managed dual-Station cleanup receipt is unsafe: could not revalidate the managed pair: managed runtime identity changed",
+    });
+    expect(mocks.preflightOwnership).not.toHaveBeenCalled();
+    expect(mocks.areContainersRunning).not.toHaveBeenCalled();
+    expect(mocks.persistRuntimeReceipt).not.toHaveBeenCalled();
+  });
+
+  it("fails closed when a recovered pair changes before locked ownership validation", async () => {
+    delete process.env.NEMOCLAW_DGX_STATION_PEER;
+    const recoveredPlan = plan();
+    mocks.recoverRuntime.mockReturnValue({ kind: "ready", plan: recoveredPlan });
+    mocks.areContainersRunning.mockReturnValue(false);
+
+    await expect(persistConfiguredDualStationVllmRuntimeReceipt()).resolves.toEqual({
+      ok: false,
+      reason: "the managed dual-Station containers changed before cleanup ownership validation",
+    });
+    expect(mocks.preflightOwnership).toHaveBeenCalledWith(recoveredPlan);
+    expect(mocks.persistRuntimeReceipt).not.toHaveBeenCalled();
+  });
 });
 
 describe("dual DGX Station vLLM install orchestration", () => {