Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1233,7 +1233,8 @@ Run `$$nemoclaw <name> shields down` before a Hermes config or inference change;

<AgentOnly variant="openclaw,hermes">

The command can fail at these layers: unsupported agent, privileged control unavailable, supervisor not running, secret-boundary refusal, unsafe config path, config hash mismatch when a strict hash is available, launch failure, health timeout, or forward recovery failure.
The command can fail at these layers: unsupported agent, privileged control unavailable, supervisor not running, secret-boundary refusal, unsafe config path, config hash mismatch when a strict hash is available, MCP reconciliation refusal, relaunch quarantined, launch failure, health timeout, or forward recovery failure.
`relaunch quarantined` means the in-sandbox supervisor stopped attempting relaunch after a startup refusal or repeated gateway exits, so restart and recovery report the supported repair, `$$nemoclaw <name> rebuild --yes`, instead of a retry.
An older direct-container image without the matching supervisor or managed controller helper reports `privileged control unavailable` and requires `$$nemoclaw <name> rebuild --yes`.
Ordinary OpenShell exec and manual in-sandbox relaunch are not fallback paths.
Terminal agents do not have a gateway runtime and fail as unsupported.
Expand Down
15 changes: 15 additions & 0 deletions docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2709,6 +2709,21 @@ The Hermes entrypoint supervisor remains responsible for the gateway, dashboard,
In the OpenShell-managed topology, that nonroot supervisor repairs failed auxiliaries continuously, recovers a gateway after four consecutive failed listener or HTTP health checks, and quarantines relaunch after five exits within 60 seconds until the sandbox is recreated.
The host only repairs the host-side OpenShell forwards after the supervised processes pass health checks.

### Restart or recovery reports `relaunch quarantined`

In the OpenShell-managed topology the strict root-owned hash is not a trust anchor for mutable config, so a direct edit of `/sandbox/.hermes/config.yaml` or `/sandbox/.hermes/.env` is not refused by the host controller.
The in-sandbox supervisor still refuses to start a gateway on configuration it cannot match to the persisted managed state, and it stops attempting relaunch once that refusal or repeated gateway exits exhaust its crash budget.
`$$nemoclaw <name> gateway restart`, `$$nemoclaw <name> recover`, and `$$nemoclaw <name> connect` then report the `relaunch quarantined` failure layer.

The refusal is deterministic, so retrying any of those commands cannot clear it.
Restore the registered configuration and refresh its integrity metadata in one transaction:

```bash
nemohermes <name> rebuild --yes
```

After the rebuild, make the intended change through a supported command such as `$$nemoclaw <name> config set` or `$$nemoclaw inference set`, which update the configuration and its hashes together.

### Port 8642 in a browser shows a blank page or `Cannot GET /`

`nemohermes onboard` forwards port `8642`, but Hermes serves an OpenAI-compatible API at that port, not a chat dashboard.
Expand Down
24 changes: 24 additions & 0 deletions src/lib/actions/sandbox/connect-boundary-refusal.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import {
type GatewayRestartFailureLayer,
gatewayIntegrityRepairLines,
isGatewayIntegrityRepairLayer,
} from "./gateway-restart";
import type { SecretBoundaryRefusalReason } from "./hermes-secret-boundary-recovery";
import {
hermesMcpReconciliationRemediationLines,
Expand All @@ -9,6 +14,25 @@ import {

type ConnectBoundaryContext = "Probe" | "Connect";

/**
* A managed recovery that failed on a deterministic integrity refusal cannot be
* retried: every relaunch re-reads the same drifted protected configuration.
* The probe path recovers quietly, so without this the operator only sees the
* generic "check the gateway log" and never learns the supported repair (#7801).
* Returns false when the layer is a retryable failure, leaving the caller's
* existing wedge diagnostics in charge.
*/
export function printGatewayIntegrityRepairGuidance(
sandboxName: string,
layer: GatewayRestartFailureLayer | null | undefined,
): boolean {
if (!isGatewayIntegrityRepairLayer(layer)) return false;
for (const line of gatewayIntegrityRepairLines(sandboxName, layer)) {
console.error(` ${line}`);
}
return true;
}

export function exitOnSecretBoundaryRefusal(
sandboxName: string,
agentName: string,
Expand Down
29 changes: 28 additions & 1 deletion src/lib/actions/sandbox/connect-flow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -460,7 +460,10 @@ describe("connectSandbox flow", () => {

await expect(harness.connectSandbox("alpha", { probeOnly: true })).resolves.toBeUndefined();

expect(harness.checkAndRecoverSpy).toHaveBeenCalledWith("alpha", { quiet: true });
expect(harness.checkAndRecoverSpy).toHaveBeenCalledWith(
"alpha",
expect.objectContaining({ quiet: true }),
);
expect(harness.runAutoPairSpy).toHaveBeenCalledWith("alpha", expect.any(Object));
expect(harness.spawnSyncSpy).not.toHaveBeenCalledWith(
"openshell",
Expand Down Expand Up @@ -489,6 +492,30 @@ describe("connectSandbox flow", () => {
);
expect(exitSpy).toHaveBeenCalledWith(1);
});
it("probe-only mode reports the supported repair when relaunch is quarantined (#7801)", async () => {
const harness = createConnectHarness({
processCheck: { checked: true, wasRunning: false, recovered: false },
});
// Managed recovery runs quiet on this path, so the classified layer only
// reaches the operator through the callback the probe passes in.
harness.checkAndRecoverSpy.mockImplementation((_sandboxName: unknown, options: unknown) => {
(
options as { onRecoveryFailureLayer?: (layer: string) => void } | undefined
)?.onRecoveryFailureLayer?.("relaunch quarantined");
return { checked: true, wasRunning: false, recovered: false };
});

await expect(harness.connectSandbox("alpha", { probeOnly: true })).rejects.toThrow(
"process.exit(1)",
);

const errorOutput = harness.errorSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
expect(errorOutput).toContain("quarantined gateway relaunch");
expect(errorOutput).toContain("nemoclaw alpha rebuild --yes");
expect(errorOutput).not.toContain("Check /tmp/gateway.log inside the sandbox for details.");
expect(exitSpy).toHaveBeenCalledWith(1);
});

it("probe-only mode exits when primary dashboard/API forward recovery fails", async () => {
const harness = createConnectHarness({
processCheck: {
Expand Down
16 changes: 15 additions & 1 deletion src/lib/actions/sandbox/connect.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ import {
import {
exitOnMcpReconciliationRefusal,
exitOnSecretBoundaryRefusal,
printGatewayIntegrityRepairGuidance,
} from "./connect-boundary-refusal";
import { prepareHermesLightTerminalSkin } from "./connect-hermes-light-skin";
import {
Expand All @@ -82,6 +83,7 @@ import { printGatewayWedgeDiagnostics } from "./gateway-wedge-diagnostics";
import {
checkAndRecoverSandboxProcesses,
executeSandboxExecCommand,
type GatewayRestartFailureLayer,
resolveSandboxDashboardPort,
} from "./process-recovery";
import { runTerminalAgentConnectProbe } from "./terminal-connect-probe";
Expand Down Expand Up @@ -245,7 +247,16 @@ async function runSandboxConnectProbe(sandboxName: string): Promise<void> {
return;
}

const processCheck = checkAndRecoverSandboxProcesses(sandboxName, { quiet: true });
// Managed recovery runs quiet here, so its classified failure layer is the
// only way this path can tell a retryable wedge apart from a deterministic
// integrity refusal that no restart, recover, or connect can clear (#7801).
let recoveryFailureLayer: GatewayRestartFailureLayer | null = null;
const processCheck = checkAndRecoverSandboxProcesses(sandboxName, {
quiet: true,
onRecoveryFailureLayer: (layer) => {
recoveryFailureLayer = layer;
},
});
if (!processCheck.checked) {
console.error(
` Probe failed: could not inspect the ${agentName} gateway inside sandbox '${sandboxName}'.`,
Expand Down Expand Up @@ -296,6 +307,9 @@ async function runSandboxConnectProbe(sandboxName: string): Promise<void> {
console.error(
` Probe failed: ${agentName} gateway is not running in '${sandboxName}' and automatic recovery failed.`,
);
if (printGatewayIntegrityRepairGuidance(sandboxName, recoveryFailureLayer)) {
process.exit(1);
}
// Surface the #4710 wedge signature: recovery ran with quiet=true, so this
// is the operator's only window into a gateway that served briefly and
// then dropped its listener.
Expand Down
149 changes: 149 additions & 0 deletions src/lib/actions/sandbox/gateway-restart-quarantine-repair.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { afterEach, describe, expect, it, vi } from "vitest";
import {
classifyGatewayRestartFailure,
gatewayIntegrityRepairLines,
isGatewayIntegrityRepairLayer,
printGatewayRestartFailure,
} from "./gateway-restart";

// The exact lines the in-sandbox Hermes supervisor emits when it stops
// attempting relaunch. `scripts/managed-gateway-control.py` allowlists these
// before forwarding them to the host as `NEMOCLAW_START_LOG=` lines.
const QUARANTINE_LINES = [
"[gateway] CRITICAL: 5 exits in 60s window — Hermes relaunch is quarantined until sandbox recreation; check /tmp/gateway.log",
"[SECURITY] Hermes automatic respawn is quarantined until MCP integrity is restored by rebuilding the sandbox",
"[gateway] CRITICAL: exact Hermes replacement could not be stopped; managed supervisor is quarantined without another launch",
"[CRITICAL] Unproven Hermes gateway child exited; managed supervisor remains quarantined until sandbox recreation",
"[CRITICAL] Newly launched Hermes gateway pid 4242 failed exact role identity capture; quarantining the managed startup supervisor without signaling the unproven child",
] as const;

// Verbatim controller output captured on a Hermes sandbox whose protected
// `config.yaml` was edited outside a supported command, then restarted.
const REPORTED_RESTART_OUTPUT = [
"GATEWAY_HEALTH_TIMEOUT",
"NEMOCLAW_CONTROL_STAGE=await-replacement",
"NEMOCLAW_SUPERVISOR_PID=42",
"NEMOCLAW_GATEWAY_PID=0",
"NEMOCLAW_START_LOG=[gateway] Hermes gateway respawned (pid 18424)",
"NEMOCLAW_START_LOG=[SECURITY] Hermes automatic respawn is quarantined until MCP integrity is restored by rebuilding the sandbox",
].join("\n");

function classify(stdout: string) {
return classifyGatewayRestartFailure({ status: 1, stdout, stderr: "" });
}

function captureStderr(run: () => void): string[] {
const lines: string[] = [];
const spy = vi.spyOn(console, "error").mockImplementation((value?: unknown) => {
lines.push(String(value));
});
run();
spy.mockRestore();
return lines;
}

afterEach(() => {
vi.restoreAllMocks();
});

describe("supervisor relaunch quarantine classification (#7801)", () => {
it.each(QUARANTINE_LINES)("classifies %s as a relaunch quarantine", (line) => {
expect(classify(line)).toMatchObject({ layer: "relaunch quarantined" });
});

it("classifies the reported restart output as a quarantine, not a health timeout", () => {
expect(classify(REPORTED_RESTART_OUTPUT)).toMatchObject({ layer: "relaunch quarantined" });
});

it("prefers the quarantine over the MCP drift it is reported through", () => {
const output = [
"HERMES_MCP_CONFIG_DRIFT",
"[SECURITY] Hermes automatic respawn is quarantined until MCP integrity is restored by rebuilding the sandbox",
].join("\n");
expect(classify(output)).toMatchObject({ layer: "relaunch quarantined" });
});

it("keeps the pre-existing layers for output without a quarantine line", () => {
expect(classify("GATEWAY_HEALTH_TIMEOUT")).toMatchObject({ layer: "health timeout" });
expect(classify("HERMES_MCP_CONFIG_DRIFT")).toMatchObject({
layer: "MCP reconciliation refusal",
});
expect(classify("GATEWAY_CONFIG_HASH_MISMATCH")).toMatchObject({
layer: "config hash mismatch",
});
expect(classify("SUPERVISOR_NOT_RUNNING")).toMatchObject({ layer: "supervisor not running" });
});

it("ignores an unrelated line that merely mentions the supervisor", () => {
expect(classify("[gateway] Hermes gateway respawned (pid 18424)")).toMatchObject({
layer: "launch failure",
});
});
});

describe("integrity repair guidance (#7801)", () => {
it("treats both deterministic integrity refusals as repairable layers", () => {
expect(isGatewayIntegrityRepairLayer("relaunch quarantined")).toBe(true);
expect(isGatewayIntegrityRepairLayer("config hash mismatch")).toBe(true);
expect(isGatewayIntegrityRepairLayer("health timeout")).toBe(false);
expect(isGatewayIntegrityRepairLayer("launch failure")).toBe(false);
expect(isGatewayIntegrityRepairLayer(null)).toBe(false);
expect(isGatewayIntegrityRepairLayer(undefined)).toBe(false);
});

it.each([
"relaunch quarantined",
"config hash mismatch",
] as const)("names the supported repair command for %s", (layer) => {
const lines = gatewayIntegrityRepairLines("repro-7801", layer).join("\n");
expect(lines).toContain("nemoclaw repro-7801 rebuild --yes");
expect(lines).toContain("Retrying the restart cannot clear it.");
expect(lines).toContain("nemoclaw repro-7801 config set");
});

it("describes the two refusals differently", () => {
const quarantined = gatewayIntegrityRepairLines("alpha", "relaunch quarantined")[0];
const drifted = gatewayIntegrityRepairLines("alpha", "config hash mismatch")[0];
expect(quarantined).not.toEqual(drifted);
expect(drifted).toContain("integrity hash");
expect(quarantined).toContain("quarantined");
});
});

describe("printGatewayRestartFailure repair guidance (#7801)", () => {
it("appends the repair to a quarantined restart failure", () => {
const lines = captureStderr(() =>
printGatewayRestartFailure("repro-7801", "relaunch quarantined", REPORTED_RESTART_OUTPUT),
).join("\n");
expect(lines).toContain("Failure layer: relaunch quarantined");
expect(lines).toContain("nemoclaw repro-7801 rebuild --yes");
});

it("still prints the repair when the controller returned no detail", () => {
const lines = captureStderr(() =>
printGatewayRestartFailure("repro-7801", "config hash mismatch", ""),
).join("\n");
expect(lines).toContain("nemoclaw repro-7801 rebuild --yes");
});

it("leaves retryable failure layers without a rebuild instruction", () => {
const timeout = captureStderr(() =>
printGatewayRestartFailure("repro-7801", "health timeout", "GATEWAY_HEALTH_TIMEOUT"),
).join("\n");
const launch = captureStderr(() =>
printGatewayRestartFailure("repro-7801", "launch failure", "GATEWAY_FAILED"),
).join("\n");
expect(timeout).not.toContain("rebuild --yes");
expect(launch).not.toContain("rebuild --yes");
});

it("keeps the MCP reconciliation remediation it already emitted", () => {
const lines = captureStderr(() =>
printGatewayRestartFailure("repro-7801", "MCP reconciliation refusal", "mcp-integrity"),
).join("\n");
expect(lines).toContain("nemoclaw repro-7801 mcp restart");
});
});
Loading
Loading