Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/manage-sandboxes/lifecycle.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,8 @@ If the gateway is already healthy, `recover` exits after the probe and does not
For built-in OpenClaw sandboxes, recovery sends an authenticated request through registry-scoped privileged direct-container control.
The request reaches either the root PID 1 supervisor in a direct root-entrypoint container or the installed root controller in an OpenShell-managed container.
It does not fall back to ordinary `openshell sandbox exec` or a manual relaunch inside the sandbox.
If the in-sandbox supervisor is no longer running, such as after a container restart, `recover` relaunches it and waits for the gateway to come back.
If the gateway still does not recover — for example when the sandbox needs credentials that only re-onboarding can restore — recreate the runtime with `$$nemoclaw <name> rebuild --yes`, or destroy and re-onboard the sandbox.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Refer to [`$$nemoclaw <name> recover`](../reference/commands#$$nemoclaw-name-recover) for details.

Use `gateway restart` when you intentionally need a supported OpenClaw gateway to reload runtime configuration or plugins.
Expand All @@ -255,6 +257,8 @@ If the gateway is already healthy, `recover` exits after the probe and does not
For built-in Hermes sandboxes, recovery sends an authenticated request through registry-scoped privileged direct-container control.
The request reaches either the root PID 1 supervisor in a direct root-entrypoint container or the installed root controller in an OpenShell-managed container.
It does not fall back to ordinary `openshell sandbox exec` or a manual relaunch inside the sandbox.
If the in-sandbox supervisor is no longer running, such as after a container restart, `recover` relaunches it and waits for the gateway to come back.
If the gateway still does not recover — for example when the sandbox needs credentials that only re-onboarding can restore — recreate the runtime with `$$nemoclaw <name> rebuild --yes`, or destroy and re-onboard the sandbox.
Refer to [`$$nemoclaw <name> recover`](../reference/commands#$$nemoclaw-name-recover) for details.

Use `gateway restart` when you intentionally need a supported Hermes gateway to reload runtime configuration or plugins.
Expand Down
1 change: 1 addition & 0 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3465,6 +3465,7 @@ The following flags change defaults for commands that manage existing sandboxes.
| `NEMOCLAW_CLEANUP_GATEWAY` | `1`, `true`, or `yes` to enable; `0`, `false`, or `no` to disable | Sets the default for whether `$$nemoclaw <name> destroy` removes the shared gateway when destroying the last sandbox. Command-line `--cleanup-gateway` and `--no-cleanup-gateway` still take precedence. |
| `NEMOCLAW_CONFIRM_LEGACY_MANAGED_RECREATE` | Exact JSON array of sandbox names | Confirms to the installer that the exact listed set of pre-fingerprint OpenClaw or Hermes sandboxes used NemoClaw-managed images, allowing recovery onto the current managed image. The normalized names must exactly match the installer's printed array. Set it only after verifying every named sandbox. Recorded custom-image evidence remains blocked. |
| `NEMOCLAW_DISABLE_INFERENCE_ROUTE_REPAIR` | `1` to enable | Skips the automatic DNS-proxy repair for stale `inference.local` routes during `$$nemoclaw <name> connect` and `$$nemoclaw <name> connect --probe-only`. Use only as a troubleshooting escape hatch. |
| `NEMOCLAW_DISABLE_SUPERVISOR_RELAUNCH` | `1` to enable | Skips the automatic in-sandbox supervisor relaunch during `$$nemoclaw <name> recover` when the supervisor is gone, for example after a container restart. Use only as a troubleshooting escape hatch; recovery then falls back to the rebuild or re-onboard guidance. |
| `NEMOCLAW_SHIELDS_ACCEPT_LEGACY_BASELINE` | `1` to opt in | Allows advanced immutable-config verification to trust the current on-disk bytes for older or partial content baselines. Use only after you have rebuilt or manually inspected the sandbox state and accepted that the baseline is operator-approved. |
| `NEMOCLAW_SHIELDS_SETTLE_MS` | milliseconds (default `750`, clamped to `0` to `10000`) | Settle window NemoClaw waits after re-applying a config lockdown (during shields auto-restore and `$$nemoclaw <name> shields up` drift remediation) before re-confirming the lock still holds. Detects when an in-sandbox reconciler changes config file permissions after lockdown and re-applies the lock; if NemoClaw cannot re-confirm the lock within the retry budget, shields stay down. This narrows the window in which a reconciler can revert permissions rather than eliminating it. The best-effort `chattr +i` immutable bit remains the only fully durable lock. Raise it on hosts where the gateway settles slowly. |
| `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP` | Exactly `1` to opt in (`true`, `yes`, `0` are not accepted) | Applies to standalone `$$nemoclaw backup-all` runs. Skips running sandboxes whose in-sandbox SSH endpoint does not answer. It does not relax the installer's strict pre-upgrade backup, which still aborts if any registered sandbox is skipped or fails. Any uncommitted state since the last successful backup is not included in the skipped backup. |
Expand Down
1 change: 1 addition & 0 deletions src/lib/actions/sandbox/gateway-restart.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ describe("gateway restart failure markers", () => {
[string, ReturnType<typeof classifyGatewayRestartFailure>["layer"]]
> = [
["PRIVILEGED_CONTROL_UNAVAILABLE", "privileged control unavailable"],
["SUPERVISOR_UNAVAILABLE", "supervisor not running"],
["SUPERVISOR_REBUILD_REQUIRED", "privileged control unavailable"],
["SUPERVISOR_BUSY", "privileged control unavailable"],
[MARKERS.SECRET_BOUNDARY_REFUSED, "secret-boundary refusal"],
Expand Down
8 changes: 7 additions & 1 deletion src/lib/actions/sandbox/gateway-restart.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ export type GatewayRestartCommandResult = {
export type GatewayRestartFailureLayer =
| "unsupported agent"
| "privileged control unavailable"
| "supervisor not running"
| "secret-boundary refusal"
| "unsafe config path"
| "config hash mismatch"
Expand Down Expand Up @@ -127,10 +128,15 @@ export function classifyGatewayRestartFailure(result: GatewayRestartCommandResul

const output = gatewayRestartOutput(result);
const detail = sanitizeGatewayRestartFailureDetail(output.trim());
if (output.includes("SUPERVISOR_UNAVAILABLE")) {
return {
layer: "supervisor not running",
detail: detail || "the in-sandbox gateway supervisor is not running",
};
}
if (
output.includes(MARKERS.ROOT_EXEC_UNAVAILABLE) ||
output.includes("PRIVILEGED_CONTROL_UNAVAILABLE") ||
output.includes("SUPERVISOR_UNAVAILABLE") ||
output.includes("SUPERVISOR_REBUILD_REQUIRED") ||
output.includes("SUPERVISOR_UNSAFE_CONTROL_DIR") ||
output.includes("SUPERVISOR_BUSY") ||
Expand Down
105 changes: 98 additions & 7 deletions src/lib/actions/sandbox/process-recovery.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,12 @@ import {
recoverMessagingHostForward,
resolveSandboxDashboardPort,
} from "./forward-recovery";
import { shouldManageDashboardForAgent } from "../../onboard/dashboard-runtime";
import { buildSandboxRuntimeEnvArgs } from "../../onboard/sandbox-create-launch";
import {
classifyGatewayRestartFailure,
type GatewayRestartDeps,
type GatewayRestartFailureLayer,
type GatewayRestartResult,
printGatewayRestartFailure,
type RestartSandboxGatewayOptions,
Expand Down Expand Up @@ -278,6 +281,72 @@ export function executeGatewaySupervisorAction(
return { status, stdout, stderr };
}

function reconstructSupervisorLaunchEnvArgs(sandboxName: string): string[] | null {
const entry = registry.getSandbox(sandboxName);
if (!entry) return null;
const agent = agentRuntime.getSessionAgent(sandboxName) ?? null;
const manageDashboard = shouldManageDashboardForAgent(agent);
const dashboardPort = String(resolveSandboxDashboardPort(sandboxName));
const chatUiUrl = manageDashboard ? `http://127.0.0.1:${dashboardPort}` : "";
const hermesDashboardEnabled = entry.hermesDashboardEnabled === true;
const { envArgs } = buildSandboxRuntimeEnvArgs({
agent,
chatUiUrl,
manageDashboard,
getDashboardForwardPort: () => dashboardPort,
hermesDashboardState: {
enabled: hermesDashboardEnabled,
config: hermesDashboardEnabled
? {
enabled: true,
port: entry.hermesDashboardPort ?? 0,
internalPort: entry.hermesDashboardInternalPort ?? 0,
tuiEnabled: entry.hermesDashboardTui === true,
}
: null,
},
extraPlaceholderKeys: [],
observabilityEnabled: entry.observabilityEnabled === true,
sandboxName,
env: process.env,
omitCredentialEnv: true,
});
return envArgs;
}

function relaunchManagedSupervisorSession(
sandboxName: string,
{ quiet }: { quiet: boolean },
): boolean {
if (process.env.NEMOCLAW_DISABLE_SUPERVISOR_RELAUNCH === "1") return false;
const envArgs = reconstructSupervisorLaunchEnvArgs(sandboxName);
if (envArgs === null) return false;
const startedMarker = "NEMOCLAW_SUPERVISOR_RELAUNCHED";
const envPrefix = envArgs.map(shellQuote).join(" ");
const daemonCommand =
`echo ${startedMarker}; ` +
`setsid nohup env ${envPrefix} nemoclaw-start ` +
">/tmp/nemoclaw-start-recover.log 2>&1 </dev/null &";
if (!quiet) console.log(" Relaunching the in-sandbox supervisor...");
let result: ReturnType<typeof spawnSync>;
try {
result = spawnSync(
getOpenshellBinary(),
["sandbox", "exec", "--name", sandboxName, "--", "sh", "-c", daemonCommand],
{
cwd: ROOT,
encoding: "utf-8",
env: buildSubprocessEnv(),
stdio: ["ignore", "pipe", "pipe"],
timeout: DEFAULT_SANDBOX_EXEC_TIMEOUT_MS,
},
);
} catch {
return false;
}
return result.status === 0 && String(result.stdout || "").includes(startedMarker);
}

async function executeSandboxExecCommandForStatus(
sandboxName: string,
command: string,
Expand Down Expand Up @@ -401,11 +470,13 @@ function recoverSandboxProcesses(
{
quiet = false,
requestGatewaySupervisorAction = executeGatewaySupervisorAction,
onFailureLayer,
}: {
quiet?: boolean;
requestGatewaySupervisorAction?: typeof executeGatewaySupervisorAction;
onFailureLayer?: (layer: GatewayRestartFailureLayer) => void;
} = {},
): "managed" | "custom" | null {
): "managed" | "custom" | "relaunched" | null {
const agent = agentRuntime.getSessionAgent(sandboxName);
const dashboardPort = resolveSandboxDashboardPort(sandboxName);
let persistedAgent: string | null;
Expand All @@ -421,7 +492,7 @@ function recoverSandboxProcesses(
}
const recoveredSsh = (result: SandboxCommandResult | null) =>
!!(result && result.status === 0 && hasGatewayRecoveryMarker(result));
const recoverManagedGateway = (): boolean => {
const recoverManagedGateway = (): "managed" | "relaunched" | null => {
const maxAttempts = 3;
const retryIntervalSeconds = readNonNegativeNumberEnv(
"NEMOCLAW_GATEWAY_RECOVERY_POLL_INTERVAL_SECONDS",
Expand All @@ -430,7 +501,7 @@ function recoverSandboxProcesses(
let execResult: SandboxCommandResult | null = null;
for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
execResult = requestGatewaySupervisorAction(sandboxName, "recover");
if (hasGatewayRecoveryMarker(execResult)) return true;
if (hasGatewayRecoveryMarker(execResult)) return "managed";

// PID 1 may replace the gateway between the host's stopped observation
// and the controller's process-tree capture. Retry only exact transient
Expand All @@ -439,16 +510,23 @@ function recoverSandboxProcesses(
sleepSeconds(retryIntervalSeconds);
}
const failure = classifyGatewayRestartFailure(execResult);
onFailureLayer?.(failure.layer);
if (
failure.layer === "supervisor not running" &&
relaunchManagedSupervisorSession(sandboxName, { quiet })
) {
return "relaunched";
}
if (!quiet) printGatewayRestartFailure(sandboxName, failure.layer, failure.detail);
return false;
return null;
};
if (persistedAgent === "hermes") {
if (!isHermesAgent(agent)) {
const detail = "Hermes agent definition could not be loaded.";
if (!quiet) printGatewayRestartFailure(sandboxName, "unsupported agent", detail);
return null;
}
return recoverManagedGateway() ? "managed" : null;
return recoverManagedGateway();
}

// A persisted non-OpenClaw runtime whose manifest cannot be loaded is not
Expand All @@ -463,7 +541,7 @@ function recoverSandboxProcesses(
}

if ((!persistedAgent || persistedAgent === "openclaw") && (!agent || agent.name === "openclaw")) {
return recoverManagedGateway() ? "managed" : null;
return recoverManagedGateway();
}

const agentScript = agentRuntime.buildRecoveryScript(agent, dashboardPort);
Expand Down Expand Up @@ -535,8 +613,17 @@ function gatewayRecoveryTimeoutSeconds(
function printHostManagedGatewayRecoveryHints(
sandboxName: string,
agent: ReturnType<typeof agentRuntime.getSessionAgent>,
failureLayer: GatewayRestartFailureLayer | null = null,
): void {
const quotedSandboxName = shellQuote(sandboxName);
if (failureLayer === "supervisor not running") {
console.error(" The in-sandbox supervisor is not running, so the managed gateway restart");
console.error(" cannot relaunch it (a container restart stops the supervisor).");
console.error(" Recreate the sandbox runtime to restore it:");
console.error(` nemoclaw ${quotedSandboxName} rebuild --yes`);
console.error(" If rebuild is blocked, destroy and re-onboard the sandbox to restore it.");
return;
}
let agentName = agent?.name ?? null;
if (!agentName) {
try {
Expand Down Expand Up @@ -848,9 +935,13 @@ function checkAndRecoverSandboxProcessesWithoutHostLock(
console.log(" Recovering...");
}

let managedRecoveryFailureLayer: GatewayRestartFailureLayer | null = null;
const recoveryKind = recoverSandboxProcesses(sandboxName, {
quiet,
requestGatewaySupervisorAction,
onFailureLayer: (layer) => {
managedRecoveryFailureLayer = layer;
},
});
if (recoveryKind !== null) {
// Wait for gateway to bind its HTTP port before declaring success. The
Expand Down Expand Up @@ -930,7 +1021,7 @@ function checkAndRecoverSandboxProcessesWithoutHostLock(
}
if (!quiet) {
console.error(` Could not restart ${recoveryDisplayName} gateway automatically.`);
printHostManagedGatewayRecoveryHints(sandboxName, recoveryAgent);
printHostManagedGatewayRecoveryHints(sandboxName, recoveryAgent, managedRecoveryFailureLayer);
}

return { checked: true, wasRunning: false, recovered: false, forwardRecovered: false };
Expand Down
30 changes: 30 additions & 0 deletions src/lib/onboard/sandbox-create-launch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
import { SANDBOX_BUILD_CONTEXT_PREFIX } from "../sandbox/build-context";
import { createOpenshellCliHelpers } from "./openshell-cli";
import {
buildSandboxRuntimeEnvArgs,
prepareSandboxCreateLaunch,
prepareSandboxCreateLaunchWithPrebuild,
} from "./sandbox-create-launch";
Expand All @@ -31,6 +32,35 @@ afterEach(() => {
}
});

describe("buildSandboxRuntimeEnvArgs", () => {
it("omits credential-bearing env when omitCredentialEnv is set", () => {
const base = {
agent: { name: "openclaw", configPaths: { dir: "/sandbox/.openclaw" } } as any,
chatUiUrl: "http://127.0.0.1:19000/",
manageDashboard: true,
getDashboardForwardPort: () => "19000",
hermesDashboardState: disabledHermesDashboardState,
extraPlaceholderKeys: ["TELEGRAM_BOT_TOKEN_AGENT_A"],
env: {
HTTPS_PROXY: "http://proxyuser:proxypass@proxy.example:8080",
NEMOCLAW_PROXY_HOST: "host.docker.internal",
NEMOCLAW_PROXY_PORT: "3129",
} as NodeJS.ProcessEnv,
};

const included = buildSandboxRuntimeEnvArgs(base).envArgs;
expect(included).toContain("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS=TELEGRAM_BOT_TOKEN_AGENT_A");
expect(included.some((arg) => arg.startsWith("HTTPS_PROXY="))).toBe(true);

const omitted = buildSandboxRuntimeEnvArgs({ ...base, omitCredentialEnv: true }).envArgs;
expect(omitted.some((arg) => arg.startsWith("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS"))).toBe(false);
expect(omitted.some((arg) => arg.includes("proxypass"))).toBe(false);
expect(omitted.some((arg) => arg.startsWith("HTTPS_PROXY="))).toBe(false);
expect(omitted).toContain("NEMOCLAW_DASHBOARD_PORT=19000");
expect(omitted).toContain("NEMOCLAW_PROXY_HOST=host.docker.internal");
});
});

describe("prepareSandboxCreateLaunch", () => {
it("builds the sandbox create command and runtime env envelope", () => {
const openshellShellCommand = vi.fn((args: string[]) => `openshell ${args.join(" ")}`);
Expand Down
52 changes: 44 additions & 8 deletions src/lib/onboard/sandbox-create-launch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,24 @@ export interface SandboxCreateLaunchWithPrebuild extends SandboxCreateLaunch {
prebuild: SandboxPrebuildResult;
}

export function prepareSandboxCreateLaunch(input: SandboxCreateLaunchInput): SandboxCreateLaunch {
const env = input.env ?? process.env;
const manageDashboard = input.manageDashboard ?? true;
export interface SandboxRuntimeEnvArgsInput {
agent: AgentDefinition | null;
chatUiUrl: string;
manageDashboard: boolean;
getDashboardForwardPort(chatUiUrl: string): string;
hermesDashboardState: HermesDashboardOnboardState;
extraPlaceholderKeys: readonly string[];
observabilityEnabled?: boolean;
sandboxName?: string;
env: NodeJS.ProcessEnv;
omitCredentialEnv?: boolean;
}

export function buildSandboxRuntimeEnvArgs(input: SandboxRuntimeEnvArgsInput): {
envArgs: string[];
effectiveDashboardPort: string;
} {
const { agent, env, manageDashboard } = input;
const envArgs = manageDashboard ? [formatEnvAssignment("CHAT_UI_URL", input.chatUiUrl)] : [];

// When manageDashboard is enabled, pass the effective dashboard port into
Expand All @@ -93,11 +108,12 @@ export function prepareSandboxCreateLaunch(input: SandboxCreateLaunchInput): San
envArgs.push(formatEnvAssignment("NEMOCLAW_DASHBOARD_PORT", effectiveDashboardPort));
}

appendOpenClawRuntimeEnvArgs(envArgs, input.agent ?? null);
appendOpenClawAutoPairRuntimeEnvArgs(envArgs, input.agent ?? null, env);
appendOpenClawRuntimeEnvArgs(envArgs, agent);
appendOpenClawAutoPairRuntimeEnvArgs(envArgs, agent, env);
appendHermesDashboardEnvArgs(envArgs, input.hermesDashboardState, formatEnvAssignment);
appendHostProxyEnvArgs(envArgs, env, {
dropCredentialBearingProxyUrls: input.agent?.name === "langchain-deepagents-code",
dropCredentialBearingProxyUrls:
agent?.name === "langchain-deepagents-code" || input.omitCredentialEnv === true,
});

// Propagate NEMOCLAW_PROXY_HOST / NEMOCLAW_PROXY_PORT to runtime containers
Expand All @@ -116,7 +132,7 @@ export function prepareSandboxCreateLaunch(input: SandboxCreateLaunchInput): San
envArgs.push(formatEnvAssignment("NEMOCLAW_PROXY_PORT", sandboxProxyPort));
}

if (input.agent?.name === "langchain-deepagents-code") {
if (agent?.name === "langchain-deepagents-code") {
const sandboxName = input.sandboxName;
if (sandboxName) {
envArgs.push(formatEnvAssignment("NEMOCLAW_SANDBOX_NAME", sandboxName));
Expand All @@ -129,7 +145,27 @@ export function prepareSandboxCreateLaunch(input: SandboxCreateLaunchInput): San
);
}

appendExtraPlaceholderKeysEnvArg(envArgs, input.extraPlaceholderKeys, formatEnvAssignment);
if (!input.omitCredentialEnv) {
appendExtraPlaceholderKeysEnvArg(envArgs, input.extraPlaceholderKeys, formatEnvAssignment);
}

return { envArgs, effectiveDashboardPort };
}

export function prepareSandboxCreateLaunch(input: SandboxCreateLaunchInput): SandboxCreateLaunch {
const env = input.env ?? process.env;
const manageDashboard = input.manageDashboard ?? true;
const { envArgs, effectiveDashboardPort } = buildSandboxRuntimeEnvArgs({
agent: input.agent ?? null,
chatUiUrl: input.chatUiUrl,
manageDashboard,
getDashboardForwardPort: input.getDashboardForwardPort,
hermesDashboardState: input.hermesDashboardState,
extraPlaceholderKeys: input.extraPlaceholderKeys,
observabilityEnabled: input.observabilityEnabled,
sandboxName: input.sandboxName,
env,
});

const sandboxEnv = (input.buildEnv ?? buildSubprocessEnv)();
// Remove host-infrastructure credentials that the generic allowlist
Expand Down
Loading
Loading