Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1160,6 +1160,15 @@ $$nemoclaw my-assistant shields down --timeout 5m --reason "maintenance"
If `shields up` reports that the config remains unlocked or drifted, confirm that the sandbox is running and ready, then retry `$$nemoclaw <name> shields up`.
If the retry still fails, rebuild a known-good baseline with `$$nemoclaw <name> rebuild --yes`.

<AgentOnly variant="deepagents">

A `CRITICAL` Deep Agents config-lock failure is not an ordinary unlocked or drifted result.
The retry and rebuild guidance above does not apply to a `CRITICAL` Deep Agents config-lock diagnostic.
Do not retry `shields up` or attempt an in-sandbox repair.
Follow [Deep Agents Config Lock Failure Recovery](troubleshooting#deep-agents-config-lock-failure-recovery) to restore a trusted snapshot or recreate the sandbox before retrying.

</AgentOnly>

Host-side config and inference writes, snapshot mutation, sandbox destruction, and shields transitions serialize per sandbox.
When a timed shields-down window reaches its deadline, auto-restore can interrupt the exact process tree holding that transition and restore lockdown.
Retry an interrupted command in a new shields-down window.
Expand Down
63 changes: 63 additions & 0 deletions docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1258,6 +1258,69 @@ Run `openshell sandbox list` on the host to check the underlying sandbox state.

## Deep Agents

### Deep Agents Config Lock Failure Recovery

A `CRITICAL` Deep Agents config-lock diagnostic can report `fail-closed containment=`, `rollback failed`, or that the lock rollback could not restore the trusted posture.
A containment result identifies one of two confirmed postures or an incomplete containment attempt.
A `rollback failed` result or lock-rollback diagnostic does not confirm containment.
Both rollback diagnostics mean NemoClaw could not restore or confirm the original trusted posture.

- **Config-root posture** (`fail-closed containment=config-root`) means NemoClaw installed fresh `0444 root:root` config and hash inodes.
NemoClaw also confirmed `0500 root:root` on `/sandbox/.deepagents` and `1775 root:sandbox` on `/sandbox`.
- **Sandbox-parent posture** (`fail-closed containment=sandbox-parent`) means NemoClaw confirmed `0700 root:root` on `/sandbox`.
NemoClaw uses this posture when it cannot confirm the complete config-root posture.
- `fail-closed containment=incomplete` means NemoClaw could not confirm either complete posture.

Preserve the complete `CRITICAL` diagnostic.
Do not retry `shields up`.
Do not run `chmod`, `chown`, or another repair inside the sandbox.
A confirmed containment posture removes the sandbox identity's access to the Deep Agents configuration.
An incomplete containment result, a `rollback failed` result, or a lock-rollback diagnostic does not establish a trustworthy boundary from which to accept the current bytes.
An ordinary `rebuild` cannot turn the current state into a trustworthy snapshot.

If you have a trusted host-side snapshot from before the failure, list the snapshots and record its selector:

```bash
$$nemoclaw <name> snapshot list
```

<Warning>
Destroying the sandbox permanently discards state newer than the selected snapshot.
Confirm that the trusted host-side snapshot exists before you destroy the sandbox.
</Warning>

Destroy the sandbox, re-onboard the same name from trusted host configuration, and restore the snapshot:

```bash
$$nemoclaw <name> destroy
$$nemoclaw onboard --name <name> --agent dcode
$$nemoclaw <name> snapshot restore <selector>
```

For snapshot contents and selector rules, refer to [Create and Restore Snapshots](../manage-sandboxes/state-and-backups/create-and-restore-snapshots).

If no trusted snapshot exists and you do not need to preserve the current state, recreate the sandbox from host-side onboarding configuration.

<Warning>
This recreation permanently discards the current sandbox state.
Continue only if you accept that loss.
</Warning>

```bash
$$nemoclaw <name> destroy
$$nemoclaw onboard --name <name> --agent dcode
```

After either recovery path, verify the recreated sandbox from the host:

```bash
$$nemoclaw <name> status
$$nemoclaw <name> shields status
```

Continue only when `status` identifies the expected Deep Agents sandbox and `shields status` returns without a `CRITICAL` or corrupt-state diagnostic.
Then retry the original `shields up` operation.

### `dcode status` reports a stale inference route

The managed `dcode` runtime reads provider and model settings from `/sandbox/.deepagents/config.toml`.
Expand Down
313 changes: 313 additions & 0 deletions nemoclaw/src/commands/migration-state-security.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,313 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import {
closeSync,
existsSync,
fstatSync,
mkdirSync,
mkdtempSync,
openSync,
readdirSync,
readFileSync,
rmSync,
statSync,
symlinkSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import type { PluginLogger } from "../index.js";
import * as credentialFilter from "../security/credential-filter.js";
import * as snapshotSanitizer from "../security/snapshot-sanitizer.js";
import * as snapshotBoundary from "../shared/snapshot-sanitizer-boundary.cjs";
import {
cleanupSnapshotBundle,
createSnapshotBundle,
type HostOpenClawState,
setConfigValue,
} from "./migration-state.js";

const roots: string[] = [];

function makeHome(): string {
const home = mkdtempSync(path.join(tmpdir(), "nemoclaw-migration-state-security-"));
roots.push(home);
return home;
}

function makeLogger(): PluginLogger {
return {
info: vi.fn(),
warn: vi.fn(),
error: vi.fn(),
debug: vi.fn(),
};
}

function makeHostState(homeDir: string, configPath: string): HostOpenClawState {
const stateDir = path.join(homeDir, ".openclaw");
return {
exists: true,
homeDir,
stateDir,
configDir: stateDir,
configPath,
workspaceDir: null,
extensionsDir: null,
skillsDir: null,
hooksDir: null,
externalRoots: [],
warnings: [],
errors: [],
hasExternalConfig: false,
};
}

function expectSnapshotBundle(
bundle: ReturnType<typeof createSnapshotBundle>,
): asserts bundle is NonNullable<ReturnType<typeof createSnapshotBundle>> {
expect(bundle).not.toBeNull();
}

function makeMinimalHostSnapshot(): {
home: string;
configPath: string;
logger: PluginLogger;
} {
const home = makeHome();
const stateDir = path.join(home, ".openclaw");
const configPath = path.join(stateDir, "openclaw.json");
mkdirSync(stateDir, { recursive: true });
writeFileSync(configPath, "{}");
return { home, configPath, logger: makeLogger() };
}

function expectSnapshotFailure(
home: string,
logger: PluginLogger,
bundle: ReturnType<typeof createSnapshotBundle>,
message: string,
): void {
expect(bundle).toBeNull();
expect(logger.error).toHaveBeenCalledWith(expect.stringContaining(message));
expect(readdirSync(path.join(home, ".nemoclaw", "staging"))).toEqual([]);
}

afterEach(() => {
vi.restoreAllMocks();
for (const root of roots.splice(0)) {
rmSync(root, { force: true, recursive: true });
}
});

describe("migration-state prepared config security", () => {
it("installs a mode-0600 config after scrubbing contextual secrets in memory", () => {
const home = makeHome();
const stateDir = path.join(home, ".openclaw");
const configPath = path.join(stateDir, "openclaw.json");
mkdirSync(stateDir, { recursive: true });
writeFileSync(
configPath,
JSON.stringify({
gateway: { auth: { token: "must-not-migrate" } },
metadata: {
environmentAssignment: "GITHUB_TOKEN=opaque-secret-value-123",
camelAssignment: "apiKey=opaque-secret-value-123",
model: "keep-me",
},
}),
);

const bundle = createSnapshotBundle(makeHostState(home, configPath), makeLogger(), {
persist: false,
});
expectSnapshotBundle(bundle);

const preparedConfigPath = path.join(bundle.preparedStateDir, "openclaw.json");
const preparedConfig = JSON.parse(readFileSync(preparedConfigPath, "utf-8")) as {
gateway?: unknown;
metadata: Record<string, string>;
};
expect(preparedConfig.gateway).toBeUndefined();
expect(preparedConfig.metadata).toEqual({
environmentAssignment: "[STRIPPED_BY_MIGRATION]",
camelAssignment: "[STRIPPED_BY_MIGRATION]",
model: "keep-me",
});
expect(statSync(preparedConfigPath).mode & 0o777).toBe(0o600);

cleanupSnapshotBundle(bundle);
});

it.runIf(process.platform !== "win32")(
"rejects an in-tree config symlink without touching its external target",
() => {
const home = makeHome();
const stateDir = path.join(home, ".openclaw");
const configPath = path.join(stateDir, "openclaw.json");
const externalConfigPath = path.join(home, "external-openclaw.json");
const original = JSON.stringify({ external: "must-remain" });
mkdirSync(stateDir, { recursive: true });
writeFileSync(externalConfigPath, original, { mode: 0o640 });
const externalConfigFd = openSync(externalConfigPath, "r");
try {
const originalMode = fstatSync(externalConfigFd).mode & 0o777;
symlinkSync(externalConfigPath, configPath);
const logger = makeLogger();

const bundle = createSnapshotBundle(makeHostState(home, configPath), logger, {
persist: false,
});

expect(bundle).toBeNull();
expect(logger.error).toHaveBeenCalled();
expect(readFileSync(externalConfigFd, "utf-8")).toBe(original);
expect(fstatSync(externalConfigFd).mode & 0o777).toBe(originalMode);
const stagingDir = path.join(home, ".nemoclaw", "staging");
expect(existsSync(stagingDir) ? readdirSync(stagingDir) : []).toEqual([]);
} finally {
closeSync(externalConfigFd);
}
},
);
});

describe("migration-state prepared config fail-closed boundaries", () => {
it("removes staging when the copied config parent cannot be inspected", () => {
const { home, configPath, logger } = makeMinimalHostSnapshot();
const inspect = vi
.spyOn(snapshotBoundary, "inspectDescriptorSnapshotRoot")
.mockReturnValue(null);

const bundle = createSnapshotBundle(makeHostState(home, configPath), logger, {
persist: false,
});

expectSnapshotFailure(home, logger, bundle, "Failed to inspect copied OpenClaw config parent");
expect(inspect).toHaveBeenCalledTimes(2);
});

it("removes staging when copied config bytes cannot be decoded", () => {
const { home, configPath, logger } = makeMinimalHostSnapshot();
const decodeDescriptorSnapshotContent = snapshotBoundary.decodeDescriptorSnapshotContent;
const decode = vi
.spyOn(snapshotBoundary, "decodeDescriptorSnapshotContent")
.mockImplementationOnce(decodeDescriptorSnapshotContent)
.mockReturnValue(null);

const bundle = createSnapshotBundle(makeHostState(home, configPath), logger, {
persist: false,
});

expectSnapshotFailure(
home,
logger,
bundle,
"Failed canonical decoding of copied OpenClaw config",
);
expect(decode).toHaveBeenCalledTimes(2);
});

it("removes staging when in-memory credential stripping returns a non-object", () => {
const { home, configPath, logger } = makeMinimalHostSnapshot();
const stripCredentials = credentialFilter.stripCredentials;
const strip = vi
.spyOn(credentialFilter, "stripCredentials")
.mockImplementationOnce(stripCredentials)
.mockReturnValue([]);

const bundle = createSnapshotBundle(makeHostState(home, configPath), logger, {
persist: false,
});

expectSnapshotFailure(
home,
logger,
bundle,
"Failed to sanitize prepared OpenClaw config in memory",
);
expect(strip).toHaveBeenCalledTimes(2);
});

it("removes staging when the prepared config cannot be installed", () => {
const { home, configPath, logger } = makeMinimalHostSnapshot();
const install = vi
.spyOn(snapshotBoundary, "installDescriptorSnapshotFile")
.mockReturnValue(false);

const bundle = createSnapshotBundle(makeHostState(home, configPath), logger, {
persist: false,
});

expectSnapshotFailure(
home,
logger,
bundle,
"Failed descriptor-bound installation of prepared OpenClaw config",
);
expect(install).toHaveBeenCalledOnce();
});

it("removes staging when the installed config cannot be sanitized", () => {
const { home, configPath, logger } = makeMinimalHostSnapshot();
const sanitize = vi
.spyOn(snapshotSanitizer, "sanitizeOpenClawConfigFile")
.mockReturnValue(false);

const bundle = createSnapshotBundle(makeHostState(home, configPath), logger, {
persist: false,
});

expectSnapshotFailure(home, logger, bundle, "Failed to sanitize prepared OpenClaw config");
expect(sanitize).toHaveBeenCalledOnce();
});
});

describe("migration-state config path security", () => {
const expectPrototypeClean = (): void => {
const probe: Record<string, unknown> = {};
for (const key of ["polluted", "isAdmin", "bar"]) {
expect(Object.prototype.hasOwnProperty.call(Object.prototype, key)).toBe(false);
expect(probe[key]).toBeUndefined();
}
};

it.each([
"__proto__",
"constructor",
"prototype",
])("rejects prototype-related config path segment: %s", (segment) => {
const doc: Record<string, unknown> = {};
expect(() => {
setConfigValue(doc, `${segment}.polluted`, "true");
}).toThrow(/Unsafe config path segment/);
expectPrototypeClean();
});

it("rejects __proto__ in nested position", () => {
const doc: Record<string, unknown> = {};
expect(() => {
setConfigValue(doc, "agents.__proto__.isAdmin", "true");
}).toThrow(/Unsafe config path segment/);
expectPrototypeClean();
});

it.each([
"foo.prototype.bar",
"foo.constructor.bar",
])("rejects prototype-related segment in nested config path: %s", (configPath) => {
const doc: Record<string, unknown> = {};
expect(() => {
setConfigValue(doc, configPath, "true");
}).toThrow(/Unsafe config path segment/);
expectPrototypeClean();
});

it("allows simple top-level keys", () => {
const doc: Record<string, unknown> = {};
setConfigValue(doc, "theme", "dark");
expect(doc.theme).toBe("dark");
});
});
Loading
Loading