Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/manage-sandboxes/backup-restore.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -258,17 +258,19 @@ Restore safe Deep Agents files only, then run the managed rebuild or onboarding

</AgentOnly>

## Back Up All Running Sandboxes
## Back Up Registered Sandboxes

To back up every registered, running sandbox in one step, run `$$nemoclaw backup-all`.
To back up every registered sandbox that is running, or whose stopped Docker-driver container can be started temporarily, run `$$nemoclaw backup-all`.
Use this host-installed command before broad maintenance such as `$$nemoclaw update`, `$$nemoclaw upgrade-sandboxes`, or an OpenShell gateway migration.

```bash
$$nemoclaw backup-all
```

`backup-all` walks the sandboxes registered on the host, creates a snapshot for each running sandbox, and stores the snapshot bundles under `~/.nemoclaw/rebuild-backups/<name>/`.
If a registered sandbox is stopped, start the sandbox or its container and rerun `$$nemoclaw backup-all` so NemoClaw can capture a fresh snapshot before maintenance.
If a registered docker-driver sandbox's container is stopped, `backup-all` starts the container for the duration of the backup and returns it to its stopped state afterward.
If the container cannot be returned to the stopped state, the backup run fails and reports that the container was left running.
If a sandbox is not running and its container cannot be started this way, start the sandbox or its container and rerun `$$nemoclaw backup-all` so NemoClaw can capture a fresh snapshot before maintenance.
Comment thread
prekshivyas marked this conversation as resolved.
Resolved
When a backup fails, NemoClaw identifies the affected state item and reports one of these causes when available: `permission denied`, `tar read error`, or `absent after extraction`.
Use `$$nemoclaw <name> snapshot list` and `$$nemoclaw <name> snapshot restore` to inspect or restore one sandbox's bundles later.

Expand Down
6 changes: 4 additions & 2 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2506,15 +2506,17 @@ A custom OpenClaw sandbox is recoverable only when the selected backup independe
### `$$nemoclaw backup-all`

Back up all registered running sandboxes to `~/.nemoclaw/rebuild-backups/`.
Sandboxes that are not running are skipped with remediation guidance.
A registered docker-driver sandbox whose container is stopped is started for the duration of the backup and returned to its stopped state afterward.
If the container cannot be returned to the stopped state, the command fails and reports that the container was left running.
Sandboxes that are not running and cannot be started this way are skipped with remediation guidance.

```bash
$$nemoclaw backup-all
```

Before an OpenShell upgrade, the installer prepares the current release CLI and uses it to run `backup-all` in strict mode.
Strict mode requires every registered sandbox to produce a fresh backup and aborts before gateway changes if any sandbox is skipped or fails.
When strict mode reports a skipped stopped sandbox, start that sandbox or its container and rerun the installer or `$$nemoclaw backup-all`.
When strict mode reports a skipped sandbox, start that sandbox or its container and rerun the installer or `$$nemoclaw backup-all`.

A running sandbox whose in-sandbox SSH endpoint does not answer fails its backup and aborts the run.
For a standalone `$$nemoclaw backup-all` run, set `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1` exactly to skip such sandboxes instead of failing.
Expand Down
164 changes: 164 additions & 0 deletions src/lib/actions/maintenance.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ const mocks = vi.hoisted(() => ({
dockerListImagesFormat: vi.fn().mockReturnValue(""),
dockerRmi: vi.fn(),
prompt: vi.fn(),
startStoppedSandboxContainerForBackup: vi.fn(),
backupStartedSandboxState: vi.fn(),
returnSandboxContainerToStopped: vi.fn(),
}));

vi.mock("../state/registry", () => ({
Expand All @@ -36,6 +39,11 @@ vi.mock("../cli/branding", () => ({
vi.mock("../credentials/store", () => ({
prompt: mocks.prompt,
}));
vi.mock("./sandbox/stopped-sandbox-backup", () => ({
startStoppedSandboxContainerForBackup: mocks.startStoppedSandboxContainerForBackup,
backupStartedSandboxState: mocks.backupStartedSandboxState,
returnSandboxContainerToStopped: mocks.returnSandboxContainerToStopped,
}));
vi.mock("../domain/lifecycle/options", () => ({
normalizeGarbageCollectImagesOptions: (o: unknown) => o || {},
}));
Expand All @@ -48,12 +56,15 @@ import { backupAll, garbageCollectImages, shouldSkipUnreachableSandboxBackup } f
describe("backupAll", () => {
beforeEach(() => {
vi.clearAllMocks();
mocks.backupStartedSandboxState.mockReset();
delete process.env.NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS;
mocks.captureSandboxListWithGatewayPreflightOrExit.mockResolvedValue({
status: 0,
output: "sb-good\nsb-bad\n",
});
mocks.parseReadySandboxNames.mockReturnValue(new Set(["sb-good", "sb-bad"]));
mocks.startStoppedSandboxContainerForBackup.mockReturnValue(null);
mocks.returnSandboxContainerToStopped.mockReturnValue(true);
});

afterEach(() => {
Expand Down Expand Up @@ -191,6 +202,159 @@ describe("backupAll", () => {
expect(errorOutput).not.toContain("prepare the upgrade manually");
});

it("starts a stopped container, backs it up, and returns it to stopped so strict mode passes (#6500)", async () => {
mocks.listSandboxes.mockReturnValue({
sandboxes: [{ name: "sb-good" }, { name: "sb-stopped" }],
defaultSandbox: null,
});
mocks.parseReadySandboxNames.mockReturnValue(new Set(["sb-good"]));
mocks.backupSandboxState.mockReturnValue({
success: true,
backedUpDirs: ["workspace"],
failedDirs: [],
backedUpFiles: [],
failedFiles: [],
manifest: { backupPath: "/backups/sb-good/timestamp" },
});
mocks.startStoppedSandboxContainerForBackup.mockImplementation((name: string) =>
name === "sb-stopped" ? { containerName: "openshell-sb-stopped-abc" } : null,
);
mocks.backupStartedSandboxState.mockResolvedValue({
success: true,
backedUpDirs: ["workspace"],
failedDirs: [],
backedUpFiles: [],
failedFiles: [],
manifest: { backupPath: "/backups/sb-stopped/timestamp" },
});
process.env.NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS = "1";
const logSpy = vi.spyOn(console, "log").mockImplementation(() => undefined);
const exitSpy = vi.spyOn(process, "exit").mockImplementation(((code?: number) => {
throw new Error(`exit:${code}`);
}) as never);

await backupAll();

expect(exitSpy).not.toHaveBeenCalled();
expect(mocks.backupStartedSandboxState).toHaveBeenCalledWith("sb-stopped");
expect(mocks.backupSandboxState).toHaveBeenCalledWith("sb-good");
expect(mocks.returnSandboxContainerToStopped).toHaveBeenCalledWith("openshell-sb-stopped-abc");
const logOutput = logSpy.mock.calls.flat().join("\n");
expect(logOutput).toContain("Starting stopped sandbox 'sb-stopped' to back it up");
expect(logOutput).toContain("Returned 'sb-stopped' to its stopped state");
expect(logOutput).toContain("2 backed up, 0 failed, 0 skipped");
expect(logOutput).not.toContain("Skipping 'sb-stopped'");
});

it("returns the container to stopped and counts a failure when the started backup fails (#6500)", async () => {
mocks.listSandboxes.mockReturnValue({
sandboxes: [{ name: "sb-stopped" }],
defaultSandbox: null,
});
mocks.parseReadySandboxNames.mockReturnValue(new Set());
mocks.startStoppedSandboxContainerForBackup.mockReturnValue({
containerName: "openshell-sb-stopped-abc",
});
mocks.backupStartedSandboxState.mockResolvedValue({
success: false,
backedUpDirs: [],
failedDirs: ["identity"],
failedDirReasons: { identity: "permission denied" },
backedUpFiles: [],
failedFiles: [],
});
process.env.NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS = "1";
const logSpy = vi.spyOn(console, "log").mockImplementation(() => undefined);
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => undefined);
vi.spyOn(process, "exit").mockImplementation(((code?: number) => {
throw new Error(`exit:${code}`);
}) as never);

await expect(backupAll()).rejects.toThrow("exit:1");

expect(mocks.returnSandboxContainerToStopped).toHaveBeenCalledWith("openshell-sb-stopped-abc");
expect(logSpy.mock.calls.flat().join("\n")).toContain("0 backed up, 1 failed, 0 skipped");
expect(errorSpy.mock.calls.flat().join("\n")).toContain(
"backup failed (identity (permission denied))",
);
});

it("fails when the started container cannot be returned to its stopped state (#6500)", async () => {
mocks.listSandboxes.mockReturnValue({
sandboxes: [{ name: "sb-stopped" }],
defaultSandbox: null,
});
mocks.parseReadySandboxNames.mockReturnValue(new Set());
mocks.startStoppedSandboxContainerForBackup.mockReturnValue({
containerName: "openshell-sb-stopped-abc",
});
mocks.backupStartedSandboxState.mockResolvedValue({
success: true,
backedUpDirs: ["workspace"],
failedDirs: [],
backedUpFiles: [],
failedFiles: [],
manifest: { backupPath: "/backups/sb-stopped/timestamp" },
});
mocks.returnSandboxContainerToStopped.mockReturnValue(false);
process.env.NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS = "1";
const logSpy = vi.spyOn(console, "log").mockImplementation(() => undefined);
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => undefined);
vi.spyOn(process, "exit").mockImplementation(((code?: number) => {
throw new Error(`exit:${code}`);
}) as never);

await expect(backupAll()).rejects.toThrow("exit:1");

expect(logSpy.mock.calls.flat().join("\n")).toContain("0 backed up, 1 failed, 0 skipped");
expect(errorSpy.mock.calls.flat().join("\n")).toContain(
"backup cleanup failed (could not return its container to the stopped state",
);
});

it("returns a started container to stopped when an orphan manifest skips backup (#6500)", async () => {
mocks.listSandboxes.mockReturnValue({
sandboxes: [{ name: "sb-stopped" }],
defaultSandbox: null,
});
mocks.parseReadySandboxNames.mockReturnValue(new Set());
mocks.startStoppedSandboxContainerForBackup.mockReturnValue({
containerName: "openshell-sb-stopped-abc",
});
mocks.backupStartedSandboxState.mockRejectedValue(
new Error("Agent 'sb-stopped' not found: /path/to/manifest.yaml"),
);
const logSpy = vi.spyOn(console, "log").mockImplementation(() => undefined);

await backupAll();

expect(mocks.returnSandboxContainerToStopped).toHaveBeenCalledWith("openshell-sb-stopped-abc");
const output = logSpy.mock.calls.flat().join("\n");
expect(output).toContain("Returned 'sb-stopped' to its stopped state");
expect(output).toContain("Skipped 'sb-stopped' (orphan manifest)");
});

it("keeps the not-running skip when no stopped container can be started (#6114)", async () => {
mocks.listSandboxes.mockReturnValue({
sandboxes: [{ name: "sb-stopped" }],
defaultSandbox: null,
});
mocks.parseReadySandboxNames.mockReturnValue(new Set());
mocks.startStoppedSandboxContainerForBackup.mockReturnValue(null);
process.env.NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS = "1";
const logSpy = vi.spyOn(console, "log").mockImplementation(() => undefined);
vi.spyOn(console, "error").mockImplementation(() => undefined);
vi.spyOn(process, "exit").mockImplementation(((code?: number) => {
throw new Error(`exit:${code}`);
}) as never);

await expect(backupAll()).rejects.toThrow("exit:1");

expect(mocks.backupStartedSandboxState).not.toHaveBeenCalled();
expect(mocks.returnSandboxContainerToStopped).not.toHaveBeenCalled();
expect(logSpy.mock.calls.flat().join("\n")).toContain("Skipping 'sb-stopped' (not running");
});

it("continues backup loop when backupSandboxState throws for one sandbox", async () => {
mocks.listSandboxes.mockReturnValue({
sandboxes: [{ name: "sb-bad" }, { name: "sb-good" }],
Expand Down
52 changes: 45 additions & 7 deletions src/lib/actions/maintenance.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ import { captureSandboxListWithGatewayPreflightOrExit } from "../openshell-sandb
import { parseReadySandboxNames } from "../runtime-recovery";
import * as registry from "../state/registry";
import * as sandboxState from "../state/sandbox";
import {
backupStartedSandboxState,
returnSandboxContainerToStopped,
type StartedForBackup,
startStoppedSandboxContainerForBackup,
} from "./sandbox/stopped-sandbox-backup";

const useColor = !process.env.NO_COLOR && !!process.stdout.isTTY;
const trueColor =
Expand Down Expand Up @@ -54,16 +60,29 @@ export async function backupAll(): Promise<void> {
let unreachableRunning = 0;
let notRunningSkipped = 0;
for (const sb of sandboxes) {
// A registered docker-driver sandbox whose container is merely stopped is
// backupable: start it for the duration of the backup and return it to
// its stopped state after (#6500). Anything else that is not Ready keeps
// the existing skip (and, under installer-strict mode, the #6114 gate).
let startedForBackup: StartedForBackup | null = null;
if (!readyNames.has(sb.name)) {
console.log(` ${D}${notRunningBackupSkipMessage(sb.name)}${R}`);
skipped++;
notRunningSkipped++;
continue;
startedForBackup = startStoppedSandboxContainerForBackup(sb.name);
if (!startedForBackup) {
console.log(` ${D}${notRunningBackupSkipMessage(sb.name)}${R}`);
skipped++;
notRunningSkipped++;
continue;
}
console.log(` Starting stopped sandbox '${sb.name}' to back it up...`);
}
console.log(` Backing up '${sb.name}'...`);
let result: sandboxState.BackupResult;
let result: sandboxState.BackupResult | null = null;
let orphanManifestMessage: string | null = null;
let returnedToStopped = true;
try {
result = sandboxState.backupSandboxState(sb.name);
result = startedForBackup
? await backupStartedSandboxState(sb.name)
: sandboxState.backupSandboxState(sb.name);
} catch (err: unknown) {
// Source-of-truth review (#5734 / #5819):
//
Expand Down Expand Up @@ -100,10 +119,29 @@ export async function backupAll(): Promise<void> {
if (!/^Agent '[^']+' not found: .+\/manifest\.yaml$/.test(msg)) {
throw err;
}
console.log(` ${YW}⚠${R} Skipped '${sb.name}' (orphan manifest): ${msg}`);
orphanManifestMessage = msg;
} finally {
if (startedForBackup) {
if (returnSandboxContainerToStopped(startedForBackup.containerName)) {
console.log(` ${D}Returned '${sb.name}' to its stopped state.${R}`);
} else {
returnedToStopped = false;
console.error(
` ${RD}✗${R} ${sb.name}: backup cleanup failed (could not return its container to the stopped state; the container was left running)`,
);
}
}
}
if (!returnedToStopped) {
failed++;
continue;
}
if (orphanManifestMessage) {
console.log(` ${YW}⚠${R} Skipped '${sb.name}' (orphan manifest): ${orphanManifestMessage}`);
skipped++;
continue;
}
if (!result) throw new Error(`Backup for '${sb.name}' completed without a result`);
if (result.success) {
console.log(
` ${G}✓${R} ${sb.name}: ${result.backedUpDirs.length} dirs, ${result.backedUpFiles.length} files → ${result.manifest?.backupPath || "unknown"}`,
Expand Down
Loading
Loading