Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions agents/hermes/manifest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,14 @@ state_files:
- path: .hermes_history
- path: runtime/state.db
strategy: sqlite_backup
# Hermes' backward-compatible default board lives in this SQLite database.
# Capture it with the online backup API like runtime/state.db; the zero-byte
# kanban.db.*.lock files are intentionally omitted. This does not cover named
# boards, attachments, worker logs, or scratch workspaces under kanban/, nor
# external dir/worktree workspace targets. Those need a separate durability
# design before the sibling kanban/ tree can safely enter the state contract.
- path: kanban.db
strategy: sqlite_backup
user_managed_files:
# Relative to /sandbox, not config.dir. Hermes stores user-edited API-key
# values in /sandbox/.hermes/.env, and rebuild should warn before dropping it.
Expand Down
9 changes: 6 additions & 3 deletions docs/manage-sandboxes/backup-restore.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,14 @@ Agent manifests can also declare durable top-level state files.
Treat snapshot directories as private local data.

<AgentOnly variant="hermes">
Hermes snapshots include `SOUL.md`, the Web Dashboard profile under `.hermes/dashboard-home/`, and the SQLite database behind `.hermes/state.db`.
NemoClaw uses SQLite's online backup API and restores the database through SQLite instead of copying a live raw database file.
Hermes snapshots include `SOUL.md`, the Web Dashboard profile under `.hermes/dashboard-home/`, the SQLite database behind `.hermes/state.db`, and the default kanban board in `.hermes/kanban.db`.
NemoClaw uses SQLite's online backup API and restores those databases through SQLite instead of copying live raw database files.

Kanban backup is limited to the backward-compatible default board in `kanban.db`.
Named boards, attachments, worker logs, scratch workspaces under `.hermes/kanban/`, and external directory or worktree targets are not included; back up that state separately.

The dashboard profile includes `MEMORY.md` and `USER.md`.
The Hermes database can contain session metadata and message history needed for a faithful restore.
The Hermes state database can contain session metadata and message history needed for a faithful restore.
</AgentOnly>
<AgentOnly variant="deepagents">
Deep Agents snapshots include manifest-declared state under `/sandbox/.deepagents`, including skills and runtime state, while omitting credential-bearing user files.
Expand Down
4 changes: 3 additions & 1 deletion docs/manage-sandboxes/workspace-files.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ Runtime state, such as logs, memory, platform sessions, and the SQLite state dat
| `/sandbox/.hermes/config.yaml` | NemoClaw-generated Hermes runtime configuration. |
| `/sandbox/.hermes/.env` | NemoClaw-generated environment and messaging placeholders. |
| `/sandbox/.hermes/state.db` | Hermes SQLite state database. |
| `/sandbox/.hermes/kanban.db` | Default Hermes kanban board database. NemoClaw snapshots preserve only this default board. |
| `/sandbox/.hermes/dashboard-home/` | Hermes Web Dashboard profile, including `MEMORY.md` and `USER.md`. |
| `/sandbox/.hermes/platforms/` | Messaging platform state, including QR-paired sessions such as WhatsApp. |
| `/sandbox/.hermes/logs/` | Hermes runtime logs. |
Expand All @@ -133,7 +134,8 @@ Runtime state, such as logs, memory, platform sessions, and the SQLite state dat

Hermes state lives in the sandbox's persistent state volume, not in the container image alone.
Normal restarts preserve that state.
Rebuilds and upgrades use NemoClaw's snapshot flow to preserve manifest-defined Hermes state, including `SOUL.md`, the Web Dashboard profile under `.hermes/dashboard-home/`, and the SQLite database behind `.hermes/state.db`.
Rebuilds and upgrades use NemoClaw's snapshot flow to preserve manifest-defined Hermes state, including `SOUL.md`, the Web Dashboard profile under `.hermes/dashboard-home/`, the SQLite database behind `.hermes/state.db`, and the default kanban board in `.hermes/kanban.db`.
Named boards, attachments, worker logs, scratch workspaces under `.hermes/kanban/`, and external directory or worktree targets are not included in the kanban backup.

Running `$$nemoclaw <name> destroy` deletes the sandbox and its persistent state volume.
Back up important state before destroying a Hermes sandbox.
Expand Down
3 changes: 2 additions & 1 deletion docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -599,7 +599,8 @@ For OpenClaw, the backed-up paths include agents, extensions, workspace, skills,
</AgentOnly>
<AgentOnly variant="hermes">

For Hermes, the backed-up paths come from `agents/hermes/manifest.yaml`, including `/sandbox/.hermes` state such as memories, sessions, skills, plugins, cron, logs, plans, workspace, messaging platform state, and `runtime/state.db`.
For Hermes, the backed-up paths come from `agents/hermes/manifest.yaml`, including `/sandbox/.hermes` state such as memories, sessions, skills, plugins, cron, logs, plans, workspace, messaging platform state, `runtime/state.db`, and the default kanban board in `kanban.db`.
Kanban backup does not include named boards, attachments, worker logs, scratch workspaces under `kanban/`, or external directory or worktree targets.

</AgentOnly>

Expand Down
53 changes: 53 additions & 0 deletions test/e2e/live/rebuild-hermes.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ SANDBOX_NAME.startsWith(TEST_SANDBOX_PREFIX) ||

const MARKER_FILE = "/sandbox/.hermes/memories/rebuild-marker.txt";
const MARKER_CONTENT = `REBUILD_HM_E2E_${Date.now()}`;
const KANBAN_TASK_TITLE = `NEMOCLAW_REBUILD_KANBAN_${Date.now()}`;
const EXCLUDED_KANBAN_FILE = "/sandbox/.hermes/kanban/excluded-rebuild-marker.txt";
const DISCORD_PLACEHOLDER = "openshell:resolve:env:DISCORD_BOT_TOKEN";
const DISCORD_FAKE_TOKEN = "test-fake-discord-token-rebuild-e2e";
const REGISTRY_FILE = path.join(os.homedir(), ".nemoclaw", "sandboxes.json");
Expand Down Expand Up @@ -736,6 +738,32 @@ test(STALE_BASE_REBUILD
);
expectExitZero(writeMarker, "write Hermes marker");

const seedKanban = await host.command(
"openshell",
[
"sandbox",
"exec",
"--name",
SANDBOX_NAME,
"--",
"sh",
"-lc",
[
"hermes kanban init",
`hermes kanban create ${shellQuote(KANBAN_TASK_TITLE)} --initial-status blocked --json`,
`mkdir -p ${shellQuote(path.dirname(EXCLUDED_KANBAN_FILE))}`,
`printf '%s' ${shellQuote(MARKER_CONTENT)} > ${shellQuote(EXCLUDED_KANBAN_FILE)}`,
].join(" && "),
],
{
artifactName: "phase-4-seed-hermes-kanban",
env: testEnv(apiKey),
redactionValues,
timeoutMs: OPENSHELL_TIMEOUT_MS,
},
);
expectExitZero(seedKanban, "seed Hermes default kanban board");

const preEnv = await host.command(
"openshell",
["sandbox", "exec", "--name", SANDBOX_NAME, "--", "cat", "/sandbox/.hermes/.env"],
Expand Down Expand Up @@ -878,6 +906,31 @@ test(STALE_BASE_REBUILD
`Hermes version output did not include expected release ${expectedVersion}: ${hermesVersionText}`,
);

const restoredKanban = await host.command(
"openshell",
["sandbox", "exec", "--name", SANDBOX_NAME, "--", "hermes", "kanban", "list", "--json"],
{
artifactName: "phase-7-list-kanban-after-rebuild",
env: testEnv(apiKey),
redactionValues,
timeoutMs: OPENSHELL_TIMEOUT_MS,
},
);
expectExitZero(restoredKanban, "list Hermes kanban tasks after rebuild");
expect(resultText(restoredKanban)).toContain(KANBAN_TASK_TITLE);

const excludedKanbanState = await host.command(
"openshell",
["sandbox", "exec", "--name", SANDBOX_NAME, "--", "test", "!", "-e", EXCLUDED_KANBAN_FILE],
{
artifactName: "phase-7-verify-excluded-kanban-state",
env: testEnv(apiKey),
redactionValues,
timeoutMs: OPENSHELL_TIMEOUT_MS,
},
);
expectExitZero(excludedKanbanState, "verify excluded Hermes kanban state was not restored");

const restoredEnv = await host.command(
"openshell",
["sandbox", "exec", "--name", SANDBOX_NAME, "--", "cat", "/sandbox/.hermes/.env"],
Expand Down
183 changes: 183 additions & 0 deletions test/hermes-kanban-snapshot.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { afterAll, expect, it } from "vitest";

// sandbox-state captures HOME when the module loads, so isolate its registry
// and rebuild backups before importing it.
const ORIGINAL_HOME = process.env.HOME;
const TMP_HOME = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-kanban-snapshot-"));
process.env.HOME = TMP_HOME;
const sandboxState = await import(
pathToFileURL(path.join(import.meta.dirname, "..", "src", "lib", "state", "sandbox.ts")).href
);

afterAll(() => {
ORIGINAL_HOME === undefined ? delete process.env.HOME : (process.env.HOME = ORIGINAL_HOME);
fs.rmSync(TMP_HOME, { recursive: true, force: true });
});

function writeExecutable(filePath: string, source: string): void {
fs.writeFileSync(filePath, source, { mode: 0o755 });
}

function writeHermesRegistry(): void {
fs.mkdirSync(path.join(TMP_HOME, ".nemoclaw"), { recursive: true });
fs.writeFileSync(
path.join(TMP_HOME, ".nemoclaw", "sandboxes.json"),
JSON.stringify({
defaultSandbox: "hermes",
sandboxes: {
hermes: {
name: "hermes",
model: "m",
provider: "p",
gpuEnabled: false,
policies: [],
agent: "hermes",
},
},
}),
);
}

it("preserves only the Hermes default-board database across rebuilds (#7095)", () => {
const fixture = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-kanban-state-"));
const oldPath = process.env.PATH;
const oldOpenshell = process.env.NEMOCLAW_OPENSHELL_BIN;
try {
const binDir = path.join(fixture, "bin");
const hermesDir = path.join(fixture, "sandbox-root", ".hermes");
const scratchFile = path.join(hermesDir, "kanban", "workspaces", "scratch", "work.txt");
const namedBoardDb = path.join(hermesDir, "kanban", "boards", "release-board", "kanban.db");
const attachmentFile = path.join(hermesDir, "kanban", "attachments", "t_1", "design.txt");
const workerLog = path.join(hermesDir, "kanban", "logs", "t_1.log");
const externalDirFile = path.join(fixture, "external-dir-workspace", "work.txt");
const externalWorktreeFile = path.join(fixture, "external-worktree", "work.txt");
const sshLog = path.join(fixture, "ssh-log.jsonl");
fs.mkdirSync(binDir, { recursive: true });
fs.mkdirSync(path.dirname(scratchFile), { recursive: true });
fs.mkdirSync(path.dirname(namedBoardDb), { recursive: true });
fs.mkdirSync(path.dirname(attachmentFile), { recursive: true });
fs.mkdirSync(path.dirname(workerLog), { recursive: true });
fs.mkdirSync(path.dirname(externalDirFile), { recursive: true });
fs.mkdirSync(path.dirname(externalWorktreeFile), { recursive: true });
fs.writeFileSync(path.join(hermesDir, "kanban.db"), "original kanban database\n");
fs.writeFileSync(scratchFile, "old scratch workspace\n");
fs.writeFileSync(namedBoardDb, "old named-board database\n");
fs.writeFileSync(attachmentFile, "old attachment\n");
fs.writeFileSync(workerLog, "old worker log\n");
fs.writeFileSync(externalDirFile, "old external dir workspace\n");
fs.writeFileSync(externalWorktreeFile, "old external worktree\n");

const openshell = path.join(binDir, "openshell");
writeExecutable(
openshell,
`#!/usr/bin/env node
const args = process.argv.slice(2);
if (args[0] === "sandbox" && args[1] === "ssh-config") {
process.stdout.write("Host openshell-hermes\\n HostName 127.0.0.1\\n User sandbox\\n");
process.exit(0);
}
process.exit(0);
`,
);

writeExecutable(
path.join(binDir, "ssh"),
`#!/usr/bin/env node
const fs = require("node:fs");
const path = require("node:path");
const cmd = process.argv[process.argv.length - 1] || "";
const hermesDir = ${JSON.stringify(hermesDir)};
fs.appendFileSync(${JSON.stringify(sshLog)}, JSON.stringify({ cmd }) + "\\n");
function readStdin() {
const chunks = [];
for (;;) {
const buffer = Buffer.alloc(65536);
const count = fs.readSync(0, buffer, 0, buffer.length, null);
if (count === 0) break;
chunks.push(buffer.subarray(0, count));
}
return Buffer.concat(chunks);
}
if (cmd.includes("[ -d ")) {
process.exit(0);
}
if (cmd.includes("nemoclaw-sqlite-backup")) {
if (!cmd.includes("kanban.db")) process.exit(2);
process.stdout.write(fs.readFileSync(path.join(hermesDir, "kanban.db")));
process.exit(0);
}
if (cmd.includes("SOUL.md") || cmd.includes(".hermes_history")) {
process.exit(2);
}
if (cmd.includes("nemoclaw-sqlite-restore")) {
fs.writeFileSync(path.join(hermesDir, "kanban.db"), readStdin());
process.exit(0);
}
process.exit(0);
`,
);

writeHermesRegistry();
process.env.NEMOCLAW_OPENSHELL_BIN = openshell;
process.env.PATH = `${binDir}${path.delimiter}${oldPath || ""}`;

const backup = sandboxState.backupSandboxState("hermes", { name: "kanban-state" });
expect(backup.success).toBe(true);
expect(backup.backedUpFiles).toEqual(["kanban.db"]);
expect(backup.failedFiles).toEqual([]);
expect(backup.backedUpDirs).not.toContain("kanban");
expect(backup.manifest?.stateDirs).not.toContain("kanban");
expect(backup.manifest?.stateFiles).toContainEqual({
path: "kanban.db",
strategy: "sqlite_backup",
});
expect(fs.readFileSync(path.join(backup.manifest!.backupPath, "kanban.db"), "utf-8")).toBe(
"original kanban database\n",
);
expect(fs.existsSync(path.join(backup.manifest!.backupPath, "kanban"))).toBe(false);

fs.writeFileSync(path.join(hermesDir, "kanban.db"), "changed kanban database\n");
fs.writeFileSync(scratchFile, "fresh scratch workspace\n");
fs.writeFileSync(namedBoardDb, "fresh named-board database\n");
fs.writeFileSync(attachmentFile, "fresh attachment\n");
fs.writeFileSync(workerLog, "fresh worker log\n");
fs.writeFileSync(externalDirFile, "fresh external dir workspace\n");
fs.writeFileSync(externalWorktreeFile, "fresh external worktree\n");

const restore = sandboxState.restoreSandboxState("hermes", backup.manifest!.backupPath);
expect(restore.success).toBe(true);
expect(restore.restoredFiles).toEqual(["kanban.db"]);
expect(restore.restoredDirs).toEqual([]);
expect(fs.readFileSync(path.join(hermesDir, "kanban.db"), "utf-8")).toBe(
"original kanban database\n",
);
expect(fs.readFileSync(scratchFile, "utf-8")).toBe("fresh scratch workspace\n");
expect(fs.readFileSync(namedBoardDb, "utf-8")).toBe("fresh named-board database\n");
expect(fs.readFileSync(attachmentFile, "utf-8")).toBe("fresh attachment\n");
expect(fs.readFileSync(workerLog, "utf-8")).toBe("fresh worker log\n");
expect(fs.readFileSync(externalDirFile, "utf-8")).toBe("fresh external dir workspace\n");
expect(fs.readFileSync(externalWorktreeFile, "utf-8")).toBe("fresh external worktree\n");

const loggedCommands = fs.readFileSync(sshLog, "utf-8");
expect(loggedCommands).toContain("sqlite3.connect");
expect(loggedCommands).not.toContain("tar -cf -");
expect(loggedCommands).not.toContain("kanban/boards/release-board");
expect(loggedCommands).not.toContain("kanban/attachments");
expect(loggedCommands).not.toContain("kanban/logs");
expect(loggedCommands).not.toContain(externalDirFile);
expect(loggedCommands).not.toContain(externalWorktreeFile);
} finally {
oldOpenshell === undefined
? delete process.env.NEMOCLAW_OPENSHELL_BIN
: (process.env.NEMOCLAW_OPENSHELL_BIN = oldOpenshell);
oldPath === undefined ? delete process.env.PATH : (process.env.PATH = oldPath);
fs.rmSync(fixture, { recursive: true, force: true });
}
});
6 changes: 3 additions & 3 deletions test/snapshot.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1323,9 +1323,7 @@ process.exit(0);
expect(loggedCommands).not.toContain(".env");
expect(loggedCommands).not.toContain(".mcp.json");
expect(loggedCommands).not.toContain(".nemoclaw-mcp.json");

// #5753 is "lost after rebuild" (backup + recreate + restore): restore
// must list agent/skills among the dirs it brings back into the sandbox.
// #5753: restore must include agent/skills after backup and recreation.
const restore = sandboxState.restoreSandboxState("deepagents", backup.manifest!.backupPath);
expect(restore.success).toBe(true);
expect(restore.restoredDirs).toEqual(
Expand Down Expand Up @@ -1398,6 +1396,7 @@ if (cmd.includes("[ -d ")) {
process.exit(0);
}
if (cmd.includes("nemoclaw-sqlite-backup")) {
if (cmd.includes("kanban.db")) process.exit(2);
process.stdout.write(fs.readFileSync(path.join(hermesDir, "runtime", "state.db")));
process.exit(0);
}
Expand Down Expand Up @@ -1455,6 +1454,7 @@ process.exit(0);
{ path: "SOUL.md", strategy: "copy" },
{ path: ".hermes_history", strategy: "copy" },
{ path: "runtime/state.db", strategy: "sqlite_backup" },
{ path: "kanban.db", strategy: "sqlite_backup" },
]);
expect(fs.readFileSync(path.join(backup.manifest!.backupPath, "SOUL.md"), "utf-8")).toBe(
"original soul\n",
Expand Down
Loading