Skip to content

fix(backup): include projects.db and kanban.db in pre-update snapshot - #52930

Closed
0xDevNinja wants to merge 1 commit into
NousResearch:mainfrom
0xDevNinja:fix/52889-snapshot-projects-kanban-db
Closed

fix(backup): include projects.db and kanban.db in pre-update snapshot#52930
0xDevNinja wants to merge 1 commit into
NousResearch:mainfrom
0xDevNinja:fix/52889-snapshot-projects-kanban-db

Conversation

@0xDevNinja

@0xDevNinja 0xDevNinja commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

The pre-update quick snapshot (_QUICK_STATE_FILES in hermes_cli/backup.py) preserves state.db, config.yaml, auth.json and the pairing stores before hermes update pulls, but it never listed projects.db or kanban.db. Both are per-profile, user-created SQLite stores that live outside the git checkout.

When a desktop upgrade restarts the backend and projects_db.connect() (or the kanban store) runs CREATE TABLE IF NOT EXISTS against a missing or zeroed file, every project, folder mapping, the active-project pointer and all board rows are silently re-created empty — with no backup and no recovery path. Every upgrade destroys all projects.

This adds both files to _QUICK_STATE_FILES so they are snapshotted alongside state.db and restored automatically if anything goes wrong, exactly as the issue suggests.

Related Issue

Fixes #52889

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • hermes_cli/backup.py: add "projects.db" and "kanban.db" to _QUICK_STATE_FILES. The existing .db branch already routes them through _safe_copy_db (WAL-safe copy), so no other code path changes.
  • tests/hermes_cli/test_backup.py: add TestQuickSnapshotProjectsKanban — asserts both files are in the manifest set, both get snapshotted (with row-level verification for projects.db), and an emptied projects.db is restored from the snapshot.

Implementation note

The snapshot list is resolved relative to HERMES_HOME (src = home / rel). projects.db always resolves to $HERMES_HOME/projects.db (per projects_db.projects_db_path), so it is covered for every profile. kanban.db sits at the shared root, which equals HERMES_HOME for the default/root profile; on non-root profiles its real path is outside HERMES_HOME and the entry is silently skipped — the same best-effort behavior the pairing entries already rely on. A fully profile-aware kanban path (resolving via kanban_db.kanban_db_path) would need separate handling outside the relative-to-home model and is left as a follow-up; this PR fixes the reported data-loss for the default install where the bug bites.

How to Test

  1. uv run python -m pytest tests/hermes_cli/test_backup.py -q → 141 passed.
  2. Manual: create a project (hermes project create), run a pre-update snapshot, empty projects.db, restore the snapshot — the project rows come back.

Checklist

  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix
  • I've run the suite and all tests pass (tests/hermes_cli/test_backup.py, 141 passed, ruff clean)
  • I've added tests for my changes
  • I've tested on my platform: macOS 14
  • Cross-platform impact considered — snapshot paths are HERMES_HOME-relative; behavior identical on Windows/macOS/Linux

@0xDevNinja
fix(backup): include projects.db and kanban.db in pre-update snapshot
f85b2e5 
The pre-update quick snapshot (_QUICK_STATE_FILES) backs up state.db,
config.yaml, auth.json and the pairing stores, but not projects.db or
kanban.db. Both are per-profile, user-created SQLite stores that live
outside the git checkout. When a desktop upgrade restarts the backend and
projects_db.connect()/kanban runs CREATE TABLE IF NOT EXISTS against a
missing or zeroed file, every project, folder mapping, active-project
pointer and board row is silently destroyed with no recovery path.

Add both to _QUICK_STATE_FILES so they are preserved alongside state.db.
projects.db always resolves under $HERMES_HOME; kanban.db sits at the
shared root for the default profile (snapshot is HERMES_HOME-relative, so
non-root profiles skip it best-effort, same as the pairing entries).

Fixes NousResearch#52889
@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard P1 High — major feature broken, no workaround labels Jun 26, 2026
@Bartok9

Bartok9 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Verified the premise against current origin/main (099df3cd8): _QUICK_STATE_FILES at hermes_cli/backup.py:755 lists state.db/config.yaml/auth.json/cron/jobs.json/pairing stores but neither projects.db nor kanban.db — both are per-profile user-created SQLite stores that live outside the git checkout (hermes_cli/projects_db.py:14 resolves $HERMES_HOME/projects.db; hermes_cli/kanban_db.py is the kanban store), so projects_db.connect()'s CREATE TABLE IF NOT EXISTS re-seeds an empty DB on restart and the pre-update snapshot has no copy to restore. This PR's two-line addition is exactly right and mirrors the existing pattern (same class of state as the cron/jobs.json entry already in the list). LGTM.

@kshitijk4poor kshitijk4poor mentioned this pull request Jun 26, 2026
Merged
kshitijk4poor pushed a commit that referenced this pull request Jun 26, 2026
@0xDevNinja @kshitijk4poor
fix(backup): include projects.db, kanban boards, and sibling stores i…
9ef49cd 
…n pre-update snapshot (#52889)

projects.db (per-profile project store) and kanban.db were missing from
_QUICK_STATE_FILES, so the pre-update quick snapshot never backed them up.
On a desktop upgrade, when the update flow removes/replaces the file and the
post-update schema-init re-creates an empty one, all user-created projects,
folder mappings, the active-project pointer, kanban board bindings, and tasks
vanish silently — no error.

Add the per-profile user-created stores to the snapshot set:
- projects.db               — project store
- response_store.db         — gateway conversation history / tool payloads (WAL)
- memory_store.db           — holographic memory facts/entities (WAL)
- verification_evidence.db  — agent verification audit trail
- kanban.db                 — default board (back-compat <root>/kanban.db)
- kanban/boards             — non-default boards (<root>/kanban/boards/<slug>/kanban.db
                              + metadata); workspaces/ and attachments/ subtrees
                              are skipped as large + regenerable.

Also: the directory-branch of create_quick_snapshot now routes *.db through the
WAL-safe _safe_copy_db (SQLite backup() API), matching the top-level file path —
previously a non-default board DB with an open WAL could be copied inconsistently.

Salvaged from #52930 by @0xDevNinja (authorship preserved via cherry-pick).
On top of the original (which covered only projects.db + the default kanban.db),
this adds: non-default-board coverage, the three sibling per-profile DBs that
meet the same upgrade-wipe criteria, WAL-safe directory copies, and a
workspaces/attachments skip to avoid snapshot bloat (×20 retained). 8 tests,
all mutation-verified; E2E verified snapshot→wipe→restore preserves all six
store types on the real code path.

Closes #52889. Supersedes #52930.
@kshitijk4poor

kshitijk4poor commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Closing in favor of #52990, which salvages this fix onto current main — your authorship is preserved on the commit (cherry-picked). #52990 carries your full change (projects.db + kanban.db in _QUICK_STATE_FILES + your round-trip tests) and extends it: non-default kanban boards (kanban/boards/<slug>/kanban.db, which your kanban.db-only entry would have missed), three sibling per-profile DBs in the same upgrade-wipe class (response_store.db, memory_store.db, verification_evidence.db), WAL-safe copies for the directory branch, and a workspaces/attachments skip to avoid snapshot bloat. Thanks @0xDevNinja — clean, well-tested fix that nailed the root cause.

pai-scaffolde pushed a commit to Scaffolde/hermes-agent that referenced this pull request Jun 28, 2026
@0xDevNinja @pai-scaffolde
fix(backup): include projects.db, kanban boards, and sibling stores i…
ade858c 
…n pre-update snapshot (NousResearch#52889)

projects.db (per-profile project store) and kanban.db were missing from
_QUICK_STATE_FILES, so the pre-update quick snapshot never backed them up.
On a desktop upgrade, when the update flow removes/replaces the file and the
post-update schema-init re-creates an empty one, all user-created projects,
folder mappings, the active-project pointer, kanban board bindings, and tasks
vanish silently — no error.

Add the per-profile user-created stores to the snapshot set:
- projects.db               — project store
- response_store.db         — gateway conversation history / tool payloads (WAL)
- memory_store.db           — holographic memory facts/entities (WAL)
- verification_evidence.db  — agent verification audit trail
- kanban.db                 — default board (back-compat <root>/kanban.db)
- kanban/boards             — non-default boards (<root>/kanban/boards/<slug>/kanban.db
                              + metadata); workspaces/ and attachments/ subtrees
                              are skipped as large + regenerable.

Also: the directory-branch of create_quick_snapshot now routes *.db through the
WAL-safe _safe_copy_db (SQLite backup() API), matching the top-level file path —
previously a non-default board DB with an open WAL could be copied inconsistently.

Salvaged from NousResearch#52930 by @0xDevNinja (authorship preserved via cherry-pick).
On top of the original (which covered only projects.db + the default kanban.db),
this adds: non-default-board coverage, the three sibling per-profile DBs that
meet the same upgrade-wipe criteria, WAL-safe directory copies, and a
workspaces/attachments skip to avoid snapshot bloat (×20 retained). 8 tests,
all mutation-verified; E2E verified snapshot→wipe→restore preserves all six
store types on the real code path.

Closes NousResearch#52889. Supersedes NousResearch#52930.
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
@0xDevNinja @kshitijk4poor
fix(backup): include projects.db, kanban boards, and sibling stores i…
d20e4ed 
…n pre-update snapshot (NousResearch#52889)

projects.db (per-profile project store) and kanban.db were missing from
_QUICK_STATE_FILES, so the pre-update quick snapshot never backed them up.
On a desktop upgrade, when the update flow removes/replaces the file and the
post-update schema-init re-creates an empty one, all user-created projects,
folder mappings, the active-project pointer, kanban board bindings, and tasks
vanish silently — no error.

Add the per-profile user-created stores to the snapshot set:
- projects.db               — project store
- response_store.db         — gateway conversation history / tool payloads (WAL)
- memory_store.db           — holographic memory facts/entities (WAL)
- verification_evidence.db  — agent verification audit trail
- kanban.db                 — default board (back-compat <root>/kanban.db)
- kanban/boards             — non-default boards (<root>/kanban/boards/<slug>/kanban.db
                              + metadata); workspaces/ and attachments/ subtrees
                              are skipped as large + regenerable.

Also: the directory-branch of create_quick_snapshot now routes *.db through the
WAL-safe _safe_copy_db (SQLite backup() API), matching the top-level file path —
previously a non-default board DB with an open WAL could be copied inconsistently.

Salvaged from NousResearch#52930 by @0xDevNinja (authorship preserved via cherry-pick).
On top of the original (which covered only projects.db + the default kanban.db),
this adds: non-default-board coverage, the three sibling per-profile DBs that
meet the same upgrade-wipe criteria, WAL-safe directory copies, and a
workspaces/attachments skip to avoid snapshot bloat (×20 retained). 8 tests,
all mutation-verified; E2E verified snapshot→wipe→restore preserves all six
store types on the real code path.

Closes NousResearch#52889. Supersedes NousResearch#52930.
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
@0xDevNinja @kshitijk4poor
fix(backup): include projects.db, kanban boards, and sibling stores i…
894f68d 
…n pre-update snapshot (NousResearch#52889)

projects.db (per-profile project store) and kanban.db were missing from
_QUICK_STATE_FILES, so the pre-update quick snapshot never backed them up.
On a desktop upgrade, when the update flow removes/replaces the file and the
post-update schema-init re-creates an empty one, all user-created projects,
folder mappings, the active-project pointer, kanban board bindings, and tasks
vanish silently — no error.

Add the per-profile user-created stores to the snapshot set:
- projects.db               — project store
- response_store.db         — gateway conversation history / tool payloads (WAL)
- memory_store.db           — holographic memory facts/entities (WAL)
- verification_evidence.db  — agent verification audit trail
- kanban.db                 — default board (back-compat <root>/kanban.db)
- kanban/boards             — non-default boards (<root>/kanban/boards/<slug>/kanban.db
                              + metadata); workspaces/ and attachments/ subtrees
                              are skipped as large + regenerable.

Also: the directory-branch of create_quick_snapshot now routes *.db through the
WAL-safe _safe_copy_db (SQLite backup() API), matching the top-level file path —
previously a non-default board DB with an open WAL could be copied inconsistently.

Salvaged from NousResearch#52930 by @0xDevNinja (authorship preserved via cherry-pick).
On top of the original (which covered only projects.db + the default kanban.db),
this adds: non-default-board coverage, the three sibling per-profile DBs that
meet the same upgrade-wipe criteria, WAL-safe directory copies, and a
workspaces/attachments skip to avoid snapshot bloat (×20 retained). 8 tests,
all mutation-verified; E2E verified snapshot→wipe→restore preserves all six
store types on the real code path.

Closes NousResearch#52889. Supersedes NousResearch#52930.
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
@0xDevNinja @kshitijk4poor
fix(backup): include projects.db, kanban boards, and sibling stores i…
b97cc41 
…n pre-update snapshot (NousResearch#52889)

projects.db (per-profile project store) and kanban.db were missing from
_QUICK_STATE_FILES, so the pre-update quick snapshot never backed them up.
On a desktop upgrade, when the update flow removes/replaces the file and the
post-update schema-init re-creates an empty one, all user-created projects,
folder mappings, the active-project pointer, kanban board bindings, and tasks
vanish silently — no error.

Add the per-profile user-created stores to the snapshot set:
- projects.db               — project store
- response_store.db         — gateway conversation history / tool payloads (WAL)
- memory_store.db           — holographic memory facts/entities (WAL)
- verification_evidence.db  — agent verification audit trail
- kanban.db                 — default board (back-compat <root>/kanban.db)
- kanban/boards             — non-default boards (<root>/kanban/boards/<slug>/kanban.db
                              + metadata); workspaces/ and attachments/ subtrees
                              are skipped as large + regenerable.

Also: the directory-branch of create_quick_snapshot now routes *.db through the
WAL-safe _safe_copy_db (SQLite backup() API), matching the top-level file path —
previously a non-default board DB with an open WAL could be copied inconsistently.

Salvaged from NousResearch#52930 by @0xDevNinja (authorship preserved via cherry-pick).
On top of the original (which covered only projects.db + the default kanban.db),
this adds: non-default-board coverage, the three sibling per-profile DBs that
meet the same upgrade-wipe criteria, WAL-safe directory copies, and a
workspaces/attachments skip to avoid snapshot bloat (×20 retained). 8 tests,
all mutation-verified; E2E verified snapshot→wipe→restore preserves all six
store types on the real code path.

Closes NousResearch#52889. Supersedes NousResearch#52930.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
@0xDevNinja @kshitijk4poor
fix(backup): include projects.db, kanban boards, and sibling stores i…
49a982e 
…n pre-update snapshot (NousResearch#52889)

projects.db (per-profile project store) and kanban.db were missing from
_QUICK_STATE_FILES, so the pre-update quick snapshot never backed them up.
On a desktop upgrade, when the update flow removes/replaces the file and the
post-update schema-init re-creates an empty one, all user-created projects,
folder mappings, the active-project pointer, kanban board bindings, and tasks
vanish silently — no error.

Add the per-profile user-created stores to the snapshot set:
- projects.db               — project store
- response_store.db         — gateway conversation history / tool payloads (WAL)
- memory_store.db           — holographic memory facts/entities (WAL)
- verification_evidence.db  — agent verification audit trail
- kanban.db                 — default board (back-compat <root>/kanban.db)
- kanban/boards             — non-default boards (<root>/kanban/boards/<slug>/kanban.db
                              + metadata); workspaces/ and attachments/ subtrees
                              are skipped as large + regenerable.

Also: the directory-branch of create_quick_snapshot now routes *.db through the
WAL-safe _safe_copy_db (SQLite backup() API), matching the top-level file path —
previously a non-default board DB with an open WAL could be copied inconsistently.

Salvaged from NousResearch#52930 by @0xDevNinja (authorship preserved via cherry-pick).
On top of the original (which covered only projects.db + the default kanban.db),
this adds: non-default-board coverage, the three sibling per-profile DBs that
meet the same upgrade-wipe criteria, WAL-safe directory copies, and a
workspaces/attachments skip to avoid snapshot bloat (×20 retained). 8 tests,
all mutation-verified; E2E verified snapshot→wipe→restore preserves all six
store types on the real code path.

Closes NousResearch#52889. Supersedes NousResearch#52930.
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
@0xDevNinja @kshitijk4poor
fix(backup): include projects.db, kanban boards, and sibling stores i…
4058c43 
…n pre-update snapshot (NousResearch#52889)

projects.db (per-profile project store) and kanban.db were missing from
_QUICK_STATE_FILES, so the pre-update quick snapshot never backed them up.
On a desktop upgrade, when the update flow removes/replaces the file and the
post-update schema-init re-creates an empty one, all user-created projects,
folder mappings, the active-project pointer, kanban board bindings, and tasks
vanish silently — no error.

Add the per-profile user-created stores to the snapshot set:
- projects.db               — project store
- response_store.db         — gateway conversation history / tool payloads (WAL)
- memory_store.db           — holographic memory facts/entities (WAL)
- verification_evidence.db  — agent verification audit trail
- kanban.db                 — default board (back-compat <root>/kanban.db)
- kanban/boards             — non-default boards (<root>/kanban/boards/<slug>/kanban.db
                              + metadata); workspaces/ and attachments/ subtrees
                              are skipped as large + regenerable.

Also: the directory-branch of create_quick_snapshot now routes *.db through the
WAL-safe _safe_copy_db (SQLite backup() API), matching the top-level file path —
previously a non-default board DB with an open WAL could be copied inconsistently.

Salvaged from NousResearch#52930 by @0xDevNinja (authorship preserved via cherry-pick).
On top of the original (which covered only projects.db + the default kanban.db),
this adds: non-default-board coverage, the three sibling per-profile DBs that
meet the same upgrade-wipe criteria, WAL-safe directory copies, and a
workspaces/attachments skip to avoid snapshot bloat (×20 retained). 8 tests,
all mutation-verified; E2E verified snapshot→wipe→restore preserves all six
store types on the real code path.

Closes NousResearch#52889. Supersedes NousResearch#52930.
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
@melon-xf
fix(backup): include projects.db, kanban boards, and sibling stores i…
278fe52 
…n pre-update snapshot (NousResearch#52889)

projects.db (per-profile project store) and kanban.db were missing from
_QUICK_STATE_FILES, so the pre-update quick snapshot never backed them up.
On a desktop upgrade, when the update flow removes/replaces the file and the
post-update schema-init re-creates an empty one, all user-created projects,
folder mappings, the active-project pointer, kanban board bindings, and tasks
vanish silently — no error.

Add the per-profile user-created stores to the snapshot set:
- projects.db               — project store
- response_store.db         — gateway conversation history / tool payloads (WAL)
- memory_store.db           — holographic memory facts/entities (WAL)
- verification_evidence.db  — agent verification audit trail
- kanban.db                 — default board (back-compat <root>/kanban.db)
- kanban/boards             — non-default boards (<root>/kanban/boards/<slug>/kanban.db
                              + metadata); workspaces/ and attachments/ subtrees
                              are skipped as large + regenerable.

Also: the directory-branch of create_quick_snapshot now routes *.db through the
WAL-safe _safe_copy_db (SQLite backup() API), matching the top-level file path —
previously a non-default board DB with an open WAL could be copied inconsistently.

Salvaged from NousResearch#52930 by @0xDevNinja (authorship preserved via cherry-pick).
On top of the original (which covered only projects.db + the default kanban.db),
this adds: non-default-board coverage, the three sibling per-profile DBs that
meet the same upgrade-wipe criteria, WAL-safe directory copies, and a
workspaces/attachments skip to avoid snapshot bloat (×20 retained). 8 tests,
all mutation-verified; E2E verified snapshot→wipe→restore preserves all six
store types on the real code path.

Closes NousResearch#52889. Supersedes NousResearch#52930.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

bug: projects.db not included in pre-update snapshot — projects lost after upgrade

4 participants

@0xDevNinja @Bartok9 @kshitijk4poor @alt-glitch