feat(ops): add verified SQLite backup and recovery rehearsal

[seonghobae]

Buyer / operator impact

ScopeWeave can create and verify a consistent live SQLite snapshot without raw-copying a WAL-backed database and without exposing a destructive restore command. Operators also get an executable stopped-writer recovery rehearsal that reopens the verified snapshot through ScopeWeave's real database bootstrap with customer-like tenant/project rows intact.

Closes #530 only when this change is actually merged into protected develop.

Exact current scope

• protected live base: develop@2c328875e00e86537df3e965170be80532571cad;
• exact contributor head: 9e3906a5f362d620ce4761ed5d5ce47d81b29411;
• exact head tree: aaadb5320ca5d7f9249d15caa2cdee6b97965165;
• branch: feat/sqlite-backup-recovery-530;
• Ready and mechanically mergeable at the latest fresh refetch;
• current changed-file count: 12;
• a fresh current-head review-thread sweep returns no unresolved threads; and
• no qualifying independent current-head approval exists.

Any base or contributor-head movement invalidates ancestry- and exact-head evidence until freshly revalidated. This body is traceability, not merge authority.

Production contract

server/sqlite_backup.mjs:

• resolves source and destination aliases canonically;
• opens the source read-only and verifies integrity and foreign keys before snapshotting;
• records application_id, user_version, and bounded canonical sqlite_schema metadata;
• streams schema metadata with StatementSync.iterate() instead of materializing .all() results;
• fails before retaining more than 100,000 non-internal schema objects or more than 8 MiB of the exact serialized JSON-array UTF-8 representation, including brackets and separators;
• reserves a unique temporary file with exclusive 0600 creation;
• executes parameterized SQLite VACUUM INTO for a consistent live snapshot;
• independently verifies produced size, integrity, foreign keys, application/user versions, and schema metadata;
• publishes by one no-overwrite hard link into the already-canonical destination directory;
• never deletes or rewrites a destination won by another process;
• cleans only attempt-owned incomplete artifacts best-effort while preserving the causal error;
• exposes stable non-secret operator JSON and no automated destructive restore operation.

TDD and review-driven hardening

The original RED contract preceded the production backup module. Subsequent regressions hardened live-WAL read-only snapshotting, metadata mismatch, parent-symlink retargeting, competing publication, owner-only permissions, portable direct-process invocation, sidecar-safe recovery, application-bootstrap recovery acceptance, bounded schema-memory behavior, foreign-key fake fidelity, and post-publication output handling.

A current-line review exposed an operator-state defect after all durable filesystem work had succeeded: if the backup command published a verified snapshot and only the success output sink failed (for example a closed stdout pipeline), the CLI returned failure and encouraged an unsafe retry against an already-existing destination.

The repair was test-first:

• tests/unit/sqlite-backup-output-failure.test.mjs creates a real SQLite database, closes the success sink, and proves a durably published backup remains verifiable;
• that regression is registered in both the canonical unit and c8 coverage case chains; and
• runSqliteBackupCli treats publication as the durable backup operation boundary: a post-publication success-output failure emits a best-effort diagnostic {ok:true, operation:'backup', warning:'success_output_failed', action:'verify_destination_before_retry'} and returns 0, rather than reclassifying the completed backup as failed.

Exact head 9e3906a5... additionally locks the intentionally different verify contract: verification is read-only/idempotent, but if its requested success output cannot be produced it returns failure with success_output_failed rather than claiming a result the caller did not receive. That final commit changes the regression expectation, not the backup production boundary.

A fresh current-head thread sweep has no unresolved review threads. Resolved review history records the intentional fail-closed concurrent-DDL mismatch boundary, the zero-length VACUUM INTO reservation dependency, read-only WAL operating constraints, and stable metadata assumptions. Those observations are not approvals and are not promoted into merge evidence.

Recovery and operator evidence

• live-WAL regression verifies committed content while a writer remains open;
• recovery acceptance requires the writer to stop before replacement, preserves the original database plus -wal, -shm, and -journal sidecars as one incident evidence set, prevents stale-sidecar mixing, and reopens a verified snapshot through server/db.mjs;
• corrupt/FK-invalid source and backup, aliases, destination collision/race, symlink retargeting, metadata mismatch, object/byte schema bounds, incomplete cleanup, CLI output, direct-process invocation, and post-publication output failure are covered;
• README.md and docs/deploy.md lead operators to npm run ops:sqlite-backup and the recovery runbook; and
• doctoring records the SQLite/Node primary basis and requirement-to-test traceability without claiming RPO/RTO or certification.

Current exact-head evidence state

For exact contributor head 9e3906a5f362d620ce4761ed5d5ce47d81b29411, the currently associated pull-request workflow runs are terminal GitHub-success:

• Server Tests 32571299640 — success;
• Dependency Review 32571299608 — success;
• Security Scan 32571299601 — success;
• OSV Scanner 32571299821 — success;
• SAST Semgrep 32571299603 — success; and
• Fuzz 32571299595 — success.

These labels are not promoted to immutable contributor-head merge authority while the known checkout-evidence defects remain unresolved. ScopeWeave #523 owns repository-native exact-head Server Tests/coverage integrity. ContextualWisdomLab/.github#1222 owns the organization-required reusable SAST/Security exact-head checkout repair through the central repository's existing dedicated writer/control path.

Historical OpenCode CHANGES_REQUESTED reviews target predecessor heads and organization-owned coverage evidence; they are dismissed/stale and do not transfer to exact head 9e3906a5.... CodeRabbit/Devin COMMENTED evidence is supplementary rather than a qualifying independent current-head approval. Pending, queued, skipped-required, cancelled, absent, neutral-required, failed, stale, predecessor, synthetic, status-only, author-only, model-only, rate-limited, or infrastructure-only evidence is non-passing.

Merge gate

Do not merge or enable auto-merge until this unchanged exact head remains reconciled to freshly resolved protected develop, #523's repository exact-head control and the central reusable exact-head SAST/Security repair are protected-shipped and regenerated on this exact head, every applicable repository/organization CI, browser, owned statement-branch-function-line coverage/docstring, CodeQL/SAST, security, dependency/supply-chain, package/provenance and required-workflow gate is substantively terminal-passing under corrected contracts, valid unresolved current-head defects are zero, and the live required approving-review rule is satisfied by a qualifying independent current-head approval (currently one approval under active ruleset 18156473; no latest-push approval is required by that ruleset). Do not self-approve or transfer predecessor evidence.

Closes #530

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

SQLite 라이브 백업 모듈과 읽기 전용 검증 CLI를 추가했습니다. VACUUM INTO 스냅샷을 검증한 뒤 기존 파일을 덮어쓰지 않고 게시합니다. 운영 문서, 회귀 테스트, 커버리지 계약과 배포 안내도 추가했습니다.

Changes

SQLite 백업 운영 경계

Layer / File(s)	Summary
검증 및 메타데이터 계약 server/sqlite_backup.mjs, tests/unit/sqlite-backup-recovery.test.mjs, tests/unit/sqlite-backup-schema-budget.test.mjs	경로, 파일 유형, SQLite 무결성, 외래 키, 메타데이터, 스키마 객체 수와 직렬화 크기를 검증합니다. 안정적인 오류 코드와 임시 파일 정리 함수를 제공합니다. 스키마 메타데이터는 iterate()로 스트리밍합니다.
검증된 스냅샷 게시 server/sqlite_backup.mjs, tests/unit/sqlite-backup-recovery.test.mjs	읽기 전용 연결에서 VACUUM INTO를 실행합니다. 임시 스냅샷을 검증한 뒤 owner-only 권한과 no-overwrite hard-link로 게시합니다. WAL, 충돌, 심볼릭 링크, 손상 데이터베이스와 복구 파일 복사를 테스트합니다.
운영 CLI 및 검증 근거 server/sqlite_backup.mjs, tests/unit/*, package.json, README.md, docs/deploy.md, docs/doctoring/sqlite-backup-recovery.md, docs/operations/sqlite-backup-recovery.md, CHANGELOG.md	backup 및 verify CLI와 JSON 결과를 추가했습니다. 단위 테스트와 커버리지 실행을 등록하고, 백업·복구 리허설 절차와 배포 안내를 문서화했습니다. @hono/node-server 버전 범위와 API 테스트 목록도 갱신했습니다.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🟡 Moderate · up to 34575

The backup feature currently materializes schema metadata instead of enforcing the required bounded inspection path, causing the schema-budget test to fail and weakening protection against oversized schemas. Merge should wait until schema inspection streams rows and reports the expected limit error.

Sequence Diagram(s)

sequenceDiagram
  participant 운영자
  participant 백업CLI
  participant createVerifiedSqliteBackup
  participant DatabaseSync
  participant 파일시스템
  운영자->>백업CLI: backup 또는 verify 실행
  백업CLI->>createVerifiedSqliteBackup: 백업 생성 요청
  createVerifiedSqliteBackup->>DatabaseSync: 무결성 검증 및 VACUUM INTO
  DatabaseSync-->>createVerifiedSqliteBackup: 임시 스냅샷 반환
  createVerifiedSqliteBackup->>DatabaseSync: 백업 무결성 및 메타데이터 검증
  createVerifiedSqliteBackup->>파일시스템: 하드 링크로 no-overwrite 게시
  파일시스템-->>백업CLI: 성공 또는 안정적인 오류 코드
  백업CLI-->>운영자: JSON 결과 및 종료 코드

Loading

Possibly related PRs

• ContextualWisdomLab/scopeweave#500: SQLite 스키마 메타데이터와 무결성 검증의 fail-closed 처리와 연결됩니다.

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	백업 구현과 검증·복구 문서는 대부분 #530 요구사항을 충족하지만, 기존 API 테스트 실행 항목 제거가 기존 게이트 보존 요구와 충돌할 수 있습니다.	기존 API 테스트 실행 항목을 제거하지 말고, 의존성 변경과 함께 해당 게이트가 계속 실행되고 통과하는지 증명하십시오.
Out of Scope Changes check	⚠️ Warning	SQLite 백업 범위와 직접 관련되지 않은 API 테스트 실행 항목 제거와 @hono/node-server 버전 변경이 포함되어 있습니다.	API 테스트 목록 제거와 의존성 버전 변경을 별도 PR로 분리하거나, SQLite 백업 요구사항에 필요한 근거를 추가하십시오.
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	제목은 검증된 SQLite 백업과 복구 리허설 추가라는 주요 변경 사항을 정확히 요약합니다. 간결하고 구체적이며 변경 내용과 직접 관련됩니다.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/sqlite-backup-recovery-530

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

@opencode-agent @cwl-noema-review

[seonghobae]

@opencode-agent review

Please submit a formal review for exact current head 0a95e21d04453c7a40248909e530a33fa9f16d20 against protected develop@44e7903cf8891c65410f7fc6ca5144de3fdb5185. Re-evaluate only current source and the resolved current threads: live-WAL read-only snapshotting, bounded schema inspection, race-safe no-overwrite publication, sidecar-safe recovery, exact metadata verification, real DB bootstrap recovery acceptance, and operator discoverability. Current opencode-review check 95237313957 is terminal success, but there is no formal exact-head approval/review submission. Review only; do not mutate, merge, weaken gates, or transfer predecessor findings.

Dismissed as predecessor-head-only coverage-control evidence for f977cd2. The current PR head is 9e3906a, so this review cannot govern the current tree. This dismissal is not an approval and does not satisfy any current-head review gate.

[seonghobae]

@opencode-agent review

Review-only request for exact head b6842e7cf60f77b1d5cee04e1c020e411afc9a6f against protected develop@2c328875e00e86537df3e965170be80532571cad. Bind any verdict to this exact head and live base. Re-evaluate the now-resolved source-replacement guard, live-WAL read-only snapshotting, bounded schema inspection, race-safe no-overwrite publication, sidecar-safe recovery, exact metadata verification, real database-bootstrap recovery acceptance, and operator discoverability. Do not transfer predecessor-head coverage failures or model/status-only evidence into current approval.