diff --git a/AGENTS.md b/AGENTS.md index e8c7b56315..7cee32d1e2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -683,6 +683,7 @@ logging and table-scoped cache invalidation stay automatic. - `deno task lint:ci` - Strict, read-only lint (`check --error-on-warnings`, no `--write`). Fails on lint warnings (e.g. cognitive complexity) and on any code that *would* be reformatted, without touching the checkout. This is the lint `deno task precommit` runs in **every** environment, so a clean `precommit` locally means the lint step will pass in CI too. Run `deno task lint` to auto-fix before re-running. - `deno task build:edge` - Build for Bunny Edge deployment - `deno task backup` - Dump the database out-of-band to a `.zip`. Uploads to the configured storage zone by default (so it appears on the Backups page and lets the next migration skip its own inline backup); pass `--out ` to write a local file. Runs in a full Deno process, so unlike the in-edge backup it has no per-request subrequest budget and can dump arbitrarily large databases. +- `deno task restore ` - Restore the database named by `DB_URL` / `DB_TOKEN` in `.env` using its `DB_ENCRYPTION_KEY`. Shows the backup details, asks for typed confirmation, and reports each restore step in the console. - `deno task snapshot --out ` - Sync the complete remote database to a standalone local SQLite file. The task prefers `DB_URL` and `DB_TOKEN` from `.env` over shell values. This developer-only task checkpoints and verifies the file, refuses to overwrite an existing path, and removes its temporary replica on success or failure. - `deno task precommit` - Run all checks (typecheck, lint, tests) - `deno task precommit:mutation` - The precommit mutation gate, runnable on its own: mutation-test every `src/` file this branch changed and demand a 100% kill rate. A source's mirror-located direct tests run first; changed tests under `test/integration/` or `test/e2e/` run only for direct-test survivors. Any other unmatched test is an error: move it to the path that mirrors its source while strengthening it to kill the mutants. The changed set is the branch's committed diff against the integration branch (`origin/main`, else a local `main`) via `base...HEAD` — three-dot/merge-base, so it's the branch's full diff vs main and stays bounded to the branch's own commits (precommit runs post-commit on a clean tree, so the index is empty). Skips cheaply when there is no base ref or no changed `src/` files (and likewise when src changed without any changed test). If a badly stale local `origin/main` balloons the changed set past `STALE_BASE_SOURCE_LIMIT`, it skips with a "run `git fetch origin main`" hint instead of mutating most of the tree. See [Mutation Testing](#mutation-testing). diff --git a/README.md b/README.md index 5da3fae1ca..5d6fd34237 100644 --- a/README.md +++ b/README.md @@ -328,6 +328,7 @@ deno task typecheck # Type check deno task build:edge # Build for Bunny Edge deno task deploy:edge # Build, upload, and publish to Bunny Edge using BUNNY_ACCESS_KEY from .env deno task backup # Dump the database out-of-band (uploads to storage; --out for a local .zip) +deno task restore # Restore the database at DB_URL from a local backup using DB_TOKEN from .env deno task snapshot --out # Copy the complete remote database to a local SQLite file deno task precommit # All checks (typecheck, lint, cpd, build:edge, test:coverage) ``` @@ -361,6 +362,13 @@ Optional: **Backups:** every table is dumped to a single `.zip`, with table reads keyset-paginated so no single response trips libsqld's "Response is too large" payload cap (the server limit behind Bunny's databases). Backups run **out-of-band**, not inside the migration: a full dump of a ~31-table schema can't fit alongside a migration within one edge request's [50-subrequest budget](https://docs.bunny.net/scripting/limits), so migrations just migrate, and a backup is taken by GitHub Actions (or `deno task backup`) beforehand. To enforce that, **`/admin/update` and the per-site update button refuse to deploy unless a backup of that database was taken in the last hour.** +Restores run out of band because realistic imports exceed Bunny's per-request +subrequest limit. Put the target database's `DB_URL`, `DB_TOKEN`, and +`DB_ENCRYPTION_KEY` in `.env`, then run `deno task restore `. The task checks +the backup, shows its available table, row, statement, and schema details, and +asks for a typed confirmation before deleting any data. It reports each restore +step and shows the backup's recorded commit when one is available. + **Local SQLite snapshots:** developers can copy a remote database into one complete SQLite file using the same `DB_URL` and `DB_TOKEN` as the app: ```bash diff --git a/deno.json b/deno.json index 286d2ded96..229130d9be 100644 --- a/deno.json +++ b/deno.json @@ -16,7 +16,7 @@ "test:raw": "deno task build:static && deno test --no-check --allow-net --allow-env --allow-read --allow-write --allow-run --allow-sys --allow-ffi test/", "lint": "deno run -A scripts/biome.ts check --write --error-on-warnings src test scripts cli e2e-payments", "lint:ci": "deno run -A scripts/biome.ts check --error-on-warnings src test scripts cli e2e-payments", - "typecheck": "deno check src/**/*.ts src/**/*.tsx test/**/*.ts cli/**/*.ts scripts/database-snapshot.ts scripts/mutation.ts scripts/mutation/**/*.ts scripts/precommit.ts scripts/precommit/**/*.ts scripts/check-copy.ts scripts/check-copy/**/*.ts scripts/test-quality-audit.ts scripts/bench/bundle-composition/**/*.ts scripts/screenshots.ts scripts/screenshots/**/*.ts && deno check --config=e2e-payments/deno.json e2e-payments/src/**/*.ts", + "typecheck": "deno check src/**/*.ts src/**/*.tsx test/**/*.ts cli/**/*.ts scripts/database-snapshot.ts scripts/restore.ts scripts/mutation.ts scripts/mutation/**/*.ts scripts/precommit.ts scripts/precommit/**/*.ts scripts/check-copy.ts scripts/check-copy/**/*.ts scripts/test-quality-audit.ts scripts/bench/bundle-composition/**/*.ts scripts/screenshots.ts scripts/screenshots/**/*.ts && deno check --config=e2e-payments/deno.json e2e-payments/src/**/*.ts", "cpd": "deno run -A scripts/cpd.ts --config .jscpd.json && deno run -A scripts/cpd.ts --config .jscpd.test.json && deno run -A scripts/cpd.ts --config .jscpd.css.json src/ui/static/style.scss", "check:copy": "deno run --allow-read=src/locales scripts/check-copy.ts", "precommit": "deno run -A scripts/precommit.ts", @@ -27,6 +27,7 @@ "unit-tests-report": "deno run --allow-read=src,test,deno.json scripts/unit-tests-report.ts", "upgrade": "deno run --allow-read --allow-write --allow-net scripts/safe-upgrade.ts", "backup": "deno run --env-file --allow-env --allow-read --allow-write --allow-net --allow-sys --allow-ffi scripts/backup.ts", + "restore": "deno run --allow-env --allow-read --allow-write --allow-net --allow-sys --allow-ffi scripts/restore.ts", "snapshot": "deno run --allow-env --allow-read --allow-write --allow-net --allow-sys --allow-ffi scripts/database-snapshot.ts", "cli:tui": "deno run --allow-env --allow-read --allow-run cli/tui.ts", "cli:api": "deno run --allow-env --allow-read --allow-run cli/api.ts", diff --git a/scripts/deploy-edge-lib.ts b/scripts/deploy-edge-lib.ts index cca4c34c87..72bb47700e 100644 --- a/scripts/deploy-edge-lib.ts +++ b/scripts/deploy-edge-lib.ts @@ -1,4 +1,5 @@ import { mapNotNullish } from "#fp"; +import type { ScriptIo } from "#scripts/script-runner.ts"; export const BUNNY_API_BASE = "https://api.bunny.net"; export const BUNDLE_PATH = "bunny-script.ts"; @@ -32,16 +33,12 @@ export interface BuildResult { success: boolean; } -export interface DeployEdgeDeps { - args: string[]; +export interface DeployEdgeDeps extends ScriptIo { bundlePath: string; cwd: string; fetchText: FetchText; - getEnv: (key: string) => string | undefined; readTextFile: (path: string) => Promise; runBuildEdge: (cwd: string) => Promise; - stderr: (line: string) => void; - stdout: (line: string) => void; } export const parseScriptIdArg = ( diff --git a/scripts/deploy-edge.ts b/scripts/deploy-edge.ts index 9f8dd62287..ecbd44d608 100644 --- a/scripts/deploy-edge.ts +++ b/scripts/deploy-edge.ts @@ -1,6 +1,7 @@ #!/usr/bin/env -S deno run --env-file --allow-env --allow-read --allow-write --allow-run --allow-net import { fromFileUrl } from "@std/path"; +import { runDenoScript } from "#scripts/script-runner.ts"; import { type FetchTextResult, runDeployEdge } from "./deploy-edge-lib.ts"; import { runBuildEdge } from "./run-build-edge.ts"; @@ -19,16 +20,13 @@ const fetchText = async ( }; }; -const exitCode = await runDeployEdge({ - args: Deno.args, - bundlePath, - cwd: repoRoot, - fetchText, - getEnv: (key) => Deno.env.get(key), - readTextFile: (path) => Deno.readTextFile(path), - runBuildEdge, - stderr: (line) => console.error(line), - stdout: (line) => console.log(line), -}); - -Deno.exit(exitCode); +await runDenoScript((io) => + runDeployEdge({ + ...io, + bundlePath, + cwd: repoRoot, + fetchText, + readTextFile: (path) => Deno.readTextFile(path), + runBuildEdge, + }), +); diff --git a/scripts/mutation/equivalent-mutants.txt b/scripts/mutation/equivalent-mutants.txt index 8e3b00c986..079758aba7 100644 --- a/scripts/mutation/equivalent-mutants.txt +++ b/scripts/mutation/equivalent-mutants.txt @@ -141,8 +141,8 @@ src/shared/db/users.ts:263:62 ?? → || # queryAll()[0] is a truthy UserAuthF src/shared/db/query-log.ts:71:62 ?? → || # queryLogScope.current() is a truthy QueryLogState object or undefined src/shared/db/query-log.ts:252:43 ?? → || # stored read counts start at 1 and remain positive, so only undefined reaches the zero fallback src/shared/db/query-log.ts:292:22 ?? → || # store is a truthy QueryLogState object or undefined -src/shared/update.ts:131:45 ?? → || # the stored marker is a string or undefined and the fallback is empty, so both operators return the same string -src/shared/update.ts:156:45 ?? → || # the stored commit is a string or undefined and the fallback is empty, so both operators return the same string +src/shared/update.ts:153:45 ?? → || # the stored commit is a string or undefined and the fallback is empty, so both operators return the same string +src/shared/update.ts:186:11 ?? → || # the primary row value is a string or absent and the fallback is empty, so both operators return the same string # Parent/child fold moved to fold-tree.ts in Phase 2a (see the fold-tree.ts # entries below); the ticket-payment.ts fold-internal equivalents went with it. @@ -1059,3 +1059,5 @@ src/shared/stripe/request.ts:152:15 === → == # Headers.get returns string|n # only for that value-less case, so every parsed result is identical. src/shared/stripe/webhook.ts:28:15 ?: → alternate only # with "=", both arms slice the same key; without it, no value can be consumed src/shared/stripe/webhook.ts:28:30 1 → 2 # this changes only the no-separator key arm; value-less recognized and unknown keys are all ignored + +# Schema indexes are an array when present, and arrays are always truthy. diff --git a/scripts/restore-lib.ts b/scripts/restore-lib.ts new file mode 100644 index 0000000000..99fbf3c275 --- /dev/null +++ b/scripts/restore-lib.ts @@ -0,0 +1,196 @@ +import { sum } from "#fp"; +import type { ScriptIo } from "#scripts/script-runner.ts"; +import { decodeKeyBytes } from "#shared/crypto/encryption.ts"; +import { + type BackupManifest, + PostResetError, + type RestoreProgress, + type RestoreProgressHandler, + type RestoreStage, +} from "#shared/db/backup.ts"; +import { SCHEMA_HASH } from "#shared/db/migrations.ts"; +import { errorMessage } from "#shared/error-message.ts"; +import { formatBytes } from "#shared/limits.ts"; + +export const RESTORE_CONFIRMATION = "RESTORE"; +export const RESTORE_USAGE = "Usage: deno task restore "; +const isFullCommitSha = (commit: string): boolean => + /^[0-9a-f]{40}$/.test(commit); +const REMOTE_DB_URL_PREFIXES = ["https://", "libsql://"]; + +export interface RestoreCliDeps extends ScriptIo { + inspectBackupZip: (data: Uint8Array) => { + manifest: BackupManifest | null; + statementCount: number; + }; + prompt: (message: string) => string | null; + readFile: (path: string) => Promise; + readRecordedScriptCommit: () => Promise; + restoreFromZip: ( + data: Uint8Array, + onProgress: RestoreProgressHandler, + ) => Promise; +} + +const progressMessages: Record< + RestoreStage, + (progress: RestoreProgress) => string +> = { + checking: ({ statementCount }) => + `Checking ${statementCount} SQL statements...`, + "clearing-caches": () => "Clearing cached database data...", + importing: ({ statementCount }) => + `Importing ${statementCount} SQL statements...`, + rebuilding: () => "Recreating the database schema...", + resetting: () => "Deleting current database data...", +}; + +const countLabel = (count: number, name: string): string => + `${count} ${name}${count === 1 ? "" : "s"}`; + +const readRestoreDbUrlOrNull = (deps: RestoreCliDeps): string | null => { + const dbUrl = deps.getEnv("DB_URL"); + if (!dbUrl?.trim()) { + deps.stderr("DB_URL is required in .env."); + return null; + } + if (dbUrl === ":memory:") { + deps.stderr( + "DB_URL cannot be :memory: for a restore. Set it to the target database in .env.", + ); + return null; + } + if ( + REMOTE_DB_URL_PREFIXES.some((prefix) => dbUrl.startsWith(prefix)) && + !deps.getEnv("DB_TOKEN")?.trim() + ) { + deps.stderr("DB_TOKEN is required in .env for a remote database."); + return null; + } + const encryptionKey = deps.getEnv("DB_ENCRYPTION_KEY"); + if (!encryptionKey?.trim()) { + deps.stderr("DB_ENCRYPTION_KEY is required in .env."); + return null; + } + try { + decodeKeyBytes(encryptionKey); + } catch (error) { + deps.stderr(errorMessage(error)); + return null; + } + return dbUrl; +}; + +const writeManifestSummary = ( + manifest: BackupManifest | null, + statementCount: number, + write: (line: string) => void, +): boolean => { + if (manifest === null) { + write("Backup manifest: not available"); + write(`Backup contents: ${countLabel(statementCount, "SQL statement")}`); + return false; + } + + const tableCount = Object.keys(manifest.tables).length; + const rowCount = sum(Object.values(manifest.tables)); + write(`Backup created: ${manifest.timestamp}`); + write( + `Backup contents: ${countLabel(tableCount, "table")}, ` + + `${countLabel(rowCount, "row")}, ` + + countLabel(statementCount, "SQL statement"), + ); + const schemaMismatch = manifest.schemaHash !== SCHEMA_HASH; + write( + schemaMismatch + ? "Schema: does not match this version of the app" + : "Schema: matches this version of the app", + ); + return schemaMismatch; +}; + +export const runRestoreCli = async (deps: RestoreCliDeps): Promise => { + if (deps.args.length !== 1 || !deps.args[0]?.trim()) { + deps.stderr(RESTORE_USAGE); + return 1; + } + + const path = deps.args[0]; + const dbUrl = readRestoreDbUrlOrNull(deps); + if (dbUrl === null) return 1; + + deps.stdout(`Reading ${path}...`); + let data: Uint8Array; + try { + data = await deps.readFile(path); + } catch (error) { + deps.stderr(`Could not read ${path}: ${errorMessage(error)}`); + return 1; + } + + let inspection: ReturnType; + try { + inspection = deps.inspectBackupZip(data); + } catch (error) { + deps.stderr(`${path} is not a valid database backup.`); + deps.stderr(errorMessage(error)); + return 1; + } + + deps.stdout(`Backup file: ${path} (${formatBytes(data.byteLength)})`); + deps.stdout(`Target database: ${dbUrl}`); + const schemaMismatch = writeManifestSummary( + inspection.manifest, + inspection.statementCount, + deps.stdout, + ); + if (schemaMismatch) { + deps.stderr( + "Warning: This backup uses a different database schema. The app may need to apply database updates after the restore.", + ); + } + deps.stdout( + "Warning: This will delete all current data in the target database. This cannot be undone.", + ); + + if ( + deps.prompt(`Type ${RESTORE_CONFIRMATION} to continue:`) !== + RESTORE_CONFIRMATION + ) { + deps.stderr("Restore cancelled. The database was not changed."); + return 1; + } + + try { + await deps.restoreFromZip(data, (progress) => + deps.stdout(progressMessages[progress.stage](progress)), + ); + } catch (error) { + deps.stderr(`Restore failed: ${errorMessage(error)}`); + if (error instanceof PostResetError) { + deps.stderr( + "The target database was reset before the restore failed. Fix the error and run the restore again.", + ); + } + return 1; + } + + deps.stdout("Database restored successfully."); + let commit: string; + try { + commit = await deps.readRecordedScriptCommit(); + } catch (error) { + deps.stderr( + "The database was restored, but its build information could not be read: " + + errorMessage(error), + ); + return 1; + } + if (isFullCommitSha(commit)) { + deps.stdout(`The backup was running commit ${commit}.`); + deps.stdout( + "Run the restore-deploy workflow with this commit if the app code must also be restored to that point in time.", + ); + } + return 0; +}; diff --git a/scripts/restore.ts b/scripts/restore.ts new file mode 100644 index 0000000000..90e46481fe --- /dev/null +++ b/scripts/restore.ts @@ -0,0 +1,12 @@ +#!/usr/bin/env -S deno run --allow-env --allow-read --allow-write --allow-net --allow-sys --allow-ffi + +import { load } from "@std/dotenv"; +import { runDenoScript } from "#scripts/script-runner.ts"; +import { runRestoreTask } from "#src/restore.ts"; + +await runRestoreTask( + await load(), + (key, value) => + value === undefined ? Deno.env.delete(key) : Deno.env.set(key, value), + runDenoScript, +); diff --git a/scripts/script-runner.ts b/scripts/script-runner.ts new file mode 100644 index 0000000000..28fae020ad --- /dev/null +++ b/scripts/script-runner.ts @@ -0,0 +1,19 @@ +export interface ScriptIo { + args: string[]; + getEnv: (key: string) => string | undefined; + stderr: (line: string) => void; + stdout: (line: string) => void; +} + +/** Give a console task its standard Deno inputs and exit with its result. */ +export const runDenoScript = async ( + run: (io: ScriptIo) => Promise, +): Promise => { + const exitCode = await run({ + args: Deno.args, + getEnv: (key) => Deno.env.get(key), + stderr: (line) => console.error(line), + stdout: (line) => console.log(line), + }); + return Deno.exit(exitCode); +}; diff --git a/src/features/admin/backup.ts b/src/features/admin/backup.ts index 4c7b8ef806..f5e035eb1b 100644 --- a/src/features/admin/backup.ts +++ b/src/features/admin/backup.ts @@ -1,31 +1,13 @@ -import { defineRoutes } from "#routes/router.ts"; -import { namedError } from "#shared/named-error.ts"; -/** - * Admin backup/restore routes — owner only - * - * Provides database backup (export to .zip) and restore (import from .zip) - * functionality for remote databases. Backups are stored unencrypted on CDN - * storage since the sensitive data is already encrypted at the field level. - */ +/** Admin backup routes. Restores run out of band through `deno task restore`. */ import { createActionHandler } from "#routes/admin/actions.ts"; -import { verifyOrRedirect } from "#routes/admin/confirmation.ts"; -/* jscpd:ignore-start */ -import { OWNER_MULTIPART, requireOwnerOr, withAuth } from "#routes/auth.ts"; -import { applyFlash, requireUploadedFile } from "#routes/csrf.ts"; -import { errorRedirect, htmlResponse, redirect } from "#routes/response.ts"; -import type { TypedRouteHandler } from "#routes/router.ts"; -/* jscpd:ignore-end */ -import { clearSessionCookie } from "#shared/cookies.ts"; +import { requireOwnerOr } from "#routes/auth.ts"; +import { applyFlash } from "#routes/csrf.ts"; +import { htmlResponse } from "#routes/response.ts"; +import { defineRoutes, type TypedRouteHandler } from "#routes/router.ts"; import { getEncryptionKeyString } from "#shared/crypto/encryption.ts"; import { formatDatetimeLabel } from "#shared/dates.ts"; -import { - countZipStatements, - createAndUploadBackup, - PostResetError, - readManifest, - restoreFromZip, -} from "#shared/db/backup.ts"; +import { createAndUploadBackup } from "#shared/db/backup.ts"; import { backupDir, backupTimestamp, @@ -34,53 +16,21 @@ import { isRemoteDatabase, parseBackupTime, } from "#shared/db/backup-storage.ts"; -import { SCHEMA_HASH } from "#shared/db/migrations.ts"; import { formatBytes, MAX_BACKUPS } from "#shared/limits.ts"; -import { flushPendingWork } from "#shared/pending-work.ts"; import { - deleteFile, downloadRaw, getBasename, isStorageEnabled, listFilesWithMeta, type StorageFileMeta, - uploadRaw, } from "#shared/storage.ts"; -import { readRecordedScriptCommit } from "#shared/update.ts"; import { adminBackupPage, - adminRestoreConfirmPage, type BackupEntry, type BackupPageState, - RESTORE_CONFIRM_PHRASE, } from "#templates/admin/backup.tsx"; -const RESTORE_PENDING_PREFIX = "restore-pending-"; - -/** Thrown by the restore execute block so onError can send post-reset failures - * (sessions and settings wiped) to /setup/, while pre-restore validation errors - * (DB intact) stay on /admin/backup. */ -class PostResetRestoreError extends namedError("PostResetRestoreError") {} - -/** A full git commit SHA (40 lowercase hex) — what restore-deploy requires and - * the only shape we echo back from restored (possibly untrusted) settings. */ -const FULL_COMMIT_SHA = /^[0-9a-f]{40}$/; - -/** - * Reject path-traversal payloads in backup filenames. - * Legitimate backup names are flat (no directory separators) and server-generated, - * so any `/`, `\`, or `..` component is always malicious input. - */ -const isSafeBackupFilename = (filename: string): boolean => - !filename.includes("/") && - !filename.includes("\\") && - !filename.includes(".."); - -/** Parse a backup file into display info (friendly date, raw timestamp, and - * human size). The download link uses the bare leaf; filenames are - * server-generated, so parseBackupTime always succeeds. The raw timestamp - * round-trips through the same codec that named the file, so the column - * always matches the filename exactly. */ +/** Parse one server-generated backup filename into its display values. */ const parseBackupEntry = (file: StorageFileMeta): BackupEntry => { const takenAt = new Date(parseBackupTime(file.name)!); return { @@ -91,25 +41,13 @@ const parseBackupEntry = (file: StorageFileMeta): BackupEntry => { }; }; -/** Pick out the backups from a folder listing, newest first. Filenames embed - * ISO timestamps, so name order is chronological. */ +/** Pick this database's backups out of a folder listing, newest first. */ const toBackupEntries = (files: StorageFileMeta[]): BackupEntry[] => files - .filter((f) => isBackupPath(f.name)) + .filter((file) => isBackupPath(file.name)) .reverse() .map(parseBackupEntry); -/** Delete stale restore-pending temp files left by abandoned uploads. - * Best-effort — allSettled swallows individual failures. */ -const cleanupStalePendingFiles = (files: StorageFileMeta[]): Promise => - Promise.allSettled( - files - .filter((f) => f.name.startsWith(RESTORE_PENDING_PREFIX)) - .map((f) => deleteFile(f.name)), - ); - -/** Build page state: this DB's backups (from its own folder) plus a best-effort - * sweep of stale restore-pending temp files at the storage root. */ const getBackupPageState = async (): Promise => { const base = { encryptionKey: getEncryptionKeyString(), @@ -120,32 +58,30 @@ const getBackupPageState = async (): Promise => { if (!isStorageEnabled()) return { ...base, backups: [] }; try { - // Backups live in this DB's folder; restore-pending temp files sit at the - // storage root. One listing each, in parallel. - const [backupFiles, rootFiles] = await Promise.all([ - listFilesWithMeta(backupDir()), - listFilesWithMeta(""), - ]); - await cleanupStalePendingFiles(rootFiles); - return { ...base, backups: toBackupEntries(backupFiles) }; + return { + ...base, + backups: toBackupEntries(await listFilesWithMeta(backupDir())), + }; } catch { - // Storage listing unavailable — still render the page (encryption key, - // forms) rather than failing the whole request on a transient CDN error. + // A transient storage listing failure must not hide the encryption key or + // the out-of-band restore instructions. return { ...base, backups: [] }; } }; -/** GET /admin/backup — show backup page */ const handleBackupGet: TypedRouteHandler<"GET /admin/backup"> = (request) => requireOwnerOr(request, async (session) => { const flash = applyFlash(request); - const state = await getBackupPageState(); return htmlResponse( - adminBackupPage(session, state, flash.error, flash.success), + adminBackupPage( + session, + await getBackupPageState(), + flash.error, + flash.success, + ), ); }); -/** POST /admin/backup/create — create a new backup */ const handleBackupCreate: TypedRouteHandler<"POST /admin/backup/create"> = createActionHandler({ auth: "owner", @@ -156,14 +92,10 @@ const handleBackupCreate: TypedRouteHandler<"POST /admin/backup/create"> = successRedirect: "/admin/backup", }); -/** GET /admin/backup/download/:filename — download a backup file */ const handleBackupDownload: TypedRouteHandler< "GET /admin/backup/download/:filename" > = (request, { filename }) => requireOwnerOr(request, async () => { - // The route param is a bare leaf. The anchored leaf check rejects anything - // that isn't "backup-{timestamp}.zip" (path separators included), and we - // resolve it inside this DB's own folder — so it can only reach our backups. if (!isBackupLeaf(filename)) { return htmlResponse("Invalid backup filename", 400); } @@ -183,142 +115,8 @@ const handleBackupDownload: TypedRouteHandler< }); }); -/** POST /admin/backup/restore — upload a .zip file for review */ -const handleBackupRestore: TypedRouteHandler<"POST /admin/backup/restore"> = ( - request, -) => - withAuth(request, OWNER_MULTIPART, async (session, formData) => { - const file = requireUploadedFile(formData, "backup_file", () => - redirect("/admin/backup", "Please select a backup file", false), - ); - if (file instanceof Response) return file; - - const bytes = new Uint8Array(await file.arrayBuffer()); - - let statementCount: number; - try { - statementCount = countZipStatements(bytes); - } catch { - return redirect( - "/admin/backup", - "Invalid backup file. Please upload a valid .zip backup.", - false, - ); - } - - // Read manifest for schema compatibility check - const manifest = readManifest(bytes); - const schemaMismatch = - manifest !== null && manifest.schemaHash !== SCHEMA_HASH; - - // Store the uploaded zip temporarily so the confirm step can use it - const tempFilename = `${RESTORE_PENDING_PREFIX}${crypto.randomUUID()}.zip`; - await uploadRaw(bytes, tempFilename); - - return htmlResponse( - adminRestoreConfirmPage( - session, - tempFilename, - statementCount, - undefined, - schemaMismatch, - ), - ); - }); - -/** POST /admin/backup/restore/confirm — execute the restore after confirmation */ -const handleBackupRestoreConfirm: TypedRouteHandler<"POST /admin/backup/restore/confirm"> = - createActionHandler({ - auth: "owner", - // After restore, the operator's current session is no longer valid (the - // backup pre-dates it). Redirect to /admin/login and clear the session - // cookie so the flash appears on the login page instead of being consumed - // by an intermediate auth-redirect that the operator never sees. - cookie: clearSessionCookie, - execute: async (_session, form) => { - const filename = form.getString("backup_filename"); - if ( - !filename?.startsWith(RESTORE_PENDING_PREFIX) || - !filename.endsWith(".zip") || - !isSafeBackupFilename(filename) - ) { - throw new Error("Invalid backup reference"); - } - - const error = verifyOrRedirect( - form, - RESTORE_CONFIRM_PHRASE, - "/admin/backup", - "Confirmation phrase", - "restore", - ); - if (error) throw new Error("Confirmation phrase does not match"); - - const data = await downloadRaw(filename); - if (!data) { - throw new Error( - "Backup file expired or not found. Please upload again.", - ); - } - - // Drain initDb's queued script-version write so it cannot land after - // the replay and clobber the restored commit that message() reads. - await flushPendingWork(); - - // Capture any restoreFromZip error so we can throw *after* the - // finally cleanup — V8's coverage source maps don't reliably track - // `throw` inside catch blocks for async functions. - let restoreErr: string | undefined; - let isPostReset = false; - try { - await restoreFromZip(data); - } catch (err) { - restoreErr = String(err); - // PostResetError means resetDatabase() already ran; route to /setup/. - // Any other error (e.g. unzipSync on invalid bytes) leaves the DB intact; - // route back to /admin/backup. - isPostReset = err instanceof PostResetError; - } finally { - // Clean up the temp file whether restore succeeds or fails - await Promise.allSettled([deleteFile(filename)]); - } - if (restoreErr !== undefined) { - if (isPostReset) throw new PostResetRestoreError(restoreErr); - throw new Error(restoreErr); - } - }, - // The restored data carries the commit the site was running when the backup - // was taken (recordScriptVersion writes it on boot). Surface the full SHA so - // the operator can redeploy that commit; restore only rolls back data, never - // code. The value comes from restored settings (an uploaded backup could - // hold anything), so only present it when it is a real 40-char SHA — both - // because the restore-deploy workflow requires one and to keep an oversized - // value out of the flash cookie. Read straight after the restore, before the - // next request's initDb re-stamps the running commit. - message: async () => { - const commit = await readRecordedScriptCommit(); - return FULL_COMMIT_SHA.test(commit) - ? `Database restored from backup. It was running commit ${commit} — run the restore-deploy workflow with that commit to restore the code to this point in time.` - : "Database restored from backup"; - }, - // Post-reset failures go to /setup/ (sessions and settings are wiped, so - // /admin/login would hit the not-activated gate and lose the flash; /setup/ - // renders flash.error directly without needing initialised settings). - // Pre-restore validation errors stay on /admin/backup. - onError: (error) => - error instanceof PostResetRestoreError - ? redirect("/setup/", error.message, false, { - cookie: clearSessionCookie(), - }) - : errorRedirect("/admin/backup", error.message), - successRedirect: "/admin/login", - }); - -/** Backup routes */ export const adminHandlers = defineRoutes({ "GET /admin/backup": handleBackupGet, "GET /admin/backup/download/:filename": handleBackupDownload, "POST /admin/backup/create": handleBackupCreate, - "POST /admin/backup/restore": handleBackupRestore, - "POST /admin/backup/restore/confirm": handleBackupRestoreConfirm, }); diff --git a/src/locales/en/backup.json b/src/locales/en/backup.json index fd37a93787..e5ec02c08f 100644 --- a/src/locales/en/backup.json +++ b/src/locales/en/backup.json @@ -1,37 +1,23 @@ { - "backup.page_title": "Database Backup", - "backup.heading": "Database Backup & Restore", + "backup.page_title": "Database backup", + "backup.heading": "Database backup and restore", "backup.guide_link": "Backup guide", "backup.local_database_warning": "Backup and restore is designed for remote databases (libsql://). You are currently using a local database.", "backup.storage_not_configured": "Storage is not configured. Backups require Bunny CDN or local storage to be enabled.", - "backup.encryption_key_heading": "Encryption Key", + "backup.encryption_key_heading": "Encryption key", "backup.encryption_key_description": "You will need this key to restore a backup to a different site. Store it securely — it cannot be recovered.", - "backup.create_backup_heading": "Create Backup", + "backup.create_backup_heading": "Create backup", "backup.create_backup_description": "Creates a .zip archive containing a .sql file for each database table. Backups are not encrypted (the sensitive contents are already encrypted at the field level).", - "backup.create_button": "Create Backup Now", - "backup.existing_backups_heading": "Existing Backups", + "backup.create_button": "Create backup now", + "backup.existing_backups_heading": "Existing backups", "backup.no_backups_found": "No backups found.", "backup.table_timestamp": "Timestamp", "backup.table_size": "Size", "backup.download_link": "Download", - "backup.restore_heading": "Restore from Backup", - "backup.restore_warning": "Warning: Restoring will delete all current data and replace it with the backup contents. This cannot be undone.", - "backup.backup_file_label": "Backup file (.zip)", - "backup.upload_button": "Upload & Review", + "backup.restore_heading": "Restore from backup", + "backup.restore_console_description": "

Run deno task restore <backup.zip> from a checkout of this app. Put the target database's DB_URL and DB_TOKEN in .env first.

A restore cannot run on this page because a large restore exceeds the edge request limit.

", "backup.retention_count": "{count, plural, one {There is 1 backup} other {There are # backups}}", "backup.retention_kept": ", shown newest first. Up to {maxBackups} are kept — ", "backup.retention_remaining": "{remaining} more can be created before the oldest is purged.", - "backup.retention_purge": "the next will purge the oldest ({oldest}).", - "backup.restore_confirm_phrase": "This will restore my whole database to an earlier state. Existing info will be lost. I understand that this is dangerous.", - "backup.restore_button": "Restore Database", - "backup.confirmation_label": "Confirmation phrase", - "backup.confirm_restore_title": "Confirm Restore", - "backup.confirm_restore_heading": "Confirm Database Restore", - "backup.schema_mismatch_warning": "Schema mismatch: This backup was created with a different database schema version. The restore will apply current migrations after importing data, but some data may be incompatible.", - "backup.restore_confirmation_intro": "You are about to restore from an uploaded backup containing {lineCount} SQL statements. This will:", - "backup.restore_step_drop_tables": "Drop all existing tables", - "backup.restore_step_recreate_schema": "Recreate the database schema", - "backup.restore_step_import_data": "Import all data from the backup", - "backup.restore_cannot_undo": "This action cannot be undone. Type", - "backup.restore_type_below": "below to confirm." + "backup.retention_purge": "the next will purge the oldest ({oldest})." } diff --git a/src/locales/en/guide-domains.json b/src/locales/en/guide-domains.json index 09a2fdc561..5704aac41d 100644 --- a/src/locales/en/guide-domains.json +++ b/src/locales/en/guide-domains.json @@ -10,7 +10,7 @@ "guide.q.available_settings": "What settings are available?", "guide.a.available_settings": "

The Settings page (owners only) lets you configure:

  • Business email — receives admin notification emails, used as reply-to on attendee confirmations, and included in webhook payloads
  • Payment provider — Stripe, Square, SumUp, or none
  • Booking fee — percentage-based fee added to ticket prices (requires a payment provider)
  • Header image — upload a logo or banner shown at the top of every page
  • Embed hosts — restrict which websites can embed your booking forms
  • Terms and conditions — attendees must agree before booking
  • Features — publish Site and show or hide optional admin tools
  • Site theme — light or dark
  • Admin password — change your login password

Your country — which sets your timezone, currency, and phone prefix — is chosen once during initial setup and cannot be changed here afterwards.

", "guide.q.advanced_settings": "What are Advanced Settings?", - "guide.a.advanced_settings": "

The main Settings page has a link to Advanced Settings for less common configuration. Advanced settings include:

  • Public API — enable the JSON API for external integrations
  • Apple Wallet — configure pass signing certificates for Add to Apple Wallet
  • Google Wallet — configure service account credentials for Add to Google Wallet
  • Email provider — choose and configure your email sending service
  • Email templates — customise confirmation and admin notification emails using Liquid syntax
  • Host subdomain — register a pretty subdomain for your site (when enabled by server administrator)
  • Custom domain — set up a custom domain for your site (Bunny CDN only)
  • Backups — create, download, and restore full database backups (requires CDN storage)
  • Software updates — check for and install new versions
  • Database reset — permanently delete all data
", + "guide.a.advanced_settings": "

The main Settings page has a link to Advanced Settings for less common configuration. Advanced settings include:

  • Public API — enable the JSON API for external integrations
  • Apple Wallet — configure pass signing certificates for Add to Apple Wallet
  • Google Wallet — configure service account credentials for Add to Google Wallet
  • Email provider — choose and configure your email sending service
  • Email templates — customise confirmation and admin notification emails using Liquid syntax
  • Host subdomain — register a pretty subdomain for your site (when enabled by server administrator)
  • Custom domain — set up a custom domain (Bunny CDN only)
  • Backups — create and download full database backups (requires CDN storage)
  • Software updates — check for and install new versions
  • Database reset — permanently delete all data
", "guide.q.what_is_debug_page": "What is the debug page?", "guide.a.what_is_debug_page": "

The debug page at /admin/debug shows the configuration status of all integrated services (payments, email, Apple Wallet, Google Wallet, storage, CDN, notifications, database) without revealing any secrets or API keys. It's useful for troubleshooting setup issues — you can quickly see which services are configured and which are missing.

The page also lists every tunable system limit (file-size caps, session and URL expiry windows, login lockout, database pruning retention) alongside its default and current value. Any limit can be overridden by setting the matching environment variable to a positive integer; overridden values are highlighted. A "Database pruning" table at the bottom shows when each short-lived table was last cleaned up.

", "guide.q.can_i_use_both_a_subdomain_and": "Can I use both a subdomain and a custom domain?", diff --git a/src/locales/en/guide-operations.json b/src/locales/en/guide-operations.json index 6a7bd3d76a..3e33838e94 100644 --- a/src/locales/en/guide-operations.json +++ b/src/locales/en/guide-operations.json @@ -11,7 +11,7 @@ "guide.q.hosting_and_images": "Do you help with hosting and images?", "guide.a.hosting_and_images": "

Yes to both. I can set you up on your own Bunny CDN account (or another host) and handle the technical configuration. I also design listing images if you need them. Get in touch and we'll figure out exactly what you need.

", "guide.q.how_do_i_restore_from_a_backup": "How do I restore from a backup?", - "guide.a.how_do_i_restore_from_a_backup": "

On the Backups page, upload a .zip backup file using the restore form. You'll see a summary of how many SQL statements it contains and whether the schema version matches. Type the full confirmation phrase to proceed. Warning: restoring drops all existing tables and replaces them with the backup contents. This cannot be undone.

", + "guide.a.how_do_i_restore_from_a_backup": "

Put the target database's DB_URL and DB_TOKEN in .env. Run deno task restore <backup.zip> from a checkout of this app. The task checks the file and asks you to confirm before it deletes any data.

", "guide.q.are_old_backups_deleted_automatically": "Are old backups deleted automatically?", "guide.a.are_old_backups_deleted_automatically": "

Yes. Only the most recent backups are kept (30 by default, configurable via MAX_BACKUPS). When a new backup is created beyond that limit, the oldest is purged automatically. The Backups page shows how many you have and when the oldest will be removed. Automatic pre-migration backups count towards this limit too.

", "guide.q.do_backups_require_any_special_configuration": "Do backups require any special configuration?", @@ -23,7 +23,7 @@ "guide.q.what_does_the_version_number_mean": "What does the version number mean?", "guide.a.what_does_the_version_number_mean": "

Versions use the format vYYYY-MM-DD-HHMMSS — the UTC date and time the release was built, to the second. Larger timestamps are newer, and your installation compares its own build time against the latest release tag to decide whether an update is available. Because the tag is generated at build time and pushed as the git tag, the version on the update page always matches the release on GitHub.

", "guide.q.what_is_the_backup_feature": "What is the backup feature?", - "guide.a.what_is_the_backup_feature": "

The Backups page (owners only, under Settings → Backups) lets you create and restore full database backups. Each backup is a .zip archive containing SQL statements for every table. Backups are stored on your configured CDN storage (Bunny CDN).

", + "guide.a.what_is_the_backup_feature": "

The Backups page (owners only, under Settings → Backups) lets you create and download full database backups. Each backup is a .zip archive containing SQL statements for every table. Backups are stored on your configured CDN storage (Bunny CDN). Restore them with the console task shown on the page.

", "guide.q.how_do_i_create_a_backup": "How do I create a backup?", "guide.a.how_do_i_create_a_backup": "

Go to Backups and click Create Backup Now. The system exports all database tables into a .zip file and uploads it to your storage. Previous backups are listed with their timestamps and can be downloaded at any time.

", "guide.q.what_is_the_encryption_key_shown_on": "What is the encryption key shown on the backup page?", diff --git a/src/restore.ts b/src/restore.ts new file mode 100644 index 0000000000..2c010f318a --- /dev/null +++ b/src/restore.ts @@ -0,0 +1,30 @@ +import { type RestoreCliDeps, runRestoreCli } from "#scripts/restore-lib.ts"; +import type { ScriptIo } from "#scripts/script-runner.ts"; +import { inspectBackupZip, restoreFromZip } from "#shared/db/backup.ts"; +import { readRecordedScriptCommit } from "#shared/update.ts"; + +type RestoreTaskDeps = Omit; +type ScriptRunner = ( + run: (io: ScriptIo) => Promise, +) => Promise; +const RESTORE_ENV_KEYS = ["DB_ENCRYPTION_KEY", "DB_TOKEN", "DB_URL"] as const; + +const productionRestoreDeps = (): RestoreTaskDeps => ({ + inspectBackupZip, + prompt, + readFile: (path) => Deno.readFile(path), + readRecordedScriptCommit, + restoreFromZip, +}); + +export const runRestoreTask = async ( + fileEnv: Record, + setEnv: (key: string, value: string | undefined) => void, + runScript: ScriptRunner, + deps: RestoreTaskDeps = productionRestoreDeps(), +): Promise => { + for (const key of RESTORE_ENV_KEYS) { + setEnv(key, fileEnv[key]); + } + await runScript((io) => runRestoreCli({ ...deps, ...io })); +}; diff --git a/src/shared/db/backup.ts b/src/shared/db/backup.ts index 4a0fe4e222..d29b389d62 100644 --- a/src/shared/db/backup.ts +++ b/src/shared/db/backup.ts @@ -4,21 +4,29 @@ * * Key design decisions: * - Tables are exported/restored in SCHEMA order (FK-dependency safe) - * - Restore runs all INSERTs in a single transaction via executeBatch - * - SQL statements are delimited by ";\n" to handle embedded newlines in values + * - Restore uses bounded transactions so large imports do not time out + * - SQL splitting preserves semicolons inside quoted multiline values * - Backups are stored unencrypted (sensitive data is already field-level encrypted) * - manifest.json enables preflight schema compatibility checks before restore */ import { unzipSync, zipSync } from "fflate"; +import * as v from "valibot"; +import { compact, reduce, sum } from "#fp"; +import { utf8ByteLength } from "#shared/bytes.ts"; import { createBackup, type TableBackup } from "#shared/db/backup-snapshot.ts"; import { backupKey, backupTimestamp, pruneOldBackups, } from "#shared/db/backup-storage.ts"; -import { executeBatch } from "#shared/db/client.ts"; +import { execute, executeBatch } from "#shared/db/client.ts"; import { MIGRATION_IDS } from "#shared/db/migrations/registry.ts"; +import { + RESTORE_DEFERRED_INDEXES, + RESTORE_DEFERRED_TRIGGERS, +} from "#shared/db/migrations/schema/restore-work.ts"; +import { noArgStatements } from "#shared/db/migrations/schema-sync.ts"; import { clearAllCaches, LATEST_UPDATE, @@ -34,6 +42,7 @@ import { import { namedError } from "#shared/named-error.ts"; import { nowIso } from "#shared/now.ts"; import { uploadRaw } from "#shared/storage.ts"; +import { integerAtLeast } from "#shared/validation/number.ts"; /** Thrown by restoreFromSql after resetDatabase() runs but a later step fails, * so callers can distinguish a post-reset failure (DB wiped) from a pre-reset @@ -42,28 +51,143 @@ export class PostResetError extends namedError("PostResetError") {} // ─── Types ────────────────────────────────────────────────────── -/** Metadata stored in manifest.json inside the backup zip */ -export type BackupManifest = { - schemaHash: string; - latestUpdate: string; - timestamp: string; - tables: Record; +/** Metadata stored in manifest.json inside the backup zip. */ +const BackupManifestSchema = v.object({ + latestUpdate: v.string(), + schemaHash: v.string(), + tables: v.record(v.string(), integerAtLeast(0)), + timestamp: v.string(), +}); + +export type BackupManifest = v.InferOutput; + +export interface BackupInspection { + manifest: BackupManifest | null; + statementCount: number; +} + +const RESTORE_STAGES = [ + "checking", + "resetting", + "rebuilding", + "importing", + "clearing-caches", +] as const; + +export type RestoreStage = (typeof RESTORE_STAGES)[number]; + +export interface RestoreProgress { + stage: RestoreStage; + statementCount: number; +} + +export type RestoreProgressHandler = (progress: RestoreProgress) => void; + +const ignoreRestoreProgress: RestoreProgressHandler = () => {}; + +// Each dump statement already contains up to 100 rows. Keep both the work and +// HTTP body of a remote libSQL transaction bounded while combining enough work +// to avoid one network round trip per statement. +const RESTORE_BATCH_STATEMENT_LIMIT = 50; +const RESTORE_BATCH_BYTE_LIMIT = 512 * 1024; +const RESTORE_CORE_TABLES = [ + "settings", + "schema_migrations", + "attendee_statuses", +] as const; + +type RestoreBatchState = { + completed: string[][]; + current: { bytes: number; statements: string[] }; +}; + +const restoreBatches = ([first, ...statements]: [ + string, + ...string[], +]): string[][] => { + const initial: RestoreBatchState = { + completed: [], + current: { bytes: utf8ByteLength(first), statements: [first] }, + }; + const result = reduce((state: RestoreBatchState, statement: string) => { + const statementBytes = utf8ByteLength(statement); + if ( + state.current.statements.length === RESTORE_BATCH_STATEMENT_LIMIT || + state.current.bytes + statementBytes > RESTORE_BATCH_BYTE_LIMIT + ) { + state.completed.push(state.current.statements); + state.current = { bytes: statementBytes, statements: [statement] }; + } else { + state.current.statements.push(statement); + state.current.bytes += statementBytes; + } + return state; + }, initial)(statements); + result.completed.push(result.current.statements); + return result.completed; +}; + +/** Remove derived write-time work while rows are loaded. Unique indexes and + * validation triggers stay active so corrupt backup rows fail immediately. */ +const removeDeferredSchemaWork = async (): Promise => { + const drops = [ + ...RESTORE_DEFERRED_INDEXES.map( + ({ name }) => `DROP INDEX IF EXISTS ${name}`, + ), + ...RESTORE_DEFERRED_TRIGGERS.map( + ({ name }) => `DROP TRIGGER IF EXISTS ${name}`, + ), + ]; + await executeBatch(noArgStatements(drops)); +}; + +/** Rebuild deferred indexes one transaction at a time so indexing a populated + * database cannot turn into one transaction that exceeds libSQL's deadline. */ +const rebuildDeferredSchemaWork = async (): Promise => { + for (const { sql } of RESTORE_DEFERRED_INDEXES) { + await executeBatch(noArgStatements([sql])); + } + for (const trigger of RESTORE_DEFERRED_TRIGGERS) { + await execute(trigger.sql); + } }; +interface OpenBackup { + decoder: TextDecoder; + files: Record; +} + +const openBackup = (zipData: Uint8Array): OpenBackup => ({ + decoder: new TextDecoder(), + files: unzipSync(zipData), +}); + // ─── Helpers ──────────────────────────────────────────────────── /** * Split SQL text into individual statements. - * Splits on ";\n" boundaries, which is the format produced by exportTable. + * Quoted values are consumed as whole tokens, so semicolons in multiline text + * stay inside their INSERT. Both SQL quote forms escape themselves by doubling. */ export const splitStatements = (sql: string): string[] => { if (sql.trim() === "") return []; - return sql - .replace(/\r\n/g, "\n") - .split(/;\s*\n/) - .map((s) => s.trim()) - .filter((s) => s !== "") - .map((s) => (s.endsWith(";") ? s : `${s};`)); + const normalized = sql.replace(/\r\n/g, "\n"); + const statements: string[] = []; + let start = 0; + const quoteOrStatementEnd = /'(?:''|[^'])*'|"(?:""|[^"])*"|;/gs; + + for (const token of normalized.matchAll(quoteOrStatementEnd)) { + if (token[0] !== ";") continue; + const statement = normalized.slice(start, token.index).trim(); + if (statement !== "") statements.push(`${statement};`); + start = token.index + 1; + } + + const remainder = normalized.slice(start).trim(); + if (remainder !== "") { + statements.push(`${remainder};`); + } + return statements; }; /** Build the manifest object for a backup */ @@ -108,50 +232,130 @@ export const createAndUploadBackup = async (): Promise => { // ─── Restore ──────────────────────────────────────────────────── -/** Validate that a parsed object has the expected BackupManifest shape */ -const isValidManifest = (v: unknown): v is BackupManifest => - typeof v === "object" && - v !== null && - typeof (v as Record).schemaHash === "string" && - typeof (v as Record).latestUpdate === "string" && - typeof (v as Record).timestamp === "string" && - typeof (v as Record).tables === "object" && - (v as Record).tables !== null; - -/** Read and parse manifest.json from a backup zip. Returns null if missing or invalid. */ -export const readManifest = (zipData: Uint8Array): BackupManifest | null => { - const files = unzipSync(zipData); +/** Read and validate manifest.json. Older backups without one remain restorable. */ +const readManifestOrNull = ( + files: Record, +): BackupManifest | null => { const manifestBytes = files["manifest.json"]; if (!manifestBytes) return null; const parsed: unknown = JSON.parse(new TextDecoder().decode(manifestBytes)); - return isValidManifest(parsed) ? parsed : null; + const result = v.safeParse(BackupManifestSchema, parsed); + if (!result.success) throw new Error("Backup manifest is invalid"); + return result.output; }; -/** Count SQL statements across all .sql files in a zip archive */ -export const countZipStatements = (zipData: Uint8Array): number => { - const files = unzipSync(zipData); - const decoder = new TextDecoder(); - let count = 0; - for (const name of Object.keys(files)) { - if (!name.endsWith(".sql")) continue; - const content = decoder.decode(files[name]!); - if (content.trim() === "") continue; - count += splitStatements(content).length; +interface InspectedSqlFile { + statementCount: number; + table: string; +} + +const insertTable = (statement: string, file: string): string => { + const match = /^INSERT\s+INTO\s+(?"?[A-Za-z_]\w*"?)\s*\(/i.exec( + statement, + ); + const table = match?.groups?.table; + if (table === undefined) { + throw new Error( + `Backup file ${file} contains a statement that is not an INSERT`, + ); } - return count; + return table.replaceAll('"', ""); }; +const inspectSqlFiles = ({ decoder, files }: OpenBackup): InspectedSqlFile[] => + Object.entries(files) + .filter(([name]) => name.endsWith(".sql")) + .map(([name, content]) => { + const statements = splitStatements(decoder.decode(content)); + if ( + statements.some( + (statement) => utf8ByteLength(statement) > RESTORE_BATCH_BYTE_LIMIT, + ) + ) { + throw new Error( + `Backup SQL statement is larger than ${RESTORE_BATCH_BYTE_LIMIT} bytes: ${name}`, + ); + } + const table = name.slice(0, -4); + if (SCHEMA_TABLE_NAMES.includes(table)) { + const wrongTable = statements + .map((statement) => insertTable(statement, name)) + .find((target) => target !== table); + if (wrongTable !== undefined) { + throw new Error( + `Backup file ${name} contains SQL for ${wrongTable} instead of ${table}`, + ); + } + } + return { statementCount: statements.length, table }; + }); + +const inspectOpenBackup = (backup: OpenBackup): BackupInspection => { + const sqlFiles = inspectSqlFiles(backup); + const unsupported = sqlFiles + .filter( + ({ statementCount, table }) => + statementCount > 0 && !SCHEMA_TABLE_NAMES.includes(table), + ) + .map(({ table }) => table); + if (unsupported.length > 0) { + throw new Error( + `Backup contains data for tables this app cannot restore: ${unsupported.join(", ")}`, + ); + } + + const statementCount = sum(sqlFiles.map((file) => file.statementCount)); + const manifest = readManifestOrNull(backup.files); + const statementsByTable = new Map( + sqlFiles.map(({ statementCount, table }) => [table, statementCount]), + ); + const legacyRequiredTables = statementCount === 0 ? [] : RESTORE_CORE_TABLES; + const requiredTables = + manifest === null + ? legacyRequiredTables + : Object.entries(manifest.tables) + .filter(([, rowCount]) => rowCount > 0) + .map(([table]) => table); + const missing = requiredTables.filter( + (table) => !statementsByTable.get(table), + ); + if (missing.length > 0) { + const reason = + manifest === null + ? "Backup without a manifest is missing required data for tables" + : "Backup is missing data for tables"; + throw new Error(`${reason}: ${missing.join(", ")}`); + } + if (statementCount === 0) { + throw new Error("Backup contains no restorable SQL statements"); + } + return { + manifest, + statementCount, + }; +}; + +/** Read the manifest and count SQL statements in one unzip pass. */ +export const inspectBackupZip = (zipData: Uint8Array): BackupInspection => + inspectOpenBackup(openBackup(zipData)); + /** * Restore the database from SQL content. * Drops all tables, reinitializes the schema, then executes all SQL - * statements in a single transaction via executeBatch. + * statements in bounded sequential transactions. * * Refuses a dump from a newer build BEFORE wiping anything: replaying it here * would silently discard newer-schema data (tables this build's schema lacks * are skipped), making an accidental rollback look like a successful restore. */ -export const restoreFromSql = async (sql: string): Promise => { +export const restoreFromSql = async ( + sql: string, + onProgress: RestoreProgressHandler = ignoreRestoreProgress, +): Promise => { const statements = splitStatements(sql); + const report = (stage: RestoreStage): void => + onProgress({ stage, statementCount: statements.length }); + report("checking"); const migrations = dumpMigrationState(statements, MIGRATION_IDS); if (migrations.fromNewerBuild.length > 0) { throw new Error( @@ -167,6 +371,7 @@ export const restoreFromSql = async (sql: string): Promise => { // sessions table already removed) are also routed as post-reset errors. let postResetErr: string | undefined; try { + report("resetting"); await resetDatabase(); // The database was just wiped, so rebuild the schema with unconditional // IF NOT EXISTS creates. Never consult the database here — neither @@ -176,54 +381,60 @@ export const restoreFromSql = async (sql: string): Promise => { // schema verification against the wiped primary ("missing table // settings") or skipped the CREATEs and died at the import ("no such // table: settings"). + report("rebuilding"); await rebuildWipedSchema(); + await removeDeferredSchemaWork(); - // Roll the seed-data deletes into the same executeBatch transaction as the - // import so that a failed import rolls the deletes back too, leaving the DB - // in the clean post-initDb state rather than a mix of empty seed tables and - // partially applied backup rows. Columns the dump writes that a migration - // the backup predates has since dropped are re-added first, so the replayed - // rows land intact for that pending migration to reshape on the next boot - // (see restore-legacy-columns.ts) — but only when the dump actually has - // pending migrations to consume them: with none pending, an unknown column - // is corruption, and the INSERT must fail loudly instead. - await executeBatch( - [ - "DELETE FROM settings", - "DELETE FROM schema_migrations", - "DELETE FROM attendee_statuses", - ...(migrations.hasPending ? legacyColumnRestores(statements) : []), - ...statements, - ].map((s) => ({ args: [], sql: s })), - ); - - // Clear all module-level caches — the backup may carry different data for - // every table, so any warm cache is now stale. clearAllCaches() covers the - // same set as resetDatabase()'s finally block, including caches (holidays, - // logistics-agents, sessions, settings) that a partial list would miss. - clearAllCaches(); + // Columns a pending migration has since dropped are re-added before replay + // so the migration can consume them on the next boot. Derived indexes and + // triggers stay absent during this load, avoiding per-row maintenance and + // preserving aggregate values already stored in the backup. Data checks + // remain active. + report("importing"); + const restoreStatements: [string, ...string[]] = [ + `DELETE FROM ${RESTORE_CORE_TABLES[0]}`, + ...RESTORE_CORE_TABLES.slice(1).map((table) => `DELETE FROM ${table}`), + ...(migrations.hasPending ? legacyColumnRestores(statements) : []), + ...statements, + ]; + for (const batch of restoreBatches(restoreStatements)) { + await executeBatch(noArgStatements(batch)); + } + await rebuildDeferredSchemaWork(); } catch (err) { postResetErr = String(err); } if (postResetErr !== undefined) { + // Batches committed before a later failure cannot be rolled back together. + // Leave a complete empty schema instead of a partial copy of the backup. + await resetDatabase(); + await rebuildWipedSchema(); throw new PostResetError(postResetErr); } + + // The database restore is complete. Cache or progress failures must surface, + // but must not enter the partial-restore cleanup and erase restored rows. + clearAllCaches(); + report("clearing-caches"); }; /** * Restore the database from a zip archive. * Files are replayed in SCHEMA order (FK-dependency safe), not alphabetically. */ -export const restoreFromZip = async (zipData: Uint8Array): Promise => { - const files = unzipSync(zipData); - const decoder = new TextDecoder(); - const allSql: string[] = []; - - // Iterate in SCHEMA order for FK safety - for (const table of SCHEMA_TABLE_NAMES) { - const content = files[`${table}.sql`]; - if (content) allSql.push(decoder.decode(content)); - } +export const restoreFromZip = async ( + zipData: Uint8Array, + onProgress: RestoreProgressHandler = ignoreRestoreProgress, +): Promise => { + const backup = openBackup(zipData); + inspectOpenBackup(backup); + const { decoder, files } = backup; + // Iterate in SCHEMA order for FK safety. + const allSql = compact( + SCHEMA_TABLE_NAMES.map((table) => files[`${table}.sql`]), + ).map((content) => decoder.decode(content)); - await restoreFromSql(allSql.join("\n")); + // Every generated table file ends its statements with semicolons, so no + // extra separator is needed between files. + await restoreFromSql(allSql.join(""), onProgress); }; diff --git a/src/shared/db/migrations/schema/restore-work.ts b/src/shared/db/migrations/schema/restore-work.ts new file mode 100644 index 0000000000..d8f295b70a --- /dev/null +++ b/src/shared/db/migrations/schema/restore-work.ts @@ -0,0 +1,38 @@ +import { compact } from "#fp"; +import { fullSchemaCreateStatements } from "#shared/db/migrations/schema-sync.ts"; +import { TRIGGERS } from "./triggers.ts"; +import type { Trigger } from "./types.ts"; + +export interface RestoreDeferredIndex { + name: string; + sql: string; +} + +const NON_UNIQUE_INDEX_PREFIX = "CREATE INDEX IF NOT EXISTS "; +const restoreDeferredIndexOrNull = ( + sql: string, +): RestoreDeferredIndex | null => { + if (!sql.startsWith(NON_UNIQUE_INDEX_PREFIX)) return null; + const nameEnd = sql.indexOf(" ON ", NON_UNIQUE_INDEX_PREFIX.length); + return { + name: sql.slice(NON_UNIQUE_INDEX_PREFIX.length, nameEnd), + sql, + }; +}; + +/** Non-unique indexes rebuilt after bulk restore rows are loaded. */ +export const RESTORE_DEFERRED_INDEXES: RestoreDeferredIndex[] = compact( + fullSchemaCreateStatements().map(restoreDeferredIndexOrNull), +); + +const RESTORE_ACTIVE_TRIGGER_NAMES: readonly string[] = [ + "trg_attendee_answers_validate_insert", + "trg_attendee_answers_validate_update", + "trg_attendees_validate_status_insert", + "trg_attendees_validate_status_update", +]; + +/** Derived-state triggers rebuilt after stored backup values are loaded. */ +export const RESTORE_DEFERRED_TRIGGERS: Trigger[] = TRIGGERS.filter( + ({ name }) => !RESTORE_ACTIVE_TRIGGER_NAMES.includes(name), +); diff --git a/src/shared/update.ts b/src/shared/update.ts index 8d8a582018..fb5b7d0510 100644 --- a/src/shared/update.ts +++ b/src/shared/update.ts @@ -13,6 +13,7 @@ import { executeBatchWithoutCacheInvalidation, inPlaceholders, queryAll, + queryOnePrimary, } from "#shared/db/client.ts"; import { denoDeployApi } from "#shared/deno-deploy-api.ts"; import { logDebug } from "#shared/logger.ts"; @@ -126,10 +127,6 @@ const readSettingMarkers = async ( return new Map(rows.map((row) => [row.key, row.value])); }; -/** Read a plaintext settings marker's value, or "" when the row is absent. */ -const readSettingMarker = async (key: string): Promise => - (await readSettingMarkers([key])).get(key) ?? ""; - /** * Record the running build's version and commit into settings so a parent * host (or a backup) can read them back. Runs on every isolate boot: one @@ -180,8 +177,13 @@ export const recordScriptVersion = async (): Promise => { * unset (older backups, or development builds). A restore reads this from the * just-restored data to tell the operator which commit to redeploy. */ -export const readRecordedScriptCommit = (): Promise => - readSettingMarker(CURRENT_SCRIPT_COMMIT_KEY); +export const readRecordedScriptCommit = async (): Promise => + ( + await queryOnePrimary<{ value: string }>( + "SELECT value FROM settings WHERE key = ?", + [CURRENT_SCRIPT_COMMIT_KEY], + ) + )?.value ?? ""; /** * Format a build timestamp for display. diff --git a/src/ui/templates/admin/backup.tsx b/src/ui/templates/admin/backup.tsx index 0591cafadc..c2fe84b49d 100644 --- a/src/ui/templates/admin/backup.tsx +++ b/src/ui/templates/admin/backup.tsx @@ -5,12 +5,9 @@ /* jscpd:ignore-start */ import { t } from "#i18n"; import { Raw } from "#shared/jsx/jsx-runtime.ts"; -import type { AdminSession } from "#shared/types.ts"; import { flashDataPage } from "#templates/admin/admin-page.tsx"; -import { ConfirmPage } from "#templates/admin/confirm-page.tsx"; import { GuideFooter } from "#templates/components/actions.tsx"; import { DataTable } from "#templates/components/data-table.tsx"; -import { ErrorNote } from "#templates/components/error.tsx"; import { SaveForm } from "#templates/components/save-form.tsx"; /* jscpd:ignore-end */ @@ -132,77 +129,17 @@ export const adminBackupPage = flashDataPage( )} - -
-
-

{t("backup.restore_heading")}

-

- -

-
- - - -
)} +
+
+

{t("backup.restore_heading")}

+ +
+
{t("backup.guide_link")} ), ); - -export const RESTORE_CONFIRM_PHRASE = - "This will restore my whole database to an earlier state. Existing info will be lost. I understand that this is dangerous."; - -export const adminRestoreConfirmPage = ( - session: AdminSession, - filename: string, - lineCount: number, - error?: string, - schemaMismatch?: boolean, -): string => - ConfirmPage({ - action: "/admin/backup/restore/confirm", - active: "/admin/backup", - buttonText: t("backup.restore_button"), - children: ( - <> -

{t("backup.confirm_restore_heading")}

- {schemaMismatch && ( - - - - )} -

- -

-
    -
  • {t("backup.restore_step_drop_tables")}
  • -
  • {t("backup.restore_step_recreate_schema")}
  • -
  • {t("backup.restore_step_import_data")}
  • -
-

- {" "} - {RESTORE_CONFIRM_PHRASE} {t("backup.restore_type_below")} -

- - ), - error, - hiddenFields: { backup_filename: filename }, - id: "backup-restore-confirm", - label: t("backup.confirmation_label"), - name: RESTORE_CONFIRM_PHRASE, - session, - title: t("backup.confirm_restore_title"), - }); diff --git a/test/e2e/booking.test.ts b/test/e2e/booking.test.ts index 871fc7c1e3..d60239f194 100644 --- a/test/e2e/booking.test.ts +++ b/test/e2e/booking.test.ts @@ -14,7 +14,7 @@ // jscpd:ignore-start import { expect } from "@std/expect"; import { describe, it as test } from "@std/testing/bdd"; -import { RESTORE_CONFIRM_PHRASE } from "#templates/admin/backup.tsx"; +import { restoreFromZip } from "#shared/db/backup.ts"; import { invalidateAllCaches, setupAndLogin, @@ -204,10 +204,10 @@ describe("e2e: full booking flow", () => { await withLocalStorageEnabled(async () => { // 14. Navigate to backup page and create a backup await browser.visit("/admin/backup"); - expect(browser.containsText("Database Backup")).toBe(true); - expect(browser.containsText("Encryption Key")).toBe(true); + expect(browser.containsText("Database backup")).toBe(true); + expect(browser.containsText("Encryption key")).toBe(true); - await browser.submitForm({}, "Create Backup Now"); + await browser.submitForm({}, "Create backup now"); expect(browser.containsText("Database backup created")).toBe(true); // 15. Download the backup zip for later restore @@ -234,31 +234,12 @@ describe("e2e: full booking flow", () => { // 20. Verify the listing and attendee are gone after reset expect(browser.containsText("Summer Concert")).toBe(false); - // 21. Navigate to backup page and restore from the saved zip - await browser.visit("/admin/backup"); - await browser.submitFormWithFile( - "backup_file", - "backup.zip", - backupZip, - {}, - "Upload", - ); - // Should show the restore confirmation page - expect(browser.containsText("Confirm Database Restore")).toBe(true); - expect(browser.containsText("SQL statements")).toBe(true); - - // 22. Confirm the restore - await browser.submitForm( - { confirm_identifier: RESTORE_CONFIRM_PHRASE }, - "Restore Database", - ); - // The restore handler clears the session cookie and redirects to - // /admin/login, so the flash appears on the login page even though the - // operator's post-reset session is no longer valid. - expect(browser.containsText("Database restored from backup")).toBe(true); + // 21. Restore out of band, as the Deno task does. A realistic restore + // cannot fit within Bunny's per-request subrequest budget. + await restoreFromZip(backupZip); invalidateAllCaches(); - // 23. Log in again (restore wiped sessions) + // 22. Log in again (restore wiped sessions) await browser.visit("/admin/"); if (browser.currentHtml.includes("Login")) { await browser.submitForm( @@ -270,7 +251,7 @@ describe("e2e: full booking flow", () => { await browser.clickLink("Back to dashboard"); } - // 24. Verify the listing and attendee are back + // 23. Verify the listing and attendee are back expect(browser.containsText("Summer Concert")).toBe(true); await browser.clickLink("Summer Concert"); const backId = browser.currentUrl.match(/\/admin\/listing\/(\d+)/)?.[1]; diff --git a/test/features/admin/backup/routes.test.ts b/test/features/admin/backup/routes.test.ts new file mode 100644 index 0000000000..5ef4fb98d0 --- /dev/null +++ b/test/features/admin/backup/routes.test.ts @@ -0,0 +1,25 @@ +import { expect } from "@std/expect"; +import { it as test } from "@std/testing/bdd"; +import { ADMIN_AREA_LOADERS } from "#routes/admin/area-loaders.ts"; +import { + getSubrequestUsage, + runWithSubrequestBudget, +} from "#shared/subrequest-budget.ts"; + +test("web restore paths are absent and use no subrequests", async () => { + const handlers = await ADMIN_AREA_LOADERS.backup.load(); + + await runWithSubrequestBudget(async () => { + for (const path of [ + "POST /admin/backup/restore", + "POST /admin/backup/restore/confirm", + ]) { + expect(Object.hasOwn(handlers, path)).toBe(false); + } + expect(getSubrequestUsage()).toEqual({ + database: 0, + external: 0, + total: 0, + }); + }); +}); diff --git a/test/features/admin/backup/server.test.ts b/test/features/admin/backup/server.test.ts new file mode 100644 index 0000000000..6759b042c9 --- /dev/null +++ b/test/features/admin/backup/server.test.ts @@ -0,0 +1,162 @@ +import { expect } from "@std/expect"; +import { describe, it as test } from "@std/testing/bdd"; +import { backupDir } from "#shared/db/backup-storage.ts"; +import { runWithStorageConfig, uploadRaw } from "#shared/storage.ts"; +import { + expectFlashRedirect, + expectHtmlResponse, + testRequiresAuth, +} from "#test-utils/assertions.ts"; +import { describeWithEnv } from "#test-utils/db.ts"; +import { createTestListing } from "#test-utils/db-helpers/listings.ts"; +import { TEST_STORAGE_ZONE } from "#test-utils/internal.ts"; +import { + awaitTestRequest, + installUrlHandler, + withFetchMock, + withStorageDisabled, +} from "#test-utils/mocks.ts"; +import { + adminFormPost, + adminGet, + createTestManagerSession, +} from "#test-utils/session.ts"; + +describeWithEnv("server (admin backup)", { db: true, storage: "local" }, () => { + describe("GET /admin/backup", () => { + testRequiresAuth("/admin/backup"); + + test("returns 403 for manager users", async () => { + const managerCookie = await createTestManagerSession(); + const response = await awaitTestRequest("/admin/backup", { + cookie: managerCookie, + }); + expect(response.status).toBe(403); + }); + + test("shows backup details and console restore instructions", async () => { + const response = await adminGet("/admin/backup"); + await expectHtmlResponse( + response, + 200, + "Database backup", + "Encryption key", + "deno task restore", + ); + }); + + test("highlights the backup navigation link", async () => { + const html = await (await adminGet("/admin/backup")).text(); + expect(html).toContain('class="active" href="/admin/backup"'); + }); + + test("shows the encryption key", async () => { + const html = await (await adminGet("/admin/backup")).text(); + expect(html).toContain("MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY="); + }); + + test("shows storage not configured when disabled", async () => { + await withStorageDisabled(async () => { + const html = await (await adminGet("/admin/backup")).text(); + expect(html).toContain("Storage is not configured"); + }); + }); + + test("shows restore instructions when storage listing fails", async () => { + await runWithStorageConfig(TEST_STORAGE_ZONE, () => + withFetchMock(async (originalFetch) => { + installUrlHandler(originalFetch, (url) => + url.includes("storage.bunnycdn.com") + ? Promise.reject(new Error("list failed")) + : null, + ); + const html = await (await adminGet("/admin/backup")).text(); + expect(html).toContain("deno task restore"); + expect(html).toContain("No backups found"); + }), + ); + }); + }); + + describe("POST /admin/backup/create", () => { + testRequiresAuth("/admin/backup/create", { + body: {}, + method: "POST", + }); + + test("creates a backup", async () => { + await createTestListing({ name: "Backup Test" }); + const { response } = await adminFormPost("/admin/backup/create"); + await expectFlashRedirect( + "/admin/backup", + "Database backup created", + )(response); + }); + + test("lists only valid backup files", async () => { + await uploadRaw(new Uint8Array(0), `${backupDir()}backup-stale.tmp`); + await adminFormPost("/admin/backup/create"); + const html = await (await adminGet("/admin/backup")).text(); + expect(html).not.toContain("backup-stale.tmp"); + expect(html).toContain(".zip"); + }); + + test("shows the timestamp from the stored filename", async () => { + await adminFormPost("/admin/backup/create"); + const html = await (await adminGet("/admin/backup")).text(); + const token = html.match( + /backup-(\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2}-\d{3}Z)\.zip/, + )?.[1]; + expect(token).toBeDefined(); + expect(html).toContain(`${token}`); + }); + + test("shows the retention summary", async () => { + await adminFormPost("/admin/backup/create"); + const html = await (await adminGet("/admin/backup")).text(); + expect(html).toContain("There is 1 backup"); + expect(html).toContain("Up to 30 are kept"); + }); + }); + + describe("GET /admin/backup/download/:filename", () => { + testRequiresAuth("/admin/backup/download/backup-local-test.zip"); + + test("rejects an invalid filename", async () => { + const response = await adminGet( + "/admin/backup/download/not-a-backup.txt", + ); + expect(response.status).toBe(400); + expect(await response.text()).toBe("Invalid backup filename"); + }); + + test("returns 404 for a missing backup", async () => { + const response = await adminGet( + "/admin/backup/download/backup-2024-01-15T12-30-00-000Z.zip", + ); + expect(response.status).toBe(404); + expect(await response.text()).toBe("Backup file not found"); + }); + + test("rejects path traversal", async () => { + const response = await adminGet( + "/admin/backup/download/backup-local-..%2F..%2Fetc.zip", + ); + expect(response.status).toBe(400); + }); + + test("downloads an existing backup", async () => { + await adminFormPost("/admin/backup/create"); + const html = await (await adminGet("/admin/backup")).text(); + const filename = html.match( + /\/admin\/backup\/download\/(backup-[^"]+\.zip)/, + )?.[1]; + expect(filename).toBeDefined(); + + const response = await adminGet(`/admin/backup/download/${filename}`); + expect(response.status).toBe(200); + expect(response.headers.get("content-type")).toBe("application/zip"); + expect((await response.arrayBuffer()).byteLength).toBeGreaterThan(0); + }); + }); +}); diff --git a/test/integration/admin-action-cookie.test.ts b/test/integration/admin-action-cookie.test.ts new file mode 100644 index 0000000000..18e631756c --- /dev/null +++ b/test/integration/admin-action-cookie.test.ts @@ -0,0 +1,28 @@ +import { expect } from "@std/expect"; +import { it as test } from "@std/testing/bdd"; +import { createActionHandler } from "#routes/admin/actions.ts"; +import { describeWithEnv } from "#test-utils/db.ts"; +import { mockFormRequest } from "#test-utils/mocks.ts"; +import { testCookie, testCsrfToken } from "#test-utils/session.ts"; + +describeWithEnv("admin action success cookie", { db: true }, () => { + test("adds the configured cookie to a successful response", async () => { + const cookie = await testCookie(); + const csrfToken = await testCsrfToken(); + const handler = createActionHandler({ + auth: "owner", + cookie: () => "restore_session=; Max-Age=0; Path=/", + execute: () => Promise.resolve(), + message: "Cookie action completed", + successRedirect: "/admin/test-cookie", + }); + + const response = await handler( + mockFormRequest("/admin/test-cookie", { csrf_token: csrfToken }, cookie), + ); + + expect(response.headers.get("set-cookie")).toContain( + "restore_session=; Max-Age=0; Path=/", + ); + }); +}); diff --git a/test/integration/code-quality.test.ts b/test/integration/code-quality.test.ts index b84821f835..c1ceb2488f 100644 --- a/test/integration/code-quality.test.ts +++ b/test/integration/code-quality.test.ts @@ -627,6 +627,7 @@ describe("code quality", () => { srcTsxContents, testContents, ALLOWED_TEST_HOOKS, + [scriptsContents, cliContents, e2eContents], ), ); } diff --git a/test/integration/confirm-page-options.test.ts b/test/integration/confirm-page-options.test.ts new file mode 100644 index 0000000000..608a80e942 --- /dev/null +++ b/test/integration/confirm-page-options.test.ts @@ -0,0 +1,21 @@ +import { expect } from "@std/expect"; +import { it as test } from "@std/testing/bdd"; +import { ConfirmPage } from "#templates/admin/confirm-page.tsx"; + +test("passes optional identifiers and hidden fields to the confirm form", () => { + const html = ConfirmPage({ + action: "/admin/example/confirm", + active: "/admin/settings", + buttonText: "Confirm", + hiddenFields: { backup_filename: "backup.zip" }, + id: "example-confirm", + label: "Name", + name: "Example", + session: { adminLevel: "owner" }, + title: "Confirm example", + }); + + expect(html).toContain('id="example-confirm"'); + expect(html).toContain('name="backup_filename"'); + expect(html).toContain('value="backup.zip"'); +}); diff --git a/test/integration/restore-task.test.ts b/test/integration/restore-task.test.ts new file mode 100644 index 0000000000..8ebb57c9a5 --- /dev/null +++ b/test/integration/restore-task.test.ts @@ -0,0 +1,304 @@ +import { expect } from "@std/expect"; +import { describe, it as test } from "@std/testing/bdd"; +import { zipSync } from "fflate"; +import { + RESTORE_CONFIRMATION, + RESTORE_USAGE, + type RestoreCliDeps, + runRestoreCli, +} from "#scripts/restore-lib.ts"; +import { inspectBackupZip, PostResetError } from "#shared/db/backup.ts"; +import { SCHEMA_HASH } from "#shared/db/migrations.ts"; +import { TEST_ENCRYPTION_KEY } from "#test-utils/internal.ts"; + +const encoder = new TextEncoder(); +const FULL_COMMIT = "0123456789abcdef0123456789abcdef01234567"; + +const cliEnv = ( + overrides: Record = {}, +): ((key: string) => string | undefined) => { + const values: Record = { + DB_ENCRYPTION_KEY: TEST_ENCRYPTION_KEY, + DB_URL: "file:target.db", + ...overrides, + }; + return (key) => values[key]; +}; + +const backupZip = ( + manifest: Record | null = { + latestUpdate: "Current database layout", + schemaHash: SCHEMA_HASH, + tables: { settings: 2 }, + timestamp: "2026-07-22T12:00:00.000Z", + }, +): Uint8Array => + zipSync({ + ...(manifest === null + ? { + "attendee_statuses.sql": encoder.encode( + "INSERT INTO attendee_statuses (id) VALUES (1);", + ), + "schema_migrations.sql": encoder.encode( + "INSERT INTO schema_migrations (id) VALUES ('initial');", + ), + } + : { "manifest.json": encoder.encode(JSON.stringify(manifest)) }), + "settings.sql": encoder.encode( + "INSERT INTO settings (key, value) VALUES ('one', '1');\n" + + "INSERT INTO settings (key, value) VALUES ('two', '2');\n", + ), + }); + +interface CliState { + deps: RestoreCliDeps; + promptMessages: string[]; + readPaths: string[]; + restored: Uint8Array[]; + stderr: string[]; + stdout: string[]; +} + +const cliState = (overrides: Partial = {}): CliState => { + const stdout: string[] = []; + const stderr: string[] = []; + const promptMessages: string[] = []; + const readPaths: string[] = []; + const restored: Uint8Array[] = []; + const zip = backupZip(); + return { + deps: { + args: ["backup.zip"], + getEnv: cliEnv(), + inspectBackupZip, + prompt: (message) => { + promptMessages.push(message); + return RESTORE_CONFIRMATION; + }, + readFile: (path) => { + readPaths.push(path); + return Promise.resolve(zip); + }, + readRecordedScriptCommit: () => Promise.resolve(""), + restoreFromZip: (data, onProgress) => { + restored.push(data); + for (const stage of [ + "checking", + "resetting", + "rebuilding", + "importing", + "clearing-caches", + ] as const) { + onProgress({ stage, statementCount: 2 }); + } + return Promise.resolve(); + }, + stderr: (line) => stderr.push(line), + stdout: (line) => stdout.push(line), + ...overrides, + }, + promptMessages, + readPaths, + restored, + stderr, + stdout, + }; +}; + +describe("restore task", () => { + test("requires one backup path", async () => { + const state = cliState({ args: [] }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toEqual([RESTORE_USAGE]); + expect(state.readPaths).toEqual([]); + }); + + test("requires DB_URL before reading the backup", async () => { + const state = cliState({ getEnv: () => undefined }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toEqual(["DB_URL is required in .env."]); + expect(state.readPaths).toEqual([]); + }); + + test("requires DB_TOKEN for every remote database URL", async () => { + for (const dbUrl of [ + "libsql://tickets.example.com", + "https://tickets.example.com", + ]) { + const state = cliState({ getEnv: cliEnv({ DB_URL: dbUrl }) }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toEqual([ + "DB_TOKEN is required in .env for a remote database.", + ]); + expect(state.readPaths).toEqual([]); + } + }); + + test("requires DB_ENCRYPTION_KEY before reading the backup", async () => { + const state = cliState({ + getEnv: (key) => (key === "DB_URL" ? "file:target.db" : undefined), + }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toEqual(["DB_ENCRYPTION_KEY is required in .env."]); + expect(state.readPaths).toEqual([]); + }); + + test("validates DB_ENCRYPTION_KEY before reading the backup", async () => { + const state = cliState({ + getEnv: cliEnv({ DB_ENCRYPTION_KEY: "c2hvcnQ=" }), + }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toEqual([ + "DB_ENCRYPTION_KEY must be 32 bytes (256 bits), got 5 bytes", + ]); + expect(state.readPaths).toEqual([]); + }); + + test("refuses an in-memory restore target", async () => { + const state = cliState({ + getEnv: cliEnv({ DB_URL: ":memory:" }), + }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toEqual([ + "DB_URL cannot be :memory: for a restore. Set it to the target database in .env.", + ]); + expect(state.readPaths).toEqual([]); + }); + + test("reports a backup file read failure", async () => { + const state = cliState({ + readFile: () => Promise.reject(new Error("permission denied")), + }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toEqual([ + "Could not read backup.zip: permission denied", + ]); + }); + + test("rejects an invalid ZIP before asking for confirmation", async () => { + const state = cliState({ + readFile: () => Promise.resolve(new Uint8Array([1, 2, 3])), + }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr[0]).toBe("backup.zip is not a valid database backup."); + expect(state.promptMessages).toEqual([]); + }); + + test("stops when the confirmation does not match", async () => { + const state = cliState({ prompt: () => "restore" }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toEqual([ + "Restore cancelled. The database was not changed.", + ]); + expect(state.restored).toEqual([]); + }); + + test("shows backup details and every restore phase", async () => { + const state = cliState({ + getEnv: cliEnv({ + DB_TOKEN: "secret", + DB_URL: "libsql://tickets.example.com", + }), + readRecordedScriptCommit: () => Promise.resolve(FULL_COMMIT), + }); + + expect(await runRestoreCli(state.deps)).toBe(0); + expect(state.stderr).toEqual([]); + expect(state.stdout).toContain( + "Target database: libsql://tickets.example.com", + ); + expect(state.stdout).toContain("Backup created: 2026-07-22T12:00:00.000Z"); + expect(state.stdout).toContain( + "Backup contents: 1 table, 2 rows, 2 SQL statements", + ); + expect(state.stdout).toContain("Schema: matches this version of the app"); + expect(state.stdout).toContain("Checking 2 SQL statements..."); + expect(state.stdout).toContain("Deleting current database data..."); + expect(state.stdout).toContain("Recreating the database schema..."); + expect(state.stdout).toContain("Importing 2 SQL statements..."); + expect(state.stdout).toContain("Clearing cached database data..."); + expect(state.stdout).toContain("Database restored successfully."); + expect(state.stdout).toContain( + `The backup was running commit ${FULL_COMMIT}.`, + ); + expect(state.stdout.join("\n")).not.toContain("secret"); + expect(state.restored).toHaveLength(1); + expect(state.promptMessages).toEqual([ + `Type ${RESTORE_CONFIRMATION} to continue:`, + ]); + }); + + test("warns when the backup schema differs", async () => { + const state = cliState({ + readFile: () => + Promise.resolve( + backupZip({ + latestUpdate: "A different layout", + schemaHash: "different", + tables: { settings: 3 }, + timestamp: "2026-07-01T00:00:00.000Z", + }), + ), + }); + + expect(await runRestoreCli(state.deps)).toBe(0); + expect(state.stderr).toContain( + "Warning: This backup uses a different database schema. The app may need to apply database updates after the restore.", + ); + }); + + test("handles an older backup without a manifest", async () => { + const state = cliState({ + readFile: () => Promise.resolve(backupZip(null)), + readRecordedScriptCommit: () => Promise.resolve("not-a-full-sha"), + }); + + expect(await runRestoreCli(state.deps)).toBe(0); + expect(state.stdout).toContain("Backup manifest: not available"); + expect(state.stdout.some((line) => line.includes("commit"))).toBe(false); + }); + + test("reports a failure that leaves the old database intact", async () => { + const state = cliState({ + restoreFromZip: () => Promise.reject(new Error("newer backup")), + }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toContain("Restore failed: newer backup"); + expect(state.stderr.some((line) => line.includes("was reset"))).toBe(false); + }); + + test("makes a post-reset restore failure explicit", async () => { + const state = cliState({ + restoreFromZip: () => + Promise.reject(new PostResetError("database import failed")), + }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toContain("Restore failed: database import failed"); + expect(state.stderr).toContain( + "The target database was reset before the restore failed. Fix the error and run the restore again.", + ); + }); + + test("reports when restored build information cannot be read", async () => { + const state = cliState({ + readRecordedScriptCommit: () => + Promise.reject(new Error("settings unavailable")), + }); + + expect(await runRestoreCli(state.deps)).toBe(1); + expect(state.stderr).toEqual([ + "The database was restored, but its build information could not be read: settings unavailable", + ]); + }); +}); diff --git a/test/lib/code-quality/detectors.ts b/test/lib/code-quality/detectors.ts index ae5ea84f2f..dcd08e9650 100644 --- a/test/lib/code-quality/detectors.ts +++ b/test/lib/code-quality/detectors.ts @@ -324,14 +324,15 @@ export const importedSymbolsOf = ( /** * Whether `symbolName` (exported from `sourceFile`) is used anywhere in * production code: within the same file, imported by another `.ts` source, or - * imported by a `.tsx` template. `srcContents`/`tsxContents` map an absolute - * path to its contents. + * imported by a `.tsx` template, or imported by another production entry point + * such as a console task. Each content map uses absolute paths as keys. */ export const isUsedInProductionCode = ( symbolName: string, sourceFile: string, srcContents: Map, tsxContents: Map, + productionEntryContents: Map[] = [], ): boolean => { // Check if it's used within the same file const sourceContent = srcContents.get(sourceFile)!; @@ -344,7 +345,10 @@ export const isUsedInProductionCode = ( // index never wrongly credits `sourceFile` itself as an importer. return ( importedSymbolsOf(srcContents).has(symbolName) || - importedSymbolsOf(tsxContents).has(symbolName) + importedSymbolsOf(tsxContents).has(symbolName) || + productionEntryContents.some((contents) => + importedSymbolsOf(contents).has(symbolName), + ) ); }; @@ -378,6 +382,7 @@ export const findTestOnlyExportViolations = ( tsxContents: Map, testContents: Map, allowedHooks: string[], + productionEntryContents: Map[] = [], ): string[] => { const violations: string[] = []; @@ -395,6 +400,7 @@ export const findTestOnlyExportViolations = ( sourceFile, srcContents, tsxContents, + productionEntryContents, ); if (!usedInProduction && isUsedInTests(exportName, testContents)) { diff --git a/test/lib/server-backup.test.ts b/test/lib/server-backup.test.ts deleted file mode 100644 index 3c46df0594..0000000000 --- a/test/lib/server-backup.test.ts +++ /dev/null @@ -1,645 +0,0 @@ -import type { InStatement } from "@libsql/client"; -import { expect } from "@std/expect"; -import { describe, it as test } from "@std/testing/bdd"; -import { zipSync } from "fflate"; -import { handleRequest } from "#routes"; -import { createBackupZip } from "#shared/db/backup.ts"; -import { backupDir } from "#shared/db/backup-storage.ts"; -import { getDb } from "#shared/db/client.ts"; -import { invalidateInitDbCache } from "#shared/db/migrations.ts"; -import { downloadRaw, uploadRaw } from "#shared/storage.ts"; -import { - readRecordedScriptCommit, - recordScriptVersion, - setBuildCommitForTest, -} from "#shared/update.ts"; -import { RESTORE_CONFIRM_PHRASE } from "#templates/admin/backup.tsx"; -import { - expectFlashRedirect, - expectHtmlResponse, - testRequiresAuth, -} from "#test-utils/assertions.ts"; -import { describeWithEnv } from "#test-utils/db.ts"; -import { createTestHoliday } from "#test-utils/db-helpers/holidays.ts"; -import { createTestListing } from "#test-utils/db-helpers/listings.ts"; -import { awaitTestRequest, withStorageDisabled } from "#test-utils/mocks.ts"; -import { statementSql, wrapDbClient } from "#test-utils/record-queries.ts"; -import { - adminFormPost, - adminGet, - createTestManagerSession, - getTestSession, -} from "#test-utils/session.ts"; - -// The whole suite runs with a per-test local storage backend so the backup -// create/restore/download paths have a real filesystem to read and write; the -// one test that asserts the disabled state opts out with `withStorageDisabled`. -describeWithEnv("server (admin backup)", { db: true, storage: "local" }, () => { - describe("GET /admin/backup", () => { - testRequiresAuth("/admin/backup"); - - test("returns 403 for manager users", async () => { - const managerCookie = await createTestManagerSession(); - const response = await awaitTestRequest("/admin/backup", { - cookie: managerCookie, - }); - expect(response.status).toBe(403); - }); - - test("shows backup page for owner", async () => { - const response = await adminGet("/admin/backup"); - await expectHtmlResponse( - response, - 200, - "Database Backup", - "Encryption Key", - ); - }); - - test("highlights the Backup sub-nav link on the backup page", async () => { - // Regression: the main /admin/backup page used the admin-page helper's - // default active route (/admin/settings), so the Backup sub-nav link - // never lit up on the page users actually land on. - const html = await (await adminGet("/admin/backup")).text(); - expect(html).toContain('class="active" href="/admin/backup"'); - }); - - test("shows encryption key on page", async () => { - const response = await adminGet("/admin/backup"); - const html = await response.text(); - expect(html).toContain("MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY="); - }); - - test("shows storage not configured when disabled", async () => { - await withStorageDisabled(async () => { - const response = await adminGet("/admin/backup"); - const html = await response.text(); - expect(html).toContain("Storage is not configured"); - }); - }); - - test("cleans up stale restore-pending files on page load", async () => { - await uploadRaw(new Uint8Array(1), "restore-pending-stale.zip"); - - await adminGet("/admin/backup"); - - // Give the fire-and-forget cleanup a moment to complete - await new Promise((r) => setTimeout(r, 50)); - - const data = await downloadRaw("restore-pending-stale.zip"); - expect(data).toBeNull(); - }); - }); - - describe("POST /admin/backup/create", () => { - testRequiresAuth("/admin/backup/create", { - body: {}, - method: "POST", - }); - - test("creates backup and redirects with success", async () => { - await createTestListing({ name: "Backup Test" }); - const { response } = await adminFormPost("/admin/backup/create"); - await expectFlashRedirect( - "/admin/backup", - "Database backup created", - )(response); - }); - - test("lists only valid backup files on backup page", async () => { - // A non-backup file sharing the folder must not appear in the list. - await uploadRaw(new Uint8Array(0), `${backupDir()}backup-stale.tmp`); - await adminFormPost("/admin/backup/create"); - const response = await adminGet("/admin/backup"); - const html = await response.text(); - expect(html).not.toContain("backup-stale.tmp"); - expect(html).toContain(".zip"); - }); - - test("lists a created backup's raw timestamp exactly as named in its file", async () => { - await adminFormPost("/admin/backup/create"); - const response = await adminGet("/admin/backup"); - const html = await response.text(); - // The timestamp column must show the same token the stored filename - // carries — the backup's unambiguous identifier. - const token = html.match( - /backup-(\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2}-\d{3}Z)\.zip/, - )?.[1]; - expect(token).toBeDefined(); - expect(html).toContain(`${token}`); - }); - - test("shows the retention summary once a backup exists", async () => { - await adminFormPost("/admin/backup/create"); - const response = await adminGet("/admin/backup"); - const html = await response.text(); - expect(html).toContain("There is 1 backup"); - expect(html).toContain("Up to 30 are kept"); - }); - }); - - describe("GET /admin/backup/download/:filename", () => { - testRequiresAuth("/admin/backup/download/backup-local-test.zip"); - - test("returns 400 for invalid filename", async () => { - const response = await adminGet( - "/admin/backup/download/not-a-backup.txt", - ); - expect(response.status).toBe(400); - }); - - test("returns 404 for missing file", async () => { - // Validly formatted leaf, but no such backup exists in this DB's folder. - const response = await adminGet( - "/admin/backup/download/backup-2024-01-15T12-30-00-000Z.zip", - ); - expect(response.status).toBe(404); - }); - - test("returns 400 for filename with path traversal", async () => { - const response = await adminGet( - "/admin/backup/download/backup-local-..%2F..%2Fetc.zip", - ); - expect(response.status).toBe(400); - }); - - test("downloads existing backup as zip", async () => { - await adminFormPost("/admin/backup/create"); - const listResp = await adminGet("/admin/backup"); - const html = await listResp.text(); - const linkMatch = html.match( - /\/admin\/backup\/download\/(backup-[^"]+\.zip)/, - ); - expect(linkMatch).toBeTruthy(); - const response = await adminGet( - `/admin/backup/download/${linkMatch![1]}`, - ); - expect(response.status).toBe(200); - expect(response.headers.get("content-type")).toBe("application/zip"); - // The response body must be the actual zip bytes, not an empty slice. - expect((await response.arrayBuffer()).byteLength).toBeGreaterThan(0); - }); - }); - - describe("POST /admin/backup/restore", () => { - testRequiresAuth("/admin/backup/restore", { - body: {}, - method: "POST", - }); - - /** POST a restore upload with `csrfToken` + optional file fields, returning - * the response. Collapses the repeated FormData + Request scaffold. */ - const postRestore = async ( - cookie: string, - csrfToken: string, - files: Record = {}, - ): Promise => { - const formData = new FormData(); - formData.append("csrf_token", csrfToken); - for (const [name, file] of Object.entries(files)) { - formData.append(name, file); - } - return handleRequest( - new Request("http://localhost/admin/backup/restore", { - body: formData, - headers: { cookie, host: "localhost" }, - method: "POST", - }), - ); - }; - - test("shows confirm page after uploading valid zip", async () => { - const zipData = await createBackupZip(); - const { cookie, csrfToken } = await getTestSession(); - const response = await postRestore(cookie, csrfToken, { - backup_file: new File([zipData.buffer as ArrayBuffer], "backup.zip"), - }); - expect(response.status).toBe(200); - const html = await response.text(); - expect(html).toContain("Confirm Database Restore"); - // The statement count must be a real number, not NaN. - expect(html).not.toContain("NaN"); - // A backup whose schema matches must NOT show the mismatch warning. - expect(html).not.toContain("Schema mismatch"); - // The uploaded zip must be stored so the confirm step can read it back. - const tempName = html.match(/value="(restore-pending-[^"]+\.zip)"/)?.[1]; - expect(tempName).toBeDefined(); - expect(await downloadRaw(tempName!)).not.toBeNull(); - }); - - test("shows schema mismatch warning for different schema", async () => { - const encoder = new TextEncoder(); - const fakeZip = zipSync({ - "manifest.json": encoder.encode( - JSON.stringify({ - latestUpdate: "", - schemaHash: "wrong", - tables: {}, - timestamp: "", - }), - ), - "settings.sql": new Uint8Array(0), - }); - const { cookie, csrfToken } = await getTestSession(); - const response = await postRestore(cookie, csrfToken, { - backup_file: new File([fakeZip.buffer as ArrayBuffer], "backup.zip"), - }); - const html = await response.text(); - expect(html).toContain("Schema mismatch"); - }); - - test("rejects missing file field", async () => { - const { cookie, csrfToken } = await getTestSession(); - const response = await postRestore(cookie, csrfToken); - await expectFlashRedirect( - "/admin/backup", - "Please select a backup file", - false, - )(response); - }); - - test("rejects invalid zip file", async () => { - const { cookie, csrfToken } = await getTestSession(); - const response = await postRestore(cookie, csrfToken, { - backup_file: new File([new ArrayBuffer(100)], "bad.zip"), - }); - await expectFlashRedirect( - "/admin/backup", - expect.stringContaining("Invalid backup file"), - false, - )(response); - }); - }); - - describe("POST /admin/backup/restore/confirm", () => { - testRequiresAuth("/admin/backup/restore/confirm", { - body: {}, - method: "POST", - }); - - test("rejects invalid filename", async () => { - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "bad.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - await expectFlashRedirect( - "/admin/backup", - "Invalid backup reference", - false, - )(response); - }); - - test("rejects a path-traversal filename that passes the prefix check", async () => { - // The leaf has the restore-pending prefix and .zip suffix, so only the - // isSafeBackupFilename guard (no "/", "\\", or "..") rejects it. - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-a/b.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - await expectFlashRedirect( - "/admin/backup", - "Invalid backup reference", - false, - )(response); - }); - - test("rejects wrong confirmation phrase", async () => { - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-x.zip", - confirm_identifier: "WRONG", - }, - ); - await expectFlashRedirect( - "/admin/backup", - "Confirmation phrase does not match", - false, - )(response); - }); - - test("rejects missing backup file", async () => { - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-gone.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - await expectFlashRedirect( - "/admin/backup", - "Backup file expired or not found. Please upload again.", - false, - )(response); - }); - - test("successfully restores from backup", async () => { - const listing = await createTestListing({ name: "Restore Me" }); - const zipData = await createBackupZip(); - // Delete the listing after capturing it in the backup so we can - // verify the restore actually writes data back to the DB. - const { getAllListings } = await import("#shared/db/listings/records.ts"); - const { deleteListing } = await import("#shared/db/listings/delete.ts"); - await deleteListing(listing.id); - expect((await getAllListings()).find((e) => e.id === listing.id)).toBe( - undefined, - ); - - await uploadRaw(zipData, "restore-pending-test.zip"); - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-test.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - await expectFlashRedirect( - "/admin/login", - "Database restored from backup", - )(response); - - const restored = (await getAllListings()).find( - (e) => e.id === listing.id, - ); - expect(restored).toBeDefined(); - expect(restored!.name).toBe("Restore Me"); - }); - - /** Withhold the boot marker read's result — a slow production round - * trip — until the restore replay starts (or a grace timer, since the - * pre-restore flush leaves nothing else to release it). */ - const delayMarkerReadUntilReplay = (): (() => void) => { - const real = getDb(); - let release = (): void => {}; - const released = new Promise((resolve) => { - release = resolve; - }); - const timer = setTimeout(release, 150); - const isMarkerRead = (statement: InStatement | string): boolean => - typeof statement !== "string" && - Array.isArray(statement.args) && - statement.args.includes("current_script_version"); - const holdMarkerRead = async (statement: InStatement) => { - const result = await real.execute(statement); - await released; - return result; - }; - const restore = wrapDbClient({ - batch: (statements) => { - if ( - statements.some((s) => statementSql(s) === "DELETE FROM settings") - ) { - release(); - } - }, - execute: (statement) => - isMarkerRead(statement) - ? holdMarkerRead(statement as InStatement) - : null, - }); - return () => { - clearTimeout(timer); - release(); - restore(); - }; - }; - - test("a deploy's first request cannot clobber the commit a restore reports", async () => { - // A new deploy's first request IS the restore POST: initDb's queued - // marker write, still in flight when the replay runs, would land after - // it and stamp the running commit over the restored one. - const backupSha = "0123456789abcdef0123456789abcdef01234567"; - const deploySha = "89abcdef0123456789abcdef0123456789abcdef"; - setBuildCommitForTest(backupSha); - try { - await getTestSession(); // ensure session row is in DB before backup - await recordScriptVersion(); - const zipData = await createBackupZip(); - await uploadRaw(zipData, "restore-pending-race.zip"); - - setBuildCommitForTest(deploySha); - invalidateInitDbCache(); - const restoreClient = delayMarkerReadUntilReplay(); - try { - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-race.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - // Read before the follow-up request, whose initDb legitimately - // restamps the running build. - expect(await readRecordedScriptCommit()).toBe(backupSha); - await expectFlashRedirect( - "/admin/login", - expect.stringContaining(backupSha), - )(response); - } finally { - restoreClient(); - } - } finally { - setBuildCommitForTest(null); - } - }); - - test("surfaces the full recorded commit so the operator can redeploy the code", async () => { - // The running build records its commit into settings; the dump carries - // it, so the restore tells the operator which commit to redeploy. The - // FULL SHA is shown because the restore-deploy workflow requires one. - const fullSha = "0123456789abcdef0123456789abcdef01234567"; - setBuildCommitForTest(fullSha); - try { - await getTestSession(); // ensure session row is in DB before backup - await recordScriptVersion(); - const zipData = await createBackupZip(); - await uploadRaw(zipData, "restore-pending-commit.zip"); - - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-commit.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - await expectFlashRedirect( - "/admin/login", - `Database restored from backup. It was running commit ${fullSha} — run the restore-deploy workflow with that commit to restore the code to this point in time.`, - )(response); - } finally { - setBuildCommitForTest(null); - } - }); - - test("omits the redeploy hint when the restored commit is not a full SHA", async () => { - // An uploaded/old backup may hold a non-SHA commit value; it must not be - // echoed into the flash (it's unusable by restore-deploy and could be - // oversized), so the message falls back to the plain confirmation. - setBuildCommitForTest("not-a-real-sha"); - try { - await getTestSession(); // ensure session row is in DB before backup - await recordScriptVersion(); - const zipData = await createBackupZip(); - await uploadRaw(zipData, "restore-pending-badsha.zip"); - - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-badsha.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - await expectFlashRedirect( - "/admin/login", - "Database restored from backup", - )(response); - } finally { - setBuildCommitForTest(null); - } - }); - - test("rejects filename with traversal despite valid prefix and suffix", async () => { - // Place a decoy at the traversal target — if traversal were allowed, - // the restore would attempt to read it. The validation layer must - // reject before any filesystem access. - await uploadRaw(new Uint8Array(1), "etc-passwd-decoy.zip"); - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-x/../../etc-passwd-decoy.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - await expectFlashRedirect( - "/admin/backup", - "Invalid backup reference", - false, - )(response); - }); - - test("routes pre-reset restore failure to backup page", async () => { - // Upload raw non-zip bytes — restoreFromZip throws from unzipSync - // before any DB reset, so the session is intact and onError redirects - // back to /admin/backup. - await uploadRaw( - new Uint8Array([0, 1, 2, 3]), - "restore-pending-badzip.zip", - ); - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-badzip.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - await expectFlashRedirect( - "/admin/backup", - expect.any(String), - false, - )(response); - }); - - test("routes post-reset restore failure to setup page", async () => { - // A valid zip whose SQL is invalid reaches restoreFromSql, which runs - // resetDatabase() before executeBatch fails — sessions and settings are - // wiped, so onError must send to /setup/ (not /admin/login or /admin/backup). - const badSqlZip = zipSync({ - "settings.sql": new TextEncoder().encode("INVALID SQL SYNTAX;"), - }); - await uploadRaw(badSqlZip, "restore-pending-badsql.zip"); - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-badsql.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - await expectFlashRedirect("/setup/", expect.any(String), false)(response); - }); - - test("cleans up temp file even on restore failure", async () => { - // Upload an invalid zip (valid zip format but contains bad SQL) - const badZip = zipSync({ - "settings.sql": new TextEncoder().encode("INVALID SQL SYNTAX;"), - }); - await uploadRaw(badZip, "restore-pending-fail.zip"); - - // Attempt restore — should fail but still clean up the temp file - try { - await adminFormPost("/admin/backup/restore/confirm", { - backup_filename: "restore-pending-fail.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }); - } catch { - // Restore failure is expected - } - - // Temp file should be cleaned up regardless - const data = await downloadRaw("restore-pending-fail.zip"); - expect(data).toBeNull(); - }); - - test("restore clears all entity caches including holidays", async () => { - const { holidays } = await import("#shared/db/holidays.ts"); - - // Snapshot before creating any holidays — backup has no holidays - const zipData = await createBackupZip(); - await uploadRaw(zipData, "restore-pending-caches.zip"); - - // Create a holiday after the backup so it exists in the cache but - // will be absent from the restored DB - await createTestHoliday({ name: "Cache Test Holiday" }); - const before = await holidays.getAll(); - expect(before.some((h) => h.name === "Cache Test Holiday")).toBe(true); - - // Restore from the pre-holiday snapshot - const { response } = await adminFormPost( - "/admin/backup/restore/confirm", - { - backup_filename: "restore-pending-caches.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }, - ); - expect(response.status).toBe(302); - - // The holidays cache must be cleared: the holiday is gone from the DB - // and must not be served from a stale cache entry - const after = await holidays.getAll(); - expect(after.some((h) => h.name === "Cache Test Holiday")).toBe(false); - }); - - test("restore clears session cache so pre-restore sessions are re-validated", async () => { - // Snapshot before the manager session is created - const zipData = await createBackupZip(); - await uploadRaw(zipData, "restore-pending-sessions.zip"); - - // Create a manager session after the backup — it will not be in the - // restored DB, so it must also be evicted from the session cache - const managerCookie = await createTestManagerSession( - "cache-test-mgr", - "cache-test-manager", - ); - - // Restore from the pre-manager snapshot - await adminFormPost("/admin/backup/restore/confirm", { - backup_filename: "restore-pending-sessions.zip", - confirm_identifier: RESTORE_CONFIRM_PHRASE, - }); - - // After restore the manager token is absent from the DB and the - // session cache has been cleared, so using it must fail auth. - // /admin/backup returns 403 for authenticated managers and 302 for - // unauthenticated requests; a 302 here proves the session was rejected. - const response = await awaitTestRequest("/admin/backup", { - cookie: managerCookie, - }); - expect(response.status).toBe(302); // redirected to login (not 403) - }); - }); -}); diff --git a/test/restore.test.ts b/test/restore.test.ts new file mode 100644 index 0000000000..ef7e05ab5b --- /dev/null +++ b/test/restore.test.ts @@ -0,0 +1,72 @@ +import { expect } from "@std/expect"; +import { it as test } from "@std/testing/bdd"; +import { RESTORE_USAGE } from "#scripts/restore-lib.ts"; +import { runRestoreTask } from "#src/restore.ts"; +import { TEST_ENCRYPTION_KEY } from "#test-utils/internal.ts"; + +test("uses only file database settings when running the restore CLI", async () => { + const env: Record = { + DB_ENCRYPTION_KEY: "ambient-key", + DB_TOKEN: "ambient-token", + DB_URL: ":memory:", + OTHER_KEY: "unchanged", + }; + const stderr: string[] = []; + let result: number | undefined; + + await runRestoreTask( + { + DB_ENCRYPTION_KEY: TEST_ENCRYPTION_KEY, + DB_URL: "libsql://tickets.example.com", + }, + (key, value) => + value === undefined + ? Reflect.deleteProperty(env, key) + : Reflect.set(env, key, value), + async (run) => { + result = await run({ + args: [], + getEnv: () => undefined, + stderr: (line) => stderr.push(line), + stdout: () => {}, + }); + }, + ); + + expect(env).toEqual({ + DB_ENCRYPTION_KEY: TEST_ENCRYPTION_KEY, + DB_URL: "libsql://tickets.example.com", + OTHER_KEY: "unchanged", + }); + expect(result).toBe(1); + expect(stderr).toEqual([RESTORE_USAGE]); +}); + +test("default restore dependencies read the requested backup path", async () => { + const path = `missing-restore-${crypto.randomUUID()}.zip`; + const env: Record = { + DB_ENCRYPTION_KEY: TEST_ENCRYPTION_KEY, + DB_URL: "file:target.db", + }; + const stderr: string[] = []; + let result: number | undefined; + + await runRestoreTask( + env, + (key, value) => + value === undefined + ? Reflect.deleteProperty(env, key) + : Reflect.set(env, key, value), + async (run) => { + result = await run({ + args: [path], + getEnv: (key) => env[key], + stderr: (line) => stderr.push(line), + stdout: () => {}, + }); + }, + ); + + expect(result).toBe(1); + expect(stderr[0]).toContain(`Could not read ${path}`); +}); diff --git a/test/routes/backup.test.ts b/test/routes/backup.test.ts deleted file mode 100644 index 6686104a42..0000000000 --- a/test/routes/backup.test.ts +++ /dev/null @@ -1,166 +0,0 @@ -import { expect } from "@std/expect"; -import { describe, it as test } from "@std/testing/bdd"; -import { handleRequest } from "#routes"; -import { signCsrfToken } from "#shared/csrf.ts"; -import { - downloadRaw, - runWithStorageConfig, - uploadRaw, -} from "#shared/storage.ts"; -import { setDeleteOverride } from "#shared/test-overrides.ts"; -import { describeWithEnv } from "#test-utils/db.ts"; -import { tempDir } from "#test-utils/files.ts"; -import { TEST_STORAGE_ZONE } from "#test-utils/internal.ts"; -import { - installUrlHandler, - mockRequest, - withFetchMock, - withLocalStorageEnabled, -} from "#test-utils/mocks.ts"; -import { testCookie } from "#test-utils/session.ts"; - -/** Request the admin backup page with the given session cookie and read its - * body, so each test can assert on the status and HTML itself. */ -const loadBackupPage = async ( - cookie: string, -): Promise<{ response: Response; html: string }> => { - const response = await handleRequest( - mockRequest("/admin/backup", { headers: { cookie } }), - ); - const html = await response.text(); - return { html, response }; -}; - -describeWithEnv("backup routes", { db: true }, () => { - describe("GET /admin/backup", () => { - test("loads backup page without storage enabled", async () => { - const cookie = await testCookie(); - const { response, html } = await loadBackupPage(cookie); - expect(response.status).toBe(200); - expect(html).toContain("Backup"); - }); - - test("loads backup page when storage cleanup fails (fire-and-forget)", async () => { - const cookie = await testCookie(); - - await withLocalStorageEnabled(async (dir) => { - // Create a stale restore-pending file - const pendingFile = `${dir}/restore-pending-stale.zip`; - await Deno.writeTextFile(pendingFile, "stale data"); - - // Make deleteFile throw - setDeleteOverride(new Error("delete failed")); - try { - const { response, html } = await loadBackupPage(cookie); - // Page should still load despite deleteFile throwing - expect(response.status).toBe(200); - expect(html).toContain("Backup"); - } finally { - setDeleteOverride(null); - } - }); - }); - - test("loads backup page when stale cleanup listing fails", async () => { - const cookie = await testCookie(); - - await runWithStorageConfig(TEST_STORAGE_ZONE, () => - withFetchMock(async (originalFetch) => { - let calls = 0; - installUrlHandler(originalFetch, (url) => { - if (!url.includes("storage.bunnycdn.com")) return null; - calls += 1; - return calls === 1 - ? Promise.reject(new Error("list failed")) - : Promise.resolve(Response.json([])); - }); - - const response = await handleRequest( - mockRequest("/admin/backup", { headers: { cookie } }), - ); - - expect(response.status).toBe(200); - expect(await response.text()).toContain("Backup"); - }), - ); - }); - }); - - describe("POST /admin/backup/restore/confirm", () => { - test("completes restore even when temp file cleanup fails", async () => { - const cookie = await testCookie(); - - // Create a valid session with owner role - const { getDb } = await import("#shared/db/client.ts"); - const { encrypt } = await import("#shared/crypto/encryption.ts"); - - // Ensure user 1 is owner - const adminLevel = await encrypt("owner"); - await getDb().execute({ - args: [adminLevel, 1], - sql: "UPDATE users SET admin_level = ? WHERE id = 1", - }); - - // Create a minimal valid backup zip - const { zipSync } = await import("fflate"); - const encoder = new TextEncoder(); - const { SCHEMA_HASH } = await import("#shared/db/migrations.ts"); - const zipData = zipSync({ - "manifest.json": encoder.encode( - JSON.stringify({ - latestUpdate: "2024-01-01T00:00:00.000Z", - schemaHash: SCHEMA_HASH, - tables: {}, - timestamp: "2024-01-01T00:00:00.000Z", - }), - ), - }); - - const temp = tempDir(); - const dir = temp.path; - try { - // Upload the zip as a pending restore file - const tempFilename = `restore-pending-${crypto.randomUUID()}.zip`; - await runWithStorageConfig( - { localPath: dir, zoneKey: "", zoneName: "" }, - async () => { - await uploadRaw(zipData, tempFilename); - await Deno.chmod(dir, 0o500); - - // Get CSRF token - const csrfToken = await signCsrfToken(); - - // Use URL-encoded form (action handlers default to form mode) - const body = new URLSearchParams({ - backup_filename: tempFilename, - confirm_phrase: "restore", - csrf_token: csrfToken, - }); - - const request = new Request( - "http://localhost/admin/backup/restore/confirm", - { - body: body.toString(), - headers: { - "content-type": "application/x-www-form-urlencoded", - cookie, - }, - method: "POST", - }, - ); - - const response = await handleRequest(request); - // Should succeed (redirect) even though deleteFile threw - expect(response.status).toBe(302); - const location = response.headers.get("location"); - expect(location).toContain("/admin/backup"); - expect(await downloadRaw(tempFilename)).not.toBeNull(); - }, - ); - } finally { - await Deno.chmod(dir, 0o700).catch(() => {}); - temp.dispose(); - } - }); - }); -}); diff --git a/test/shared/db/backup.test.ts b/test/shared/db/backup.test.ts index 1bc3a2cc67..46b968508c 100644 --- a/test/shared/db/backup.test.ts +++ b/test/shared/db/backup.test.ts @@ -5,9 +5,9 @@ import { stub } from "@std/testing/mock"; import { unzipSync, zipSync } from "fflate"; import { type BackupManifest, - countZipStatements, createBackupZip, - readManifest, + inspectBackupZip, + PostResetError, restoreFromSql, restoreFromZip, splitStatements, @@ -37,6 +37,12 @@ describeWithEnv("backup", { db: true }, () => { expect(stmts[1]).toBe("INSERT INTO b VALUES (2);"); }); + test("normalizes Windows newlines inside quoted values", () => { + expect( + splitStatements("INSERT INTO a VALUES ('first\r\nsecond');"), + ).toEqual(["INSERT INTO a VALUES ('first\nsecond');"]); + }); + test("returns empty array for empty input", () => { expect(splitStatements("")).toHaveLength(0); expect(splitStatements(" ")).toHaveLength(0); @@ -46,6 +52,31 @@ describeWithEnv("backup", { db: true }, () => { const stmts = splitStatements("INSERT INTO a VALUES (1);\n"); expect(stmts).toHaveLength(1); }); + + test("adds a terminator to a final statement that has none", () => { + expect(splitStatements("mutated")).toEqual(["mutated;"]); + }); + + test("keeps a terminator on the final statement", () => { + expect(splitStatements("SELECT 1;")).toEqual(["SELECT 1;"]); + }); + + test("ignores empty statements", () => { + expect(splitStatements(";\nSELECT 1;")).toEqual(["SELECT 1;"]); + }); + + test("keeps semicolon-newlines inside quoted CSS values", () => { + const cssInsert = + "INSERT INTO settings (key, value) VALUES ('custom_css', '/* 🐸 Splish-Splash — a bright kids'' theme 🐳 */\n" + + ":root {\n --color-bg: #f2fcff;\n --color-text: #0d3b45;\n}');"; + const nextInsert = + "INSERT INTO settings (key, value) VALUES ('site_name', 'Frog pond');"; + + expect(splitStatements(`${cssInsert}\n${nextInsert}`)).toEqual([ + cssInsert, + nextInsert, + ]); + }); }); describe("createBackupZip", () => { @@ -70,48 +101,104 @@ describeWithEnv("backup", { db: true }, () => { manifest.timestamp, ); expect(manifest.tables.listings).toBe(1); + expect(new TextDecoder().decode(files["manifest.json"]!)).toContain( + '\n "latestUpdate"', + ); + // ZIP method 8 is DEFLATE; method 0 would store every entry uncompressed. + expect(new DataView(zipData.buffer).getUint16(8, true)).toBe(8); }); }); - describe("readManifest", () => { + describe("inspectBackupZip", () => { test("reads manifest from backup zip", async () => { - const manifest = readManifest(await createBackupZip()); + const { manifest } = inspectBackupZip(await createBackupZip()); expect(manifest).not.toBeNull(); expect(manifest!.schemaHash).toBe(SCHEMA_HASH); }); test("returns null for zip without manifest", () => { - expect(readManifest(zipSync({ "a.sql": new Uint8Array(0) }))).toBeNull(); + expect( + inspectBackupZip( + zipSync({ + "attendee_statuses.sql": new TextEncoder().encode( + "INSERT INTO attendee_statuses (id) VALUES (1);", + ), + "schema_migrations.sql": new TextEncoder().encode( + "INSERT INTO schema_migrations (id) VALUES ('initial');", + ), + "settings.sql": new TextEncoder().encode( + "INSERT INTO settings (key, value) VALUES ('k', 'v');", + ), + }), + ).manifest, + ).toBeNull(); }); - test("returns null for manifest with invalid shape", () => { + test("rejects a manifest with an invalid shape", () => { const encoder = new TextEncoder(); const zip = zipSync({ "manifest.json": encoder.encode(JSON.stringify({ wrong: "shape" })), }); - expect(readManifest(zip)).toBeNull(); + expect(() => inspectBackupZip(zip)).toThrow("Backup manifest is invalid"); }); - test("returns null for manifest missing required fields", () => { + test("rejects a manifest missing required fields", () => { const encoder = new TextEncoder(); const zip = zipSync({ "manifest.json": encoder.encode( JSON.stringify({ latestUpdate: "ok", schemaHash: "ok" }), ), }); - expect(readManifest(zip)).toBeNull(); + expect(() => inspectBackupZip(zip)).toThrow("Backup manifest is invalid"); + }); + + test("rejects an archive with no restorable SQL", () => { + expect(() => inspectBackupZip(zipSync({}))).toThrow( + "Backup contains no restorable SQL statements", + ); + }); + + test("lists every populated table whose data is missing", () => { + const cases = [ + [{ listings: 1 }, "listings"], + [{ attendees: 2, listings: 1 }, "attendees, listings"], + ] as const; + + for (const [tables, names] of cases) { + const zip = zipSync({ + "manifest.json": new TextEncoder().encode( + JSON.stringify({ + latestUpdate: LATEST_UPDATE, + schemaHash: SCHEMA_HASH, + tables, + timestamp: "2026-07-22T00:00:00.000Z", + }), + ), + }); + expect(() => inspectBackupZip(zip)).toThrow( + `Backup is missing data for tables: ${names}`, + ); + } }); - }); - describe("countZipStatements", () => { test("counts SQL statements across files in zip", async () => { await createTestListing({ name: "Count Test" }); - const count = countZipStatements(await createBackupZip()); - expect(count).toBeGreaterThanOrEqual(2); + const inspection = inspectBackupZip(await createBackupZip()); + expect(inspection.statementCount).toBeGreaterThanOrEqual(2); }); }); describe("restoreFromSql", () => { + const dumpWithFutureMigrations = async (ids: string[]): Promise => { + const recorded = await exportTable("schema_migrations"); + return `${recorded.sql}\n${ids + .map( + (id) => + `INSERT INTO "schema_migrations" ("id", "description", "applied_at") VALUES ('${id}', 'Future change', '2099-01-01T00:00:00.000Z');`, + ) + .join("\n")}`; + }; + test("restores data from SQL statements", async () => { await createTestListing({ name: "Before Restore" }); const { sql } = await exportTable("listings"); @@ -147,32 +234,95 @@ describeWithEnv("backup", { db: true }, () => { ); expect(result.rows[0]?.value).toBe(SCHEMA_HASH); }); + + test("refuses one future migration before wiping existing data", async () => { + await createTestListing({ name: "Still here" }); + + await expect( + restoreFromSql( + await dumpWithFutureMigrations(["2099-01-01_from_the_future"]), + ), + ).rejects.toThrow("2099-01-01_from_the_future"); + expect((await listingsTable.findAll()).map(({ name }) => name)).toEqual([ + "Still here", + ]); + }); + + test("explains every future migration and how to proceed", async () => { + const ids = [ + "2099-01-01_first_future_change", + "2099-01-02_second_future_change", + ]; + const error = await restoreFromSql( + await dumpWithFutureMigrations(ids), + ).catch((caught: unknown) => caught); + + expect((error as Error).message).toBe( + "Backup is from a newer version of the app: it records migration(s) " + + `newer than this build knows (${ids.join(", ")}). ` + + "Update the site to that version or newer, then restore this backup.", + ); + }); + + test("marks failures after reset with their exact cause", async () => { + const error = await restoreFromSql("", ({ stage }) => { + if (stage === "rebuilding") throw new Error("rebuild stopped"); + }).catch((caught: unknown) => caught); + + expect(error).toBeInstanceOf(PostResetError); + expect((error as Error).name).toBe("PostResetError"); + expect((error as Error).message).toBe("Error: rebuild stopped"); + }); }); describe("restoreFromZip", () => { test("round-trips backup and restore", async () => { await createTestListing({ name: "Zip Restore Test" }); - await restoreFromZip(await createBackupZip()); + const zip = await createBackupZip(); + const progress: Array<{ stage: string; statementCount: number }> = []; + await restoreFromZip(zip, (event) => progress.push(event)); const listings = await listingsTable.findAll(); expect(listings.length).toBe(1); expect(listings[0]!.name).toBe("Zip Restore Test"); + expect(progress.map(({ stage }) => stage)).toEqual([ + "checking", + "resetting", + "rebuilding", + "importing", + "clearing-caches", + ]); + expect(progress.map(({ statementCount }) => statementCount)).toEqual( + Array(5).fill(inspectBackupZip(zip).statementCount), + ); }); - test("preserves newlines in values through roundtrip", async () => { - await createTestListing({ - description: "first\nsecond\nthird", - name: "Newline Zip", + test("preserves CSS with emojis and semicolon-newlines through roundtrip", async () => { + const css = + "/* 🐸 Splish-Splash — a bright kids' theme 🐳 */\n\n" + + ":root {\n --color-bg: #f2fcff;\n --color-text: #0d3b45;\n}"; + await getDb().execute({ + args: ["custom_css", css], + sql: "INSERT OR REPLACE INTO settings (key, value) VALUES (?, ?)", }); await restoreFromZip(await createBackupZip()); - const listings = await listingsTable.findAll(); - expect(listings[0]!.description.replace(/\r\n/g, "\n")).toBe( - "first\nsecond\nthird", + const restored = await queryAll<{ value: string }>( + "SELECT value FROM settings WHERE key = 'custom_css'", ); + expect(restored).toEqual([{ value: css }]); }); - test("handles zip with missing table files", async () => { + test("allows a manifest backup to omit empty table files", async () => { + const encoder = new TextEncoder(); const partial = zipSync({ - "settings.sql": new TextEncoder().encode( + "manifest.json": encoder.encode( + JSON.stringify({ + latestUpdate: LATEST_UPDATE, + schemaHash: SCHEMA_HASH, + tables: { settings: 1 }, + timestamp: "2026-07-22T00:00:00.000Z", + }), + ), + "settings.sql": encoder.encode( "INSERT INTO settings (key, value) VALUES ('k', 'v');", ), }); @@ -183,6 +333,33 @@ describeWithEnv("backup", { db: true }, () => { expect(rows[0]!.value).toBe("v"); }); + test("lists unsupported table data before wiping the database", async () => { + await createTestListing({ name: "Still here" }); + const cases = [ + [["events"], "events"], + [["event_attendees", "events"], "event_attendees, events"], + ] as const; + + for (const [tables, names] of cases) { + const unsupported = zipSync( + Object.fromEntries( + tables.map((table) => [ + `${table}.sql`, + new TextEncoder().encode( + `INSERT INTO "${table}" ("id") VALUES (1);`, + ), + ]), + ), + ); + await expect(restoreFromZip(unsupported)).rejects.toThrow( + `Backup contains data for tables this app cannot restore: ${names}`, + ); + } + expect((await listingsTable.findAll()).map(({ name }) => name)).toEqual([ + "Still here", + ]); + }); + const sqlOf = (stmt: string | { sql: string }): string => typeof stmt === "string" ? stmt : stmt.sql; diff --git a/test/shared/db/backup/import-batches.test.ts b/test/shared/db/backup/import-batches.test.ts new file mode 100644 index 0000000000..791ebc3986 --- /dev/null +++ b/test/shared/db/backup/import-batches.test.ts @@ -0,0 +1,110 @@ +import type { InStatement, TransactionMode } from "@libsql/client"; +import { expect } from "@std/expect"; +import { it as test } from "@std/testing/bdd"; +import { stub } from "@std/testing/mock"; +import { PostResetError, restoreFromSql } from "#shared/db/backup.ts"; +import { getDb, queryAll } from "#shared/db/client.ts"; +import { verifyCurrentAppSchema } from "#shared/db/migrations/schema-sync.ts"; +import { describeWithEnv } from "#test-utils/db.ts"; + +describeWithEnv("backup import batches", { db: true }, () => { + const restoreAndInspectImport = async ( + sql: string, + ): Promise<{ schemaObjects: string[] | null; sizes: number[] }> => { + const client = getDb(); + const originalBatch = client.batch.bind(client); + const sizes: number[] = []; + let schemaObjects: string[] | null = null; + let importing = false; + const batchStub = stub(client, "batch", (async ( + statements: InStatement[], + mode?: TransactionMode, + ) => { + const sqls = statements.map((statement) => + typeof statement === "string" ? statement : statement.sql, + ); + if (sqls.includes("DELETE FROM settings")) { + importing = true; + const result = await client.execute( + "SELECT name FROM sqlite_master WHERE name LIKE 'idx_%' OR name LIKE 'trg_%'", + ); + schemaObjects = result.rows.map(({ name }) => String(name)); + sizes.push(statements.length); + return originalBatch(statements, mode); + } + if (importing && !sqls[0]!.includes(" INDEX IF NOT EXISTS ")) { + sizes.push(statements.length); + } + return originalBatch(statements, mode); + }) as never); + + try { + await restoreFromSql(sql); + } finally { + batchStub.restore(); + } + return { schemaObjects, sizes }; + }; + + const settingsDump = (values: string[]): string => + values + .map( + (value, index) => + `INSERT INTO settings (key, value) VALUES ('large_${index}', '${value}');`, + ) + .join("\n"); + + test("limits the number of statements in one transaction", async () => { + const { sizes } = await restoreAndInspectImport( + settingsDump(Array.from({ length: 51 }, (_, index) => String(index))), + ); + + expect(sizes).toEqual([50, 4]); + expect( + await queryAll<{ count: number }>( + "SELECT COUNT(*) AS count FROM settings WHERE key LIKE 'large_%'", + ), + ).toEqual([{ count: 51 }]); + }); + + test("limits the SQL bytes in one transaction", async () => { + const { sizes } = await restoreAndInspectImport( + settingsDump(["a".repeat(300_000), "b".repeat(300_000)]), + ); + + expect(sizes).toEqual([4, 1]); + }); + + test("keeps data checks while deferring derived database work", async () => { + const { schemaObjects } = await restoreAndInspectImport( + settingsDump(["v"]), + ); + + expect(schemaObjects).toContain("idx_listings_slug_index"); + expect(schemaObjects).toContain("trg_attendee_answers_validate_insert"); + expect(schemaObjects).not.toContain("idx_maintenance_tasks_due"); + expect(schemaObjects).not.toContain( + "trg_listing_attendees_aggregates_insert", + ); + await verifyCurrentAppSchema(); + }); + + test("clears rows committed before a later import batch fails", async () => { + const committedPrefix = settingsDump( + Array.from({ length: 47 }, (_, index) => String(index)), + ); + const invalidStatement = + "INSERT INTO holidays (id, date, stray_col) VALUES (1, '2026-01-01', 'bad');"; + + await expect( + restoreFromSql(`${committedPrefix}\n${invalidStatement}`), + ).rejects.toThrow(PostResetError); + + expect( + await queryAll<{ count: number }>( + "SELECT COUNT(*) AS count FROM settings WHERE key LIKE 'large_%'", + ), + ).toEqual([{ count: 0 }]); + await verifyCurrentAppSchema(); + }); +}); diff --git a/test/shared/db/backup/restore.test.ts b/test/shared/db/backup/restore.test.ts index 0118dff575..23a8378ec4 100644 --- a/test/shared/db/backup/restore.test.ts +++ b/test/shared/db/backup/restore.test.ts @@ -1,5 +1,6 @@ import { expect } from "@std/expect"; import { describe, it as test } from "@std/testing/bdd"; +import { zipSync } from "fflate"; import { createBackupZip, PostResetError, @@ -8,8 +9,11 @@ import { } from "#shared/db/backup.ts"; import { exportTable } from "#shared/db/backup-snapshot.ts"; import { getDb, queryAll, queryOne } from "#shared/db/client.ts"; +import { listingsTable } from "#shared/db/listings/records.ts"; +import { verifyCurrentAppSchema } from "#shared/db/migrations/schema-sync.ts"; import { initDb } from "#shared/db/migrations.ts"; import { describeWithEnv } from "#test-utils/db.ts"; +import { bookTestAttendee } from "#test-utils/db-helpers/attendees.ts"; import { createTestListing } from "#test-utils/db-helpers/listings.ts"; /** @@ -25,6 +29,15 @@ import { createTestListing } from "#test-utils/db-helpers/listings.ts"; describeWithEnv("db > backup restore", { db: true, triggers: true }, () => { const listingCount = async (): Promise => (await queryOne<{ n: number }>("SELECT COUNT(*) AS n FROM listings"))!.n; + const zipSql = (file: string, sql: string): Uint8Array => + zipSync({ [file]: new TextEncoder().encode(sql) }); + const listingTotals = ( + listingId: number, + ): Promise<{ booked_quantity: number; tickets_count: number } | null> => + queryOne( + "SELECT booked_quantity, tickets_count FROM listings WHERE id = ?", + [listingId], + ); const listingColumnNames = async (): Promise => { const rows = await queryAll<{ name: string }>( @@ -33,6 +46,131 @@ describeWithEnv("db > backup restore", { db: true, triggers: true }, () => { return rows.map((row) => row.name); }; + test("preserves stored listing totals while replaying booking rows", async () => { + const listing = await createTestListing({ name: "Stored totals" }); + await bookTestAttendee([listing.id]); + const zip = await createBackupZip(); + + await restoreFromZip(zip); + + expect(await listingTotals(listing.id)).toEqual({ + booked_quantity: 1, + tickets_count: 1, + }); + }); + + test("reinstalls listing total triggers after importing rows", async () => { + const listing = await createTestListing({ name: "Restored triggers" }); + await restoreFromZip(await createBackupZip()); + + const attendee = await getDb().execute( + "INSERT INTO attendees (created, pii_blob) VALUES ('2026-07-22T00:00:00.000Z', '')", + ); + await getDb().execute({ + args: [listing.id, Number(attendee.lastInsertRowid)], + sql: "INSERT INTO listing_attendees (listing_id, attendee_id) VALUES (?, ?)", + }); + + expect(await listingTotals(listing.id)).toEqual({ + booked_quantity: 1, + tickets_count: 1, + }); + }); + + test("keeps validation triggers active while importing rows", async () => { + await expect( + restoreFromSql("INSERT INTO attendee_answers (attendee_id) VALUES (1);"), + ).rejects.toThrow("invalid attendee answer"); + + expect( + await queryOne<{ count: number }>( + "SELECT COUNT(*) AS count FROM attendee_answers", + ), + ).toEqual({ count: 0 }); + await verifyCurrentAppSchema(); + }); + + test("keeps unique indexes active while importing rows", async () => { + const listing = + "INSERT INTO listings (created, max_attendees, slug_index) " + + "VALUES ('2026-07-22T00:00:00.000Z', 1, 'duplicate');"; + + await expect(restoreFromSql(`${listing}\n${listing}`)).rejects.toThrow( + "UNIQUE constraint failed: listings.slug_index", + ); + + expect(await listingCount()).toBe(0); + await verifyCurrentAppSchema(); + }); + + test("rejects one oversized statement before wiping existing data", async () => { + await createTestListing({ name: "Still here" }); + const statement = `INSERT INTO "settings" ("key", "value") VALUES ('oversized', '${"x".repeat( + 600_000, + )}');`; + + await expect( + restoreFromZip(zipSql("settings.sql", statement)), + ).rejects.toThrow("Backup SQL statement is larger than 524288 bytes"); + + expect(await listingCount()).toBe(1); + }); + + test("rejects a statement for the wrong table before wiping existing data", async () => { + await createTestListing({ name: "Still here" }); + const statement = 'INSERT INTO "events" ("id") VALUES (1);'; + + await expect( + restoreFromZip(zipSql("settings.sql", statement)), + ).rejects.toThrow( + "Backup file settings.sql contains SQL for events instead of settings", + ); + + expect(await listingCount()).toBe(1); + }); + + test("rejects non-insert SQL before wiping existing data", async () => { + await createTestListing({ name: "Still here" }); + + await expect( + restoreFromZip(zipSql("settings.sql", "DELETE FROM settings;")), + ).rejects.toThrow( + "Backup file settings.sql contains a statement that is not an INSERT", + ); + + expect(await listingCount()).toBe(1); + }); + + test("rejects a partial backup without a manifest before wiping existing data", async () => { + await createTestListing({ name: "From partial backup" }); + const { sql } = await exportTable("listings"); + await createTestListing({ name: "Must stay" }); + + await expect(restoreFromZip(zipSql("listings.sql", sql))).rejects.toThrow( + "Backup without a manifest is missing required data for tables: settings, schema_migrations, attendee_statuses", + ); + + expect((await listingsTable.findAll()).map(({ name }) => name)).toEqual([ + "From partial backup", + "Must stay", + ]); + }); + + test("keeps restored data when the final progress callback fails", async () => { + await createTestListing({ name: "From backup" }); + const zip = await createBackupZip(); + await createTestListing({ name: "Not in backup" }); + + const error = await restoreFromZip(zip, ({ stage }) => { + if (stage === "clearing-caches") throw new Error("progress stopped"); + }).catch((caught: unknown) => caught); + + expect(error).toEqual(new Error("progress stopped")); + expect((await listingsTable.findAll()).map(({ name }) => name)).toEqual([ + "From backup", + ]); + }); + describe("backups taken before a column-dropping migration", () => { const FIRST_CLASS_IMAGES_ID = "2026-07-05_first_class_images"; @@ -92,22 +230,6 @@ describeWithEnv("db > backup restore", { db: true, triggers: true }, () => { }); describe("backups this build cannot replay", () => { - test("a backup from a newer build is refused before anything is wiped", async () => { - await createTestListing({ name: "Still Here" }); - const recorded = await exportTable("schema_migrations"); - const dump = - `${recorded.sql}\n` + - `INSERT INTO "schema_migrations" ("id", "description", "applied_at") ` + - `VALUES ('2099-01-01_from_the_future', 'Future change', '2099-01-01T00:00:00.000Z');\n`; - - await expect(restoreFromSql(dump)).rejects.toThrow( - "2099-01-01_from_the_future", - ); - - // Refused up front: the database was never reset, so the data survives. - expect(await listingCount()).toBe(1); - }); - test("an orphaned marker from a historically renamed migration is not refused", async () => { // Real databases carry schema_migrations rows whose migration was later // renamed (e.g. 2026-06-18_answer_price_modifiers); the old marker is diff --git a/test/shared/db/migrations/schema/restore-work.test.ts b/test/shared/db/migrations/schema/restore-work.test.ts new file mode 100644 index 0000000000..a479470dc2 --- /dev/null +++ b/test/shared/db/migrations/schema/restore-work.test.ts @@ -0,0 +1,38 @@ +import { expect } from "@std/expect"; +import { it as test } from "@std/testing/bdd"; +import { + RESTORE_DEFERRED_INDEXES, + RESTORE_DEFERRED_TRIGGERS, +} from "#shared/db/migrations/schema/restore-work.ts"; +import { TRIGGERS } from "#shared/db/migrations/schema/triggers.ts"; + +test("defers non-unique indexes but keeps unique checks active", () => { + expect( + RESTORE_DEFERRED_INDEXES.find( + ({ name }) => name === "idx_maintenance_tasks_due", + ), + ).toEqual({ + name: "idx_maintenance_tasks_due", + sql: "CREATE INDEX IF NOT EXISTS idx_maintenance_tasks_due ON maintenance_tasks(next_run_at, lease_expires_at, name)", + }); + expect( + RESTORE_DEFERRED_INDEXES.some( + ({ name }) => name === "idx_listings_slug_index", + ), + ).toBe(false); +}); + +test("keeps only validation triggers active during restore imports", () => { + const deferredNames = RESTORE_DEFERRED_TRIGGERS.map(({ name }) => name); + + expect( + TRIGGERS.filter(({ name }) => !deferredNames.includes(name)).map( + ({ name }) => name, + ), + ).toEqual([ + "trg_attendee_answers_validate_insert", + "trg_attendee_answers_validate_update", + "trg_attendees_validate_status_insert", + "trg_attendees_validate_status_update", + ]); +}); diff --git a/test/shared/update.test.ts b/test/shared/update.test.ts index 515fe467db..2fafed2617 100644 --- a/test/shared/update.test.ts +++ b/test/shared/update.test.ts @@ -2,7 +2,7 @@ import { expect } from "@std/expect"; import { afterEach, describe, it as test } from "@std/testing/bdd"; import { stub } from "@std/testing/mock"; import { bunnyCdnApi } from "#shared/bunny-cdn.ts"; -import { execute, queryAll, queryOne } from "#shared/db/client.ts"; +import { execute, getDb, queryAll, queryOne } from "#shared/db/client.ts"; import { denoDeployApi } from "#shared/deno-deploy-api.ts"; import { getSubrequestUsage, @@ -273,6 +273,25 @@ describeWithEnv("recordScriptVersion", { db: true }, () => { ).toBeNull(); }); + test("reads the restored commit from the primary", async () => { + setBuildCommitForTest("abc123def4567890"); + await recordScriptVersion(); + const client = getDb(); + const originalBatch = client.batch.bind(client); + const modes: unknown[] = []; + const batchStub = stub(client, "batch", ((statements, mode) => { + modes.push(mode); + return originalBatch(statements, mode); + }) as typeof client.batch); + + try { + expect(await readRecordedScriptCommit()).toBe("abc123def4567890"); + } finally { + batchStub.restore(); + } + expect(modes).toEqual(["write"]); + }); + test("stores the exact rows for each available build marker", async () => { await clearMarkers(); setBuildTimestampForTest(""); diff --git a/test/ui/templates/admin/backup.test.ts b/test/ui/templates/admin/backup.test.ts index dd826974b2..360ae006c6 100644 --- a/test/ui/templates/admin/backup.test.ts +++ b/test/ui/templates/admin/backup.test.ts @@ -2,9 +2,7 @@ import { expect } from "@std/expect"; import { it as test } from "@std/testing/bdd"; import { adminBackupPage, - adminRestoreConfirmPage, type BackupPageState, - RESTORE_CONFIRM_PHRASE, } from "#templates/admin/backup.tsx"; import { describeWithEnv } from "#test-utils/db.ts"; @@ -26,12 +24,14 @@ const baseState: BackupPageState = { describeWithEnv("backup template", { encryptionKey: true }, () => { test("renders page title", () => { const html = adminBackupPage(mockSession, baseState); - expect(html).toContain("Database Backup"); + expect(html).toContain("Database backup"); + expect(html).toContain('class="active" href="/admin/backup"'); }); test("displays encryption key", () => { const html = adminBackupPage(mockSession, baseState); expect(html).toContain(baseState.encryptionKey); + expect(html).toContain('

Encryption key'); }); test("shows local database warning when not remote", () => { @@ -57,17 +57,20 @@ describeWithEnv("backup template", { encryptionKey: true }, () => { test("shows create backup form when storage enabled", () => { const html = adminBackupPage(mockSession, baseState); - expect(html).toContain("Create Backup Now"); + expect(html).toContain("Create backup now"); expect(html).toContain("/admin/backup/create"); + expect(html).toContain('class="no-bg"'); + expect(html).toContain('id="backup-create"'); + expect(html).toContain('

Create backup'); }); - test("hides backup forms when storage disabled", () => { + test("hides the create form when storage is disabled", () => { const html = adminBackupPage(mockSession, { ...baseState, storageEnabled: false, }); - expect(html).not.toContain("Create Backup Now"); - expect(html).not.toContain("Restore from Backup"); + expect(html).not.toContain("Create backup now"); + expect(html).toContain("Restore from backup"); }); test("shows no backups message when list is empty", () => { @@ -110,7 +113,7 @@ describeWithEnv("backup template", { encryptionKey: true }, () => { ], maxBackups: 30, }); - expect(html).toContain('class="prose"'); + expect(html).toContain('

There is 1 backup'); expect(html).toContain("There is 1 backup"); expect(html).toContain("Up to 30 are kept"); expect(html).toContain( @@ -118,6 +121,23 @@ describeWithEnv("backup template", { encryptionKey: true }, () => { ); }); + test("retention keeps the last available backup slot", () => { + const html = adminBackupPage(mockSession, { + ...baseState, + backups: [ + { + filename: "backup-2024-01-15T12-00-00-000Z.zip", + label: "Monday 15 January 2024 at 12:00 UTC", + sizeLabel: "1MB", + timestamp: "2024-01-15T12-00-00-000Z", + }, + ], + maxBackups: 2, + }); + expect(html).toContain("1 more can be created before the oldest is purged"); + expect(html).not.toContain("the next will purge the oldest"); + }); + test("retention note warns the oldest is purged next when at capacity", () => { const entry = (n: number): BackupPageState["backups"][number] => { const day = String(n).padStart(2, "0"); @@ -138,11 +158,15 @@ describeWithEnv("backup template", { encryptionKey: true }, () => { expect(html).toContain("the next will purge the oldest (backup 1)"); }); - test("renders restore form with file upload for .zip", () => { + test("shows the out-of-band restore command without a web form", () => { const html = adminBackupPage(mockSession, baseState); - expect(html).toContain("Restore from Backup"); - expect(html).toContain('type="file"'); - expect(html).toContain('accept=".zip"'); + expect(html).toContain("deno task restore <backup.zip>"); + expect(html).toContain("large restore exceeds the edge request limit"); + expect(html).not.toContain("/admin/backup/restore"); + expect(html).toContain( + '

Restore from backup', + ); + expect(html).toContain('href="/admin/guide#backups"'); }); test("shows error message when provided", () => { @@ -165,64 +189,4 @@ describeWithEnv("backup template", { encryptionKey: true }, () => { expect(html).toContain("Backup created"); expect(html).toContain("success"); }); - - test("restore confirm page shows statement count", () => { - const html = adminRestoreConfirmPage(mockSession, "test.zip", 42); - expect(html).toContain("42"); - expect(html).toContain("SQL statements"); - }); - - test("restore confirm page shows confirmation phrase", () => { - const html = adminRestoreConfirmPage(mockSession, "test.zip", 10); - expect(html).toContain(RESTORE_CONFIRM_PHRASE); - }); - - test("restore confirm page includes hidden backup filename", () => { - const html = adminRestoreConfirmPage( - mockSession, - "restore-pending-abc.zip", - 5, - ); - expect(html).toContain("restore-pending-abc.zip"); - expect(html).toContain('name="backup_filename"'); - }); - - test("restore confirm page renders error as HTML not escaped text", () => { - const html = adminRestoreConfirmPage( - mockSession, - "test.zip", - 10, - "Phrase mismatch", - ); - expect(html).toContain("Phrase mismatch"); - // Should be rendered as HTML div, not escaped - expect(html).toContain('class="error"'); - }); - - test("restore confirm page shows schema mismatch warning", () => { - const html = adminRestoreConfirmPage( - mockSession, - "test.zip", - 10, - undefined, - true, - ); - expect(html).toContain("Schema mismatch"); - expect(html).toContain("different database schema version"); - }); - - test("restore confirm page hides schema warning when schemas match", () => { - const html = adminRestoreConfirmPage( - mockSession, - "test.zip", - 10, - undefined, - false, - ); - expect(html).not.toContain("Schema mismatch"); - }); - - test("RESTORE_CONFIRM_PHRASE describes the danger", () => { - expect(RESTORE_CONFIRM_PHRASE).toContain("dangerous"); - }); });