Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,7 @@ logging and table-scoped cache invalidation stay automatic.
- `deno task lint:ci` - Strict, read-only lint (`check --error-on-warnings`, no `--write`). Fails on lint warnings (e.g. cognitive complexity) and on any code that *would* be reformatted, without touching the checkout. This is the lint `deno task precommit` runs in **every** environment, so a clean `precommit` locally means the lint step will pass in CI too. Run `deno task lint` to auto-fix before re-running.
- `deno task build:edge` - Build for Bunny Edge deployment
- `deno task backup` - Dump the database out-of-band to a `.zip`. Uploads to the configured storage zone by default (so it appears on the Backups page and lets the next migration skip its own inline backup); pass `--out <path>` to write a local file. Runs in a full Deno process, so unlike the in-edge backup it has no per-request subrequest budget and can dump arbitrarily large databases.
- `deno task restore <backup.zip>` - Restore the database named by `DB_URL` / `DB_TOKEN` in `.env` using its `DB_ENCRYPTION_KEY`. Shows the backup details, asks for typed confirmation, and reports each restore step in the console.
- `deno task snapshot --out <path.sqlite>` - Sync the complete remote database to a standalone local SQLite file. The task prefers `DB_URL` and `DB_TOKEN` from `.env` over shell values. This developer-only task checkpoints and verifies the file, refuses to overwrite an existing path, and removes its temporary replica on success or failure.
- `deno task precommit` - Run all checks (typecheck, lint, tests)
- `deno task precommit:mutation` - The precommit mutation gate, runnable on its own: mutation-test every `src/` file this branch changed and demand a 100% kill rate. A source's mirror-located direct tests run first; changed tests under `test/integration/` or `test/e2e/` run only for direct-test survivors. Any other unmatched test is an error: move it to the path that mirrors its source while strengthening it to kill the mutants. The changed set is the branch's committed diff against the integration branch (`origin/main`, else a local `main`) via `base...HEAD` — three-dot/merge-base, so it's the branch's full diff vs main and stays bounded to the branch's own commits (precommit runs post-commit on a clean tree, so the index is empty). Skips cheaply when there is no base ref or no changed `src/` files (and likewise when src changed without any changed test). If a badly stale local `origin/main` balloons the changed set past `STALE_BASE_SOURCE_LIMIT`, it skips with a "run `git fetch origin main`" hint instead of mutating most of the tree. See [Mutation Testing](#mutation-testing).
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,7 @@ deno task typecheck # Type check
deno task build:edge # Build for Bunny Edge
deno task deploy:edge <script-id> # Build, upload, and publish to Bunny Edge using BUNNY_ACCESS_KEY from .env
deno task backup # Dump the database out-of-band (uploads to storage; --out <path> for a local .zip)
deno task restore <backup.zip> # Restore the database at DB_URL from a local backup using DB_TOKEN from .env
deno task snapshot --out <path.sqlite> # Copy the complete remote database to a local SQLite file
deno task precommit # All checks (typecheck, lint, cpd, build:edge, test:coverage)
```
Expand Down Expand Up @@ -361,6 +362,13 @@ Optional:

**Backups:** every table is dumped to a single `.zip`, with table reads keyset-paginated so no single response trips libsqld's "Response is too large" payload cap (the server limit behind Bunny's databases). Backups run **out-of-band**, not inside the migration: a full dump of a ~31-table schema can't fit alongside a migration within one edge request's [50-subrequest budget](https://docs.bunny.net/scripting/limits), so migrations just migrate, and a backup is taken by GitHub Actions (or `deno task backup`) beforehand. To enforce that, **`/admin/update` and the per-site update button refuse to deploy unless a backup of that database was taken in the last hour.**

Restores run out of band because realistic imports exceed Bunny's per-request
subrequest limit. Put the target database's `DB_URL`, `DB_TOKEN`, and
`DB_ENCRYPTION_KEY` in `.env`, then run `deno task restore <backup.zip>`. The task checks
the backup, shows its available table, row, statement, and schema details, and
asks for a typed confirmation before deleting any data. It reports each restore
step and shows the backup's recorded commit when one is available.

**Local SQLite snapshots:** developers can copy a remote database into one complete SQLite file using the same `DB_URL` and `DB_TOKEN` as the app:

```bash
Expand Down
3 changes: 2 additions & 1 deletion deno.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"test:raw": "deno task build:static && deno test --no-check --allow-net --allow-env --allow-read --allow-write --allow-run --allow-sys --allow-ffi test/",
"lint": "deno run -A scripts/biome.ts check --write --error-on-warnings src test scripts cli e2e-payments",
"lint:ci": "deno run -A scripts/biome.ts check --error-on-warnings src test scripts cli e2e-payments",
"typecheck": "deno check src/**/*.ts src/**/*.tsx test/**/*.ts cli/**/*.ts scripts/database-snapshot.ts scripts/mutation.ts scripts/mutation/**/*.ts scripts/precommit.ts scripts/precommit/**/*.ts scripts/check-copy.ts scripts/check-copy/**/*.ts scripts/bench/bundle-composition/**/*.ts scripts/screenshots.ts scripts/screenshots/**/*.ts && deno check --config=e2e-payments/deno.json e2e-payments/src/**/*.ts",
"typecheck": "deno check src/**/*.ts src/**/*.tsx test/**/*.ts cli/**/*.ts scripts/database-snapshot.ts scripts/restore.ts scripts/mutation.ts scripts/mutation/**/*.ts scripts/precommit.ts scripts/precommit/**/*.ts scripts/check-copy.ts scripts/check-copy/**/*.ts scripts/bench/bundle-composition/**/*.ts scripts/screenshots.ts scripts/screenshots/**/*.ts && deno check --config=e2e-payments/deno.json e2e-payments/src/**/*.ts",
"cpd": "deno run -A scripts/cpd.ts --config .jscpd.json && deno run -A scripts/cpd.ts --config .jscpd.test.json && deno run -A scripts/cpd.ts --config .jscpd.css.json src/ui/static/style.scss",
"check:copy": "deno run --allow-read=src/locales scripts/check-copy.ts",
"precommit": "deno run -A scripts/precommit.ts",
Expand All @@ -27,6 +27,7 @@
"unit-tests-report": "deno run --allow-read=src,test,deno.json scripts/unit-tests-report.ts",
"upgrade": "deno run --allow-read --allow-write --allow-net scripts/safe-upgrade.ts",
"backup": "deno run --env-file --allow-env --allow-read --allow-write --allow-net --allow-sys --allow-ffi scripts/backup.ts",
"restore": "deno run --allow-env --allow-read --allow-write --allow-net --allow-sys --allow-ffi scripts/restore.ts",
"snapshot": "deno run --allow-env --allow-read --allow-write --allow-net --allow-sys --allow-ffi scripts/database-snapshot.ts",
"cli:tui": "deno run --allow-env --allow-read --allow-run cli/tui.ts",
"cli:api": "deno run --allow-env --allow-read --allow-run cli/api.ts",
Expand Down
7 changes: 2 additions & 5 deletions scripts/deploy-edge-lib.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { mapNotNullish } from "#fp";
import type { ScriptIo } from "#scripts/script-runner.ts";

export const BUNNY_API_BASE = "https://api.bunny.net";
export const BUNDLE_PATH = "bunny-script.ts";
Expand Down Expand Up @@ -32,16 +33,12 @@ export interface BuildResult {
success: boolean;
}

export interface DeployEdgeDeps {
args: string[];
export interface DeployEdgeDeps extends ScriptIo {
bundlePath: string;
cwd: string;
fetchText: FetchText;
getEnv: (key: string) => string | undefined;
readTextFile: (path: string) => Promise<string>;
runBuildEdge: (cwd: string) => Promise<BuildResult>;
stderr: (line: string) => void;
stdout: (line: string) => void;
}

export const parseScriptIdArg = (
Expand Down
24 changes: 11 additions & 13 deletions scripts/deploy-edge.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env -S deno run --env-file --allow-env --allow-read --allow-write --allow-run --allow-net

import { fromFileUrl } from "@std/path";
import { runDenoScript } from "#scripts/script-runner.ts";
import { type FetchTextResult, runDeployEdge } from "./deploy-edge-lib.ts";
import { runBuildEdge } from "./run-build-edge.ts";

Expand All @@ -19,16 +20,13 @@ const fetchText = async (
};
};

const exitCode = await runDeployEdge({
args: Deno.args,
bundlePath,
cwd: repoRoot,
fetchText,
getEnv: (key) => Deno.env.get(key),
readTextFile: (path) => Deno.readTextFile(path),
runBuildEdge,
stderr: (line) => console.error(line),
stdout: (line) => console.log(line),
});

Deno.exit(exitCode);
await runDenoScript((io) =>
runDeployEdge({
...io,
bundlePath,
cwd: repoRoot,
fetchText,
readTextFile: (path) => Deno.readTextFile(path),
runBuildEdge,
}),
);
3 changes: 3 additions & 0 deletions scripts/mutation/equivalent-mutants.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1049,3 +1049,6 @@ src/shared/stripe/request.ts:152:15 === → == # Headers.get returns string|n
# only for that value-less case, so every parsed result is identical.
src/shared/stripe/webhook.ts:28:15 ?: → alternate only # with "=", both arms slice the same key; without it, no value can be consumed
src/shared/stripe/webhook.ts:28:30 1 → 2 # this changes only the no-separator key arm; value-less recognized and unknown keys are all ignored

# Schema indexes are an array when present, and arrays are always truthy.
src/shared/db/migrations/schema-sync.ts:149:20 ?? → || # table.indexes is Index[]|undefined, so every present value uses the left side and undefined uses [] with either operator
196 changes: 196 additions & 0 deletions scripts/restore-lib.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
import { sum } from "#fp";
import type { ScriptIo } from "#scripts/script-runner.ts";
import { decodeKeyBytes } from "#shared/crypto/encryption.ts";
import {
type BackupManifest,
PostResetError,
type RestoreProgress,
type RestoreProgressHandler,
type RestoreStage,
} from "#shared/db/backup.ts";
import { SCHEMA_HASH } from "#shared/db/migrations.ts";
import { errorMessage } from "#shared/error-message.ts";
import { formatBytes } from "#shared/limits.ts";

export const RESTORE_CONFIRMATION = "RESTORE";
export const RESTORE_USAGE = "Usage: deno task restore <backup.zip>";
const isFullCommitSha = (commit: string): boolean =>
/^[0-9a-f]{40}$/.test(commit);
const REMOTE_DB_URL_PREFIXES = ["https://", "libsql://"];

export interface RestoreCliDeps extends ScriptIo {
inspectBackupZip: (data: Uint8Array) => {
manifest: BackupManifest | null;
statementCount: number;
};
prompt: (message: string) => string | null;
readFile: (path: string) => Promise<Uint8Array>;
readRecordedScriptCommit: () => Promise<string>;
restoreFromZip: (
data: Uint8Array,
onProgress: RestoreProgressHandler,
) => Promise<void>;
}

const progressMessages: Record<
RestoreStage,
(progress: RestoreProgress) => string
> = {
checking: ({ statementCount }) =>
`Checking ${statementCount} SQL statements...`,
"clearing-caches": () => "Clearing cached database data...",
importing: ({ statementCount }) =>
`Importing ${statementCount} SQL statements...`,
rebuilding: () => "Recreating the database schema...",
resetting: () => "Deleting current database data...",
};

const countLabel = (count: number, name: string): string =>
`${count} ${name}${count === 1 ? "" : "s"}`;

const readRestoreDbUrlOrNull = (deps: RestoreCliDeps): string | null => {
const dbUrl = deps.getEnv("DB_URL");
if (!dbUrl?.trim()) {
deps.stderr("DB_URL is required in .env.");
return null;
}
if (dbUrl === ":memory:") {
deps.stderr(
"DB_URL cannot be :memory: for a restore. Set it to the target database in .env.",
);
return null;
}
if (
REMOTE_DB_URL_PREFIXES.some((prefix) => dbUrl.startsWith(prefix)) &&
!deps.getEnv("DB_TOKEN")?.trim()
) {
deps.stderr("DB_TOKEN is required in .env for a remote database.");
return null;
}
const encryptionKey = deps.getEnv("DB_ENCRYPTION_KEY");
if (!encryptionKey?.trim()) {
deps.stderr("DB_ENCRYPTION_KEY is required in .env.");
return null;
}
try {
decodeKeyBytes(encryptionKey);
} catch (error) {
deps.stderr(errorMessage(error));
return null;
}
return dbUrl;
};

const writeManifestSummary = (
manifest: BackupManifest | null,
statementCount: number,
write: (line: string) => void,
): boolean => {
if (manifest === null) {
write("Backup manifest: not available");
write(`Backup contents: ${countLabel(statementCount, "SQL statement")}`);
return false;
}

const tableCount = Object.keys(manifest.tables).length;
const rowCount = sum(Object.values(manifest.tables));
write(`Backup created: ${manifest.timestamp}`);
write(
`Backup contents: ${countLabel(tableCount, "table")}, ` +
`${countLabel(rowCount, "row")}, ` +
countLabel(statementCount, "SQL statement"),
);
const schemaMismatch = manifest.schemaHash !== SCHEMA_HASH;
write(
schemaMismatch
? "Schema: does not match this version of the app"
: "Schema: matches this version of the app",
);
return schemaMismatch;
};

export const runRestoreCli = async (deps: RestoreCliDeps): Promise<number> => {
if (deps.args.length !== 1 || !deps.args[0]?.trim()) {
deps.stderr(RESTORE_USAGE);
return 1;
}

const path = deps.args[0];
const dbUrl = readRestoreDbUrlOrNull(deps);
if (dbUrl === null) return 1;

deps.stdout(`Reading ${path}...`);
let data: Uint8Array;
try {
data = await deps.readFile(path);
} catch (error) {
deps.stderr(`Could not read ${path}: ${errorMessage(error)}`);
return 1;
}

let inspection: ReturnType<RestoreCliDeps["inspectBackupZip"]>;
try {
inspection = deps.inspectBackupZip(data);
} catch (error) {
deps.stderr(`${path} is not a valid database backup.`);
deps.stderr(errorMessage(error));
return 1;
}

deps.stdout(`Backup file: ${path} (${formatBytes(data.byteLength)})`);
deps.stdout(`Target database: ${dbUrl}`);
const schemaMismatch = writeManifestSummary(
inspection.manifest,
inspection.statementCount,
deps.stdout,
);
if (schemaMismatch) {
deps.stderr(
"Warning: This backup uses a different database schema. The app may need to apply database updates after the restore.",
);
}
deps.stdout(
"Warning: This will delete all current data in the target database. This cannot be undone.",
);

if (
deps.prompt(`Type ${RESTORE_CONFIRMATION} to continue:`) !==
RESTORE_CONFIRMATION
) {
deps.stderr("Restore cancelled. The database was not changed.");
return 1;
}

try {
await deps.restoreFromZip(data, (progress) =>
deps.stdout(progressMessages[progress.stage](progress)),
);
} catch (error) {
deps.stderr(`Restore failed: ${errorMessage(error)}`);
if (error instanceof PostResetError) {
deps.stderr(
"The target database was reset before the restore failed. Fix the error and run the restore again.",
);
}
return 1;
}

deps.stdout("Database restored successfully.");
let commit: string;
try {
commit = await deps.readRecordedScriptCommit();
Comment thread
stefan-burke marked this conversation as resolved.
Resolved
} catch (error) {
deps.stderr(
"The database was restored, but its build information could not be read: " +
errorMessage(error),
);
return 1;
}
if (isFullCommitSha(commit)) {
deps.stdout(`The backup was running commit ${commit}.`);
deps.stdout(
"Run the restore-deploy workflow with this commit if the app code must also be restored to that point in time.",
);
}
return 0;
};
12 changes: 12 additions & 0 deletions scripts/restore.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/usr/bin/env -S deno run --allow-env --allow-read --allow-write --allow-net --allow-sys --allow-ffi

import { load } from "@std/dotenv";
import { runDenoScript } from "#scripts/script-runner.ts";
import { runRestoreTask } from "#src/restore.ts";

await runRestoreTask(
await load(),
(key, value) =>
value === undefined ? Deno.env.delete(key) : Deno.env.set(key, value),
runDenoScript,
);
19 changes: 19 additions & 0 deletions scripts/script-runner.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
export interface ScriptIo {
args: string[];
getEnv: (key: string) => string | undefined;
stderr: (line: string) => void;
stdout: (line: string) => void;
}

/** Give a console task its standard Deno inputs and exit with its result. */
export const runDenoScript = async (
run: (io: ScriptIo) => Promise<number>,
): Promise<never> => {
const exitCode = await run({
args: Deno.args,
getEnv: (key) => Deno.env.get(key),
stderr: (line) => console.error(line),
stdout: (line) => console.log(line),
});
return Deno.exit(exitCode);
};
Loading