Repository navigation
Restore database backups safely and quickly from the console #1886
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 9 commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
fa46dd1
Add safe out-of-band database restore
stefan-burke 879ccbc
Place restore coverage with integration tests
stefan-burke 7f64315
Test the restore task entry point
stefan-burke 1fd0882
Speed up out-of-band restores
stefan-burke 6763a76
Keep restore batching in the restore path
stefan-burke 076cc0a
Fix restore review findings
stefan-burke 2a9bde2
Merge main into restore backup task
stefan-burke 40620c9
Keep restore integrity checks active
stefan-burke bc2a068
Derive restore trigger groups
stefan-burke 142d5de
Isolate restore schema work
stefan-burke cf39824
Validate restore archives before reset
stefan-burke 7288dca
Merge remote-tracking branch 'origin/main' into restore-backup-task
stefan-burke 9e5565b
Name nullable manifest lookup clearly
stefan-burke 5599efb
Validate legacy restores and primary reads
stefan-burke c5f9fa6
Merge main into restore backup task
stefan-burke File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,196 @@ | ||
| import { sum } from "#fp"; | ||
| import type { ScriptIo } from "#scripts/script-runner.ts"; | ||
| import { decodeKeyBytes } from "#shared/crypto/encryption.ts"; | ||
| import { | ||
| type BackupManifest, | ||
| PostResetError, | ||
| type RestoreProgress, | ||
| type RestoreProgressHandler, | ||
| type RestoreStage, | ||
| } from "#shared/db/backup.ts"; | ||
| import { SCHEMA_HASH } from "#shared/db/migrations.ts"; | ||
| import { errorMessage } from "#shared/error-message.ts"; | ||
| import { formatBytes } from "#shared/limits.ts"; | ||
|
|
||
| export const RESTORE_CONFIRMATION = "RESTORE"; | ||
| export const RESTORE_USAGE = "Usage: deno task restore <backup.zip>"; | ||
| const isFullCommitSha = (commit: string): boolean => | ||
| /^[0-9a-f]{40}$/.test(commit); | ||
| const REMOTE_DB_URL_PREFIXES = ["https://", "libsql://"]; | ||
|
|
||
| export interface RestoreCliDeps extends ScriptIo { | ||
| inspectBackupZip: (data: Uint8Array) => { | ||
| manifest: BackupManifest | null; | ||
| statementCount: number; | ||
| }; | ||
| prompt: (message: string) => string | null; | ||
| readFile: (path: string) => Promise<Uint8Array>; | ||
| readRecordedScriptCommit: () => Promise<string>; | ||
| restoreFromZip: ( | ||
| data: Uint8Array, | ||
| onProgress: RestoreProgressHandler, | ||
| ) => Promise<void>; | ||
| } | ||
|
|
||
| const progressMessages: Record< | ||
| RestoreStage, | ||
| (progress: RestoreProgress) => string | ||
| > = { | ||
| checking: ({ statementCount }) => | ||
| `Checking ${statementCount} SQL statements...`, | ||
| "clearing-caches": () => "Clearing cached database data...", | ||
| importing: ({ statementCount }) => | ||
| `Importing ${statementCount} SQL statements...`, | ||
| rebuilding: () => "Recreating the database schema...", | ||
| resetting: () => "Deleting current database data...", | ||
| }; | ||
|
|
||
| const countLabel = (count: number, name: string): string => | ||
| `${count} ${name}${count === 1 ? "" : "s"}`; | ||
|
|
||
| const readRestoreDbUrlOrNull = (deps: RestoreCliDeps): string | null => { | ||
| const dbUrl = deps.getEnv("DB_URL"); | ||
| if (!dbUrl?.trim()) { | ||
| deps.stderr("DB_URL is required in .env."); | ||
| return null; | ||
| } | ||
| if (dbUrl === ":memory:") { | ||
| deps.stderr( | ||
| "DB_URL cannot be :memory: for a restore. Set it to the target database in .env.", | ||
| ); | ||
| return null; | ||
| } | ||
| if ( | ||
| REMOTE_DB_URL_PREFIXES.some((prefix) => dbUrl.startsWith(prefix)) && | ||
| !deps.getEnv("DB_TOKEN")?.trim() | ||
| ) { | ||
| deps.stderr("DB_TOKEN is required in .env for a remote database."); | ||
| return null; | ||
| } | ||
| const encryptionKey = deps.getEnv("DB_ENCRYPTION_KEY"); | ||
| if (!encryptionKey?.trim()) { | ||
| deps.stderr("DB_ENCRYPTION_KEY is required in .env."); | ||
| return null; | ||
| } | ||
| try { | ||
| decodeKeyBytes(encryptionKey); | ||
| } catch (error) { | ||
| deps.stderr(errorMessage(error)); | ||
| return null; | ||
| } | ||
| return dbUrl; | ||
| }; | ||
|
|
||
| const writeManifestSummary = ( | ||
| manifest: BackupManifest | null, | ||
| statementCount: number, | ||
| write: (line: string) => void, | ||
| ): boolean => { | ||
| if (manifest === null) { | ||
| write("Backup manifest: not available"); | ||
| write(`Backup contents: ${countLabel(statementCount, "SQL statement")}`); | ||
| return false; | ||
| } | ||
|
|
||
| const tableCount = Object.keys(manifest.tables).length; | ||
| const rowCount = sum(Object.values(manifest.tables)); | ||
| write(`Backup created: ${manifest.timestamp}`); | ||
| write( | ||
| `Backup contents: ${countLabel(tableCount, "table")}, ` + | ||
| `${countLabel(rowCount, "row")}, ` + | ||
| countLabel(statementCount, "SQL statement"), | ||
| ); | ||
| const schemaMismatch = manifest.schemaHash !== SCHEMA_HASH; | ||
| write( | ||
| schemaMismatch | ||
| ? "Schema: does not match this version of the app" | ||
| : "Schema: matches this version of the app", | ||
| ); | ||
| return schemaMismatch; | ||
| }; | ||
|
|
||
| export const runRestoreCli = async (deps: RestoreCliDeps): Promise<number> => { | ||
| if (deps.args.length !== 1 || !deps.args[0]?.trim()) { | ||
| deps.stderr(RESTORE_USAGE); | ||
| return 1; | ||
| } | ||
|
|
||
| const path = deps.args[0]; | ||
| const dbUrl = readRestoreDbUrlOrNull(deps); | ||
| if (dbUrl === null) return 1; | ||
|
|
||
| deps.stdout(`Reading ${path}...`); | ||
| let data: Uint8Array; | ||
| try { | ||
| data = await deps.readFile(path); | ||
| } catch (error) { | ||
| deps.stderr(`Could not read ${path}: ${errorMessage(error)}`); | ||
| return 1; | ||
| } | ||
|
|
||
| let inspection: ReturnType<RestoreCliDeps["inspectBackupZip"]>; | ||
| try { | ||
| inspection = deps.inspectBackupZip(data); | ||
| } catch (error) { | ||
| deps.stderr(`${path} is not a valid database backup.`); | ||
| deps.stderr(errorMessage(error)); | ||
| return 1; | ||
| } | ||
|
|
||
| deps.stdout(`Backup file: ${path} (${formatBytes(data.byteLength)})`); | ||
| deps.stdout(`Target database: ${dbUrl}`); | ||
| const schemaMismatch = writeManifestSummary( | ||
| inspection.manifest, | ||
| inspection.statementCount, | ||
| deps.stdout, | ||
| ); | ||
| if (schemaMismatch) { | ||
| deps.stderr( | ||
| "Warning: This backup uses a different database schema. The app may need to apply database updates after the restore.", | ||
| ); | ||
| } | ||
| deps.stdout( | ||
| "Warning: This will delete all current data in the target database. This cannot be undone.", | ||
| ); | ||
|
|
||
| if ( | ||
| deps.prompt(`Type ${RESTORE_CONFIRMATION} to continue:`) !== | ||
| RESTORE_CONFIRMATION | ||
| ) { | ||
| deps.stderr("Restore cancelled. The database was not changed."); | ||
| return 1; | ||
| } | ||
|
|
||
| try { | ||
| await deps.restoreFromZip(data, (progress) => | ||
| deps.stdout(progressMessages[progress.stage](progress)), | ||
| ); | ||
| } catch (error) { | ||
| deps.stderr(`Restore failed: ${errorMessage(error)}`); | ||
| if (error instanceof PostResetError) { | ||
| deps.stderr( | ||
| "The target database was reset before the restore failed. Fix the error and run the restore again.", | ||
| ); | ||
| } | ||
| return 1; | ||
| } | ||
|
|
||
| deps.stdout("Database restored successfully."); | ||
| let commit: string; | ||
| try { | ||
| commit = await deps.readRecordedScriptCommit(); | ||
| } catch (error) { | ||
| deps.stderr( | ||
| "The database was restored, but its build information could not be read: " + | ||
| errorMessage(error), | ||
| ); | ||
| return 1; | ||
| } | ||
| if (isFullCommitSha(commit)) { | ||
| deps.stdout(`The backup was running commit ${commit}.`); | ||
| deps.stdout( | ||
| "Run the restore-deploy workflow with this commit if the app code must also be restored to that point in time.", | ||
| ); | ||
| } | ||
| return 0; | ||
| }; | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| #!/usr/bin/env -S deno run --allow-env --allow-read --allow-write --allow-net --allow-sys --allow-ffi | ||
|
|
||
| import { load } from "@std/dotenv"; | ||
| import { runDenoScript } from "#scripts/script-runner.ts"; | ||
| import { runRestoreTask } from "#src/restore.ts"; | ||
|
|
||
| await runRestoreTask( | ||
| await load(), | ||
| (key, value) => | ||
| value === undefined ? Deno.env.delete(key) : Deno.env.set(key, value), | ||
| runDenoScript, | ||
| ); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| export interface ScriptIo { | ||
| args: string[]; | ||
| getEnv: (key: string) => string | undefined; | ||
| stderr: (line: string) => void; | ||
| stdout: (line: string) => void; | ||
| } | ||
|
|
||
| /** Give a console task its standard Deno inputs and exit with its result. */ | ||
| export const runDenoScript = async ( | ||
| run: (io: ScriptIo) => Promise<number>, | ||
| ): Promise<never> => { | ||
| const exitCode = await run({ | ||
| args: Deno.args, | ||
| getEnv: (key) => Deno.env.get(key), | ||
| stderr: (line) => console.error(line), | ||
| stdout: (line) => console.log(line), | ||
| }); | ||
| return Deno.exit(exitCode); | ||
| }; |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.