Skip to content

fix(db): harden migration recovery snapshots #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Bl0ck154 wants to merge 2 commits into
base: integrate/upstream-db-safety
Choose a base branch
from
Draft

fix(db): harden migration recovery snapshots #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9b1ce6c
fix(db): harden migration recovery snapshots (#12435)
diegosouzapw Sep 4, 2026
874299f
fix(db): adapt migration hardening to production branch
Bl0ck154 Sep 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,11 @@ INITIAL_PASSWORD=CHANGEME
# executor's on-disk thread-sticky session cache. Leave unset to rely on DATA_DIR.
# OMNIROUTE_DATA_DIR=/var/lib/omniroute

# Escape hatch for the test-context DATA_DIR guard (#10428). A test run that never
# chose a DATA_DIR is redirected to a throwaway temp dir so it cannot open the
# operator's real database. Set to 1 only for a deliberate run against the real
# DATA_DIR — never for CI. Used by: src/lib/dataPaths.ts
# Escape hatch for the test/eval DATA_DIR guard (#10428). A test run or Node
# eval/print probe (-e/--eval/-p/--print) that never chose a DATA_DIR is redirected
# to a throwaway temp dir so it cannot open the operator's real database.
# Set to 1 only for a deliberate run against the real DATA_DIR — never for CI.
# Used by: src/lib/dataPaths.ts
# OMNIROUTE_ALLOW_DEFAULT_DATA_DIR=1

# Build provenance (#10427). OMNIROUTE_BUILD_SHA lets a container inject the artifact's git
Expand Down
1 change: 1 addition & 0 deletions changelog.d/fixes/migration-151-152-safety.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Harden SQLite upgrades around the historical migration-074 version collision: missing discovery and inspector tables are replayed atomically, pre-existing databases (including setup-created skeletons) receive reusable content-addressed safety snapshots, and Node test/eval probes without `DATA_DIR` are isolated from the operator database.
2 changes: 1 addition & 1 deletion docs/guides/DOCKER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -612,7 +612,7 @@ In-process density (compression off the HTTP isolate) is [#11023](https://github
## Important Notes

- **SQLite WAL Mode:** `docker stop` should be allowed to finish so OmniRoute can checkpoint the latest changes back into `storage.sqlite`. The bundled Compose files already set a 40s stop grace period. If you run the image directly, keep `--stop-timeout 40`.
- **`DISABLE_SQLITE_AUTO_BACKUP`:** Set to `true` if backups are managed externally.
- **`DISABLE_SQLITE_AUTO_BACKUP`:** Set to `true` if routine/pre-write backups are managed externally. Existing-database migrations still require their own durable safety snapshot and mass-migration guard.
- **Data Persistence:** Always mount a volume to `/app/data` to persist your database, keys, and configurations across container restarts.
- **Port Configuration:** Override `PORT` environment variable to change the default `20128` port.

Expand Down
18 changes: 14 additions & 4 deletions docs/reference/ENVIRONMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ OmniRoute uses **SQLite** (via `better-sqlite3`) for all persistence. These vari
| Variable | Default | Source File | Description |
| -------------------------------------- | -------------------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DATA_DIR` | `~/.omniroute/` | `src/lib/db/core.ts` | Root directory for SQLite DB, backups, and data files. Override for Docker volumes or custom paths. |
| `OMNIROUTE_ALLOW_DEFAULT_DATA_DIR` | _(unset)_ | `src/lib/dataPaths.ts` | Escape hatch for the test-context DATA_DIR guard (#10428). Test runs with no `DATA_DIR` are redirected to a throwaway temp dir so they cannot open the operator's real database; set to `1` to opt back in to the real directory. |
| `OMNIROUTE_ALLOW_DEFAULT_DATA_DIR` | _(unset)_ | `src/lib/dataPaths.ts` | Escape hatch for the test/eval DATA_DIR guard (#10428). Tests and Node eval/print probes (`-e`/`--eval`/`-p`/`--print`, including `--eval=`/`--print=` forms) with no `DATA_DIR` are redirected to a throwaway temp dir so they cannot open the operator's real database; set to `1` to opt back in to the real directory. |
| `OMNIROUTE_BUILD_SHA` | _(unset)_ | `src/lib/monitoring/buildSha.ts` | Git SHA of the running artifact. Stamped by `npm run build:release`; injectable in containers that ship without the `dist/BUILD_SHA` sentinel. Surfaced as `system.buildSha` on `/api/monitoring/health`. |
| `OMNIROUTE_RELEASE_REF` | `origin/main` | `scripts/build/buildProvenance.ts` | Ref the pack-artifact provenance gate checks the build SHA against (#10427). |
| `OMNIROUTE_ALLOW_CANARY_BUILD` | _(unset)_ | `scripts/build/buildProvenance.ts` | Set to `1` to allow packing a build whose SHA is not on the release line, recording it as a deliberate canary instead of failing the gate (#10427). |
Expand All @@ -95,7 +95,7 @@ OmniRoute uses **SQLite** (via `better-sqlite3`) for all persistence. These vari
| `OMNIROUTE_DATA_DIR` | _(unset)_ | `open-sse/executors/promptql/threadSticky.ts` | **Fallback alias** for `DATA_DIR`, checked only when `DATA_DIR` is unset. Used to locate the PromptQL executor's on-disk thread-sticky session cache (`<dir>/promptql-thread-sessions.json`); if neither var is set, the cache stays in-memory only (not persisted across restarts). |
| `STORAGE_ENCRYPTION_KEY` | _(empty = disabled)_ | `src/lib/db/encryption.ts` | AES key for full SQLite database encryption at rest. Generate with `openssl rand -hex 32`. |
| `STORAGE_ENCRYPTION_KEY_VERSION` | `v1` | `scripts/build/bootstrap-env.mjs`, `electron/main.js` | Version label for the encryption key. Increment when performing key rotation to support decryption of old backups. |
| `DISABLE_SQLITE_AUTO_BACKUP` | `false` | `src/lib/db/backup.ts` | When `true`, skips automatic + pre-write SQLite file backups (startup, models.dev pricing save/clear, settings writes). Manual and pre-restore backups still run. Non-manual backups are also **throttled to at most once per 60 minutes** so hourly models.dev sync does not copy the whole DB on every pricing write. Dashboard **Settings → Storage** can disable auto-backup independently. |
| `DISABLE_SQLITE_AUTO_BACKUP` | `false` | `src/lib/db/backup.ts` | When `true`, skips routine/pre-write SQLite file backups (models.dev pricing save/clear, settings writes). Manual and pre-restore backups still run. It does **not** disable the migration runner's mandatory durable safety snapshot or mass-migration guard for an existing persistent DB. Non-manual backups are throttled to at most once per 60 minutes. Dashboard **Settings → Storage** can disable routine auto-backup independently. |
| `OMNIROUTE_CRYPT_KEY` | _(unset)_ | `src/lib/db/encryption.ts` | **Legacy alias** for `STORAGE_ENCRYPTION_KEY`. Accepted as a fallback when the primary variable is absent. |
| `OMNIROUTE_API_KEY_BASE64` | _(unset)_ | `src/lib/db/encryption.ts` | **Legacy alias** (Base64-encoded form) accepted as a fallback. Decoded automatically before use. |
| `OMNIROUTE_DB_HEALTHCHECK_INTERVAL_MS` | _(unset)_ | `src/lib/db/core.ts` | Override the periodic SQLite healthcheck interval (ms). When unset, defaults are derived from `NODE_ENV`. |
Expand All @@ -121,6 +121,16 @@ OmniRoute uses **SQLite** (via `better-sqlite3`) for all persistence. These vari
| `BATCH_BACKOFF_MAX_MS` | `3600000` (1h) | `open-sse/services/batchProcessor.ts` | Cap (ms) for exponential backoff between batch item retries. |
| `BATCH_MAX_CONCURRENT` | `1` | `open-sse/services/batchProcessor.ts` | Maximum number of batches processed concurrently. Raise to increase throughput; keep low to avoid rate-limit storms. |

> [!IMPORTANT]
> Before changing an existing persistent database, the migration runner publishes a complete,
> content-addressed snapshot under `DATA_DIR/db_backups/`. Publication requires a filesystem
> that supports same-filesystem, no-overwrite hard links plus durable file sync. POSIX hosts also
> require directory sync; on Windows, Node may reject directory handles, so OmniRoute flushes the
> published file and treats directory-entry sync as best effort.
> If the mounted `DATA_DIR` cannot provide those guarantees, startup fails closed before applying
> a migration. Move `DATA_DIR` to a volume with those primitives; do not use
> `DISABLE_SQLITE_AUTO_BACKUP` to bypass migration safety.

### Scenarios

| Scenario | Configuration |
Expand Down Expand Up @@ -1306,8 +1316,8 @@ Provider quota endpoints, network tunnels (Tailscale, Ngrok, MITM debug proxy),
| `TAILSCALED_BIN` | _(auto-detect)_ | `src/lib/tailscaleTunnel.ts` | Explicit path to the `tailscaled` daemon binary. |
| `TAILSCALE_AUTHKEY` | _(unset)_ | `src/lib/tailscaleTunnel.ts` | Pre-shared Tailscale auth key for non-interactive / headless `tailscale up` (passed via `--auth-key=`). When unset, login falls back to the interactive browser auth URL. |
| `NGROK_AUTHTOKEN` | _(unset)_ | `src/lib/ngrokTunnel.ts` | Authenticates outbound ngrok tunnels. |
| `DB_BACKUP_MAX_FILES` | `20` | `src/lib/db/backup.ts`, `src/lib/db/migrationRunner.ts` | Maximum SQLite backup files retained on disk. Applies to manual/scheduled backups and to pre-migration snapshots. Overrides the value saved from Settings → Database backup retention. |
| `DB_BACKUP_RETENTION_DAYS` | `0` | `src/lib/db/backup.ts`, `src/lib/db/migrationRunner.ts` | Maximum age (days) of retained backups. `0` disables age-based pruning. Applies to manual/scheduled backups and to pre-migration snapshots. Overrides the value saved from Settings → Database backup retention. |
| `DB_BACKUP_MAX_FILES` | `20` | `src/lib/db/backup.ts` | Maximum SQLite backup files retained by manual/scheduled backup cleanup. Migration snapshots are content-addressed and reused for an identical DB state; they are not pruned inside the concurrent migration window. Overrides the value saved from Settings → Database backup retention. |
| `DB_BACKUP_RETENTION_DAYS` | `0` | `src/lib/db/backup.ts` | Maximum age (days) retained by manual/scheduled backup cleanup. `0` disables age-based pruning. Migration snapshots are not pruned inside the concurrent migration window. Overrides the value saved from Settings → Database backup retention. |
| `OMNIROUTE_BACKUP_SCHEDULE_JOB_INTERVAL_MS` | `30000` | `src/lib/jobs/backupScheduleJob.ts` | Tick interval (ms) of the server-side job that executes `backup-schedule.json`. Must stay well under the 1-minute cron granularity; values below `5000` or unparseable fall back to `30000`. |
| `CONTAINER_HOST` | `docker` | `scripts/check-permissions.sh` | Container runtime hint for the entrypoint permission check. Set to `podman` for any Podman topology. Because the container cannot determine whether the engine is local or reached through Podman Machine, the warning stays topology-neutral and points to `contrib/podman/README.md`. |
| `QUOTA_STORE_DRIVER` | `sqlite` | `src/lib/quota/storeFactory.ts` | Quota-share consumption store backend: `sqlite` (default) or `redis`. |
Expand Down
49 changes: 44 additions & 5 deletions src/lib/dataPaths.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,38 +101,77 @@ export function isTestContext(): boolean {
);
}

/**
* `node --eval` / `node -e` (and their print variants) are common shapes used by
* one-off import probes.
* Such a process has no application entry point from which to establish storage intent,
* so defaulting it to the operator's durable database is unsafe. A deliberate production
* inspection can still opt in with an explicit DATA_DIR (preferred) or
* OMNIROUTE_ALLOW_DEFAULT_DATA_DIR=1.
*/
function isEvalProbeContext(): boolean {
return process.execArgv.some(
(arg) =>
arg === "--eval" ||
arg === "-e" ||
arg === "-pe" ||
arg === "-ep" ||
arg.startsWith("--eval=") ||
arg === "--print" ||
arg === "-p" ||
arg.startsWith("--print=")
);
}

/** Process-wide redirect target, so repeated calls share one DB instead of one per call. */
let testContextDataDir: string | null = null;
let testContextCleanupRegistered = false;

export function resolveWritableDataDir({ isCloud = false }: { isCloud?: boolean } = {}): string {
const resolved = resolveDataDir({ isCloud });
const configured = normalizeConfiguredPath(process.env.DATA_DIR);

// Cloud/serverless never owns a writable home dir; leave its sentinel alone.
if (isCloud) return resolved;

// #10428: a test/ad-hoc run that never chose a DATA_DIR would otherwise open the
// #10428: a test/eval-probe run that never chose a DATA_DIR would otherwise open the
// OPERATOR'S REAL database (~/.omniroute/storage.sqlite — live provider credentials).
// Redirect to a throwaway dir instead of throwing: the documented single-file command
// (`node --import tsx/esm --test tests/unit/x.test.ts`) does not load the isolation
// setup, and a hard failure there would only teach people to disable the guard.
// `OMNIROUTE_ALLOW_DEFAULT_DATA_DIR=1` opts back in, so the intent is recorded.
if (
!process.env.DATA_DIR &&
isTestContext() &&
!configured &&
(isTestContext() || isEvalProbeContext()) &&
process.env.OMNIROUTE_ALLOW_DEFAULT_DATA_DIR !== "1"
) {
if (!testContextDataDir) {
testContextDataDir = fs.mkdtempSync(path.join(os.tmpdir(), `${APP_NAME}-testctx-`));
if (!testContextCleanupRegistered) {
testContextCleanupRegistered = true;
process.once("exit", () => {
if (!testContextDataDir) return;
try {
fs.rmSync(testContextDataDir, {
recursive: true,
force: true,
maxRetries: 5,
retryDelay: 25,
});
} catch {
// An unclean exit is left to the operating system's temp-directory policy.
}
});
}
console.warn(
`[DATA_DIR] test context without DATA_DIR → using '${testContextDataDir}' instead of ` +
`[DATA_DIR] test/eval context without DATA_DIR → using '${testContextDataDir}' instead of ` +
`'${resolved}'. Set DATA_DIR explicitly (or load tests/_setup/isolateDataDir.ts) to silence this.`
);
}
return testContextDataDir;
}

// No explicit override → already the default user dir; nothing to fall back to.
const configured = normalizeConfiguredPath(process.env.DATA_DIR);
if (!configured) return resolved;

try {
Expand Down
Loading
Loading