fix(db): converge the install and upgrade schemas; stop ENOSPC from faking a divergence by diegosouzapw · Pull Request #11845 · cheaper-inference/OmniRoute

diegosouzapw · 2026-08-27T21:40:51Z

The reported defect was mostly a false alarm — and the log said so

The v3.8.50 npm publish failed check:install-upgrade with what looked like a serious migration gap: 15 tables a clean install creates that an upgrade does not.

It reported four failures, not one, and the second invalidates the third:

✗ clean: health reports version undefined, expected 3.8.50
✗ upgraded: exited with code 0 before serving        ← this voids the next line
✗ schema divergence — agentic_conversations, ccr_blocks, … (15 tables)
✗ NEW residual table(s): model_capabilities

Between them sit 5,611 lines of:

npm warn tar TAR_ENTRY_ERROR ENOSPC: no space left on device, write

Phase B's npm install of the 275 MB tarball filled the runner's disk, npm reported ENOSPC as a warning and exited 0, the truncated package made omniroute serve exit before serving, runMigrations() never ran — and the gate compared a healthy clean install against an upgrade that never happened. The 15 missing tables were the intact 3.8.49 schema.

Proven, not assumed: a real booted 3.8.49 database (115 tables, 130 migrations applied) replayed through the current runner applies 29 migrations and reaches 130 tables. Full convergence. Packaging and the mass-migration guard were both ruled out by experiment.

Two genuine defects, both fixed

1. model_capabilities had no migration. It was created only by ensureCapabilitiesTable() (src/lib/modelsDevSync.ts) on the first models.dev sync — so whether a database had it depended on timing, not schema version. Now migration 163, with ensureCapabilitiesTable() kept as an idempotent no-op safety net and a test pinning the two definitions against drift.

2. The gate's version assertion could never pass on 3.8.50. GHSA-mvf8-qc78-5mxm made /api/monitoring/health withhold version from anonymous callers; the probe read body.version with no credential. check:pack-boot had already solved this with a token — check:install-upgrade was left behind.

Gate hardened

ENOSPC now fails instead of fabricating a phantom divergence (needed spawnSync: execFileSync piped stderr straight to the parent, so the script never saw the warnings)
health probe authenticated with a self-generated internal token
Phase A's 3 GB tree freed before Phase B
warns when the tmpfs cannot hold the run
prints the failing server's output
convergence is not computed when a phase never served — the run still fails on the boot, it just stops inventing 15 tables

Also

The cache_metrics allowlist justification was factually wrong ("a CREATE that left the migration set"); it is lazily created in src/lib/semanticCache.ts, the same class as model_capabilities. Corrected, with which cause needs a migration rather than an allowlist. Nothing was added to the allowlist and the gate was not weakened.

Tests

tests/unit/db-install-upgrade-schema-parity.test.ts (new, TDD — red before, green after) plus 3 ENOSPC-guard tests. Green: db-* (986), check-* (487), typecheck:core, migration-numbering (160), changelog-integrity, docs-sync, docs-counts.

Honest caveats

The full gate did not run locally. npm pack of the 2.8 GB dist/ takes >68 min on the devbox versus 4.5 min on the CI runner. And ENOSPC is not reproducible here by construction — 209 GB free. CI is the real proof for this PR.
tests/unit/check-deps.test.ts fails on @tensorflow/tfjs — pre-existing base-red, no manifest touched.
10 other tables are still created lazily outside migrations (audit_log, cache_metrics, combo_health, prompt_templates, …) — the same latent class, able to fake this alarm in either direction on a future run. Deliberately out of scope; the pattern is documented.

…aking a divergence The v3.8.50 publish run failed `check:install-upgrade` with "15 tables a CLEAN install creates but an UPGRADE does not" (agentic_conversations, ccr_blocks, the whole Radar set, jobs/job_runs, exclusive_connection_leases, …). None of them was missing. Root cause, from the CI log (run 33104507735): the Phase B upgrade `npm install` hit `npm warn tar TAR_ENTRY_ERROR ENOSPC: no space left on device` 5611 times, npm still exited 0, and the resulting truncated package made `omniroute serve` "exit with code 0 before serving". No migration ever ran, so the database still held the 3.8.49 schema (115 tables) and every post-133 migration table read as a divergence. Verified against the real thing: booting the published omniroute@3.8.49 and replaying that database through the current runner applies exactly 29 migrations and lands on the same table set a clean install produces — the migration set was never at fault. What changes: - `163_model_capabilities.sql` — the one genuine convergence defect. The table was only ever created by `ensureCapabilitiesTable()` on the first models.dev sync, so whether a database has it depends on timing, not on the schema version. It is the residual the gate reported. A migration makes both install paths deterministic. - `check:install-upgrade` now fails on an ENOSPC-truncated install instead of measuring a broken tree; authenticates its health probe with a minted internal-service token, so the version assertion works against the health payload hardened by GHSA-mvf8-qc78-5mxm (an anonymous caller gets no version — the same run also failed with "health reports version undefined"); frees the ~3 GB clean-install tree before the upgrade phase; warns when the temp filesystem cannot hold the run; prints the failing server's output; and skips the convergence verdict when a phase never served, so a broken boot can no longer manufacture a schema divergence on top of the real failure. Tests: `tests/unit/db-install-upgrade-schema-parity.test.ts` pins the deterministic half of the gate in milliseconds (every migration reachable on a clean install; model_capabilities comes from the migration set; its DDL does not drift from the runtime helper), and the ENOSPC guard is covered in the existing gate test.

…allowlist The allowlist described every residual as "a CREATE that left the migration set in some past cycle". cache_metrics never was in the migration set: it is created lazily by ensureCacheMetricsTable() (src/lib/semanticCache.ts:34) the first time the semantic cache runs, which is the same class as the model_capabilities divergence that blocked the v3.8.50 publish. Document both causes so the next residual is fixed with a migration where that is the right answer, instead of reflexively allowlisted.

check:docs-counts-sync enforces the shipped migration count as a STRICT claim in README.md, AGENTS.md and llm.txt.

…bump

github-actions · 2026-08-27T21:51:09Z

CI Coverage Report

Coverage job: skipped
PR test policy: success

Coverage artifact was not available for this run.

…11855) * fix(ci): run the install-upgrade gate on disk, not on the /tmp tmpfs The v3.8.50 publish failed this gate again, and this time it said why: free space in /tmp: 2.9 GB ⚠️ only 2.9 GB free — this gate needs roughly 12 GB crashed: upgrade install ran out of disk space (58269 ENOSPC errors) On the self-hosted runner `/tmp` is a **12 GB tmpfs backed by RAM**, while the root filesystem had 66 GB free. The gate builds two ~3 GB install trees, installs the second one over twice, and packs a 275 MB tarball — roughly 12 GB, all of it demanded from the wrong filesystem. This is why freeing disk never fixed it: 84 GB were freed on `/`, and none of it ever reached the volume the gate was using. The check even measured the right number and reported it against the wrong path, so the warning read as "the disk is full" when the disk was fine. - work in `<repo>/.install-upgrade/` (gitignored) instead of `os.tmpdir()`, overridable with `OMNIROUTE_INSTALL_UPGRADE_WORKDIR` - the free-space log and the ENOSPC crash message now name the directory the run actually uses, so the next reader is sent to the filesystem that ran out Phase A already passes on the current main: clean install healthy, version reported correctly, 130 tables — the authentication fix and migration 163 from #11845 both hold. Only Phase B was starved. * docs(env): document OMNIROUTE_INSTALL_UPGRADE_WORKDIR The workdir override introduced in this branch is a new `process.env.*` read, and two gates caught it immediately: `issue #7793: real .env.example is in sync with process.env.* reads in code` and `check:env-doc-sync` (Docs Sync STRICT). Both were right — an env var that exists only in code is an env var nobody can find. Documented in `.env.example` and `docs/reference/ENVIRONMENT.md` with the reason it exists: the gate needs ~12 GB and must not land on a small tmpfs.

…nc-back step 2/2 The eighteen commits main carries beyond the cycle branch, and what each one became here: already in release/v3.8.51 by its own PR (no-op, verified by content): b090b60 / 026e1ca deps: nanoid 3.3.18 equal, dompurify 3.4.14 newer 918fba5 .gitignore: /_tasks already anchored 5f0a394 #10026 hide health-check-excluded models — same helper, 5 call sites c68cda7 #11075 shared passthrough providers — superseded by #11071/#11078 superseded, one piece kept: ca23eed #10055 memoize models.dev pricing — the cycle memoizes on the catalog cache version already; only the resetDbInstance() hook is ported, wired to that memo applied as-is: 8778ea7 stamp dist/BUILD_SHA before the npm provenance gate (#11721) aa52351 decouple the Bun image from the release manifest (#11724) 925feb2 let the bun digest artifact be absent (#11740) b65ef33 size the install-upgrade gate to a measured run 0ce2123 #11845 converge install/upgrade schemas (migration renumbered in the next commit: 163 collides with 163_radar_feed_cache_generated_at) b7c07ed #11855 install-upgrade gate on disk, not tmpfs 8e2fb04 #11864 drop *.nft.json from the npm tarball (413) dea6bb8 #11877 publish npm from a hosted runner (provenance 422) handled by the sync script that follows (CHANGELOG protocol): b4ec780 Release v3.8.50 — squash of content this branch already carries 5458026 / c44c0a2 CHANGELOG aggregation, stats and top-25 applied separately (its own commit, ten files): 65e8115 #11088 Ollama capability routing — a 5,094-file squash from a stale base; only the Ollama files are the change Every cherry-pick that touched a file this branch had also changed was resolved by hand and re-run through the tests both sides own for it.

…nch collision with 163_radar_feed_cache_generated_at) #11845 landed on main as 163 while release/v3.8.51 had already used 163–168. Per the cross-PR collision precedent (#3365/#3371) the later arrival takes the next free number; the SQL is idempotent, so installs that already ran it as 163 on main are unaffected.

@MumuTW

…ments, cover every cycle commit, credit every contributor (#12971) Mid-cycle reconciliation of `## [3.8.51]` against the full cycle range `release/v3.8.50..release/v3.8.51` (0915890..d6f3150, 696 non-merge commits): - fold the 366 `changelog.d/` fragments into the section (features had been landing under `[Unreleased]` because the aggregator appends at the FIRST matching heading) and delete them - drop 25 fragments that duplicate bullets already shipped in `[3.8.50]` / `[Unreleased]` (six phantom fragments re-added by branches cut before the v3.8.50 aggregation; 19 pre-cycle PR fragments) - generate one bullet per cycle commit that had no fragment (350 commits: 44 features / 196 fixes / 91 maintenance, 17 Dependabot bumps rolled up), carrying the merged PR link and `— thanks @author` - credit audit: link every fragment to the PR that actually landed it (`git log --diff-filter=A`), fix three `#PR_NUMBER` placeholders and two misnumbered fragments (#11845, #11864), credit the recreated PRs to their original authors (#11887–#11892 → @MumuTW, #12255 → @backryun, #11771 → @Rahulsharma0810), the nine hartmark co-authored fixes, and the deleted PR #11370 to @kriptoburak - add the "Release by the numbers" block, the Top-25 ranking (mailmap + GitHub login) and the mandatory `### 🙌 Contributors` hall (111 external contributors + maintainer, generated by `release:contributors --inject`) - resync the 41 i18n CHANGELOG mirrors Gates: check:changelog-integrity OK, check:docs-sync PASS.

…nc-back step 2/2 The eighteen commits main carries beyond the cycle branch, and what each one became here: already in release/v3.8.51 by its own PR (no-op, verified by content): 7124dd3 / bae33e2 deps: nanoid 3.3.18 equal, dompurify 3.4.14 newer ceb8e3e .gitignore: /_tasks already anchored bf87c7d cheaper-inference#10026 hide health-check-excluded models — same helper, 5 call sites 3816ade cheaper-inference#11075 shared passthrough providers — superseded by cheaper-inference#11071/cheaper-inference#11078 superseded, one piece kept: b735371 cheaper-inference#10055 memoize models.dev pricing — the cycle memoizes on the catalog cache version already; only the resetDbInstance() hook is ported, wired to that memo applied as-is: a93d123 stamp dist/BUILD_SHA before the npm provenance gate (cheaper-inference#11721) dd329d7 decouple the Bun image from the release manifest (cheaper-inference#11724) a5e649b let the bun digest artifact be absent (cheaper-inference#11740) e9045c4 size the install-upgrade gate to a measured run f5addae cheaper-inference#11845 converge install/upgrade schemas (migration renumbered in the next commit: 163 collides with 163_radar_feed_cache_generated_at) a38590e cheaper-inference#11855 install-upgrade gate on disk, not tmpfs 8588804 cheaper-inference#11864 drop *.nft.json from the npm tarball (413) 28bde09 cheaper-inference#11877 publish npm from a hosted runner (provenance 422) handled by the sync script that follows (CHANGELOG protocol): fc15eef Release v3.8.50 — squash of content this branch already carries 2b80128 / 4398260 CHANGELOG aggregation, stats and top-25 applied separately (its own commit, ten files): 837b36a cheaper-inference#11088 Ollama capability routing — a 5,094-file squash from a stale base; only the Ollama files are the change Every cherry-pick that touched a file this branch had also changed was resolved by hand and re-run through the tests both sides own for it.

…nch collision with 163_radar_feed_cache_generated_at) cheaper-inference#11845 landed on main as 163 while release/v3.8.51 had already used 163–168. Per the cross-PR collision precedent (cheaper-inference#3365/cheaper-inference#3371) the later arrival takes the next free number; the SQL is idempotent, so installs that already ran it as 163 on main are unaffected.

…aking a divergence (cheaper-inference#11845) * fix(db): converge the install and upgrade schemas; stop ENOSPC from faking a divergence The v3.8.50 publish run failed `check:install-upgrade` with "15 tables a CLEAN install creates but an UPGRADE does not" (agentic_conversations, ccr_blocks, the whole Radar set, jobs/job_runs, exclusive_connection_leases, …). None of them was missing. Root cause, from the CI log (run 33104507735): the Phase B upgrade `npm install` hit `npm warn tar TAR_ENTRY_ERROR ENOSPC: no space left on device` 5611 times, npm still exited 0, and the resulting truncated package made `omniroute serve` "exit with code 0 before serving". No migration ever ran, so the database still held the 3.8.49 schema (115 tables) and every post-133 migration table read as a divergence. Verified against the real thing: booting the published omniroute@3.8.49 and replaying that database through the current runner applies exactly 29 migrations and lands on the same table set a clean install produces — the migration set was never at fault. What changes: - `163_model_capabilities.sql` — the one genuine convergence defect. The table was only ever created by `ensureCapabilitiesTable()` on the first models.dev sync, so whether a database has it depends on timing, not on the schema version. It is the residual the gate reported. A migration makes both install paths deterministic. - `check:install-upgrade` now fails on an ENOSPC-truncated install instead of measuring a broken tree; authenticates its health probe with a minted internal-service token, so the version assertion works against the health payload hardened by GHSA-mvf8-qc78-5mxm (an anonymous caller gets no version — the same run also failed with "health reports version undefined"); frees the ~3 GB clean-install tree before the upgrade phase; warns when the temp filesystem cannot hold the run; prints the failing server's output; and skips the convergence verdict when a phase never served, so a broken boot can no longer manufacture a schema divergence on top of the real failure. Tests: `tests/unit/db-install-upgrade-schema-parity.test.ts` pins the deterministic half of the gate in milliseconds (every migration reachable on a clean install; model_capabilities comes from the migration set; its DDL does not drift from the runtime helper), and the ENOSPC guard is covered in the existing gate test. * docs(db): record the real cause of the cache_metrics residual in the allowlist The allowlist described every residual as "a CREATE that left the migration set in some past cycle". cache_metrics never was in the migration set: it is created lazily by ensureCacheMetricsTable() (src/lib/semanticCache.ts:34) the first time the semantic cache runs, which is the same class as the model_capabilities divergence that blocked the v3.8.50 publish. Document both causes so the next residual is fixed with a migration where that is the right answer, instead of reflexively allowlisted. * docs: bump the migration count to 160 after 163_model_capabilities check:docs-counts-sync enforces the shipped migration count as a STRICT claim in README.md, AGENTS.md and llm.txt. * docs(i18n): re-sync the 42 llm.txt mirrors after the migration-count bump

…heaper-inference#11855) * fix(ci): run the install-upgrade gate on disk, not on the /tmp tmpfs The v3.8.50 publish failed this gate again, and this time it said why: free space in /tmp: 2.9 GB ⚠️ only 2.9 GB free — this gate needs roughly 12 GB crashed: upgrade install ran out of disk space (58269 ENOSPC errors) On the self-hosted runner `/tmp` is a **12 GB tmpfs backed by RAM**, while the root filesystem had 66 GB free. The gate builds two ~3 GB install trees, installs the second one over twice, and packs a 275 MB tarball — roughly 12 GB, all of it demanded from the wrong filesystem. This is why freeing disk never fixed it: 84 GB were freed on `/`, and none of it ever reached the volume the gate was using. The check even measured the right number and reported it against the wrong path, so the warning read as "the disk is full" when the disk was fine. - work in `<repo>/.install-upgrade/` (gitignored) instead of `os.tmpdir()`, overridable with `OMNIROUTE_INSTALL_UPGRADE_WORKDIR` - the free-space log and the ENOSPC crash message now name the directory the run actually uses, so the next reader is sent to the filesystem that ran out Phase A already passes on the current main: clean install healthy, version reported correctly, 130 tables — the authentication fix and migration 163 from cheaper-inference#11845 both hold. Only Phase B was starved. * docs(env): document OMNIROUTE_INSTALL_UPGRADE_WORKDIR The workdir override introduced in this branch is a new `process.env.*` read, and two gates caught it immediately: `issue cheaper-inference#7793: real .env.example is in sync with process.env.* reads in code` and `check:env-doc-sync` (Docs Sync STRICT). Both were right — an env var that exists only in code is an env var nobody can find. Documented in `.env.example` and `docs/reference/ENVIRONMENT.md` with the reason it exists: the gate needs ~12 GB and must not land on a small tmpfs.

@MumuTW

…ments, cover every cycle commit, credit every contributor (cheaper-inference#12971) Mid-cycle reconciliation of `## [3.8.51]` against the full cycle range `release/v3.8.50..release/v3.8.51` (104d4f8..8da2a33, 696 non-merge commits): - fold the 366 `changelog.d/` fragments into the section (features had been landing under `[Unreleased]` because the aggregator appends at the FIRST matching heading) and delete them - drop 25 fragments that duplicate bullets already shipped in `[3.8.50]` / `[Unreleased]` (six phantom fragments re-added by branches cut before the v3.8.50 aggregation; 19 pre-cycle PR fragments) - generate one bullet per cycle commit that had no fragment (350 commits: 44 features / 196 fixes / 91 maintenance, 17 Dependabot bumps rolled up), carrying the merged PR link and `— thanks @author` - credit audit: link every fragment to the PR that actually landed it (`git log --diff-filter=A`), fix three `#PR_NUMBER` placeholders and two misnumbered fragments (cheaper-inference#11845, cheaper-inference#11864), credit the recreated PRs to their original authors (cheaper-inference#11887–cheaper-inference#11892 → @MumuTW, cheaper-inference#12255 → @backryun, cheaper-inference#11771 → @Rahulsharma0810), the nine hartmark co-authored fixes, and the deleted PR cheaper-inference#11370 to @kriptoburak - add the "Release by the numbers" block, the Top-25 ranking (mailmap + GitHub login) and the mandatory `### 🙌 Contributors` hall (111 external contributors + maintainer, generated by `release:contributors --inject`) - resync the 41 i18n CHANGELOG mirrors Gates: check:changelog-integrity OK, check:docs-sync PASS.

diegosouzapw added 4 commits August 27, 2026 17:16

docs: bump the migration count to 160 after 163_model_capabilities

b95fedf

check:docs-counts-sync enforces the shipped migration count as a STRICT claim in README.md, AGENTS.md and llm.txt.

docs(i18n): re-sync the 42 llm.txt mirrors after the migration-count …

8506966

…bump

diegosouzapw merged commit 0ce2123 into main Aug 27, 2026
45 of 48 checks passed

diegosouzapw mentioned this pull request Aug 28, 2026

fix(ci): run the install-upgrade gate on disk, not on the /tmp tmpfs #11855

Merged

This was referenced Sep 7, 2026

docs(changelog): reconcile the v3.8.51 living section — fold 366 fragments, cover every cycle commit, credit every contributor #12971

Merged

Release v3.8.51 #11442

Merged

webzidev mentioned this pull request Sep 28, 2026

fix(compression): 3.8.50 npm hotfix — terminate idle compression workers on eviction (#13091 → main) #14988

Merged

diegosouzapw deleted the fix/upgrade-schema-divergence branch October 18, 2026 23:51

Navigation Menu

Repository navigation

fix(db): converge the install and upgrade schemas; stop ENOSPC from faking a divergence - #11845

fix(db): converge the install and upgrade schemas; stop ENOSPC from faking a divergence#11845
diegosouzapw merged 4 commits into
mainfrom
fix/upgrade-schema-divergence

diegosouzapw commented Aug 27, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

diegosouzapw commented Aug 27, 2026

The reported defect was mostly a false alarm — and the log said so

Two genuine defects, both fixed

Gate hardened

Also

Tests

Honest caveats

Uh oh!

github-actions Bot commented Aug 27, 2026

CI Coverage Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant