fix(db): bound health scans and isolate native diagnostics - #13717
diegosouzapw merged 64 commits into
Conversation
|
Updated in 6a6daa3 to address the three CI regressions attributed to this PR:
Verification on the reviewed files:
The baseline failures identified in the previous CI run remain outside this correction. The new GitHub checks are not yet claimed green. No deployment or merge performed. |
Migration 178 requires the FTS table created by earlier optional migrations. Defer it on drivers without FTS5 so fallback startup works and a later native restart can apply it. Signed-off-by: Minxi Hou <houminxi@gmail.com>
The asynchronous logger can recreate log directories while the fixture removes its data directory. Disable this unrelated writer before imports to keep cleanup deterministic. Signed-off-by: Minxi Hou <houminxi@gmail.com>
Large quota histories exhausted memory and blocked request handling during synchronous health checks. Page exact SQLite IDs, reuse statements, and run file-backed native checks in a cancellable child with coalesced results. Require a successful snapshot before repair, preserve the sql.js owner connection, and cancel diagnostic work before shutdown. Ship the worker through both packaging paths and await results at HTTP and MCP callers. Signed-off-by: Minxi Hou <houminxi@gmail.com>
The provider boundary fixture starts background health work, so its final cleanup must drain it before closing the database. Keep synchronous restore guards intact. Extract snapshot ownership and quota validation helpers to meet the new-code complexity limits without changing behavior. Signed-off-by: Minxi Hou <houminxi@gmail.com>
Signed-off-by: Minxi Hou <houminxi@gmail.com>
6a6daa3 to
fbc5b67
Compare
|
Thank you for this — the paginated, int64-safe quota-snapshot scan and moving native Two things we'll handle on our side before merging, no action needed from you:
Coordination note: #13149 edits the same route + sanitization test (your cache/child-process |
…kup and wire the deferral flag
…apw#15/diegosouzapw#17) runManagedDbHealthCheck() now forks native diagnostics into a child process via healthCheckRunner.ts. Every process-spawning route must be classified isLocalOnlyPath() so a leaked JWT via tunnel cannot trigger process spawning, matching the /api/db-backups/exportAll precedent. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
# Conflicts: # src/lib/db/core.ts # src/lib/db/migrationRunner/constants.ts # tests/unit/db-migrationrunner-constants-split.test.ts
… sync + stryker tap.testFiles) (diegosouzapw#13826) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…iegosouzapw#12370) (diegosouzapw#13824) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…uzapw#13679) (diegosouzapw#13813) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…NGEME remote login (diegosouzapw#13679) (diegosouzapw#13812) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
PR F) (diegosouzapw#13811) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
diegosouzapw#13652) (diegosouzapw#13808) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
… follow-ups (diegosouzapw#13599) (diegosouzapw#13807) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…ate peek (diegosouzapw#13620) (diegosouzapw#13806) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…zapw#13680, diegosouzapw#13681) (diegosouzapw#13805) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…ck (diegosouzapw#13679 PR A) (diegosouzapw#13804) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…all_logs (diegosouzapw#13544) (diegosouzapw#13803) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…pen (diegosouzapw#13597) (diegosouzapw#13802) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…souzapw#13591) (diegosouzapw#13801) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
…ma (diegosouzapw#13562) (diegosouzapw#13800) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
… of reasoning_content (diegosouzapw#13558) (diegosouzapw#13799) Merged in the 2026-09-16 sweep of the maintainer's own open PRs, at the owner's explicit instruction. No push was made to the PR branch: the merge took the head as the owning session left it (verified OPEN, non-draft and MERGEABLE against the release tip immediately before merging).
… failing the job (diegosouzapw#13889) "Fast Quality Gates" is red on every open PR against release/v3.8.51. The failing step is `forgotten-sibling-tests`, which is explicitly advisory — its own output says "Report-only calibration: these findings do not fail the job" — yet it exits 1. When a PR diff touches a hub module (`open-sse/config/providerRegistry.ts` in the current reds), the analysis walks every import edge in the repo and multiplies each consumer by its candidate tests. The result reaches millions of rows, and `lines.join("\n")` then exceeds V8's maximum string length. The throw lands in main()'s catch, which exits 1 — so an advisory report takes the whole job down. Measured with a synthetic hub cross-product, before the change: 3,000,000 findings -> a 435 MB report string (no throw, but absurd) 4,500,000 findings -> Invalid string length (the CI failure, verbatim) After: the same 4,500,000 findings render as 27 KB. The fix bounds only the ENUMERATION. The header keeps the exact totals, so the signal ("this diff has N unreviewed sibling tests") is unchanged; at most 200 rows per section are listed, followed by a line naming how many were withheld. The JSON artifact gets the same treatment (5,000 items per array) plus an explicit `totals` object, since `JSON.stringify` would throw on the same input for the same reason. `markdown()` is exported so the bound is testable without a CI-sized diff.
…, stryker registry, env/docs contract (diegosouzapw#13834) * docs(skills): regenerate omni-settings SKILL.md for the egress-observation endpoint * fix(ci): register provider-401-ambiguous-runtime test in stryker tap.testFiles * fix(ci): drain current base drift — regenerate cli-mcp SKILL.md, register 4 stryker tests * fix(ci): document OMNIROUTE_CLOUD_SYNC_ENFORCE_SIGNATURE and CDP_PROXY_TOKEN (env/docs contract)
…ication from the rebased head of diegosouzapw#13717
…tions) + review-locale script (diegosouzapw#13885) Reviewer pass (native-speaker prompt) over the 1,865 pt-BR leaves retranslated in diegosouzapw#13782: 172 corrections applied; new scripts/i18n/review-locale.mjs with tests.⚠️ base-red inherited: diegosouzapw#12732
…the site catalogs (diegosouzapw#13886) scripts/i18n/retranslate-site.mjs + untranslatable-site-keys.json (22 keys) + tests; the run landed on the site as OmniRouteSite#8 (2,059 leaves, mean English residue 10.3 % → 6.3 %).⚠️ base-red inherited: diegosouzapw#12732
… empty) (diegosouzapw#13892) sync-ui-keys --catalog=cli + blocking i18n:check-keys:cli gate; 65 CLI catalogs synced (52,000 strings, 0 __MISSING__, all 830 keys, placeholders verified).⚠️ base-red inherited: diegosouzapw#12732
# Conflicts: # .github/workflows/ci.yml # config/quality/file-size-baseline.json # docs/architecture/QUALITY_GATES.md # docs/guides/I18N.md # docs/i18n/am/llm.txt # docs/i18n/ar/llm.txt # docs/i18n/az/llm.txt # docs/i18n/bg/llm.txt # docs/i18n/bn/llm.txt # docs/i18n/cs/llm.txt # docs/i18n/da/llm.txt # docs/i18n/de/llm.txt # docs/i18n/el/llm.txt # docs/i18n/es/llm.txt # docs/i18n/et/llm.txt # docs/i18n/fa/llm.txt # docs/i18n/fi/llm.txt # docs/i18n/fr/llm.txt # docs/i18n/ga/llm.txt # docs/i18n/gu/llm.txt # docs/i18n/ha/llm.txt # docs/i18n/he/llm.txt # docs/i18n/hi/llm.txt # docs/i18n/hr/llm.txt # docs/i18n/hu/llm.txt # docs/i18n/hy/llm.txt # docs/i18n/id/llm.txt # docs/i18n/ig/llm.txt # docs/i18n/it/llm.txt # docs/i18n/ja/llm.txt # docs/i18n/ka/llm.txt # docs/i18n/km/llm.txt # docs/i18n/kn/llm.txt # docs/i18n/ko/llm.txt # docs/i18n/lt/llm.txt # docs/i18n/lv/llm.txt # docs/i18n/ml/llm.txt # docs/i18n/mr/llm.txt # docs/i18n/ms/llm.txt # docs/i18n/mt/llm.txt # docs/i18n/my/llm.txt # docs/i18n/ne/llm.txt # docs/i18n/nl/llm.txt # docs/i18n/no/llm.txt # docs/i18n/or/llm.txt # docs/i18n/pa/llm.txt # docs/i18n/phi/llm.txt # docs/i18n/pl/llm.txt # docs/i18n/pt-BR/llm.txt # docs/i18n/pt/llm.txt # docs/i18n/ro/llm.txt # docs/i18n/ru/llm.txt # docs/i18n/si/llm.txt # docs/i18n/sk/llm.txt # docs/i18n/sl/llm.txt # docs/i18n/sr/llm.txt # docs/i18n/sv/llm.txt # docs/i18n/sw/llm.txt # docs/i18n/ta/llm.txt # docs/i18n/te/llm.txt # docs/i18n/th/llm.txt # docs/i18n/tr/llm.txt # docs/i18n/uk-UA/llm.txt # docs/i18n/ur/llm.txt # docs/i18n/uz/llm.txt # docs/i18n/vi/llm.txt # docs/i18n/yo/llm.txt # docs/i18n/zh-CN/llm.txt # docs/i18n/zh-TW/llm.txt # package.json # scripts/i18n/check-key-completeness.mjs # src/i18n/messages/am.json # src/i18n/messages/ar.json # src/i18n/messages/az.json # src/i18n/messages/bg.json # src/i18n/messages/bn.json # src/i18n/messages/cs.json # src/i18n/messages/da.json # src/i18n/messages/de.json # src/i18n/messages/el.json # src/i18n/messages/en.json # src/i18n/messages/es.json # src/i18n/messages/et.json # src/i18n/messages/fa.json # src/i18n/messages/fi.json # src/i18n/messages/fr.json # src/i18n/messages/ga.json # src/i18n/messages/gu.json # src/i18n/messages/ha.json # src/i18n/messages/he.json # src/i18n/messages/hi.json # src/i18n/messages/hr.json # src/i18n/messages/hu.json # src/i18n/messages/hy.json # src/i18n/messages/id.json # src/i18n/messages/ig.json # src/i18n/messages/it.json # src/i18n/messages/ja.json # src/i18n/messages/ka.json # src/i18n/messages/km.json # src/i18n/messages/kn.json # src/i18n/messages/ko.json # src/i18n/messages/lt.json # src/i18n/messages/lv.json # src/i18n/messages/ml.json # src/i18n/messages/mr.json # src/i18n/messages/ms.json # src/i18n/messages/mt.json # src/i18n/messages/my.json # src/i18n/messages/ne.json # src/i18n/messages/nl.json # src/i18n/messages/no.json # src/i18n/messages/or.json # src/i18n/messages/pa.json # src/i18n/messages/phi.json # src/i18n/messages/pl.json # src/i18n/messages/pt-BR.json # src/i18n/messages/pt.json # src/i18n/messages/ro.json # src/i18n/messages/ru.json # src/i18n/messages/si.json # src/i18n/messages/sk.json # src/i18n/messages/sl.json # src/i18n/messages/sr.json # src/i18n/messages/sv.json # src/i18n/messages/sw.json # src/i18n/messages/ta.json # src/i18n/messages/te.json # src/i18n/messages/th.json # src/i18n/messages/tr.json # src/i18n/messages/uk-UA.json # src/i18n/messages/ur.json # src/i18n/messages/uz.json # src/i18n/messages/vi.json # src/i18n/messages/yo.json # src/i18n/messages/zh-CN.json # src/i18n/messages/zh-TW.json # src/shared/constants/featureFlagDefinitions.ts # tests/unit/feature-flags-settings.test.ts # tests/unit/i18n-key-completeness.test.ts
…, stale suppressions)
e7999c4
into
diegosouzapw:release/v3.8.51
…ouzapw#13717 managed health check Merge fallout only: runManagedDbHealthCheck moved behind the health coordinator on the release tip, so the per-call skipIntegrityCheck now travels through it. A waived integrity scan is part of the job identity, so it is never replayed from the 60s diagnosis cache to a caller that asked for the full scan. Co-authored-by: cryptiklemur <cryptiklemur@users.noreply.github.com>
* fix(db): skip integrity scans during health polling * test(db): update health error fixture for scan-free polling * docs(changelog): add fragment for health poll integrity skip * fix(db): keep the #13149 dashboard skip inside the #13717 managed health check Merge fallout only: runManagedDbHealthCheck moved behind the health coordinator on the release tip, so the per-call skipIntegrityCheck now travels through it. A waived integrity scan is part of the job identity, so it is never replayed from the 60s diagnosis cache to a caller that asked for the full scan. Co-authored-by: cryptiklemur <cryptiklemur@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: cryptiklemur <cryptiklemur@users.noreply.github.com>
The release tip's latest migration is 179 (diegosouzapw#13717 folded the former 180 into 178), so 185 opened a 180–184 gap that check-migration-numbering rejects. check-migration-numbering.test.ts: 15/15. The 176→177 count in README/AGENTS/ llm.txt is a maintainer-side bump (protected surfaces) applied at merge time. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
…apw#13717) Merged after a maintainer rework that kept every one of @HouMinXi's commits intact. **What the rework added on top of the contribution:** the new DB health-check behaviour is gated behind a default-off feature flag (`src/shared/constants/featureFlagDefinitions.ts`, `defaultValue: "false"`), documented in `docs/reference/FEATURE_FLAGS.md` with the description key carried into all 66 locales, so the release default is unchanged and the new bounds only apply when an operator opts in. The optional-FTS5 migration set was reconciled by hand with the "180" entry that landed meanwhile (`src/lib/db/migrationRunner/constants.ts`). **Carried from your rebased head:** the `/api/db/health` local-only classification in `src/server/authz/routeGuard.ts` plus its `routeGuard` assertion — `runManagedDbHealthCheck()` forks native diagnostics into a child process, so Hard Rules diegosouzapw#15/diegosouzapw#17 apply. Re-verified here: 37 pass / 0 fail. Validated as a combined board first (this PR merged with the 21 siblings of the same wave on the release tip): eslint with the frozen suppressions, typecheck:core, check:open-sse-typecheck, complexity, cognitive-complexity, changelog-integrity, i18n new-key coverage, docs-counts, docs-sync, migration-numbering, provider-consistency and a duplicate-identifier audit all green, plus 176 passing / 0 failing focused node:test cases across the 25 test files the wave touches and the dashboard test under Vitest (2/0). Then re-validated alone on the fresh tip before this merge: conflicts re-resolved, file sizes rebaselined for this PR's own growth, eslint and this PR's focused tests re-run. Thank you for the depth of this one — the resource-bounds suite and the sql.js startup/backup coverage are the kind of tests that keep a database layer honest.
* fix(db): skip integrity scans during health polling * test(db): update health error fixture for scan-free polling * docs(changelog): add fragment for health poll integrity skip * fix(db): keep the diegosouzapw#13149 dashboard skip inside the diegosouzapw#13717 managed health check Merge fallout only: runManagedDbHealthCheck moved behind the health coordinator on the release tip, so the per-call skipIntegrityCheck now travels through it. A waived integrity scan is part of the job identity, so it is never replayed from the 60s diagnosis cache to a caller that asked for the full scan. Co-authored-by: cryptiklemur <cryptiklemur@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: cryptiklemur <cryptiklemur@users.noreply.github.com>
Current behavior
Database health checks load all quota snapshots at once and repeatedly prepare connection lookups. A large history can block request handling and consume substantial native memory. The endpoint is not the only caller: startup, periodic repair and the MCP health tool share this path.
Problem
The previous synchronous scan couples database maintenance to the request thread. Moving it into a child alone would leave unbounded reads, overlapping checks, unsafe repair ordering and shutdown races unresolved.
Changes
Two independently reproduced baseline failures are separate commits: defer dependent FTS migration 178 on drivers without FTS5; isolate the XP cleanup fixture from its unrelated asynchronous file logger. The startup test verifies a subsequent native restart applies the deferred migrations.
Verification
[KNOWN/HIGH] Local Linux, Node 22.23.1:
Known limits: no full Next/Electron build or cross-platform certification. sql.js work remains synchronous and its main-schema snapshot excludes attached/TEMP schemas. Existing ESLint debt remains; comparison against the base found zero new lint errors, not a clean full-repository lint run.
Result
Native health maintenance no longer performs the quota scan on the request thread. sql.js automatic repair remains supported with backup-before-write protection. No production deployment is included in this PR.