Skip to content

fix(lcm): structural FTS rebuilds log why + how long; undecryptable-row fixture (follow-up to #966) - #1022

Merged
Kyzcreig merged 1 commit into
mainfrom
fix/lcm-undecryptable-row-fixture
Sep 25, 2026
Merged

Kyzcreig merged 1 commit into
mainfrom
fix/lcm-undecryptable-row-fixture

Conversation

@Kyzcreig

@Kyzcreig Kyzcreig commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #966 (card t_d3963974, Apollo boot-cost freeze #3).

Why

The two structural FTS rebuilds on 2026-09-24 (05:07→05:20 = 13 min, 05:22→05:28 = 6.5 min, each under _LOAD_LOCK + the SQLite write lock, up to 12 turns queued) left no log line at all — the only trace was PHASE=context_engine_load_slow minutes later, and py-spy. repair_external_content_fts decided to DROP + INSERT ... VALUES('rebuild') silently.

What

  • repair_external_content_fts: when a structural rebuild is about to run, log why (missing table / missing shadow <name> / not fts5 / column missing / schema-probe error: <exc> — the last one covers a transient SQLITE_BUSY on the sqlite_master probe, which the old code also answered with a rebuild) plus an O(1) size hint (max(rowid) of the content table), then log the duration after the rebuild commits. _fts_needs_rebuild_structural now records its reason in _last_structural_reason for the caller.
  • Test fixture (test_lcm_init_cost_regression.py::_fill): one AEAD-prefixed row the (disabled) test cipher cannot decrypt — the fleet DB has 3 such rows — so test_engine_construction_needs_no_write_lock now also gates the per-boot NULL-over-NULL msg_fts_update trigger write that fix(lcm): FTS parity COUNT(*) ran under _LOAD_LOCK on every engine load (freeze #3) #966 removed (4–57 s per boot on the fleet DB, 3 row(s) undecryptable × 813 in the gateway log).

Evidence

No behavior change to the rebuild decision itself (that is #966); this is observability + a test gap.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…ble-row fixture

Follow-up to #966 (card t_d3963974). Today's two structural rebuilds
(13 min + 6.5 min under _LOAD_LOCK) left no log line; the only trace was
a PHASE=context_engine_load_slow WARNING minutes later. Now the decision
site logs the reason (missing table / missing shadow / wrong column /
schema-probe exception incl. transient SQLITE_BUSY) and an O(1) size
hint before the rebuild, and the duration after it.

Test fixture: one AEAD-prefixed row the (disabled) cipher cannot
decrypt, so the write-lock test also gates the per-boot NULL-over-NULL
FTS trigger write that #966 removed.
@Kyzcreig
Kyzcreig force-pushed the fix/lcm-undecryptable-row-fixture branch from dafad80 to 30d8083 Compare September 25, 2026 11:56
@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

🤖 merged-by: apollo · lane: boil-ocean-aged · gate: BYPASS: FR PAUSED by Ace ruling 2026-09-22 (state/fleetreview-pause-20260922.md); Apollo-reviewed lands via bypass · why: 0/1 commits on main by patch-id; content not in #1023 (no _fts_structural_problem on main); rebased clean onto 7ed45d9, re-authored to Kyzcreig noreply for check-attribution; 19 passed lcm fts/init_cost tests

@Kyzcreig
Kyzcreig enabled auto-merge September 25, 2026 11:56
@Kyzcreig
Kyzcreig added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit bd9af01 Sep 25, 2026
58 checks passed
@Kyzcreig
Kyzcreig deleted the fix/lcm-undecryptable-row-fixture branch September 25, 2026 16:26
@Kyzcreig Kyzcreig added the fleetreview:post-merge Ask FleetReview to review this MERGED pull (merge commit vs first parent) label Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fleetreview:post-merge Ask FleetReview to review this MERGED pull (merge commit vs first parent)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant