fix(repair): check the FTS5 content table before rebuilding from it (#2278)

[mvalentsev]

Fixes #2278.

What does this PR do?

An isolated malformed inverted index for FTS5 table … says the index and embedding_fulltext_search_content disagree, not which one is wrong. maybe_autoheal_fts5_index rebuilds the index from that content table anyway and calls it "rebuilt from intact content".

Chroma stores each document twice — in embedding_metadata under chroma:document, and in the FTS5 table at rowid = embeddings.id — and reads return the metadata copy. The heal now checks the content table against it, restores the rows that disagree and rebuilds, in one transaction under the mine lock, and declines when the check cannot conclude. Rows the authority cannot speak for keep their content and are named; the rebuild still indexes them unchecked, which this PR does not fix. The ABORT banner it stands in front of already said to "Recreate the FTS5 virtual table from intact embedding_metadata rows".

On a mined 30-drawer palace with 12 drawers carrying one word, those 12 content rows rewritten: lexical_search returns 12, still 12 while quick_check is dirty, then 0 after develop's heal — reported as success, the word still in embedding_metadata. Here it stays 12. mine is where that is permanent, and there the heal prints nothing.

Cost: two scans before a full reindex — 4.9s and 3.1s against a 232s rebuild at 325,451 documents, one run, and the ratio moves with corpus size and page cache. _vacuum_and_rebuild_fts5 rebuilds unchecked too and is left alone — both its call sites run after the collection was re-filed.

How to test

uv run pytest tests/test_repair.py -q -k "fts5 or autoheal"
uv run pytest tests/test_cli.py tests/test_miner_fts5_validation.py -q

Verification

claim	how
content damage produces the accepted wording; the check still separates the two sides	4 tokenizers × 5 damage modes × 3 builds: on trigram every content case is accepted on 3.45.1/3.47.1/3.51.2 and reports rows to restore where index damage reports none
the heal costs the palace its lexical reach	real palace: lexical_search 12 → 12 → 0 on develop, 12 → 12 → 12 here; a real mine prints no FTS5 line either way, exit 0
declining and failing write nothing	authority dropped, its key renamed, rebuild forced to raise via a Connection subclass: errors unchanged, file byte-identical each time
a NULL metadata id cannot stall the heal	NULL into an INTEGER PRIMARY KEY auto-assigns instead of conflicting, so the row never reconciles and every later mine fails; typeof(m.id) = 'integer' skips it
the tests pin this, and not to one SQLite	14 mutations, 13 fail the suite; the fixture uses trigram because unicode61 calls content damage fts5: checksum mismatch from 3.51.2 on, which the classifier does not match

4289 passed, 31 skipped; six spawn-based lock and timing tests flake here under load, and develop fails the same suites at the same rate. ruff check and ruff format --check clean on 0.16.1 and 0.15.14.

Checklist

• Tests pass (python -m pytest tests/ -v)
• No hardcoded paths
• Linter passes (ruff check .)

[igorls]

Verified against current develop (merges cleanly; full suite 4423 passed, 31 skipped, ruff clean).

Approving. The framing is what makes this one land: malformed inverted index says the index and the content table disagree, not which side is wrong — and rebuilding from the content table anyway assumes the answer. When the content side is the damaged one, the current heal destroys lexical reach and reports success. Your measurement of that is the part worth quoting: lexical_search 12 → 12 → 0, called "rebuilt from intact content", with the word still sitting in embedding_metadata the whole time. Silent loss reported as a repair is the worst failure mode a repair path can have.

Checking against the copy that reads actually return is the right authority to pick, and declining when the check cannot conclude is the right default — an inconclusive repair that does nothing beats a confident one that deletes.

Two things I appreciated in the writeup:

You were explicit about what this does not fix: rows the authority cannot speak for keep their content, are named, and are still indexed unchecked. Scoping that honestly rather than implying a full guarantee is more useful than a bigger claim would have been.

And the cost is stated with its shape, not just a number — 4.9s and 3.1s against a 232s rebuild at 325k documents, one run, ratio moving with corpus size and page cache. Two extra scans in front of a repair that only runs on damage is an easy trade.

Leaving _vacuum_and_rebuild_fts5 alone because both its call sites run after the collection was re-filed is the right call too; widening the blast radius of a repair fix is how repair fixes turn into incidents.