fix(state): do not stamp empty FTS after interrupted optimize-storage demote (salvage #72717)

[kshitijk4poor]

Salvages #72717 by @Adolanium — cherry-picked to preserve authorship, with one efficiency follow-up commit on top.

Context — what this fixes, for whom

Any long-time user who runs hermes sessions optimize-storage (the v22→v23 FTS storage migration) and has the process die mid-run — laptop lid close, ctrl-C, OOM — can hit a crash window where the re-run permanently destroys search over their entire message history while reporting success. The interrupted demote leaves empty v23 index tables with no backfill markers; the re-run tears down the old index (the only remaining copy of the indexed data), stamps fts_storage_version as optimized, and fts_optimize_available() returns False forever after. session_search silently returns nothing for all historical messages, and the user is never offered the repair.

Verified side-by-side on current main (0a62610) vs this branch, identical simulated crash-window state, then optimize_fts_storage(vacuum=False):

	main	this PR
result	{"ok": true}	{"ok": true}
indexed rows after (3 messages)	0	3
search_messages('deployment') after	0 hits	2 hits
offered again via fts_optimize_available()	False (stuck forever)	False (genuinely done)

What the fix does (all from #72717, kept verbatim)

1. Closes the crash window: demote now commits the rebuild markers in the same BEGIN IMMEDIATE as the demote, before executescript creates the empty v23 schema (which implicitly commits and therefore runs outside the write transaction, same rule as the CJK recreate path). A crash at any point now leaves a resumable claim.
2. Refuses to stamp incomplete work: both the pre-vacuum check and the settle write transaction re-verify no pending markers / trash / empty-index-with-messages before stamping; refusal returns {"ok": False, "reason": ...} instead of silently succeeding.
3. Heals already-damaged DBs: _repair_optimize_bookkeeping() runs at the top of every optimize — re-seeds a full backfill claim for the unmarked-empty-index shape (clearing a premature stamp), and repairs orphaned high_water-without-progress meta (resetting a partially-populated index to empty first, because the chunk worker replays whole id ranges without an anti-join — otherwise every surviving row would be duplicated).
4. The _init_schema stamp condition and fts_optimize_available() also gained the empty-index leg, so all three fts_storage_version writers are guarded and a damaged DB keeps being offered the repair.

Follow-up commit (ours)

_fts_external_index_empty_with_messages is evaluated on every writable open via the _init_schema stamp condition. The original used COUNT(*) on both messages and messages_fts_docsize — a full b-tree scan measured at ~100ms per open on a 2M-row DB (the heavy installs this subsystem exists for are 25GB). The function only compares against zero, so the follow-up switches to EXISTS(SELECT 1 ...) — identical boolean, O(1) (measured 0.05ms).

Verification

• tests/test_hermes_state.py: 147 passed (includes the PR's 313 lines of new tests: crash-window restore, premature-stamp heal, orphan-marker repair, no-duplicate partial rebuild, markers-before-schema ordering, settle refusal — plus FTS5 integrity-check assertions)
• FTS consumer suites (tests/state/test_fts_runtime_rebuild.py, tests/test_fts_cjk_bigram.py, tests/test_session_db_read_path_split.py): 18 passed, 4 pre-existing environmental skips
• Mutation check on the final stack: reverting hermes_state_search.py/hermes_state_schema.py to main makes all 7 new guard tests fail; restoring goes green — the EXISTS follow-up does not defeat the fix's tests
• Caller audit: the only production caller of optimize_fts_storage is hermes_cli/sessions_cmd.py (optimize-storage command), which already handles ok: False; no gateway/web/TUI callers
• ruff clean on all changed files

Closes #72717 (superseded by this salvage — original author credited via cherry-pick authorship).

Post-review follow-up (commit 3)

The simplify pass caught that _reset_fts_index_to_empty's plain no-WHERE DELETE is O(rows) on external-content FTS5 (measured ~12µs/row — 0.22s @100k, 5.2s @400k, ~25s projected @2m — while holding the write lock), and its 'efficient drop-all' claim only holds for ordinary rowid tables. Swapped to the FTS5 'delete-all' special command: the documented O(1) truncate for external-content tables (measured 1.6ms @100k) that also truncates safely when indexed rows have diverged from messages — exactly the broken-bookkeeping shape this repair path handles. Mutation check re-run on the final stack: all 7 guard tests fail on reverted production code, pass restored.