fix(state): retire orphaned FTS v2 during storage optimize

[Soju06]

Problem

hermes sessions optimize-storage aborts on this host:

Error: optimization failed: error in trigger messages_fts_v2_delete after rename:
       vtable constructor failed: messages_fts_v2
No data was lost. Re-run to resume.

messages_fts_v2 (tokenize='cjk_unicode61') was superseded by messages_fts_cjk in f13f845116. A DB that stopped mid-transition keeps the v2 vtable, its three AFTER INSERT/UPDATE/DELETE ON messages triggers, and its fts_v2_* state_meta markers — while no code references it any more (grep -c messages_fts_v2 *.py = 0 on HEAD).

Two defects turn that residue into a hard stop:

1. Shadow collection was name-based. _demote_legacy_fts_to_trash gathered shadow tables with a messages_fts_% LIKE glob, which also matches every sibling generation's shadows (messages_fts_v2_data, messages_fts_v2_idx, ...). Renaming them re-parses the v2 triggers; without the loadable cjk_unicode61 tokenizer on that connection the vtable cannot be constructed, and the whole migration rolls back.
2. Nothing retired the orphan. So the failure recurred on every re-run and the DB stayed on the pre-v23 layout — 20.0 GB for 4 GB of messages here, with ~14 GB of duplicate index copies.

Worth calling out separately: while the orphan exists, plain PRAGMA integrity_check fails with no such tokenizer: cjk_unicode61, so the database cannot be verified with stock sqlite3 at all. The size is a symptom; the stuck generation is the disease.

Fix

Both changes live in the single shared path, so sync and async callers inherit them.

• Exact shadow matching. The shadow set is derived from the vtable names actually being demoted, combined with FTS5's fixed shadow suffixes (_data, _idx, _content, _docsize, _config), and matched with IN (...). A sibling generation can no longer be swept in by name resemblance.
• Explicit orphan retirement before the demote. messages_fts_v2 is retired via a named allowlist, not a prefix scan — deliberately, so that a future generation cannot be dropped for merely looking similar. Trigger and vtable definitions are removed through writable_schema, which is why this works without the tokenizer present; the shadow tables are handed to the existing chunked teardown rather than deleted inline.
• Fail-closed preflight. Retirement proceeds only when: messages is non-empty, a live base index exists and covers every row, the integrity check is clean, and rowcount parity holds. Otherwise the orphan is left untouched and the run behaves as before.

Verification

Tests fail on unpatched code and pass with the patch — verified by reverting hermes_state_search.py alone while keeping the new tests:

	result
RED (fix reverted, tests kept)	3 failed — test_demote_only_stages_exact_legacy_generation, test_optimize_retires_poisoned_v2_but_preserves_live_cjk, test_orphaned_v2_alone_is_offered_and_retired
GREEN (on base 3c27eb623)	4 files, 27 tests passed, 0 failed incl. test_state_db_malformed_repair.py, test_fts_runtime_rebuild.py, test_fts_update_of_narrowing.py

Live run on the affected 20 GB database, gateway serving traffic throughout:

• 20041.3 MB → 8164.3 MB (11.9 GB reclaimed)
• PRAGMA integrity_check (tokenizer loaded): ok
• No messages_fts_v2* or fts_v22_trash* residue; state_meta reduced to fts_storage_version=1
• No data loss — messages 1,268,987 = messages_fts 1,268,987, parity holding while rows kept arriving mid-run
• messages_fts_cjk present and serving: EN 87,848 hits / CJK 2-char 811 / trigram 3,100, all sub-10 ms

Notes

• hermes_state_search.py is untouched by every other patch in the stack, so this applies directly on base and does not interact with the assembly fixes.
• Upstream candidate: the defect is generic to any install that carries the cjk_unicode61 generation, not host-specific. Filed as local-only for now since it needs the v2 generation to exist to be observable.

[chatgpt-codex-connector]

💡 Codex Review

https://github.com/Soju06/hermes-agent/blob/32f562255969e62f4001be25145814a9883f4806/.github/workflows/upload_to_pypi.yml#L1
Restore the PyPI publishing workflow

This deletion removes the only repository workflow that builds and uploads distributions on CalVer tag pushes; a repo-wide search of .github and scripts finds no replacement publisher. Consequently, the next tagged release will create no new PyPI package or signed distribution artifacts. This FTS maintenance fix also changes roughly 5,000 unrelated files relative to its parent, so these inherited changes should be removed by rebasing the focused three-file patch onto the intended base.

AGENTS.md reference: AGENTS.md:L170-L174

---

hermes-agent/hermes_state_search.py

Lines 337 to 341 in 32f5622

	if indexed_rows != source_rows:
	raise sqlite3.OperationalError(
	"refusing to retire messages_fts_v2: live base FTS "
	f"row-count mismatch ({indexed_rows} != {source_rows})"
	)

Verify indexed row IDs before retiring v2

When the external base index has one missing message row and one stale row, its internal FTS integrity check succeeds and messages_fts_docsize still has the same count as messages, so this condition retires v2 and reports a successful optimization even though the live index does not cover every source message. This can occur after an interrupted or faulty repair and leaves historical terms absent from search; validate row-ID coverage in both directions rather than relying only on equal counts.

AGENTS.md reference: AGENTS.md:L80-L83

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".