Skip to content

fix(state): sessions repair should detect incomplete FTS schema, not just unopenable DBs - #71933

Open
Kyzcreig wants to merge 1 commit into
NousResearch:mainfrom
ANG-Ventures:up/repair-detects-fts-schema
Open

fix(state): sessions repair should detect incomplete FTS schema, not just unopenable DBs#71933
Kyzcreig wants to merge 1 commit into
NousResearch:mainfrom
ANG-Ventures:up/repair-detects-fts-schema

Conversation

@Kyzcreig

Copy link
Copy Markdown
Contributor

fix(state): sessions repair must not report a write-broken FTS schema as healthy

Summary

hermes sessions repair --check-only reports

✓ /Users/…/.hermes/state.db opens cleanly — no repair needed.

on a database that cannot record a single new message. Every
INSERT INTO messages fails with no such table: messages_fts_trigram, but the health
probe returns "clean", so the one diagnostic a user is told to reach for actively points
them away from the real fault.

The bug

_db_opens_cleanly() deliberately drives a rolled-back message write through the FTS
triggers, precisely so that FTS damage which leaves reads and integrity_check passing is
still caught (#50502). That probe works. The problem is how its failure is classified:

msg = str(exc).lower()
if "no such table" in msg or "no such column" in msg:
    return None          # assumed: brand-new file mid-init

The assumption holds for a genuinely fresh file, which has no sessions/messages tables
yet. But the same error text is produced by a structurally broken store: an FTS
trigger that outlived the virtual table it writes to. That happens after an interrupted
migration, a partially-applied schema change, or a manual DROP — and in that state the
store is completely unable to accept writes.

Both cases returned None, so a fully-populated database whose every message write fails
was reported as healthy.

The probe already validates openability; it did not validate that the FTS schema is
internally consistent. That gap is what makes this failure class hard to diagnose: the
user runs the recommended repair command, is told nothing is wrong, and has no next step.

Reproduction on main

Seed a normal store, drop messages_fts_trigram but leave its triggers, then ask the
repair probe what it thinks:

[broken] trigram table present: False; surviving triggers:
         ['messages_fts_trigram_insert', 'messages_fts_trigram_delete',
          'messages_fts_trigram_update']
[ground truth] INSERT INTO messages -> OperationalError: no such table: main.messages_fts_trigram

[_db_opens_cleanly] -> None
RED  ✗ repair reports 'opens cleanly — no repair needed'
        …while every message write fails. False negative.

With this patch, unchanged harness:

[_db_opens_cleanly] -> 'incomplete FTS schema — message writes fail: no such table:
  main.messages_fts_trigram. An FTS trigger references a table that no longer exists
  (interrupted migration or partial schema change).'
GREEN ✓ repair flags the DB as needing repair and names the cause

The fix

Narrow the no such table / no such column branch so it only absolves a database that
is actually mid-init. If the base tables exist, the store is populated and a failing
write probe is a real, actionable fault:

try:
    base_tables = {r[0] for r in conn.execute(
        "SELECT name FROM sqlite_master WHERE type = 'table' "
        "AND name IN ('sessions', 'messages')").fetchall()}
except sqlite3.DatabaseError:
    return None
if {"sessions", "messages"} - base_tables:
    return None                      # genuinely mid-init
return (f"incomplete FTS schema — message writes fail: {exc}. "
        "An FTS trigger references a table that no longer exists "
        "(interrupted migration or partial schema change).")

Two deliberate details:

  • The cjk_unicode61 check moved above this branch. That error is a
    capability gap on the probing process, not damage, and must keep returning None
    regardless of whether the base tables exist. Reordering keeps it unconditional.
  • The returned string names the missing table, so the reason surfaced by
    sessions repair is actionable rather than a bare sqlite error.

Once the check reports the fault, the existing repair pipeline handles it: Strategy 0
(rebuild_fts) re-creates the FTS schema from the canonical messages table, which is
the correct, least-destructive recovery for this shape.

Blast radius

This makes the health probe stricter, so the risk is false positives on databases that
are fine. Guarded by construction and by test:

  • The sessions/messages existence check means fresh and mid-init databases are
    unchanged.
  • A healthy store with the trigram index cleanly removed (table and triggers — the
    supported trigram-disabled shape) still reports clean, because nothing writes to a
    missing table. That is asserted explicitly, and it is the configuration that would
    otherwise be most at risk of a spurious flag.
  • The cjk_unicode61 capability path is preserved.

Tests

TestRepairDetectsIncompleteFtsSchema in tests/test_hermes_state.py:

  • test_repair_check_detects_trigger_without_its_fts_table — establishes ground truth
    first (asserts the raw INSERT INTO messages genuinely fails), then asserts
    _db_opens_cleanly() returns a reason naming the missing table. Fails on main.
  • test_repair_check_still_passes_a_healthy_store — the false-positive guard. Asserts a
    normal store reports clean, and that the same store with the trigram index cleanly
    removed still reports clean. Passes both before and after, by design: it exists to
    constrain this change, not to demonstrate it.

tests/test_hermes_state.py passes in full. ruff clean.

Scope

Detection only. This PR does not change what repair does once a fault is found — the
existing rebuild_fts strategy already covers this shape.

Companion PR: fix(state): skip the trigram sweep in _fts_rebuild_finish when the trigram index is unavailable fixes the optimize-storage crash that produces this schema state
in the first place. Independent; either can land first.

…hema as healthy

## Summary

`hermes sessions repair --check-only` reports

```
✓ /Users/…/.hermes/state.db opens cleanly — no repair needed.
```

on a database that **cannot record a single new message**. Every
`INSERT INTO messages` fails with `no such table: messages_fts_trigram`, but the health
probe returns "clean", so the one diagnostic a user is told to reach for actively points
them away from the real fault.

## The bug

`_db_opens_cleanly()` deliberately drives a rolled-back message write through the FTS
triggers, precisely so that FTS damage which leaves reads and `integrity_check` passing is
still caught (NousResearch#50502). That probe works. The problem is how its failure is classified:

```python
msg = str(exc).lower()
if "no such table" in msg or "no such column" in msg:
    return None          # assumed: brand-new file mid-init
```

The assumption holds for a genuinely fresh file, which has no `sessions`/`messages` tables
yet. But the **same error text** is produced by a structurally broken store: an FTS
trigger that outlived the virtual table it writes to. That happens after an interrupted
migration, a partially-applied schema change, or a manual `DROP` — and in that state the
store is completely unable to accept writes.

Both cases returned `None`, so a fully-populated database whose every message write fails
was reported as healthy.

The probe already validates **openability**; it did not validate that the FTS schema is
internally consistent. That gap is what makes this failure class hard to diagnose: the
user runs the recommended repair command, is told nothing is wrong, and has no next step.

## Reproduction on `main`

Seed a normal store, drop `messages_fts_trigram` but leave its triggers, then ask the
repair probe what it thinks:

```
[broken] trigram table present: False; surviving triggers:
         ['messages_fts_trigram_insert', 'messages_fts_trigram_delete',
          'messages_fts_trigram_update']
[ground truth] INSERT INTO messages -> OperationalError: no such table: main.messages_fts_trigram

[_db_opens_cleanly] -> None
RED  ✗ repair reports 'opens cleanly — no repair needed'
        …while every message write fails. False negative.
```

With this patch, unchanged harness:

```
[_db_opens_cleanly] -> 'incomplete FTS schema — message writes fail: no such table:
  main.messages_fts_trigram. An FTS trigger references a table that no longer exists
  (interrupted migration or partial schema change).'
GREEN ✓ repair flags the DB as needing repair and names the cause
```

## The fix

Narrow the `no such table` / `no such column` branch so it only absolves a database that
is *actually* mid-init. If the base tables exist, the store is populated and a failing
write probe is a real, actionable fault:

```python
try:
    base_tables = {r[0] for r in conn.execute(
        "SELECT name FROM sqlite_master WHERE type = 'table' "
        "AND name IN ('sessions', 'messages')").fetchall()}
except sqlite3.DatabaseError:
    return None
if {"sessions", "messages"} - base_tables:
    return None                      # genuinely mid-init
return (f"incomplete FTS schema — message writes fail: {exc}. "
        "An FTS trigger references a table that no longer exists "
        "(interrupted migration or partial schema change).")
```

Two deliberate details:

- **The `cjk_unicode61` check moved above this branch.** That error is a
  *capability* gap on the probing process, not damage, and must keep returning `None`
  regardless of whether the base tables exist. Reordering keeps it unconditional.
- **The returned string names the missing table**, so the reason surfaced by
  `sessions repair` is actionable rather than a bare sqlite error.

Once the check reports the fault, the existing repair pipeline handles it: Strategy 0
(`rebuild_fts`) re-creates the FTS schema from the canonical `messages` table, which is
the correct, least-destructive recovery for this shape.

## Blast radius

This makes the health probe **stricter**, so the risk is false positives on databases that
are fine. Guarded by construction and by test:

- The `sessions`/`messages` existence check means fresh and mid-init databases are
  unchanged.
- A **healthy store with the trigram index cleanly removed** (table *and* triggers — the
  supported trigram-disabled shape) still reports clean, because nothing writes to a
  missing table. That is asserted explicitly, and it is the configuration that would
  otherwise be most at risk of a spurious flag.
- The `cjk_unicode61` capability path is preserved.

## Tests

`TestRepairDetectsIncompleteFtsSchema` in `tests/test_hermes_state.py`:

- `test_repair_check_detects_trigger_without_its_fts_table` — establishes **ground truth**
  first (asserts the raw `INSERT INTO messages` genuinely fails), then asserts
  `_db_opens_cleanly()` returns a reason naming the missing table. Fails on `main`.
- `test_repair_check_still_passes_a_healthy_store` — the false-positive guard. Asserts a
  normal store reports clean, *and* that the same store with the trigram index cleanly
  removed still reports clean. Passes both before and after, by design: it exists to
  constrain this change, not to demonstrate it.

`tests/test_hermes_state.py` passes in full. `ruff` clean.

## Scope

Detection only. This PR does not change what `repair` *does* once a fault is found — the
existing `rebuild_fts` strategy already covers this shape.

Companion PR: `fix(state): skip the trigram sweep in _fts_rebuild_finish when the trigram
index is unavailable` fixes the `optimize-storage` crash that produces this schema state
in the first place. Independent; either can land first.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/sessions Session lifecycle, resume, persistence, history P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Jul 26, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for isolating a real false negative in the write-health probe. Current main still returns None for this class at hermes_state.py:1201-1202, so the detector change remains needed.

Problems

  • The changed tests stop at _db_opens_cleanly(). The PR body says Strategy 0 repairs the missing trigram table, but current Strategy 0 skips absent tables at hermes_state.py:1275-1285; recovery instead falls through to the FTS-schema drop at hermes_state.py:1352-1360 and is rebuilt on a later SessionDB open. The patch needs an end-to-end repair regression proving preserved rows, resumed writes, and usable FTS after reopening.

Suggested changes

  • Salvage the narrow classifier change into current hermes_state.py and add the recovery test near the existing repair coverage in tests/test_state_db_malformed_repair.py:272-297.

This is an automated hermes-sweeper review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants