Skip to content

fix(state): avoid WAL unlink race during state.db repair - #101085

Closed
JoaoMarcos44 wants to merge 2 commits into
NousResearch:mainfrom
JoaoMarcos44:fix/98552-telegram-truncated-preview-ack
Closed

JoaoMarcos44 wants to merge 2 commits into
NousResearch:mainfrom
JoaoMarcos44:fix/98552-telegram-truncated-preview-ack

Conversation

@JoaoMarcos44

Copy link
Copy Markdown
Contributor

Summary

  • Restores the pre-repair journal mode through the exclusive repair connection.
  • Prevents a second connection from recreating state.db-wal after the old WAL was unlinked.

Root cause
Journal-mode restoration opened a new connection after the exclusive repair guard had released the live database. In WAL mode, a writer could retain the unlinked WAL inode while that second connection created a fresh state.db-wal path.

Bottleneck scan
The focused repair path remains serialized by _exclusive_repair_db_guard. No additional correctness bottleneck was found in hermes_state.py or the regression test; unrelated checkpoint/VACUUM paths were not changed.

Tests

  • scripts/run_tests.sh tests/state/test_state_db_wal_unlink_race.py -q
  • 1 passed, 0 failed

Limitations

  • Focused regression covers connection reuse; no live multi-process corruption reproduction was available.

Closes #101064

JoaoMarcos44 and others added 2 commits September 1, 2026 09:20
…ccess

An oversized progressive edit is clipped to one message rather than split:
splitting mid-stream moves the editable message id, so the next accumulated
-token edit re-splits and the reply duplicates (NousResearch#48648). The clipped edit
succeeded, so `edit_message` returned a bare `SendResult(success=True)` and
the caller had no way to learn that Telegram stored less than it was given.

`GatewayStreamConsumer._send_or_edit` then advanced `_last_sent_text` from
the text it *sent*. Every "what has the user already seen" derivation is
built on that field — `_visible_prefix()`, `_continuation_text()`, and the
turn-final payload recorded by `_mark_skip_redundant_finalize()` and the
failed-final-edit guard. With an over-long value all three describe text no
API call ever carried: a fallback send skips the un-stored middle of the
answer, and `delivered_final_matches()` reconciles the clipped preview
against the completed `final_response`, so `gateway/run.py` logs "final
delivery already confirmed ... content_delivered=True" and suppresses the
corrective send. The user is left with a frozen preview and a stuck cursor.

Telegram now returns `raw_response={"stream_preview_truncated": True,
"delivered_prefix": <stored text>}` on all three clipped-preview returns
(pre-flight clip, saturated dedup, and the reactive message_too_long
retry) — the same `delivered_prefix` field the existing `partial_overflow`
contract carries on the failure branch. The consumer reads it through
`_delivered_text_for()` and tracks what the platform stored rather than
what it requested. Adapters that deliver exactly what they were given
report nothing and are unaffected.

Refs NousResearch#98552, NousResearch#48648, NousResearch#71643, NousResearch#78541, NousResearch#82656.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins platform/telegram Telegram bot adapter area/sessions Session lifecycle, resume, persistence, history sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Sep 2, 2026
@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference; please use your judgment.

PR #101085 — fix(state): avoid WAL unlink race during state.db repair + truncated Telegram preview fix (#98552)

  • Title describes the WAL branch; diff also bundles the Telegram stream fix under the same commit (branch name fix/98552-...). Both slices are independently correct — see also standalone fix(state): refuse SessionDB open and writes on a deleted WAL generation #101081 for the WAL-generation guard.
  • WAL repair slice (hermes_state.py:57-119, tests/state/test_state_db_wal_unlink_race.py:499-510): _restore_journal_mode_after_repair(db_path, before_mode, *, conn=None) now optionally reuses an already-open exclusive repair connection (owned_conn flag). repair_state_db_schema captures before_mode = _probe_journal_mode_for_repair and _repair_state_db_schema_locked passes it down; after a successful scratch-copy repair _restore_journal_mode_after_repair(..., conn=live_guard) re-applies WAL/DELETE inside the same live_guard that still holds the DB lock. Prevents the window where a second connection creates a fresh state.db-wal after the old one was unlinked while the guard had released. New test test_wal_restoration_reuses_exclusive_repair_connection asserts _connect_repair_durable is never called.
  • Stream slice (plugins/platforms/telegram/adapter.py:131-155,174-191, gateway/stream_consumer.py:10-45,43-45, tests/gateway/test_truncated_preview_ack.py:195-475): Telegram's saturated progressive edits (≥4096 UTF-16) are clipped to one message instead of split (avoiding fix(telegram): Infinite streamed message duplication loop during 4096-char overflow #48648 duplication loop). Previously the clipped edit returned bare SendResult(success=True) so GatewayStreamConsumer advanced _last_sent_text to the requested text, not the shorter prefix Telegram stored — every "already seen" derivation (_visible_prefix, _continuation_text, turn-final payload reconciliation) then described text no API call carried, causing delivered_final_matches to falsely report delivery and suppressing the gateway's corrective send (frozen clipped preview, Telegram streaming finalize fails — message truncated well under 4096 chars, content_delivered=True false positive #98552). Adapter now returns SendResult(raw_response={"stream_preview_truncated": True, "delivered_prefix": truncated}) on all three paths (saturation dedup, clipped send, truncated overflow). Consumer adds _delivered_text_for(result, sent_text) and advances self._last_sent_text from result.raw_response["delivered_prefix"] when present. Tests pin both layers and the end-to-end ClippingAdapter scenario (clipped preview + failed final edit → must not claim final delivery).

Non-blocking:

  • _restore_journal_mode_after_repair is still best-effort (WARNING on failure, never raises) — repair success is not retroactively failed by journal-mode re-apply.
  • delivered_prefix == "" is treated as absent so under-cap edits keep bare-success contract (no false positives).

Verdict: LGTM. Two leakage fixes in one PR: exclusive-connection WAL restore closes the post-repair split-brain, and delivered-prefix tracking closes the clipped-preview silent-loss.

@kshitijk4poor

Copy link
Copy Markdown
Contributor

Salvaged via #101736 (merged; the WAL commit cherry-picked with authorship preserved). The bundled Telegram preview commit was dropped — #98552 was already closed via #100533. Confirmed not redundant: main still reopened state.db after the guard released. Your unit test could only run on non-WAL-reset SQLite builds (never on macOS 3.46), so I added an end-to-end test through repair_state_db_schema that traces every connection against the guard's enter/exit and fails on main's shape on every platform. Thanks @JoaoMarcos44. Closing in favour of the salvage PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P1 High — major feature broken, no workaround platform/telegram Telegram bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

4 participants