fix(session): keep state db replays out of sidecar tail

[ai-ag2026]

Summary

• Prevent replayed state.db rows from being appended after an already-correct sidecar transcript tail.
• Track sidecar visible-message occurrences so replay duplicates are skipped without collapsing legitimate repeated assistant/user messages.
• Add regressions for both full-replay and middle-segment replay shapes.

Root cause

merge_session_messages_append_only() already tried to skip state.db rows that replay the sidecar transcript, but two edge cases still leaked through:

1. The final row of a replayed sidecar prefix was not skipped because the replay index had reached the sidecar sequence length.
2. A replayed middle segment was not considered prefix replay, so old state.db rows could be appended after the saved assistant tail.

That made /api/session appear to end on an old user prompt even when the saved sidecar already ended on the real assistant answer.

The fix uses a visible-identity replay budget based on the sidecar's visible message counts. This skips replayed state rows up to the number already represented in the sidecar, while preserving additional legitimate repeats from state.db.

Related reconnaissance

• Complements fix: prevent replayed context in reconciliation and metering #2685, which also touches replay/reconciliation paths but still contains the edge case fixed here.
• Distinct from fix: drop stale cached user tail after saved assistant #2733, which handles stale in-process cached user tails; this PR fixes state.db replay rows in the display merge itself.

Test plan

• python3 -m pytest tests/test_session_lost_response_regression.py -q -o 'addopts='
• python3 -m pytest tests/test_webui_state_db_reconciliation.py tests/test_session_lost_response_regression.py -q -o 'addopts='
• python3 -m py_compile api/models.py
• git diff --check
• Added-line hygiene scan for private/local markers: added_marker_hits 0

[nesquena-hermes]

Summary

Pulled the branch and read api/models.py:3023-3120 at HEAD vs origin/master, plus the two new tests in tests/test_session_lost_response_regression.py:198-251. The diagnosis is correct: the previous replays_sidecar_prefix and state_replay_idx < len(sidecar_visible_sequence) gate had two real gaps that allowed restamped state.db rows to land after the saved assistant tail. The new visible-key budget closes both.

Code reference

The new _matching_visible_duplicate() returns the matched key tuple, and the merge loop tracks per-key skip counts against per-key sidecar counts at api/models.py:3070-3119:

if replays_sidecar_prefix:
    matched_visible_key = _matching_visible_duplicate(visible_key, sidecar_visible_keys)
    if matched_visible_key is not None:
        skipped_state_visible_counts[matched_visible_key] = (
            skipped_state_visible_counts.get(matched_visible_key, 0) + 1
        )
    continue
...
matched_visible_key = _matching_visible_duplicate(visible_key, sidecar_visible_keys)
if matched_visible_key is not None:
    skipped_count = skipped_state_visible_counts.get(matched_visible_key, 0)
    sidecar_count = sidecar_visible_counts.get(matched_visible_key, 0)
    if skipped_count < sidecar_count:
        skipped_state_visible_counts[matched_visible_key] = skipped_count + 1
        continue

The two key behavioral fixes:

1. The original gate dropped its state_replay_idx < len(sidecar_visible_sequence) clause and unconditionally continues once replays_sidecar_prefix is true. That solves the "final replayed row of a full prefix replay" case in test_state_db_full_replay_does_not_append_after_sidecar_tail — previously, state_replay_idx reached the sequence length on the last assistant tail row, the gate evaluated false, and the row was appended.

2. The post-prefix block now also runs a budget check: any state row whose visible key matches a sidecar key gets skipped, but only up to the count the sidecar already carries. That's what catches the middle-segment replay in test_state_db_middle_segment_replay_does_not_append_after_sidecar_tail where the replayed segment doesn't form a prefix at all.

Diagnosis / Recommendation

The "preserve legitimate repeats" property is what makes this fix non-destructive. If a user genuinely sends the same text twice and the assistant genuinely answers the same way twice, sidecar_visible_counts[(role, content)] records 2; state.db replay of those same two rows skips 2 (skipped_count < sidecar_count); a third state.db occurrence of the same content (a real new turn) falls past the budget and is appended. That's the right semantics.

One thing I'd verify before merge:

Workspace-tagged user content. The middle-segment test injects [Workspace::v1: /tmp/project-workspace]\nErstelle deine Version on the state side against the bare Erstelle deine Version on the sidecar. _matching_visible_duplicate() walks the substring/loose-content paths to match these — that's the existing loose-match behavior, not new in this PR, but it's the load-bearing matcher. Skim _loose_session_message_content() to confirm it strips [Workspace::v1: ...] tags consistently with how _session_message_visible_key() keys are produced on both sides. If a future workspace-tag format change desyncs those two helpers, the budget will under- or over-count and replays will leak again. A tiny additional test that asserts loose-match contains the workspace-tag prefix strip would lock the contract in place.

Interaction with the existing <= timestamp gate at lines 3097-3110. The new post-prefix budget check sits between the message_id dedup and the timestamp gate. For state rows that match a sidecar visible-key, the budget either skips (if under budget) or proceeds to the <= max_sidecar_timestamp block. With 100.1, 100.2, 100.3, 100.4 all above sidecar 100, none of them hit the <= short-circuit, so the budget alone has to do the work. That looks correct given the assertion in the test that the merged transcript ends exactly on the sidecar tail. Good.

Verification

The two new tests directly exercise the two failure modes. Worth running pytest tests/test_webui_state_db_reconciliation.py tests/test_session_lost_response_regression.py -q against the branch and confirming no prior reconciliation case regressed — the visible-key budget is a stricter filter than the old prefix-only gate, so any test that relied on "state.db row with sidecar-matching content appears in the merged output a second time" would catch a real semantic change. From the PR description ("addopts=' runs pass"), that already held.

The narrative reference to #2685 is accurate — #2685 (Stage 396 / Stage 399) shipped the dedup-restamped-replay class for the strict-timestamp path, and this PR closes the residual edge case where prefix-replay-by-visible-key didn't fully cover the tail. The PR is appropriately scoped: 35 lines in models.py, 56 in the regression test.

LGTM as the narrow fix it claims to be. The two notes above are about future-proofing, not blocking.

(File:line citations: api/models.py:3023-3120 on PR HEAD, tests/test_session_lost_response_regression.py:198-251 on PR HEAD, prior Stage 399 / PR #2685 lineage.)

[nesquena-hermes]

Shipped via cherry-pick in v0.51.111 (commit dd07334 via release PR #2759). Closing the original PR — the change landed under your authorship via git cherry-pick 39cadf4933 onto a fresh stage from current master. Thanks for the fix @ai-ag2026!