fix: prevent message duplication after context compression

[jasonjcwu]

Problem

After multiple context compression cycles, the UI transcript accumulates duplicate messages. In a session with 137 messages, 89 were duplicates from repeated history replays. The user sees old conversation fragments re-appear, and the actual new messages get buried — looking like messages are "swallowed".

Root Cause

Two issues in api/streaming.py:

1. _find_current_user_turn returns first match (not last)

When the user asks similar questions across turns (e.g., "推荐一下股票" twice), this function scans forward and returns the first matching user message. After context compression, result_messages contains the full history, so turn_candidates = result_messages[first_match:] includes the entire old conversation.

2. Merge dedup only handles 3 specific cases

The dedup loop in _merge_display_messages_after_agent_result only skips:

• Current user turn checkpoint duplicates
• Adjacent same assistant messages
• Compression marker duplicates

Old messages from prior compression cycles with matching identity but not fitting these 3 cases were appended as new messages.

Fix

1. _find_current_user_turn: Changed to return the last matching user turn instead of the first. This ensures turn_candidates only includes messages from the current turn onward.

2. General seen-set dedup: Added a safety net check — skip any candidate whose _message_identity is already in the seen set (unless it's the current user turn, which has its own checkpoint handling).

Testing

• All 63 existing compression/dedup/compaction tests pass
• Manually tested with realistic multi-compression scenario (duplicate user messages across turns)
• Verified no false dedup of legitimate identical messages in separate turns

[nesquena-hermes]

Reading the diff at api/streaming.py against origin/master, plus the full _merge_display_messages_after_agent_result body and the existing merge tests, I think the last-match fix is correct but the "general dedup safety net" introduces a regression in the prefix (non-compression) merge path.

The two changes do different things

Change 1 — _find_current_user_turn returning the last match (streaming.py:3023) — is well-targeted. After compression, result_messages holds the full history, and turn_candidates = result_messages[current_user_idx:] should start at the current turn. First-match anchored it at an old similar turn and replayed everything after. Last-match fixes that. Good.

Change 2 — the unconditional seen-set skip — is the problem. It lives in the shared candidate loop, so it runs in both branches, including the prefix branch:

if (
    key is not None
    and key in seen
    and not ((key == current_user_key) or is_current_user_turn)
):
    continue

seen is seeded from previous_display (streaming.py:3343), and _message_identity (streaming.py:2815) is content-based, not turn-aware — role + normalized text + tool ids. So any assistant/tool message whose text matches something already in the visible transcript gets dropped, regardless of which turn it belongs to.

Why this breaks an existing invariant

The immediately-preceding adjacent-assistant check (streaming.py:3393-3403) is deliberately adjacency-scoped, with this comment:

# Treat only adjacent identity matches as replay duplicates so
# identical answers in separate user turns remain visible.

The new block discards exactly that scoping. Trace test_same_assistant_text_across_different_turns_is_preserved (tests/test_session_save_mode.py:199):

• previous_display = [user "first prompt", assistant "same answer"]
• result_messages = previous_context + [user "second prompt", assistant "same answer"], msg_text="second prompt"

This takes the prefix branch (_messages_have_prefix is true). _strip_replayed_prefix finds no suffix/prefix overlap (candidate prefix is user "second prompt"), so candidates = [user "second prompt", assistant "same answer"]. The user turn is the current turn (kept). The trailing assistant "same answer" has key in seen (it matches the first turn's answer) and is neither the current user key nor a current user turn — so the new block continues and drops it. Expected [..., "second prompt", "same answer"] becomes [..., "second prompt"].

So I'd expect this test (and likely test_context_message_dedup.py:39's sibling assertions about identical answers) to fail. The PR body says all 63 pass — worth re-running tests/test_session_save_mode.py::test_same_assistant_text_across_different_turns_is_preserved specifically, since that's the one the safety net collides with.

Recommendation

After the last-match fix, turn_candidates already excludes old history, so the only pre-current-turn candidates are marker_candidates (compression markers, already deduped at streaming.py:3405). The general net is mostly redundant in the else branch and actively harmful in the prefix branch. Two options:

1. Scope the net to the compression branch only, and only to the pre-current-turn region — i.e. apply it to marker_candidates/history slices, not to the tail at/after current_user_idx. That kills replayed history without touching legitimate separate-turn repeats.
2. Track origin explicitly: compute prev_display_keys = set(seen) once before the loop and only skip a candidate if its index falls before the current-turn boundary and its key was in prev_display_keys. Newly-seen-this-merge identities (the actual current-turn answer) then survive.

A regression test mirroring test_same_assistant_text_across_different_turns_is_preserved but routed through a post-compression replay (full history in result_messages, repeated answer in the current turn) would lock both behaviors down at once.

[jasonjcwu]

Great catch, thank you. You're right — the general seen-set net was breaking the prefix branch invariant.

I reproduced the exact failure you described:

previous_display = [user "first prompt", assistant "same answer"]
result_messages = previous_context + [user "second prompt", assistant "same answer"]
# Got 3 messages — second "same answer" dropped

Removed the over-broad safety net in 59f36a2. The last-match fix to _find_current_user_turn already ensures turn_candidates starts at the current turn, so old history is excluded from the delta without needing a general net.

All 81 tests pass including test_same_assistant_text_across_different_turns_is_preserved and the compression replay scenario.

[nesquena-hermes]

Shipped in v0.51.235 (Release HC) via release PR #3490 — thank you @jasonjcwu! 🙏

Your fix to _find_current_user_turn (return the last matching user turn so post-compression result_messages doesn't replay the whole conversation history) is now on master and tagged.

One adjustment absorbed on the way in: the pre-release Codex regression gate found that a naive last-match could be overridden by a synthetic role:"user" continuation prompt (the agent loop injects "Continue"/empty-recovery nudges after the real turn — conversation_loop.py:1763/4183/4356) that only substring-matches the user text, which would anchor the merge past the real turn and drop the assistant/tool output in between. So the merged-in version tracks strong (exact _looks_like_current_user_turn) and weak (substring) matches separately and returns last_strong → last_weak → fallback — the real turn always wins over a later synthetic continuation. A 7-case regression test pins the behavior (verified to fail against the pre-fix logic).

Note: the PR body also described a "general seen-set dedup" change, but that wasn't in the committed diff — only the _find_current_user_turn change shipped. If you intended that second change, feel free to open a follow-up.

Authorship preserved via Co-authored-by and credited in the CHANGELOG. Closing as merged-via-release-stage (the change is in master, recommitted on the stage branch rather than via this branch directly).