fix(hindsight): flush only the un-retained delta on session switch

[Vesna-9]

In append-capable mode (Hindsight >= 0.5.0) sync_turn ships only the
turns past _last_retained_turn_count on each retain and advances that
watermark, so already-appended turns are never re-sent. The
flush-on-switch path in on_session_switch, however, snapshotted the
full _session_turns (old_turns = list(self._session_turns)) and
re-appended every turn under update_mode='append', ignoring the
watermark. With the default retain_every_n_turns=1 every turn has
already been appended by the time the session rotates, so each
/resume, /branch, /new, /reset, and context compression appended
a duplicate copy of the entire session to the stored document — silent
memory-store corruption that skews recall and reflect.

The flush now mirrors sync_turn: in append mode it sends only
self._session_turns[self._last_retained_turn_count:] and skips the
retain entirely when that delta is empty, while the legacy/overwrite
path still resends the whole session (each retain replaces the
document). message_count metadata is derived from the trimmed delta.

What does this PR do?

Fixes duplicate-turn accumulation in the Hindsight memory provider. The
flush that runs when the agent rotates its session_id was re-appending
the whole buffered session even though sync_turn had already appended
each turn incrementally, so the backing document grew a fresh duplicate
of the conversation on every session switch.

Before: switching sessions in append mode re-appended every turn already
written by sync_turn; with retain_every_n_turns=1 that meant a full
duplicate of the session per /resume, /branch, /new, /reset, and
compression.

After: the flush sends only the turns past _last_retained_turn_count,
and fires no retain at all when the watermark has caught up to the
buffer. The legacy/overwrite path is unchanged.

Related Issue

N/A

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• plugins/memory/hindsight/__init__.py: in on_session_switch, resolve
  the old session's update_mode before computing the flush payload; in
  append mode slice _session_turns[self._last_retained_turn_count:]
  instead of copying the full buffer, skip the retain when the delta is
  empty, and size message_count from the trimmed delta. The
  legacy/overwrite branch still flushes the entire session.
• tests/plugins/memory/test_hindsight_provider.py: add
  test_session_switch_does_not_reflush_already_appended_turns (every
  turn already retained -> switch fires no flush) and
  test_session_switch_flush_ships_only_unretained_delta_in_append_mode
  (partially-buffered -> flush carries only the un-retained tail, never
  the already-appended turns).

How to Test

1. Run the targeted regression tests:
   scripts/run_tests.sh tests/plugins/memory/test_hindsight_provider.py
2. Both new tests pass against the fix; reverting the slice back to
   old_turns = list(self._session_turns) makes them fail because the
   flush re-ships turn1/turn2 under update_mode='append'.
3. Full file is green: 104 passed.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 15 (Darwin 25.5)

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

[tonydwb]

Code Review Summary

Verdict: Approved

Analysis

Correctness ✅

• Root cause clearly identified: sync_turn in append mode ships each turn as a delta and advances _last_retained_turn_count, but on_session_switch was flushing the entire _session_turns buffer — causing silent duplicate appends on every switch, resume, branch, reset, and context compression.
• Fix correctly slices self._session_turns[self._last_retained_turn_count:] to get only the un-retained delta.
• The if old_turns: guard correctly skips the flush entirely when the watermark has caught up (all turns already retained).

Testing ✅

• test_session_switch_does_not_reflush_already_appended_turns: verifies that when retain_every_n_turns=1, no retain call is made at all (watermark == buffer len).
• test_session_switch_flush_ships_only_unretained_delta_in_append_mode: verifies that when retain_every_n_turns=2, only turn 3 is flushed (turns 1+2 were already appended).

Code Quality ✅

• The restructured code is clearer — the delta-slicing logic is upfront and obvious.

Recommendation
Approve — well-scoped fix that eliminates the duplicate data corruption with thorough tests.

[teknium1]

Thanks for the focused regression fix. The premise holds on current main: sync_turn uses the append delta at plugins/memory/hindsight/__init__.py:1631-1640 and advances _last_retained_turn_count at :1693-1696, while on_session_switch still snapshots the entire buffer at :1822-1823 and sends it with the resolved append mode at :1855-1859.

The proposed branch mirrors the existing append/legacy split, skips an empty append delta, retains the legacy full-buffer overwrite behavior, and tests both the fully retained and partially buffered cases. GitHub currently reports the PR as mergeable.

Automated hermes-sweeper review.

[mlapida]

Independent reproduction on a separate deployment, plus one adjacent gap worth flagging.

I ran into this while investigating duplicate facts accumulating in a Hindsight bank, traced it to the same root cause, and wrote essentially the same patch before finding this PR — same delta slice, same skip-when-empty, same "legacy/overwrite still resends the whole session", same message_count derived from the trimmed delta. Converging on an identical fix independently is at least weak evidence the approach is the right one.

Reproduction, driving turns through sync_turn and then firing the hook, counting what actually reaches aretain_batch:

23 turns, retain_every_n_turns=5

                       shipped  unique  dupes
pre-fix                     43      23     20
patched                     23      23      0

Turns 1–20 committed twice — every turn up to the last retain_every_n_turns boundary. With the default retain_every_n_turns=1 it's the entire session, as your description says.

I also checked the legacy path doesn't regress, since that's the easy thing to break here:

legacy (no append support), 23 turns:
  shipped=73  unique=23   all turns present

Repeated full-buffer sends, which is correct under overwrite semantics — each retain replaces the document.

Still applies to current main. plugins/memory/hindsight/__init__.py:1829 on origin/main is still old_turns = list(self._session_turns). The PR is from early June and main has moved a long way since, so it likely needs a rebase, but the defect it fixes is untouched.

Adjacent gap: #55936 adds an on_session_end hook that snapshots the buffer the same way (old_turns = list(self._session_turns)). If that lands after this one, it reintroduces the identical bug at session end rather than session switch. I've left a note over there pointing back here. Whichever merges second should pick up the delta slice — or the slice could be factored into a small shared helper, since both call sites want byte-identical logic.