fix(hindsight): acknowledge append retains after success

[Tosko4]

What does this PR do?

Hindsight append-mode retains currently advance the turn watermark as soon as a background write is queued. If that write fails, the writer logs the exception and drops the job, while the advanced watermark prevents the failed turns from being included in the next retain. That silently loses conversation turns from Hindsight.

This PR gives append retains acknowledgement-on-success semantics:

• queued append jobs capture a target turn boundary;
• turns are removed from the in-memory append buffer only after aretain_batch succeeds;
• a failed prefix remains pending and is included in the next append boundary;
• FIFO jobs resolve only the still-unacknowledged part of their own target, preventing loss, duplication, or reordering while writes are in flight;
• session switches replace the active append state while queued old-session jobs continue to reference only the old state.

sync_turn() remains non-blocking, successful append retains still bound the buffer, and overwrite/legacy retention behavior is unchanged.

Related Issue

Related to #62977.

That PR identifies the unbounded append buffer, but its enqueue-time clear still makes queued turns unrecoverable when the background Hindsight write fails. This PR addresses the missing delivery/acknowledgement semantics while preserving the bounded-memory goal.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• Added per-session append-retain state with lock-protected pending-turn accounting in plugins/memory/hindsight/__init__.py.
• Changed append queue jobs to acknowledge and prune only the prefix confirmed by a successful aretain_batch call.
• Kept failed append prefixes available for a later boundary or old-session switch flush.
• Isolated queued old-session work from newly switched session state.
• Synchronized switch flush publication and state rotation with shutdown ownership.
• Reset append target accounting when a provider is reinitialized.
• Added deterministic regression coverage for failed retries, successful and failed in-flight writes, exact batch partitioning, post-ACK compaction, reinitialization, and session-switch/shutdown races.
• Updated the hermetic session-switch test provider to initialize the new append state.

How to Test

1. Run the Hindsight provider and session-switch suites:

   python -m pytest \
     tests/agent/test_memory_session_switch.py \
     tests/plugins/memory/test_hindsight_provider.py -q

   Result: 137 passed.

2. Run the append acknowledgement regressions directly:

   python -m pytest tests/plugins/memory/test_hindsight_provider.py \
     -q -k 'TestAppendRetainAcknowledgements'

   Result: 9 passed. The delivery-failure, concurrent shutdown, and reinitialization regressions were observed failing against the corresponding pre-fix behavior before their implementations were added.

3. Attempt the repository test suite:

   scripts/run_tests.sh tests

   The run was interrupted by the host's /tmp tmpfs user quota (OSError: [Errno 122] Disk quota exceeded) after the relevant Hindsight files had passed. Re-running the affected files with TMPDIR on the root filesystem eliminated most failures; the remaining unrelated files either hardcode /tmp/hermes_test or depend on the runner's original temp environment. The focused Hindsight/session-switch suite above is green; CI remains the authoritative repository-wide gate.

4. Run static and compatibility checks:

   ruff check \
     plugins/memory/hindsight/__init__.py \
     tests/agent/test_memory_session_switch.py \
     tests/plugins/memory/test_hindsight_provider.py
   python3 -m compileall -q plugins/memory/hindsight
   python3 scripts/check-windows-footguns.py \
     plugins/memory/hindsight/__init__.py \
     tests/agent/test_memory_session_switch.py \
     tests/plugins/memory/test_hindsight_provider.py
   git diff --check

   Result: all checks passed. Scoped ty reports the same existing diagnostic categories and counts as clean main; this change introduces no new type diagnostics.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: Ubuntu Linux, Python 3.11

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — N/A; no user-facing configuration or API changed
• I've updated cli-config.yaml.example if I added/changed config keys — N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide; the diff uses only cross-platform Python threading/queue primitives and passes the repository footgun check
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

Not applicable; this is a background memory-provider correctness fix with deterministic regression tests.

[Tosko4]

CI is green except for Python tests / Run tests slice 2/8, which failed twice on the timing-sensitive tests/tools/test_async_delegation.py::test_interrupt_all_signals_running_children assertion (expected "interrupted", got "completed").

This PR only changes the Hindsight provider and its session-switch/provider tests. The failing delegation test passes locally on both this PR head and the current main head (df5700ebe). I do not have permission to rerun upstream Actions; could a maintainer please rerun the failed job?

[teknium1]

Thanks for the focused acknowledgement fix. The current PR base still advances the append watermark at queue time (plugins/memory/hindsight/__init__.py:1638-1696), while the writer logs and drops a later failed job (plugins/memory/hindsight/__init__.py:1107-1128), so the reported loss path is real.

The new prefix state acknowledges only after aretain_batch() returns (plugins/memory/hindsight/__init__.py:1613-1673) and isolates old session state during rotation (plugins/memory/hindsight/__init__.py:1908-2015). The regression suite covers retry after failure, in-flight ordering, session switch, shutdown, reinitialization, and compaction (tests/plugins/memory/test_hindsight_provider.py:1497-1771). No verified correctness or design-fit issue found.

Automated hermes-sweeper review.

[GottZ]

This was generated by AI during triage.

Summary

The 26 listed PRs address or reference a memory-lifecycle complex spanning stale flush-agent writes, provider shutdown reachability, Hindsight final-batch loss, and append-write acknowledgement. The direct provider fixes add session-end flushing for buffered Hindsight turns, while #64599 independently prevents failed queued append writes from being acknowledged and discarded prematurely.

Related pull requests

• fix: prevent stale memory overwrites on session reset (#2670) #2675 [closed] partial — (+20/-0) — n/a: Skips cron flushes and adds a flush opt-out for stale-memory overwrites; closed because its cron bypass was incorporated into fix(gateway): prevent stale memory overwrites by flush agent (#2670) #2687.
• fix: Prevent memory overwrite by flush agent (add-only constraint) #2676 [closed] partial — (+5/-0) — n/a: Constrains the legacy flush agent to add-only writes; closed after fix(gateway): prevent stale memory overwrites by flush agent (#2670) #2687 superseded it with live-memory injection.
• fix(gateway): prevent stale memory overwrites by flush agent (#2670) #2687 [merged] best fix — (+204/-0) — n/a: Merged reference implementation for Memory flush agent overwrites live memory on session reset/gateway restart #2670, combining the cron bypass with live MEMORY.md/USER.md injection and focused stale-overwrite tests.
• Fix/flush agent terminal output #3045 [closed] related — (+116/-40) — n/a: Silences temporary gateway agents and spinner output rather than changing memory retention; closed after its commits were salvaged into fix(gateway): silence background agent terminal output #3297.
• fix(gateway): silence background agent terminal output #3297 [merged] related — (+116/-40) — n/a: Merged reference implementation for background-agent output suppression, adjacent to but distinct from the reported memory-loss paths.
• fix(gateway): trigger memory provider shutdown on /new and /reset #7762 [closed] best fix — (+5/-0) — n/a: Adds the missing shutdown_memory_provider() call for /new and /reset; closed after the targeted fix was incorporated into fix(openviking): salvage three OpenViking fixes — search ordering, gateway reset commit, setup env vars #9796.
• fix: close leaked cron and gateway resources #8331 related — (+154/-40) — n/a: Gateway and cron cleanup hunks are superseded, but the diff still contains an actionable ProcessRegistry pipe-close fix and test; this matches the keep_open verdict's narrow salvage path.
• fix(gateway): dispatch memory session-end hooks on expiry #11304 [closed] partial — (+72/-0) — n/a: Dispatches on_session_end through the former flush path but cannot cover an already soft-evicted agent; closed in favor of fix(gateway): fire on_session_end across all agent-cache eviction paths (salvage #31856) #57378.
• fix(gateway): fire MemoryProvider.on_session_end on session expiry #11410 [closed] partial — (+206/-0) — n/a: Adds session-end dispatch to the removed _flush_memories_for_session path; closed because current cleanup and fix(gateway): fire on_session_end across all agent-cache eviction paths (salvage #31856) #57378 supersede that integration point.
• fix(gateway): finalize session-end memory providers with transcript #12261 [closed] partial — (+180/-34) — n/a: Introduces transcript-aware expiry/reset finalization on the old flush architecture; closed as overlapping lifecycle work.
• fix(gateway): clean up cached agents on shutdown (#11205) #12917 [closed] partial — (+227/-0) — n/a: Sweeps idle cached agents during gateway shutdown so providers are finalized; closed after its implementation was salvaged into fix(gateway): tear down cached agents on stop (salvage #12917) #16207.
• fix(gateway): pass session messages to shutdown_memory_provider (#15165) #15481 [closed] partial — (+163/-1) — n/a: Forwards _session_messages during gateway provider shutdown; closed after fix(memory): pass session transcript to shutdown_memory_provider on gateway + CLI (#15165) #16571 incorporated it and added the CLI sibling fix.
• fix(gateway): tear down cached agents on stop (salvage #12917) #16207 [merged] partial — (+227/-0) — n/a: Merged shutdown reference that finalizes idle cached agents, making provider hooks reachable at gateway stop without implementing Hindsight's own final-buffer hook.
• fix(memory): pass session transcript to shutdown_memory_provider on gateway + CLI (#15165) #16571 [merged] best fix — (+285/-4) — n/a: Merged transcript-forwarding reference for gateway and CLI teardown; it fixes empty hook input but not Hindsight's missing buffered-tail flush.
• fix(hindsight): implement on_session_end to flush buffered turns at session exit #16697 [closed] partial — (+197/-0) — n/a: Adds an early Hindsight session-end remainder flush, but its _sync_thread wait and retention model predate the current writer and append behavior; closed and later resubmitted as fix(hindsight): implement on_session_end to flush buffered turns at session exit #28845.
• feat(hindsight): smart retain pipeline with client-side extraction #20262 [closed] related — (+911/-25) — n/a: Adds a broad smart-retain pipeline rather than the narrow final-buffer fix; closed after a contributor blocked plugin-specific core changes and requested framework-level design first.
• fix(hindsight): implement on_session_end to flush buffered turns at session exit #28845 partial — (+257/-0) — n/a: Implements append-aware versus legacy session-end flushing and queue draining, but overlaps fix(hindsight): flush buffered turns on session end #55046 and does not show end-then-switch coordination; despite its keep_open review, fix(hindsight): flush buffered turns on session end #55046 now carries the tested tail-only path.
• Feat/hindsight smart retain pipeline pluggable #29857 related — (+947/-25) — n/a: Adds the plugin-scoped smart-retain pipeline, but its keep_open review identifies ineffective main-model fallback, bypassed auxiliary settings, mismatched configuration wiring, and absent behavioral tests.
• fix(gateway): keep idle cached agents alive until session expires #31856 [closed] partial — (+73/-0) — n/a: Defers idle eviction until true session expiry so the live provider remains available; closed after fix(gateway): fire on_session_end across all agent-cache eviction paths (salvage #31856) #57378 incorporated it and added mode-none and LRU-cap handling.
• fix(hindsight): flush buffered turns on session end #36219 partial — (+164/-1) — n/a: Adds the missing writer-queue session-end hook but appends the full buffer after prior append retains and tests queue occupancy rather than payloads; despite its keep_open review, fix(hindsight): flush buffered turns on session end #55046 supplies the requested delta and executed regressions.
• fix(hindsight): flush buffered turns on session end to prevent silent data loss #36988 [closed] partial — (+145/-0) — n/a: Adds the same full-buffer session-end flush and therefore risks append duplication after prior batches; closed by contributors as a duplicate of fix(hindsight): flush buffered turns on session end #36219.
• fix(gateway): call _cleanup_agent_resources before idle-TTL eviction (#11205) #54293 [closed] partial — (+5/-0) — n/a: Calls full cleanup at idle-TTL eviction, which can terminate resources for a still-resumable session; closed in favor of the non-destructive fix(gateway): fire on_session_end across all agent-cache eviction paths (salvage #31856) #57378 design.
• fix(hindsight): flush buffered turns on session end #55046 best fix — (+265/-0) — n/a: Recorded best existing fix for [Bug]: Gateway restart drops session memory — shutdown_memory_provider receives empty messages #15165, [Bug]: Hindsight drops buffered turns on session end when retain_every_n_turns > 1 #36216, and Hindsight memory provider has no on_session_end override - retain_every_n_turns batching silently loses short sessions #53625: its current diff sends only the unacknowledged watermark tail in append mode, preserves full legacy overwrite payloads, and executes payload, guard, and FIFO tests consistent with its keep_open verdict.
• fix(memory/hindsight): flush buffered turns on session exit #55936 partial — (+109/-59) — n/a: Refactors end/switch flushing and slices the append tail, but on_session_end immediately preceding on_session_switch can enqueue that tail twice; despite its keep_open review, the visible diff has no regression covering this boundary.
• fix(gateway): fire on_session_end across all agent-cache eviction paths (salvage #31856) #57378 [merged] best fix — (+359/-2) — n/a: Merged lifecycle reference that preserves agents until finite-session expiry and commits memory before LRU soft eviction, covering hook reachability without implementing Hindsight's provider-side tail flush.
• fix(hindsight): acknowledge append retains after success #64599 related — (+613/-151) — n/a: Adds acknowledgement-on-success for queued append retains so failed prefixes remain retryable and old-session state stays isolated during rotation; the keep_open verdict reports high salvageability and no verified correctness or design-fit issue, but this complements rather than replaces fix(hindsight): flush buffered turns on session end #55046's session-end enqueue.

Duplicates

#2675 and #2676 were superseded by #2687; #3045 by #3297; #12917 by #16207; #15481 by #16571; and #31856 by #57378. #16697, #28845, #36219, #36988, #55046, and #55936 are competing Hindsight session-end flushes, with #55046 the recorded best existing fix and the others duplicate or partial variants.

Suggested consolidation

Keep #64599 open with the salvage path validated by its keep_open review: preserve acknowledgement-after-success, failed-prefix retry, FIFO target resolution, old-session isolation, and the supplied regression coverage. Separately keep #55046 open as the recorded best session-end-flush track; close #28845, #36219, and #55936 as duplicate/partial variants of #55046 only with the review deviations above recorded, while #8331 and #29857 remain separate author-action tracks for their contributor-reviewed fixes.

Cross-PR triage: Reviewed 26 pull requests and 6 issues in this complex. Each diff was read against this issue; Assessment working set: 372 kB of PR diffs, 95 kB of issue/PR text, 29 kB of discussion (43 comments), 35 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.