fix: dedupe tool-only partial recovery markers

[Michaelyklam]

Thinking Path

• Cancelled and interrupted WebUI streams preserve partial assistant output so reloads do not lose already-visible progress.
• Tool-only turns can have empty assistant content but still carry reasoning text and _partial_tool_calls metadata.
• The existing duplicate guard only ran for non-empty text content, so repeated recovery/cancel handling could keep inserting identical empty-content partial markers.
• The fix needs to deduplicate by the full partial payload and repair sessions that already accumulated adjacent duplicate partials.

What Changed

• Added a stable partial-marker signature based on content, reasoning, and _partial_tool_calls metadata.
• Updated cancel_stream() to skip inserting an equivalent partial marker within the current user turn, including tool-only/reasoning-only markers with empty content.
• Added a full-load session self-heal that collapses adjacent duplicate _partial markers and persists the repaired session without changing recency/index ordering.
• Added regression coverage for tool-only partial dedupe, current-turn scoping, and persisted session-load cleanup.
• Added an Unreleased changelog entry.

Why It Matters

This prevents a single failed or cancelled tool-only stream from bloating a session with thousands of identical _partial messages and rendering thousands of duplicate Thinking cards on reload.

Closes #2592

Verification

• /home/michael/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_issue2592_partial_dedupe.py -q — 3 passed
• /home/michael/.hermes/hermes-agent/venv/bin/python -m py_compile api/streaming.py api/models.py — passed
• /home/michael/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_issue2592_partial_dedupe.py tests/test_sprint49.py tests/test_session_save_mode.py -q — 31 passed
• git diff --check — passed

Risks / Follow-ups

• The load-time repair intentionally collapses only adjacent identical partial markers so repeated but separate later turns are preserved.
• The shrink writes a .bak through the existing session-save safeguard before replacing the bloated session file.

Model Used

AI-assisted change with repository inspection, targeted editing, and shell-based test verification.

[nesquena-hermes]

Fix matches the #2592 root-cause exactly

Reading the diff at 0736e454 against origin/master, this is a targeted fix for the bug I (we) flagged on #2592. The reproduction trail was clear: a tool-only failed turn has _stripped == '', the existing dedup guard at api/streaming.py:5507-5524 (master) was content-gated by if _stripped:, so for tool-only turns the dedup never ran and every cancel/replay path re-inserted the same partial. This PR walks that down to root.

Diff at the streaming side

The new signature-based guard at api/streaming.py:2593-2622:

def _partial_message_signature(message: dict) -> tuple:
    ...
    return (
        str(message.get('content') or '').strip(),
        str(message.get('reasoning') or '').strip(),
        tuple(tool_sig),
    )

is exactly the (content, reasoning_signature, tool_call_signature) shape #2592 called for. The _partial_marker_already_present helper at lines 2625-2641 also scopes the search to the current user turn only by walking backwards to the most recent role=='user' message — this is important because the same tool call ("retry the failing thing") is a legitimate distinct partial in the next turn, not a duplicate. The test at tests/test_issue2592_partial_dedupe.py:40-50 covers that scoping correctly.

The cancel branch at api/streaming.py:5605-5630 drops the content-gated _partial_already_present block and replaces it with the signature-aware call:

if not _partial_marker_already_present(
    _cs.messages,
    _partial_msg,
    before_idx=_cancel_marker_idx,
):
    _cs.messages.insert(_cancel_marker_idx, _partial_msg)

That's the surgical fix. Good.

Self-heal on load

api/models.py:565-575 adds a one-shot collapse on Session.load() for sessions already corrupted in the wild:

data['messages'], _collapsed_partials = _collapse_adjacent_duplicate_partials(data.get('messages'))
session = cls(**data)
if _collapsed_partials:
    try:
        # Self-heal bloated sessions on first full load without touching
        # recency/index ordering; save() creates a .bak because this
        # intentionally shrinks the transcript (#2592).
        session.save(touch_updated_at=False, skip_index=True)
    except Exception:
        ...

This is the right shape. Two things I checked:

1. .bak is created by save() on shrink. api/models.py:444-498 defines save() with touch_updated_at and skip_index params, and the per-P0 DATA LOSS: metadata-only Session.save() wipes conversation history (v0.50.279+) #1558 backup safeguard already writes a <sid>.json.bak whenever the incoming messages array is shorter than the existing one. So collapsing 2,047 duplicates produces a recoverable snapshot. The test at tests/test_issue2592_partial_dedupe.py:84 asserts (session_dir / f"{sid}.json.bak").exists() — good.

2. skip_index=True is correct. Sidebar recency/index ordering uses updated_at. The self-heal sets touch_updated_at=False so the timestamp is preserved (test at line 83 asserts persisted["updated_at"] == 200.0), and skip_index=True avoids a redundant index write since the metadata didn't change. That keeps the sidebar exactly as the user left it.

The _collapse_adjacent_duplicate_partials algorithm only collapses adjacent duplicates (line 779: previous_partial_sig == sig), which means a partial with a different _partial_tool_calls payload between two duplicates will break the collapse run. That's the right shape — if two partials are not adjacent, they came from genuinely distinct streams and should be preserved. The test seeds three adjacent identical partials and asserts only one survives.

One small note

The dedup signature includes 'preview'/'snippet' from _partial_tool_calls:

str(tool_call.get('preview') or tool_call.get('snippet') or ''),

That's correct for the reported case (preview is empty/identical for an is_error: True tool call that never produced a result), but a tool that produces a different preview on retry — e.g. a flaky terminal call that succeeds the second time and writes a real snippet — would intentionally not be deduped. I think that's the right behavior (different result = different turn), but worth noting explicitly in the docstring so future readers know it's intentional.

CHANGELOG entry

Added at CHANGELOG.md under [Unreleased] / Fixed. Good.

Verdict

CI green on 3.11/3.12/3.13. The test suite (tests/test_issue2592_partial_dedupe.py:23-86) exercises both the in-flight dedup path and the self-heal on load, including the .bak invariant. Self-contained, minimal, doesn't change semantics for the non-failed path, and matches the issue's root cause analysis precisely.

I'd land this. Once it lands, the user reporting #2592 can do a single full-load of each affected session and the disk bloat self-heals — the comment on #2592 should mention that so they don't reach for a manual cleanup tool.