fix(session): use second-level timestamp granularity in legacy dedup key

[bengdan]

Summary

Fix duplicate messages in chat (closes #2616).

_normalized_message_timestamp_for_key was preserving microsecond precision (%.6f). When the same message is persisted by both the WebUI sidecar JSON writer and the Hermes agent state.db writer, their timestamps can differ by a few microseconds. This caused _session_message_merge_key to produce different keys for the same logical message, letting both copies survive the dedup pass in merge_session_messages_append_only — resulting in every user message and AI response appearing twice in the UI.

Fix

Truncate to second-level granularity (int(timestamp)) so sub-second drift between the two writers collapses to the same key.

Reproduction

1. Run WebUI v0.51.95
2. Send any message
3. Both the user message and AI response appear twice

v0.51.90 is not affected.

Closes #2616

@nesquena Could you review and merge? One-line fix for the regression introduced in stage-387/388.

[nesquena-hermes]

Summary

Reading the two-dot diff origin/master..HEAD -- api/models.py (what would actually land if this merges), the change is a clean one-liner that drops the microsecond fallback in _normalized_message_timestamp_for_key. The three-dot stat looked alarming (~615 insertions) because the branch was cut against a03cd131 — well before stage-388 — and the commit re-applies what looks like a full snapshot of api/models.py. But the actual merge delta is tiny and surgical, which is good.

Code reference

What the merge actually changes in api/models.py:2495:

     except (TypeError, ValueError):
         return str(value)
-    if timestamp.is_integer():
-        return str(int(timestamp))
-    return ("%.6f" % timestamp).rstrip("0").rstrip(".")
+    # Drop sub-second precision so that timing drift between
+    # the sidecar JSON write and the state.db created_at write does not cause
+    # the legacy dedup key to differ for the same logical message.
+    return str(int(timestamp))

The helper feeds into the legacy 5-tuple at api/models.py:2515-2528 (_session_message_merge_key), which is the dedup key used by merge_session_messages_append_only at api/models.py:2531+. The legacy tuple is only consulted when the message lacks both id and message_id — i.e. exactly the case where the sidecar JSON writer and the agent state.db writer each persist the turn with their own created_at/timestamp and no shared identity.

Diagnosis

The fix is correct for the symptom in #2616. Sub-second drift between the two stores is the failure mode, and the timestamp string is one component of a 6-tuple that already includes role, content, tool_call_id, and tool_name. Collapsing to second-level granularity could in principle merge two genuinely distinct messages with identical role+content+timestamp-bucket, but two messages with identical role and identical full content arriving inside the same second is essentially indistinguishable from "the same message persisted twice" anyway. The other guard at merge_session_messages_append_only (max_sidecar_timestamp cutoff with explicit-id escape, api/models.py:2553) still preserves retries that carry their own message_id.

Worth noting alongside this PR: the follow-up comment on #2616 confirms the bug reproduces on fresh v0.51.95 sessions (not only legacy-upgraded ones), which strongly implies state.db is persisting created_at with sub-second precision from a separate time.time() call than the sidecar uses. This PR does not fix that root mismatch, but it neutralizes it on the read path, which is the right layer for a v0.51.95 hotfix.

Gap — no test

The PR does not add a regression test. A small addition to tests/test_webui_state_db_reconciliation.py would lock the contract:

def test_legacy_dedup_collapses_subsecond_drift():
    sidecar = [{"role": "user", "content": "hi", "timestamp": 1000.000123}]
    state   = [{"role": "user", "content": "hi", "timestamp": 1000.000789}]
    assert len(merge_session_messages_append_only(sidecar, state)) == 1

Without that, a future refactor of _normalized_message_timestamp_for_key could silently regress #2616.

Recommendation

The fix is sound. I would ask @bengdan to add the regression test above before merge so the dedup contract is pinned. Pairs naturally with PR #2618 (which closes a parallel stale-index gap on the metadata-only polling path) — together they cover both halves of the v0.51.95 reconciliation contract.

[heagandev]

Is this a backwards compatible fix, do existing chats with duplication get fixed or only does it only prevent duplication in future chats?

[nesquena-hermes]

@heagandev good question — the answer is "yes, retroactively, for the dominant failure mode in #2616, and no rewrite-on-disk is needed."

Why

_normalized_message_timestamp_for_key is consulted by the read-side reconciler merge_session_messages_append_only (api/models.py:2531+), not by any persistence step. Every full session load runs this merge:

# api/routes.py:3749-3758
_all_msgs = merge_session_messages_append_only(s.messages, state_db_messages)
...
_all_msgs = merge_session_messages_append_only(cli_messages, sidecar_messages)
...
_all_msgs = merge_session_messages_append_only(_metadata_sidecar, state_db_messages)

So as soon as you pull this fix, every subsequent session open re-merges the existing sidecar + state.db rows through the new second-granularity legacy key. Pairs that previously diverged only because the sidecar wrote 1730000000.000123 and state.db wrote 1730000000.000789 now key the same and collapse to one. No data migration, no state.db rewrite, no sidecar rewrite — the on-disk artifacts stay untouched and the dedup happens at the reconciliation layer on every load.

What it does NOT fix retroactively

Two narrow cases survive:

1. Duplicates persisted into the sidecar JSON itself. Look at the sidecar loop in merge_session_messages_append_only (api/models.py:2543-2550):

for msg in sidecar_messages:
    ...
    key = _session_message_merge_key(msg)
    seen_message_keys.add(key)
    merged_messages.append(msg)

Sidecar messages are appended unconditionally — the dedup is only applied to the state_messages loop. So if, on some past run, two near-identical assistant turns landed inside the same messages.json, they remain on disk and continue to render duplicated. That is rare in practice for #2616's symptom (the bug is sidecar-vs-state-db drift, not intra-sidecar), but worth noting.

2. Duplicates whose drift crosses a whole-second boundary (e.g. sidecar 1730000000.999 vs state.db 1730000001.001). Second-granularity won't collapse those. Empirically these are also rare; the observed drift in Duplicate messages in chat - both user and AI responses are repeated identically #2616 was sub-millisecond.

TL;DR for users

Pull the fix, hard refresh, reopen the duplicated session — duplicates created by sub-second drift between sidecar and state.db will disappear from the rendered view without any cleanup step. New sessions won't accumulate them. If a small handful of pre-existing duplicates remain after this and PR #2618 land, they're almost certainly the intra-sidecar variant and would need a separate one-shot cleanup pass (worth filing as a follow-up if it shows up).