fix: gateway persists transcript to state.db itself instead of relying on agent's internal flush

[rayjun]

Root Cause

gateway/run.py uses skip_db=True on all append_to_transcript() calls, relying on the agent's internal _flush_messages_to_session_db() to persist messages. When pm2/systemd restart or OOM intervenes between the agent's flush and the gateway's response delivery, the assistant message is permanently lost from state.db — leaving a session of consecutive role=user messages with no replies.

Fix

Remove the skip_db mechanism and let append_to_transcript always write to SQLite. The gateway and agent already share the same SessionDB instance; the gateway writing its own entries is idempotent for messages the agent already persisted, and a safety net for the ones it didn't.

Changes

• gateway/run.py: Remove agent_persisted flag and all skip_db=agent_persisted arguments
• gateway/session.py: Update append_to_transcript docstring to reflect new semantics

Testing

Pre-existing test suite passes with no new failures.

Closes #46088

[tonydwb]

Code Review Summary

Verdict: Approved

Gateway now persists transcript to state.db directly instead of relying on the agent's internal flush. More reliable persistence. Small, focused change.

Looks Good

• Single fix, clean implementation
• Moves persistence responsibility to the correct layer

---

Reviewed by Hermes Agent

[liuhao1024]

Regression Risk: Duplicate DB Writes

This PR removes skip_db=agent_persisted from four append_to_transcript() calls in _handle_message_with_agent, but does not modify the agent's own _flush_messages_to_session_db() path. The skip_db parameter was introduced specifically to prevent the duplicate-write bug documented in #860 / #42039 — the agent already persists messages to SQLite via its internal flush, and having the gateway also write them caused duplicate rows.

Current state on main: lines ~9149–9193 still use skip_db=agent_persisted in other code paths within the same file, so this PR creates an inconsistent policy: some append_to_transcript calls skip the DB (when the agent has already persisted), while these four no longer do.

Concrete risk: After this PR, for every agent turn, both the gateway's append_to_transcript and the agent's _flush_messages_to_session_db write the same user+assistant messages to session_messages. This doubles the row count and may confuse downstream consumers (e.g., get_messages_as_conversation, session search, compression heuristics that count messages).

Suggested approach: If the goal is to make the gateway the single source of truth for persistence, the agent's _flush_messages_to_session_db should be gated/removed for the same message set — not just the gateway's skip_db. Alternatively, dedup can be added at the append_to_transcript level (e.g., check for an existing row with the same (session_id, role, content_hash, timestamp) before inserting).

The other skip_db=agent_persisted calls at lines ~9149–9193 in gateway/run.py should also be updated for consistency if this approach is adopted.

[rayjun]

Good catch, thanks for the review. I've revised the approach:

Instead of removing skip_db without counterpart: Gateway always writes to the DB, and now has a last-message dedup — before inserting, it checks whether the most recent message in the session already has the same role + content. If the agent's already persisted it, we skip (no duplicate row). If the agent was killed mid-flush and never wrote it, the gateway's write is the safety net.

Changes:

• : Still removes the flag (gateway always writes), but the dedup is now in itself
• : calls before inserting
• : New method

The inconsistency concern is also addressed — there are no other call sites left in the file after this PR (the flag variable was only used in the four calls being changed).

[teknium1]

Thanks for this @rayjun — the bug in #46088 is real and well-diagnosed: if the gateway process dies (pm2/OOM) between the agent's _flush_messages_to_session_db() and response delivery, the assistant reply is lost and the session degrades to all-user turns.

Closing this particular fix though, because the dedup mechanism doesn't hold up and the direction fights an intentional (recently re-hardened) design:

1. The last-message role+content dedup fails for the common multi-message path. The agent flushes a whole turn [user, assistant, tool…], then the gateway loops and appends each new message. The guard only compares against the current DB tail: the first gateway write (user) doesn't match the tail (assistant) so it writes a duplicate — and that write shifts the tail, so the next (assistant) doesn't match either → another duplicate. Net effect is the exact #860/#42039 duplicate-row bug returning on every normal turn.

2. Content-shape mismatch. _flush_messages_to_session_db() rewrites content before persisting (multimodal → text summary, list content-parts → stripped text). The gateway passes raw content, so even the tail compare mismatches on those turns.

3. skip_db=agent_persisted is deliberate and was just reinforced. Commit 8e4c447 (Bug: User messages stored twice in state.db when agent and gateway both write to SQLite #42039) added it to the remaining fallback paths with a dedicated guard test, specifically to stop double-writes. Removing it reopens that.

The right fix targets the actual gap — the flush/delivery ordering window — using a per-message idempotency key rather than a tail role+content compare. has_platform_message_id() (#47237) already exists as exactly that kind of key and is the better primitive here. We'll take that route in a follow-up.

Your reproduction and root-cause writeup were spot on — thank you.