fix(agent): inject streamed final_response into messages before persistence (#31269)

[xxxigm]

What does this PR do?

Fixes #31269: bridge-worker silently drops assistant replies from state.db.

The bridge worker (hermes_bridge.py) and state.db flush by index — _flush_messages_to_session_db writes messages[_last_flushed_db_idx:], so anything not in the structured messages list silently never reaches disk. A few break paths in run_conversation set final_response from already-streamed bytes WITHOUT appending the matching {"role": "assistant", "content": ...} dict:

• Partial-stream recovery (conversation_loop.py:~3543) — stream died mid-flight, the streamed buffer is recovered as the reply, break.
• Prior-turn-content fallback (conversation_loop.py:~3569) — empty follow-up after housekeeping tools, the earlier turn's text is reused, break.

In both cases the user saw the reply in the WebUI (it streamed through the Socket.IO callback) but state.db ended up with only the user message. The diagnostic in the issue captured this exactly:

[hermes-bridge-worker:default] [DBG-BUG2] NOTHING TO FLUSH! flush_from=490 msg_len=490 last_asst_role=assistant

— the slice was empty because the dict never landed.

Two small additions wired together:

1. Safety-net helper (agent/conversation_loop.py) — adds module-level _ensure_final_response_in_messages(messages, final_response) that appends a structured {"role": "assistant", "content": <text>, "_injected_from_final_response": True} dict when the messages tail doesn't already carry the streamed reply. Idempotent on the happy text-response path (the loop already appended its own dict at line ~3833). Whitespace-tolerant matching avoids spurious double-injection. Skipped for None / empty / whitespace final_response and for the "(empty)" user-facing failure sentinel — those have their own persistence semantics. Wired in right before the FINAL _persist_session (after scaffolding cleanup so it doesn't fight the empty-response sentinel pop).

2. Regression tests (tests/run_agent/test_final_response_injection_31269.py) — 15 focused tests including the bridge-worker diagnostic replay (_last_flushed_db_idx=1 covering the user turn, no assistant dict, asserts the SQLite append actually receives the assistant row).

Related Issue

Fixes #31269

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• agent/conversation_loop.py — new _EMPTY_RESPONSE_SENTINEL constant and _ensure_final_response_in_messages helper; integration call between _drop_trailing_empty_response_scaffolding and the FINAL _persist_session at run_conversation's tail; debug-level log when injection happened so post-mortems can correlate with the turn's exit reason.
• tests/run_agent/test_final_response_injection_31269.py — 15 unit + end-to-end regression tests (NEW FILE).

Backwards compatible: every existing exit path keeps working, the helper is idempotent, the new _injected_from_final_response flag is a transparent debugging marker that no existing consumer reads.

How to Test

# New regression suite
python -m pytest tests/run_agent/test_final_response_injection_31269.py -v
# expected: 15 passed

# Adjacent run_agent suites — confirms no regression in persistence /
# dedup / scaffolding-cleanup paths
python -m pytest tests/run_agent/test_run_agent.py tests/run_agent/test_860_dedup.py tests/run_agent/test_final_response_injection_31269.py -q
# expected: 362 passed

# Focused conversation-loop / partial-stream / persist sweep
python -m pytest tests/run_agent -k 'conversation_loop or partial_stream or persist or session_db' -q
# expected: 33 passed

End-to-end behaviour after the fix (TypeScript WebUI → bridge worker → state.db):

User sends \"what's 2+2?\"
  ↓
Bridge worker → AIAgent.run_conversation()
  ↓ partial-stream recovery: stream died, _current_streamed_assistant_text = \"2+2 is 4.\"
  ↓ final_response = \"2+2 is 4.\"; break
  ↓ _drop_trailing_empty_response_scaffolding(messages)  # no scaffolding present
  ↓ _ensure_final_response_in_messages(messages, \"2+2 is 4.\")  # injects
  ↓     messages = [user(\"what's 2+2?\"), assistant(\"2+2 is 4.\", _injected=True)]
  ↓ _persist_session(messages, ...)
  ↓ _flush_messages_to_session_db: messages[1:] writes 1 row → state.db ✅

Checklist

• Conventional Commits (fix(agent):, test(agent):)
• 2 focused commits, single author
• 15 new tests pass; 362 adjacent run_agent tests pass; 33 conversation-loop / persist tests pass
• Tested on macOS 15.6 (darwin 24.6.0)
• No new config keys, no schema migration
• Idempotent on the happy text-response path (no duplicate writes)
• Skips empty / sentinel final_response so empty-response semantics are preserved

[alt-glitch]

Competing fix with #31291 for the same issue #31269 — both inject _ensure_final_response_in_messages() into conversation_loop.py before persistence to fix bridge-worker silently dropping assistant replies from state.db. Recommend consolidating into a single PR.

[teknium1]

Thanks @xxxigm — clean root-cause writeup, and the two break paths you identified (partial_stream_recovery, fallback_prior_turn_content in conversation_loop.py) are exactly right. Closing because the mechanism this PR proposes — injecting the recovered final_response into messages before persistence (_ensure_final_response_in_messages()) — is one we evaluated and deliberately did not adopt.

The origin issue #31269 was closed NOT_PLANNED, and the competing fix #31291 (same helper, same two paths) was closed as implemented_on_main with the note: "The PR's proposed _ensure_final_response_in_messages() helper was not adopted, and the linked issue discussion correctly identified the bridge/cursor path as the real failure. Since current main now fixes that observable persistence failure with stronger flush semantics, this PR is redundant."

The real bug was at the flush layer, and that's where it was fixed:

• fix(agent): persist repaired-turn responses #46071 (merged) — identity-based SessionDB flushing, so a repair-shrunk messages list can't drop the assistant tail (this was the _last_flushed_db_idx cursor failure the issue diagnostic captured).
• fix(gateway): close silent response loss after agent tool calls (#29346) #34336 / fix(agent): surface silent partial-stream recovery (salvage #41498) #53987 (merged) — scope stream-delivery flags at tool boundaries and deliver recovered text via the gateway, with a visible "send continue" explainer on partial_stream_recovery.

Backfilling the recovered fragment into durable history as a complete assistant turn also works against #53987, which intentionally leaves response_was_previewed=False so the gateway surfaces the truncation + continue-flow rather than persisting a dead-mid-sentence turn as final.

Closing as superseded — the observable persistence failure is covered on current main. Same close we already made on the identical #31291. Appreciated the diligence.