fix: flatten sync_turn content to enable turn-aware chunking for hindsight plugin

[qxxaa]

Problem

sync_turn() produces a nested JSON array when building the retain payload:

# Each turn stored as a JSON array string:
turn = json.dumps(self._build_turn_messages(...))  # '[{user},{asst}]'

# At retain time, joined into:
content = "[" + ",".join(turns_to_retain) + "]"
# Produces: [[{user},{asst}],[{user},{asst}],...]

Hindsight's server-side chunk_text() (in engine/retain/fact_extraction.py) detects conversation format via:

parsed = json.loads(text)
if isinstance(parsed, list) and all(isinstance(turn, dict) for turn in parsed):
    return _chunk_conversation(parsed, max_chars)

This detection has no fallback handling for nested arrays. It silently fails the isinstance(turn, dict) check (outer elements are lists, not dicts) and drops through to RecursiveCharacterTextSplitter. The text splitter has no awareness of conversation turn boundaries and can split mid-message, producing chunks that begin without speaker attribution.

When an extraction LLM receives a chunk with no role prefix, it has no basis for determining who said what. Any first-person statement in such a fragment will be attributed to whichever speaker the model guesses, and that guess propagates through consolidation as a persistent fact.

Fix

One-line change in sync_turn(). Store each turn as comma-joined JSON objects instead of a JSON array:

# Before:
turn = json.dumps(self._build_turn_messages(...), ensure_ascii=False)

# After:
turn = ",".join(json.dumps(m, ensure_ascii=False) for m in self._build_turn_messages(...))

The two join sites (sync_turn line 1571, on_session_switch line 1759) already wrap with "[" + ",".join(...) + "]" and now produce a flat array:

[{"role":"user","content":"Q: ...","timestamp":"..."},{"role":"assistant","content":"JARVIS: ...","timestamp":"..."},...]

This passes the all(isinstance(turn, dict)) check and routes to _chunk_conversation(), which packs complete turns without splitting mid-message.

Affected code paths

Site	Line	Context
sync_turn()	1544	Turn serialisation at storage time

The join sites at lines 1571 (sync_turn retain) and 1759 (on_session_switch flush) are unchanged. They produce correct output once the stored elements are flat.

Regression risk

None. The change only affects the internal serialisation format of _session_turns entries. The final content string sent to aretain_batch is still a valid JSON array of message dicts. The only difference is nesting depth. Hindsight's extraction pipeline processes the same message objects regardless of how they arrived.

Test changes

• Updated test_sync_turn_retains_metadata_rich_turn - assertions now reference flat indices (content[0] not content[0][0]).
• Updated test_sync_turn_every_n_turns - len(content) == 6 (3 turns x 2 messages) instead of len(content) == 3 (3 pairs).
• Added test_sync_turn_produces_flat_conversation_array - explicitly verifies the flat-dict contract that enables Hindsight's turn-aware chunking path.

[qxxaa]

Tested on my environment with a live Hindsight deployment. After applying this fix, conversation chunks now consistently start with [{"role": and route through _chunk_conversation correctly.

Before the fix, the nested [[{...}]] structure caused Hindsight's server-side chunker to fall through to sentence-boundary splitting, producing chunks that started mid-sentence with no speaker attribution. With the flat serialization, multiple append cycles produce valid JSON arrays and chunking respects turn boundaries.

There's a complementary server-side fix I've submitted separately to Hindsight (vectorize-io/hindsight#2412) to handle the "\n".join() corruption on their end. The two fixes are independent but together resolve the full data path..

[qxxaa]

Hi, the complementary server-side fix for this (vectorize-io/hindsight#2412) merged back in June. Without this PR, the client is still sending nested arrays that bypass Hindsight's turn-aware chunking, causing it to fall back to blind text splitting and misattribute speakers.

The data integrity issue is on the sending side. Would appreciate this getting another look

[qxxaa]

Hi @alt-glitch - would you consider bumping this to P2? I don't think P3/cosmetic is the right fit here.

This is a data integrity bug, not a cosmetic issue. Without this fix, every Hindsight retain cycle sends nested arrays ([[{user},{asst}]]) that fail the all(isinstance(turn, dict)) check in chunk_text(). The fallback RecursiveCharacterTextSplitter splits mid-message with no turn awareness, producing chunks without speaker attribution. The extraction LLM then has to guess who said what, and wrong guesses persist through consolidation as permanent facts in the user's memory store.

The server-side companion fix (vectorize-io/hindsight#2412) merged back in June. That handles the "\n".join() corruption on the Hindsight side, but the hermes client is still the one sending malformed nested arrays. Both halves need to be fixed for the full data path to work correctly.

It's a one-line change with three updated tests and zero regression risk. Happy to address any feedback if needed.

[teknium1]

Thanks for the focused fix. The premise is verified on current main: sync_turn() stores each turn as a JSON array at plugins/memory/hindsight/__init__.py:1621, then wraps the joined fragments in another array at :1648. The same stored fragments feed the session-switch flush at :1836, so the proposed change correctly flattens both outbound paths. The linked upstream Hindsight companion fix merged as 78d32cd16cc10af20f552f0158ef4ee409d67e10 and explicitly recognizes flat arrays of message dictionaries.

Problems

• The new flat-array assertion covers the normal sync_turn() retain path, but TestSessionSwitchBufferFlush::test_buffered_turns_flushed_before_clear only checks that content strings are present. It does not protect the flat top-level-dictionary contract for the independent flush assembly at plugins/memory/hindsight/__init__.py:1836.

Suggested changes

• Parse the flushed item["content"] in that session-switch test and assert all top-level elements are dictionaries in user/assistant order.

Automated hermes-sweeper review.

[qxxaa]

Added the flat-dict contract assertion to TestSessionSwitchBufferFlush::test_buffered_turns_flushed_before_clear as suggested - parses the flushed item["content"] and asserts all top-level elements are dicts in user/assistant order.

Full test suite passes (117/117).

Note: the unrelated test_normalize_codex_response_salvage_is_xai_scoped failure in CI is pre-existing on main - not related to this change.

[qxxaa]

Rebased onto current main to absorb the issuer_kind pinning fix (8fa8aabb, #64844) - clears the unrelated test_normalize_codex_response_salvage_is_xai_scoped CI failure.