fix(retain): merge JSON arrays in append mode to preserve conversation-aware chunking

[qxxaa]

Problem

When update_mode=append is used with conversation-format content (flat JSON arrays of {role, content, timestamp} dicts), original_text becomes invalid JSON after the second retain cycle.

In _streaming_retain_batch, combined_content is built with:

combined_content = "\n".join([c.get("content", "") for c in contents_dicts])

For conversation content, this produces [...]\n[...] - two JSON arrays joined by a newline. This is neither valid JSON nor valid JSONL (lines are arrays, not dicts).

On subsequent append cycles, chunk_text() fails to parse the corrupted original_text. _chunk_jsonl() also rejects it. The text falls through to RecursiveCharacterTextSplitter, which splits on sentence boundaries (. ) inside message content with no awareness of conversation turn structure. Chunks begin mid-sentence without speaker attribution, and the extraction LLM misattributes statements.

Fix

In orchestrator.py, in _streaming_retain_batch, after the append-mode block assembles contents_dicts = [existing_content, *contents_dicts]: detect when all content items are JSON arrays of dicts, parse them, merge into a single flat array, and re-serialize.

Non-conversation content (plain text, JSONL, single JSON objects) is unaffected - the isinstance check fails and the block is skipped.

Tests

• New chunking and integration tests in test_chunking.py and test_retain_append_mode.py verify both the corrupted path (regression guard) and the merged path (correct routing through _chunk_conversation).
• test_newline_joined_json_arrays_bypass_conversation_chunking is intentionally a regression document (proves the broken path exists), not a behavioural assertion. The second test (test_merged_json_array_routes_to_conversation_chunking) validates the correct routing.
• max_chars=120 in the routing test is budget-sensitive but intentionally tight to force a multi-chunk split with short test messages.

Closes #2409

[nicoloboschi]

Confirmed the bug and traced it through the code. Forward fix is correct: merging the JSON arrays keeps original_text valid so subsequent appends route back through _chunk_conversation. Tests pass locally. Healing of pre-existing corrupted docs is intentionally out of scope.