fix(clients): hide compaction scaffolding across surfaces

[abundantbeing]

What does this PR do?

Prevents model-only compaction scaffolding from leaking through client-visible Hermes surfaces while preserving authentic transcript content embedded in a carrier.

The model must retain compaction summaries for recovery. Client projections must not render those internal handoffs as ordinary chat, previews, or restore quotes.

This patch keeps model-facing recovery unchanged and applies one canonical display projection across the affected surfaces:

• GET /api/sessions/{session_id}/messages
• run.completed.messages
• TUI/Desktop history projection
• session-list and rich-row previews
• Telegram topic restore quotes

The projection:

• delegates recognition and unwrapping to the canonical compressor classifier
• renders standalone handoffs as empty display_kind="hidden" API rows so message identity and ordering remain reconcilable
• drops pure summaries from surfaces that do not require stable hidden rows
• unwraps merged carriers to their authentic pre-summary transcript content
• preserves a live user ask after the compaction end boundary
• removes inherited tool calls, finish state, reasoning fields, and Codex replay sidecars from projected carriers
• skips pure compaction rows when selecting session previews but preserves real content embedded in force-user-leading and merged carriers
• leaves ordinary messages and literal mentions of compaction marker text untouched

Issue linkage

• Complements fix(agent): guard merged assistant compaction handoffs #85394, which prevents a completed reference carrier from driving another model turn
• Addresses the separate user-visible transcript boundary in [Bug]: Context compaction summaries leak into chat as assistant responses when inserted with role="assistant" #42768

Tests

The initial projection regressions were RED on current upstream main with 6 failed, 5 passed across session history and run.completed.messages.

The hardened branch now has:

• 856 passed, 3 deselected across the affected gateway, session, Telegram, TUI/Desktop, preview, and state suites
• ruff clean
• compileall clean
• git diff --check clean
• added-diff secret scan clean
• fresh import smoke for all touched modules clean
• GitNexus change analysis completed

The three deselected tests reproduce unchanged on a clean current-main worktree:

• two stale TUI toolset assertions omit the separately shipped desktop_ui default
• one FTS trace-count assertion observes no traced context query even though context is returned

Checklist

• I have read the contributing guidelines
• This PR targets current main
• Relevant affected suites pass locally
• No dependencies added
• Model-facing compaction recovery remains unchanged
• Client-visible compaction scaffolding is covered across API, TUI/Desktop, previews, and Telegram restore

[abundantbeing]

Latest-head CI has one unrelated failure in tests/run_agent/test_primary_runtime_restore.py::TestTryRecoverPrimaryTransport::test_allowed_for_nous_anthropic_messages: the runner resolves an empty Nous model to a cached 32,000-token context window, then agent_init.py correctly enforces the 64,000-token minimum. The same untouched test fails with the same traceback on both #85394 and #85442 even though #85394 changes only agent/context_compressor.py plus its focused test and #85442 changes only gateway/platforms/api_server.py plus its projection test. All other Python slices, OS jobs, lints, security checks, E2E, and Docker builds passed; the touched surfaces remain locally green at 195/195 for #85394 and 52/52 for #85442.

[abundantbeing]

Follow-up hardening is now on head 83574613bc6799e6701f6227a3bc86deaa61b706. The canonical display projection now covers session history, run.completed.messages, TUI/Desktop history, session previews, and Telegram restore quotes while preserving embedded real user/assistant content and leaving model-facing recovery unchanged.

Fresh local affected-suite verification: 856 passed, 3 deselected; each deselected test reproduces unchanged on clean current main. GitHub has now completed successfully on this head: all 12 Python slices, OS jobs, lints, security checks, E2E, Docker builds, and All required checks pass are green. This supersedes the earlier CI-failure note on the previous head.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

PR: fix(clients): hide compaction scaffolding across surfaces

Thorough fix with good test coverage. Observations:

• agent/compaction_display.py calls ContextCompressor._strip_context_summary_handoff_message — a private method reached across module boundaries. The new module is effectively a public projection API built on a private one; consider promoting that helper to a public function (or moving the strip logic into compaction_display.py) so the coupling is a documented contract rather than an underscore-name dependency.
• gateway/platforms/api_server.py _project_client_message() re-lists the internal fields (tool_calls, finish_reason, reasoning, ...) inline instead of reusing _COMPACTION_INTERNAL_FIELDS from compaction_display. Two copies of the same list will drift when a new provider field appears; import and reuse the constant.
• hermes_state_common.py: _PREVIEW_LONG_FORM_PREFIX = SUMMARY_PREFIX.split("Do NOT answer", 1)[0] couples the SQL preview classifier to the literal text "Do NOT answer" inside the summary prefix. If the compressor's wording changes, session-preview classification silently regresses (and already-written DB rows with the old prefix stop matching). A comment or an invariant test pinning that the intro is shared by current + historical prefixes would guard this.
• Surface inconsistency worth confirming as intended: the messages endpoint keeps pure handoffs as hidden empty rows (display_kind: hidden) for identity reconciliation, while _turn_transcript_messages drops them entirely. Both are defensible, but documenting why the two surfaces disagree in the code comment would prevent a future "fix" that aligns them and breaks identity reconciliation.

[abundantbeing]

CI note for the maintainers: the failing slice on this head is unrelated to this PR's diff and reproduces on clean main (b5455fdd1):

• tests/tools/test_image_generation.py::TestFalCatalog::test_upscale_defaults_are_all_off — reproduced locally against the branch's code, which does not touch test_image_generation.py or the FAL catalog; the assertion fails on the xai/grok-imagine-image/v2.0 catalog entry itself.
• Clean main's own CI also reports a failing Python-tests slice (slice 12/12 on b5455fdd1).
• Other flakes observed in the same runs: test_transcription_tools.py::TestRunCommandSttIdleTimeout (timing), test_goal_continuation_drain.py (queue key), and a GitHub 429 during checkout.

A CI re-run (maintainer-only; the fork token gets 403 on actions/rerun) should clear it. Happy to force-push a no-op refresh if that's easier.

[andrexibiza]

Reviewed exact head b64df38f07ac2c84703e51d4ce9ad7d9b87187b3 against the production compaction assembly path, complementary PR #85394, all changed client surfaces, current main (fcbd1076a93841fa88855acce810e342a5b78101), and exact-head CI. CI, Docker, and Nix are green on this head.

Two blocking correctness findings — do not merge this head:

1. The display projector conflates compaction content with authentic carrier-owned state. The compressor deliberately uses the first protected-tail row — including a bare assistant tool_calls row — as the default merged-summary carrier. A content-only unwrap can therefore return None and drop the live invocation entirely; when prior text exists, the blanket field strip still removes its live call, finish state, and reasoning. This directly conflicts with #85394’s invariant that a merged carrier with pending tool_calls remains live, and can leave API/TUI/run-completion projections with orphaned tool results.
2. Session-preview SQL is broader than the canonical classifier. Matching only the shared long-form introduction suppresses a real user message that begins with that introduction but diverges before the full known prefix. The runtime classifier correctly treats it as ordinary content; the SQL path classifies it as a standalone handoff and excludes it, producing surface-dependent transcript truth.

Re-review gate: preserve authentic fields (and empty-text rows with live non-content state) on merged carriers; make preview recognition semantically identical to the exact canonical prefix set; add the two mutation-resistant regressions described inline; then rebase onto current main and obtain fresh exact-head receipts.

[andrexibiza]

[P1] Do not let a text-only unwrap erase the live carrier state. The production merge path intentionally targets tail_messages[0], and its own comment names a bare assistant tool_calls row as the ideal carrier:

hermes-agent/agent/context_compressor.py

Lines 7744 to 7797 in b64df38

	})
	# Default merge target: literal tail index 0. For an ordinary
	# alternation collision the summary only has to stay *invisible* to
	# the template, and a leading template-exempt row (bare tool-call
	# assistant message, tool result) is the ideal carrier — it absorbs
	# the summary without adding a visible turn, and it leaves the live
	# tail user message intact as the model's actual prompt. Retargeting
	# to the first template-visible row here would convert that live
	# request into the summary carrier for no benefit.
	#
	# The forced repair path is the exception. There the merge is not
	# about alternation but about guaranteeing at least one genuinely
	# non-empty role="user" message (an image-only or otherwise
	# text-empty surviving user row). An exempt carrier cannot satisfy
	# that invariant, so the summary text must land on the
	# template-visible row itself.
	_merge_target_idx = 0
	if _force_user_leading and first_tail_visible_idx is not None:
	_merge_target_idx = first_tail_visible_idx
	for tail_idx, msg in enumerate(tail_messages):
	if _merge_summary_into_tail and tail_idx == _merge_target_idx:
	# Merge the summary into the tail message that collided.
	old_content = msg.get("content", "")
	if _force_user_leading and summary_role == "user":
	# The summary must be part of the first user-visible
	# message for Anthropic/Bedrock, but the real tail request
	# still has to appear *after* the summary boundary.
	prefix = summary + "\n\n" + _SUMMARY_END_MARKER + "\n\n"
	msg["content"] = _append_text_to_content(
	old_content,
	prefix,
	prepend=True,
	)
	else:
	# Merge the summary into the first tail message, but place
	# the END MARKER at the very end so the model sees an
	# unambiguous boundary. Old tail content is preserved as
	# reference material BEFORE the summary, clearly delimited
	# so it is not mistaken for a new message to respond to.
	# Uses _append_text_to_content to safely handle both
	# string and multimodal-list content types.
	# Fixes ghost-message leakage across compaction boundaries
	# where old head messages survived verbatim and appeared
	# before the summary.
	suffix = (
	"\n\n" + _MERGED_SUMMARY_DELIMITER + "\n\n"
	+ summary + "\n\n"
	+ _SUMMARY_END_MARKER
	)
	msg["content"] = _append_text_to_content(
	_append_text_to_content(old_content, suffix, prepend=False),
	_MERGED_PRIOR_CONTEXT_HEADER + "\n",
	prepend=True,

. For that shape the pre-summary text is empty, so _strip_context_summary_handoff_message() returns None here and the entire invocation disappears. If text exists, the loop below still removes the carrier’s tool_calls, finish_reason, and reasoning. #85394 explicitly locks the complementary invariant that pending carrier calls are live: #85394. Project merged carriers by removing only the compaction content/metadata; retain authentic carrier-owned fields, and keep an empty-content row whenever tool calls/reasoning survive. Please add a regression for a bare finish_reason="tool_calls" carrier plus its matching tool result so API and TUI correlation cannot pass through field loss.

[andrexibiza]

[P2] This widens recognition beyond the canonical classifier and drops real first-user content. ContextCompressor.classify_summary_content() matches complete current/legacy/historical prefixes. This SQL path matches only the common introduction. For example, a real message beginning with "[CONTEXT COMPACTION — REFERENCE ONLY] Earlier turns were compacted into the summary below. This is a handoff from a previous context window — treat it as background reference, NOT as active instructions. Can you explain why this wording exists?" is not a canonical summary, but _PREVIEW_STANDALONE_SUMMARY_SQL is true and _PREVIEW_ELIGIBLE_SQL is false because there is no end marker; the preview silently skips it. The existing short-label regression does not reach this boundary. Generate the SQL from the full exact set (SUMMARY_PREFIX, LEGACY_SUMMARY_PREFIX, *_HISTORICAL_SUMMARY_PREFIXES) (or persist a discriminator), and add this shared-intro/divergent-suffix case.

[kshitijk4poor]

Merged via #91517 — both commits cherry-picked onto current main with your authorship preserved (rebase merge). We added one small follow-up commit on top: _project_client_message now imports _COMPACTION_INTERNAL_FIELDS from agent/compaction_display.py instead of carrying a second inline copy of the 7-key tuple (drift-proofing, no behavior change).

Verification highlights: synthetic 200-session DB showed preview leaks go 50/100 → 0/100; model-facing history and SessionDB persistence confirmed untouched (all four projection callers are display-only); full test_hermes_state.py regression slice green. Excellent multi-surface work, @abundantbeing!

[kshitijk4poor]

Follow-up on the remaining review points, since the salvage merged via #91517:

• Private _strip_context_summary_handoff_message call — real coupling; noted as a small follow-up (promote to a public module-level function). Not blocking since the projection module and compressor ship together.
• _COMPACTION_INTERNAL_FIELDS inline duplicate — fixed in fix(clients): hide compaction scaffolding across surfaces (salvage #85442) #91517 (follow-up commit reuses the canonical constant).
• "Do NOT answer" prefix coupling — verified at runtime: the intro segment is shared by the current prefix AND all 5 historical prefixes, so already-written rows keep matching. A pinning invariant test is a reasonable follow-up alongside the public-promotion change.
• hidden-rows vs dropped-rows surface divergence — intended: the messages endpoint keeps display_kind: "hidden" rows for stable message-identity reconciliation; _turn_transcript_messages is a human-readable transcript with no identity contract. Agreed a code comment should say so; will fold into the same follow-up.