feat(compression): add keep_history context engine — request-only compaction that preserves the visible chat history

[skyzea1]

Problem

When auto-compression triggers, the entire visible chat history disappears from the UI. Everything from the beginning of the conversation is wiped out and replaced with just a summary card — the user can no longer scroll up to review the original past messages.

Root cause

The built-in ContextCompressor.compress() rewrites the live transcript on every compaction: archive_and_compact() soft-archives every active row (active=0, compacted=1) and re-inserts head + LLM summary + tail as fresh rows. UIs that render the session store (desktop app, dashboard, web) show only active=1 rows, so after a compaction the visible timeline collapses to a summary card plus the recent tail.

Real-world evidence from a desktop session (20260813_065425_b72b43): 1503 stored messages, only 197 active — 1044 rows soft-archived by repeated compressions. The archived rows are still in the DB (searchable via session_search), but the UI cannot show them.

Related: #45117 (make pre-compression history discoverable/resumable), #82462, #78484.

Fix

Adds a bundled keep_history context engine that takes the Codex-Desktop approach: compact background context (tool logs, terminal dumps, file arrays) only in the per-request message list via the existing select_context() hook, which is request-only by contract — the persisted transcript is never rewritten, so the visible chat history stays fully intact and scrollable at all times while the model still receives a bounded, compacted context.

• should_compress() always returns False — the destructive LLM-summary + archive_and_compact path never auto-fires.
• select_context() deterministically prunes old tool results (dedup, one-line summaries for large outputs, tool_call-arg truncation) when the request crosses the token trigger, with a rearm watermark so prompt-cache breaks stay episodic, plus a hard-ceiling trim so the request still fits the window in pathological sessions.
• compress() (manual /compress, gateway hygiene) performs the same deterministic prune — every user/assistant message survives verbatim and no summary card is produced.
• Honors the existing compression.* config block (threshold, protect_last_n, min_tail_user_messages, proactive_prune_tokens, proactive_prune_min_result_chars, proactive_prune_min_reclaim_tokens).
• Exposed in the desktop Settings → Context Engine dropdown; documented in the developer guide.

Opt-in, additive: the default compressor engine is untouched. Select via:

context:
  engine: "keep_history"

Tests

tests/agent/test_keep_history_engine.py — 8 tests covering discovery, never-triggering destructive compression, loop-prune no-op, request-only pruning (transcript untouched), small-request no-op, rearm gating, and compress() preserving every chat turn.

23 passed (engine tests + existing context-engine/prune suites)

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

feat(compression): add keep_history context engine — request-only compaction that preserves the visible chat history

1. Integration risk — no safety valve when budget_tokens is missing: select_context returns None whenever ctx <= 0 (i.e. context_length is unset and budget_tokens == 0). Combined with should_compress() == False and prune_tool_results_only being a no-op, a call site that fails to pass budget_tokens leaves the session with NOTHING firing — the request can grow past the window with no compaction at all. Verify every call site in run_agent.py/gateway always supplies budget_tokens, or make the engine fall back to context_length (or a hard default) when the budget is absent.

2. Doc/behavior mismatch: the module docstring and the website doc claim "only old tool-result bodies are shortened... never drops chat turns", but _trim_request_to_budget (the hard-ceiling fallback inside select_context) DOES drop whole messages from the REQUEST, including middle user/assistant turns. The claim holds only for compress(). The wording should distinguish request-drop from persisted-drop so users understand mid-chat turns can vanish from what the model sees in pathological sessions (even though the stored transcript stays intact).

3. Config resolution: _load_compression_config re-implements config loading with a direct YAML read and a broad except Exception: pass, bypassing the standard loaders (load_config / load_cli_config). It will silently swallow config errors and can drift from loader semantics (env overrides, merging). Also, the docs claim threshold feeds the request-prune trigger calculations, but the code uses proactive_prune_tokens else ctx * 0.55 — threshold_percent is parsed but never consulted. Align the docs with the code (or actually use threshold).

4. The rearm watermark and the request-only contract are well-designed and the tests (no in-place mutation, episodic rearm, all chat turns surviving compress()) cover the key invariants well.

[skyzea1]

Thanks for the thorough review!

1. Good catch — will add a safety valve so the engine falls back to context_length (or a hard default) when budget_tokens is absent, instead of silently leaving the session with no compaction at all.
2. Will clarify the docs: compress() never drops chat turns, but the hard-ceiling fallback (_trim_request_to_budget) can drop mid-chat messages from the request only — the persisted transcript stays intact. The wording will distinguish request-level drops from persisted drops.
3. Will switch _load_compression_config to the standard loaders (load_config / load_cli_config) and reconcile threshold_percent — either actually consult it or remove it from the docs.
4. Glad the rearm watermark and the invariants landed well — thanks!

[skyzea1]

Follow-up pushed (47c32b8) addressing points 1–3:

1. Safety valve: select_context no longer returns None when the context window is unknown — it falls back to a conservative 128k default, so a call site that skips budget_tokens can no longer leave the session with no compaction at all. Covered by a new test (test_select_context_falls_back_when_budget_missing).
2. Docs now distinguish request-level drops from persisted drops: the hard-ceiling trim inside select_context may drop mid-chat messages from the request the model sees, while compress() never drops turns and the stored transcript always keeps everything.
3. _load_compression_config goes through the standard hermes_cli.config.load_config (env overrides, managed scope, profile switches, mtime cache) with debug logging on failure, matching the other bundled plugins. The threshold claim is fixed too: the request-prune trigger is proactive_prune_tokens else a named 0.55 window ratio; threshold is documented as accepted-but-not-consulted.

Full engine suite: 8 passed. ruff check/format clean.