feat(compression): proactive tool-result pruning for large-window models

[Kolektori]

Summary

An opt-in pass that trims old tool-result payloads out of the re-sent history on a low token trigger, separate from the full-compression trigger. It makes no LLM call and is off by default, so nothing changes unless you turn it on.

Problem

Every turn re-sends the whole message list, so a single large tool output (a terminal dump, a read_file, a web_extract) keeps getting re-billed on every turn after it lands. Profiling real sessions, re-sent history was over 70% of input tokens, almost all of it old tool output that never gets trimmed.

The compressor already has a cheap, deterministic pass for exactly this: _prune_old_tool_results dedups identical results, summarizes old ones, and truncates oversized tool-call args. The catch is that it only runs as phase 1 of compress(), which fires at roughly 50% of the context window. On a 1M-token window that's ~500K, and real sessions rarely come close, so the prune sits there and never runs.

Change

Run that same phase-1 prune on its own low trigger, proactive_prune_tokens, decoupled from compression:

• ContextCompressor.prune_tool_results_only() runs the prune without the LLM summary phase.
• It's wired into the loop as an elif on the compression branch, so the two never both fire in a turn. When should_compress() is False (the usual case on a large window), the cheap prune gets its shot.
• The recent tail is protected by message count (protect_last_n), not by a token budget. A token-budget tail on a 1M window would cover the whole session and prune nothing.
• The method is declared on the ContextEngine base class as a no-op default, so a pluggable context engine that doesn't implement it inherits the no-op instead of raising AttributeError on the post-tool-call path. The built-in compressor supplies the real implementation.

Configuration

Opt-in, under the top-level compression: block (same place as threshold, protect_last_n, etc.):

compression:
  proactive_prune_tokens: 48000           # run the tool-result prune once re-sent history
                                          # reaches this many tokens. 0 (default) = off.
  proactive_prune_min_result_chars: 8000  # only summarize tool results larger than this. Clamped to >= 200.

Both keys are registered in the config defaults and documented in the configuration guide. compression.* keys hot-reload on a running gateway, so tuning the trigger takes effect on the next message.

Safety and behavior

• It never calls the model, so there's no summary-quality risk; it only dedups, writes one-line summaries, and truncates big tool-call args.
• The last protect_last_n messages are never touched by the summarize or truncate passes. Dedup keeps the newest full copy and only back-references byte-identical older ones, so nothing unique is lost.
• Persistence is untouched. Pruned messages are copies that keep the internal persisted marker, so the session DB isn't double-written or stripped, and the full output is still on disk for reload.
• It's idempotent. The proactive_prune_min_result_chars floor is clamped to 200 so a generated summary can't itself be re-summarized into garbage.
• Pluggable context engines are unaffected: they inherit the base no-op and never see this behavior unless they implement it.
• If compression backs off (its anti-thrash guard), the prune still reclaims tokens.

Performance

On real large sessions this reclaimed 18–30% of snapshot input tokens with no quality change I could see, and should_compress() stayed False throughout, so the prune did the work rather than compaction.

Testing

New tests/agent/test_proactive_tool_result_pruning.py (9 tests) plus a regression test in tests/agent/test_context_engine.py that a minimal engine implementing only the required interface inherits the base no-op without raising.

pytest tests/agent/test_proactive_tool_result_pruning.py tests/agent/test_context_compressor.py tests/agent/test_context_engine.py -q
# 188 passed

Backward compatibility

Off by default (proactive_prune_tokens: 0). _prune_old_tool_results gets a new min_prune_chars argument defaulting to the old hard-coded 200, so the existing compression path is byte-for-byte identical. The new ContextEngine.prune_tool_results_only is a no-op default, so existing engines keep their current behavior.

[alt-glitch]

This was generated by AI during triage.

Competing implementation with #62389 for feature #513 (two-phase context management). Both add an opt-in, default-off proactive tool-result prune decoupled from the summarization trigger: this one triggers on proactive_prune_tokens, #62389 on an absolute prune_protect_tokens budget. Cross-linking so a maintainer can pick the canonical approach; also relates to #20717.

[teknium1]

Thanks for isolating the no-LLM prune path; the current main premise is valid: agent/context_compressor.py:2853-2856 only prunes inside full compress(), while agent/conversation_loop.py:4769-4778 enters that path only after should_compress().

Problems

• agent/conversation_loop.py:4802 unconditionally calls prune_tool_results_only() whenever normal compression does not fire. The pluggable ContextEngine contract only requires update_from_response, should_compress, and compress (agent/context_engine.py:70-106); the repository's conforming StubEngine lacks this method (tests/agent/test_context_engine.py:15-48). An active external engine will therefore raise AttributeError after a tool call.
• The public settings need the established top-level compression: configuration surface. Defaults/docs currently enumerate that block in hermes_cli/config.py:1419-1432 and website/docs/user-guide/configuration.md:738-745, but this PR changes neither.

Suggested changes

• Gate this behavior to the built-in compressor, or add a backwards-compatible optional context-engine hook plus an external-engine regression test.
• Document and default the two settings under compression:; correct the PR example from agent.compression.

Automated hermes-sweeper review.

[teknium1]

agent.context_compressor may be a configured external ContextEngine, whose required interface does not define this method (agent/context_engine.py:70-106). Gate this to the built-in compressor or add a safe optional interface/no-op, otherwise any external engine reaches an AttributeError on this post-tool path.

[Kolektori]

Thanks for the review. Both are fixed in 037ca72.

For the external-engine case: prune_tool_results_only is now a no-op on the ContextEngine base (returns (messages, 0)), so an engine that doesn't implement it inherits that and won't raise on the post-tool-call path. Only the built-in ContextCompressor carries the real prune. I added a regression test in tests/agent/test_context_engine.py using a minimal engine that implements just the required interface.

On the config surface: both keys now live in the top-level compression: defaults in hermes_cli/config.py, with docs in configuration.md. I also corrected the agent.compression example in the description; the real key path is top-level compression:, which is where the code actually reads them.

Tests: pytest tests/agent/test_proactive_tool_result_pruning.py tests/agent/test_context_compressor.py tests/agent/test_context_engine.py -q → 188 passed.

[teknium1]

Merged via salvage PR #70254 with your commit cherry-picked and authorship preserved — thanks @Kolektori, including your 037ca72 fixes for both review blockers! Your prune_tool_results_only + opt-in low-trigger design is on main, extended with a measured-savings commit gate (adopted from #62389's review, credited) so prompt-cache breaks stay episodic. The bake-off rationale vs #62389 is in the salvage body.