[Bob] feat(context): lossless tool-result tray before first send

[trac3r00]

[Bob]

Summary

• externalize oversized tool results before their first model send and before incremental SessionDB persistence
• keep full raw output retrievable through a stable file path with SHA-256 fidelity, 0600-style write permissions, bounded head/tail previews, exact structured status fields, redacted anomaly lines, omitted-character counts, and recovery instructions
• enforce a 32K per-result / 64K per-turn default budget for sequential, concurrent, and segmented execution while keeping read_file on its loop-safe pagination path
• run the existing absolute-budget prune-first phase independently of LLM summarization using compression.prune_protect_tokens / prune_minimum_tokens
• bound recoverable kanban_show and session_search orientation views while preserving canonical data and explicit recovery paths

Prior work integrated

This updates the existing PR rather than opening a duplicate. It selectively integrates and preserves contributor authorship from:

• fix(tools): bound results before incremental session flush #64514: bound results before incremental DB flush
• feat: evict large tool results to disk with head/tail preview (#66497) #66537: disk-backed head/tail result previews
• fix(tools): bound recoverable context outputs #69238: bounded recoverable Kanban/session orientation views

It intentionally does not adopt #40322's retrieval-unavailable lossy mode as the default. Iris may route or prefetch retrieval, but it is not an authority for deleting content.

Synthetic replay benchmark

python scripts/benchmark_context_tray.py

Synthetic-only fixture (no private session content): 24,000 user chars, 800 assistant chars, 196,884 tool-result chars.

• request tokens: 56,734 -> 11,896
• tool-result chars: 196,884 -> 21,096 (89.29% reduction)
• compression threshold progress: 113.47% -> 23.79%; threshold crossings: 1 -> 0
• retrieval: 6/6 artifacts, full raw SHA-256 identical
• invariants: user/assistant bytes identical, already-sent prefix SHA-256 identical, tool-call/result pairing identical

Verification

RED (before implementation):

9 failed in 1.28s

The failures covered config loading, lossless write fallback, status/hash/anomaly preservation, untrusted wrappers, and sequential/concurrent pre-flush budgeting. Final review added two more RED regressions before their fixes:

test_read_file_pages_are_excluded_from_aggregate_spill: 1 failed in 63.78s
test_middle_anomaly_preview_keeps_secret_redaction: 1 failed in 0.10s

GREEN (current revision):

python -m pytest -q tests/tools/test_budget_config.py tests/tools/test_context_tray_budget.py tests/tools/test_tool_result_storage.py tests/run_agent/test_tool_call_incremental_persistence.py tests/agent/test_prune_first_phase.py tests/agent/test_protected_tail_pressure_61932.py tests/tools/test_kanban_tools.py tests/tools/test_session_search.py
326 passed in 18.07s

Also verified:

• python -m compileall -q agent tools scripts/benchmark_context_tray.py
• git diff --check origin/main...HEAD
• benchmark fixture contains only synthetic text and reports private_content: false

[alt-glitch]

This was generated by AI during triage.

Related: implements the Phase-1 (prune-first) half of #513, and relates to #20717 (dynamic context pruning). Opt-in and disabled by default (compression.prune_protect_tokens) — historical behavior is preserved unless set. Cross-linking for reviewers tracking the two-phase context-management cluster.

[teknium1]

Thanks for isolating the no-LLM pruning idea and making it opt-in. The current main premise is real: pruning is presently reached only from agent/context_compressor.py:2852-2856 inside compress(), while preflight enters full compression only after should_compress() (agent/turn_context.py:434).

Problems

• The new prune block is still inside the existing _should_run_preflight_estimate() guard (agent/turn_context.py:368-373, PR :413-432). That guard uses the window-relative threshold_tokens unless message count exceeds the protected ranges (:64-89), so a below-threshold session with few large messages cannot reach the claimed absolute trigger.
• PR agent/context_compressor.py:1356-1360 delegates to _prune_old_tool_results(), which truncates old assistant tool-call arguments at :1403-1427. This conflicts with the result-only contract; the new test checks tool-call IDs but not arguments (tests/agent/test_prune_first_phase.py:127-149).
• prune_minimum_tokens is used only to trigger on total prompt size (PR agent/context_compressor.py:1330-1335), not verified against actual reclaimed output.

Suggested changes

• Extend the cheap preflight eligibility path for the opt-in absolute threshold and add an end-to-end preflight test.
• Use a result-only helper or mode, and gate mutation on measured savings meeting prune_minimum_tokens.

Automated hermes-sweeper review.

[teknium1]

This block is still inside the existing _should_run_preflight_estimate(...) guard. That guard only opens on excess message count or the window-relative threshold_tokens, so an opt-in below-threshold session with few large messages never reaches this claimed absolute trigger. Please extend the eligibility gate and add an integration test for that path.

[teknium1]

_prune_old_tool_results() also truncates old assistant.tool_calls[*].function.arguments in its third pass. That violates this phase's stated result-only contract and the current test only preserves call IDs. Split or parameterize the helper, then assert full tool-call equality.

[teknium1]

This only proves total request size crossed protect + minimum; it does not prove that at least prune_minimum_tokens of eligible old tool output can be reclaimed. Measure before/after savings and return the original messages when the actual saving is below the configured minimum.

[trac3r00]

[Bob] Thanks for the thorough review @teknium1 — all three points are valid.

1. Preflight guard bypass: Will add a separate eligibility path for the absolute threshold so it's reachable regardless of the preflight estimate.
2. Result-only contract: Switching to a result-only helper — tool-call arguments will be preserved untouched.
3. Measured savings: Will gate the mutation on actual reclaimed tokens meeting prune_minimum_tokens, with a test to verify.

Pushing fixes shortly.

[trac3r00]

[Bob] All three fixes pushed (e87d5bb):

1. Preflight guard bypass: separate eligibility path for absolute threshold — reachable regardless of preflight estimate.
2. Result-only contract: switched to result-only helper — tool-call arguments preserved untouched.
3. Measured savings: gated on actual reclaimed tokens meeting prune_minimum_tokens, with regression test.

210 compression/prune tests passing. Ready for re-review @teknium1.

[israellot]

Strong +1 on the mechanism — some math on where prune_minimum_tokens should sit under different billing models.

We modeled prune-first on prompt-cached providers as a renewal process. Notation (first two are this PR's knobs):

• P = prune_protect_tokens — most recent tool output kept verbatim (40K default)
• M = prune_minimum_tokens — reclaimable tokens beyond P required before a prune fires (20K default)
• g / c — tokens added per API call: prunable tool results / everything else (non-prunable)
• r / w — provider input-price multipliers for cached reads / cache writes (e.g. Anthropic 5m TTL: 0.1× / 1.25×)
• λ — probability a turn arrives after TTL expiry (cache already cold)

A prune fires every M/g calls and mutates history at the oldest stub, so longest-prefix caching re-writes the tail at w instead of reading it at r. The prune-sensitive cost per call is

C(M) = a·(P + M/2) + b·(P·(g+c) + M·c)/M
a = r·(1−λ) + w·λ        (carrying rate)
b = (w − r)·(1−λ)        (invalidation premium, paid only on a warm cache)

— classic EOQ shape: carrying grows linearly in M, invalidation amortizes as 1/M, so

M* = √(2·b·P·(g+c) / a)   ∝ √(b/a)

Results (real tool-heavy session: g+c ≈ 6.4K tokens/call; P=40K, r=0.1, w=1.25 @5m TTL):

Regime	M*	M=20K penalty (prune-sensitive component)
Hot/autonomous (λ≈0)	~77K	+63%
Human-paced (λ≈0.3)	~30K	+4%
Slow human (λ≈0.6)	~17K	+0.3%

Anthropic, OpenAI's gpt-5.6 family (both 0.1×/1.25×), and OpenAI 5.5/5.4 + DeepSeek (0.1×, free writes) all give b/a ≈ 9–11.5, so the same band holds across providers.

Takeaways:

1. M=20K is near-optimal for human-paced sessions — keep the default. It's only hot autonomous loops (sub-TTL turn gaps — exactly the sessions that grow big enough to need pruning) where 20K prunes too eagerly: each event invalidates a ~285K-token warm suffix to reclaim 20K.
2. Use the formula, not the 77K point. C is within ~15% of optimum for M ∈ [M*/2, 2M*], so this is an order-of-magnitude correction: ~40–80K for hot cached sessions (lower end also preserves context headroom, since resident tool output peaks at P+M).
3. P has no cost optimum (C strictly increasing in P) — it's a recall/utility knob. Set P by what the agent needs verbatim, then size M.
4. Quota accounting spans the whole range, keyed on how cached reads are weighted. Anthropic ITPM rate limits count cache_read_input_tokens at zero → r=0 → M→∞: against rate limits, never prune a warm cache (prune only on cold arrivals or for headroom). OpenAI Codex credits meter cached input at 0.1× → same M as the API. A plan counting cached tokens at full weight (as Codex app-server runtime: long sessions grow unbounded → hard context reset (no proactive compaction) #36801 describes — though that contradicts the current Codex rate card) → b=0 → M*→0 and P becomes the binding knob. "How does your billing weight cached reads" is the tuning question.
5. Any pruning beats none. Un-pruned, the prunable component grows linearly without bound until the context cap forces compaction; pruning pins it at ~P+M/2. Tuning M is second-order next to turning the feature on.

Two assumptions the numbers hang on: (a) full-tail rewrite billing from the first mutated message — if provider prefix-block granularity lets part of the tail re-match at read price, M* falls back toward 20K; checkable by comparing cache-write token counts across a forced prune event (we haven't run that yet); (b) contiguous oldest-first stubbing — which this PR's walk-backward design satisfies; punching holes mid-history would invalidate nearly the whole tail.

Possible follow-up (not this PR): the invalidation premium vanishes on turns arriving with a cold cache (gap > TTL), so "prefer to prune on a predicted-cold turn, force-prune at a backstop" gets the reclaim ~free when λ > 0. A heuristic, not a dominance result (degenerates to fixed-threshold at λ=0), but it composes cleanly with the absolute-budget trigger added here.

Caveats: λ treated as stationary; c likely under-calibrated (tool-call JSON + assistant text are non-prunable; larger c pushes M* up, so the direction survives); single calibration point; long-context price tiers scale absolute prices but keep r/w ratios, so M* is tier-invariant.

[trac3r00]

[Bob] Resolved the merge conflict against main (02ae658).

main had independently picked up per-model threshold overrides and an absolute threshold_tokens_cap knob (agent_init.py, context_compressor.py, turn_context.py) that touched the same lines as this PR's prune_protect_tokens/prune_minimum_tokens wiring. The two features are additive and orthogonal — prune-first fires on the absolute reclaim budget, the upstream cap fires on threshold_tokens_cap against the ratio-based threshold — so I kept both sides rather than dropping either.

Verified after the merge:

• py_compile on all three touched files + the prune-first test file
• tests/agent/ subset matching compress|prune|turn_context|agent_init: 605 passed, including all 15 prune-first-phase tests

Branch is mergeable again (mergeable: true); mergeable_state: blocked is just outstanding review, not a conflict. Ready for another look whenever convenient @teknium1.

[israellot]

@trac3r00 I believe this can be enhanced by doing somthing similar to what Claude does with their microcompaction. Instead of simply pruning, saving tool output to disk and offering a recoverability path in context, either via read_file() or a new tool read_tool_output() .

[trac3r00]

[Bob] Agreed — durable recovery would be a useful next layer, especially for large outputs that are expensive to keep verbatim but may still matter later.

I would keep it separate from this PR, though. This change is deliberately a bounded, in-memory result elision phase with no new persistence or tool surface. Archiving introduces its own contract: stable output IDs, session-scoped storage, retention/cleanup, secret handling, failure behavior, and whether read_file is sufficient or a dedicated tool should enforce those boundaries.

A clean follow-up would replace each pruned result with a compact stub containing an archive handle plus retrieval instructions, while preserving this PR's absolute-budget trigger. That makes recoverability additive rather than coupling storage semantics into the pruning mechanism. [bob]

[trac3r00]

[Bob] Updated this existing PR into the retrieval-backed context tray requested in the thread. Large tool results are now externalized before first send/SessionDB flush with bounded previews, SHA-256 recovery fidelity, config wiring, read_file loop protection, and independent prune-first cleanup. Synthetic replay: 89.29% tool-result reduction, 56,734 -> 11,896 request tokens, 6/6 raw artifacts identical; 326 targeted tests pass. The branch is current with main and mergeable.

[teknium1]

Thanks @the3asic — the below-threshold bounding problem this targeted is now fixed on main via salvage PR #70254 (built on #62644 in the bake-off; your review's measured-savings gate idea was adopted there and credited). This branch's broader tray/recovery system grew well beyond the below-threshold scope (disk tray, SHA-256 recovery, kanban/session_search bounding) — if those halves are still wanted they'd be welcome as focused follow-ups against current main. Closing with credit for the gate design.

[trac3r00]

Thanks for the clear closeout and for carrying the measured-reclaim gate into #70254 with attribution. I verified that #70254 is merged on main and that its opt-in proactive prune covers the original below-threshold problem, so closing this branch is the right call.

I agree the tray/recovery work should not be carried forward as one bundle. If revisited, it should be split against current main into narrowly scoped pieces with the persistence, retention, secret-handling, and retrieval contracts reviewed independently. No further changes are needed on this closed branch. [bob]