fix(compression): separate provider-exact vs projected token state

[heathley]

Summary

Fixes premature context compression caused by conflating two different token states in last_prompt_tokens.

Today Hermes writes two semantically different values into the same field:

• provider-exact prompt_tokens after update_from_response()
• projected local request size after _compress_context()

The compression decision later reads last_prompt_tokens as if it were provider-exact in both cases, which can trigger compression early when the post-compression estimate is higher than the last API-reported value.

Resolves #23902.

Root cause

last_prompt_tokens was carrying both:

1. exact provider-reported prompt usage
2. estimated current request pressure

That worked poorly after compression, because _compress_context() overwrote the exact value with an estimate, and the next compression check treated that estimate as authoritative.

What changed

ContextCompressor state split

Separated token tracking into:

• last_provider_prompt_tokens
• projected_prompt_tokens
• projected_prompt_tokens_source
• _transcript_mutated_since_api

For backward compatibility, last_prompt_tokens now mirrors projected_prompt_tokens, so existing gateway/CLI consumers still read a current projected context value.

Exact vs projected request pressure

Added get_current_request_pressure(messages, system_prompt, tools).

Behavior:

• if the transcript has not changed since the last API response, use provider-exact prompt tokens
• otherwise, recompute a fresh full request estimate using:
  • messages
  • system prompt
  • tool schemas

This preserves the post-#14695 requirement that tool schema tokens remain part of request pressure estimation.

Dirty-state invalidation

Added coarse invalidation when local transcript state changes, including:

• user message appended
• assistant turn committed
• tool execution batch completed
• active system prompt swapped
• post-compression state

This avoids per-message delta accounting while still preventing stale provider-exact values from being reused after local mutations.

Post-compression behavior

_compress_context() no longer overwrites provider-exact token state.

Instead it updates projected pressure as an estimate with source estimated_post_compression, leaving the last provider-exact value intact for diagnostics while ensuring the next compression check does
not mistake stale exact usage for current request pressure.

Why this shape

This intentionally does not revert #14695 / #18265 behavior.

Tool schemas are still included in estimates, and the fix avoids reintroducing the “compression triggers too late because tools were excluded” bug.

It also does not add per-message delta accounting or low-level append hooks. The implementation uses coarse invalidation plus fresh full-request estimation when the transcript is dirty.

Tests

Added/updated regression coverage for:

• update_from_response() setting provider-exact + projected state consistently
• projected state aliasing through last_prompt_tokens
• using provider-exact tokens only when the transcript is still clean
• switching to fresh full estimation when the transcript is dirty
• keeping tools schema in estimates (BUG: Post-compression token estimate excludes tools schema, delaying next compression cycle #14695 guard)
• preserving provider-exact state across _compress_context()
• avoiding reuse of stale provider-exact state after compression
• resetting the new token-state fields on session reset

Validation

Passed:

• pytest tests/agent/test_context_compressor.py tests/agent/test_context_engine.py tests/run_agent/test_projected_token_state.py -q
• python3 -m py_compile agent/context_compressor.py run_agent.py tests/run_agent/test_projected_token_state.py

Related

• Fixes Context compression fires prematurely — estimated token count overwrites precise API value #23902
• Related to BUG: Post-compression token estimate excludes tools schema, delaying next compression cycle #14695
• Related to fix(compression): include system prompt + tool schemas in token estimates #18265

[teknium1]

Automated hermes-sweeper review: this compression-token-state bug is already fixed on current main.

• agent/conversation_compression.py:960-963 retains the post-compression rough estimate separately, sets last_prompt_tokens = -1, and waits for real provider usage rather than treating the rough value as provider usage.
• agent/context_compressor.py:1170-1224 records provider-reported prompt usage separately and defers known-noisy rough preflight estimates after compression.
• agent/turn_context.py:387-405 preserves the sentinel and skips a repeat preflight compaction when the provider-reported request fit; tests/run_agent/test_413_compression.py:644-683 covers that behavior.
• This shipped in v2026.6.5 via e38b0b55d12cfa39a6ac71d553d224c0711856f2, with additional sentinel guarding in b2c84a16267245dfb34b2c497113b425542ef446.

The cross-referenced competing fix discussion on #28057 identified the same post-compression rough-estimate failure mode; current main now addresses it without reusing the rough estimate as provider usage.