fix(agent): defer preflight compression by projecting real usage (salvage #80997) - #81069
Merged
Merged
Conversation
…owth tolerance The rough preflight estimate intentionally overestimates, but not by a fixed margin: CJK text is counted at ~1.7x its o200k cost and Responses-mode reasoning replay blobs at several times their billed cost. Heavy sessions show rough estimates 2-3x real usage and compact at 35-55% of the real window, stalling turns for minutes and discarding detail (churn), because the defer guard only tolerated 5% rough growth and sessions that never compressed had no baseline at all. Pair every request's rough estimate (note_request_rough_estimate, recorded in the conversation loop right after the pressure estimate) with the provider's real prompt_tokens in update_from_response(), then defer preflight while projected real usage — last real + rough growth since that reading — stays under the threshold. Rough growth is itself an overestimate of real growth, so the projection is an upper bound and deferring below the threshold is safe; the provider's context-overflow handler remains the backstop. The baseline no longer ratchets on defer: it is refreshed by the response pairing, and advancing it without a matching real reading would shrink apparent growth and defer on stale data.
…dings) Two docstring corrections on top of the salvaged NousResearch#80997 fix — behavior unchanged, both verified against the code: - The 'rough growth over-counts every content class' claim is false for Cyrillic/Greek/Thai/Arabic (chars/4 vs ~2-3 chars/token on o200k): growth there can under-count up to ~2x (NousResearch#62605's direction). Document the real backstops instead: an at/over-threshold real reading clears the baseline (post-response gate fires on real usage within one call) and the provider overflow handler compacts reactively. - Document the two measurement bases (turn-prologue raw messages vs the loop's fully assembled request that seeds the baseline) and why the prologue's smaller basis can only OVER-defer — the loop's pre-API pressure check re-runs the projection with the aligned basis before every provider call, so a prologue over-defer never skips a needed compaction.
alt-glitch
added
type/bug
14 tasks
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Preflight compression no longer fires at 35–55% of the model's real context window on sessions where the rough token estimate runs far ahead of provider-reported usage (CJK text ~1.7x, Responses-mode reasoning replay blobs severalfold). The defer guard now pairs every request's rough estimate with the provider's real
prompt_tokensfor that same request and defers while the projection stays under the threshold:Salvage of #80997 by @chesterXalan — cherry-picked with authorship preserved, plus one docs follow-up from review.
Who hits this
Anyone running long CJK-heavy or reasoning-replay-heavy sessions: before, the rough estimate could reach ~413K while the provider had just reported 150K real prompt tokens, so the session compacted minutes-long summary passes at half the real window, discarded detail (220 → 71 messages), regrew, and re-fired within the hour. Fresh sessions had no defer baseline at all (it was only seeded post-compaction) and could be force-compacted 24 minutes in at ~35% real usage.
Changes
agent/context_compressor.py—note_request_rough_estimate()records the request-pressure estimate;update_from_response()pairs it with real usage on every fitting response (post-compaction pairing still wins, Bug: Context compression triggers repeatedly after fresh compress — last_prompt_tokens=-1 not updated until next API call #36718);should_defer_preflight_to_real_usage()projects real usage instead of tolerating fixed growth; pending state reset at all four reset sites.agent/conversation_loop.py— record the estimate right after it is computed.Follow-up commit (review findings, docs-only): the docstring's "rough growth over-counts every content class" claim was false for Cyrillic/Greek/Thai/Arabic (chars/4 vs ~2-3 chars/token on o200k — #62605's direction); it now documents the real backstops (at/over-threshold real reading clears the baseline; provider overflow handler compacts reactively) and the two measurement bases (turn-prologue raw messages vs the loop's fully-assembled request) and why the prologue's smaller basis can only over-defer, never skip a needed compaction.
Validation
agent/)last_real_prompt_tokensassigned in exactly one place, same call that consumes the noteRelated: #36718 / #50762 (defer mechanism this generalizes), #62605 (under-count direction, backstops documented), #58784, #14695.
Based on #80997 by @chesterXalan — commit cherry-picked to preserve authorship.