fix(agent): align the output-fit usable floor with the continuation guard - #308
Merged
Merged
Conversation
…uard _OUTPUT_FIT_MIN_USABLE was 512, picked as "a short answer or a tool call is still usable". conversation_loop already answers the same question with a different number: _LENGTH_CONTINUE_MIN_HEADROOM_TOKENS = 2048, the point below which the length-continuation path refuses to scaffold another round because there is no room left for a useful completion. Two guards, one question, two answers — and below 2048 they actively fight. A clamped-to-527 request truncates, finish_reason="length" starts a continuation round, the boost there sizes the next cap to the real context head-room (~26,000 tokens at 86% fill, since the provider's true prompt count is well under our 1.15x reservation), and _preflight_clamp_output_tokens shrinks it straight back to ~527. Every round burns two more messages in an already-full window for output the continuation guard itself considers too small to write into. Raise the floor to 2048 and document the coupling at both ends. No import — conversation_loop imports model_metadata, not the other way round. Cost, measured over a 200,000-token window: the covered band shrinks at the top. None now starts at 85.8% fill rather than 86.6%, so the 85.8%-86.6% slice goes back to no clamp and the extra round-trip. That slice is exactly where the old floor was producing caps of 512-2047 — precisely the ones that would have fought the continuation guard — so the trade is a strict improvement in kind, not a compromise. Below 81.8% fill nothing changes; the tier switch moves from 82.5% to 81.8% and the step size at the switch is unchanged (it is 0.05 * est + 1023 either way). Re-verified by brute force over every estimate in a 200,000-token window: zero contract violations, zero sub-floor caps, and the largest contract head-room ever declined is 2,559 tokens — under the usable floor plus the window margin, so the no-cliff property still holds. Tests: tests/test_output_fit_preflight_clamp.py gains a case pinning the invariant directly (no reported fit may fall below the continuation guard's minimum, swept in 500-token steps) and asserting the two constants agree. test_band_is_covered_without_a_hole now sweeps 83%-85%, the band that still clears the raised floor; it caught this change by failing at 86% fill, which is the ~0.8% of coverage the higher floor gives up. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
OmarB97
force-pushed
the
fix/output-fit-graceful-degradation-fork-20260801
branch
from
August 2, 2026 13:31
4d07514 to
4ce2148
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #302 (merged), which reworked
output_tokens_that_fit()to degrade the input reservation at high fill instead of falling off a cliff toNone. This PR fixes one number in it.The problem
#302 introduced
_OUTPUT_FIT_MIN_USABLE = 512— the smallest output cap worth reporting, below whichNone("no usable cap, leavemax_tokensto your own budget logic") is the honest answer. 512 was picked on the reasoning that a short answer or a tool call is still usable.conversation_loopalready answers the same question with a different number._LENGTH_CONTINUE_MIN_HEADROOM_TOKENS = 2048is the point below which the length-continuation path refuses to scaffold another round, on the grounds that there is no room left for a useful completion.Two guards, one question, two answers — and below 2048 they actively fight each other:
finish_reason="length";_preflight_clamp_output_tokensshrinks that boost straight back to ~527.Every round burns two more messages in an already-full window to produce output the continuation guard itself considers too small to write into.
The fix
Raise
_OUTPUT_FIT_MIN_USABLEto 2048 and document the coupling at both ends. No import —conversation_loopimportsmodel_metadata, not the other way round — so a test pins the two constants together instead.Cost, measured
Over a 200,000-token window, the covered band shrinks at the top.
Nonenow starts at 85.8% fill rather than 86.6%, so that slice goes back to no clamp and the extra round-trip:That slice is exactly where the old floor was producing caps of 512–2047 — precisely the ones that would have fought the continuation guard — so the trade drops the cases that misbehave rather than trading good behaviour for good behaviour. Below 81.8% fill nothing changes at all; the tier switch moves from 82.5% to 81.8%, and the step size at the switch is unchanged (it is
0.05 * est + 1023either way).Verification
Brute-forced over every estimate in a 200,000-token window: zero safety-contract violations, zero sub-floor caps, and the largest contract head-room ever declined is 2,559 tokens — under the usable floor plus the window margin deliberately held back, so #302's no-cliff property still holds.
Tests gain a case pinning the invariant directly (no reported fit may fall below the continuation guard's minimum, swept in 500-token steps) and asserting the two constants agree.
test_band_is_covered_without_a_holenow sweeps 83%–85%, the band that still clears the raised floor — it caught this change by failing at 86% fill, which is precisely the ~0.8% of coverage the higher floor gives up.ruff checkclean. Rebased onto currentmain(post-#302), so this PR is the single follow-up commit.🤖 Generated with Claude Code