Skip to content

fix(agent): align the output-fit usable floor with the continuation guard - #308

Merged
OmarB97 merged 1 commit into
mainfrom
fix/output-fit-graceful-degradation-fork-20260801
Aug 2, 2026
Merged

fix(agent): align the output-fit usable floor with the continuation guard#308
OmarB97 merged 1 commit into
mainfrom
fix/output-fit-graceful-degradation-fork-20260801

Conversation

@OmarB97

@OmarB97 OmarB97 commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Follow-up to #302 (merged), which reworked output_tokens_that_fit() to degrade the input reservation at high fill instead of falling off a cliff to None. This PR fixes one number in it.

The problem

#302 introduced _OUTPUT_FIT_MIN_USABLE = 512 — the smallest output cap worth reporting, below which None ("no usable cap, leave max_tokens to your own budget logic") is the honest answer. 512 was picked on the reasoning that a short answer or a tool call is still usable.

conversation_loop already answers the same question with a different number. _LENGTH_CONTINUE_MIN_HEADROOM_TOKENS = 2048 is the point below which the length-continuation path refuses to scaffold another round, on the grounds that there is no room left for a useful completion.

Two guards, one question, two answers — and below 2048 they actively fight each other:

  1. the pre-flight clamp sizes a request to, say, 527 output tokens;
  2. the response truncates, finish_reason="length";
  3. the continuation path checks context head-room (~26,000 tokens at 86% fill, since the provider's true prompt count is well under our 1.15x reservation), decides there is plenty of room, and boosts the next cap accordingly;
  4. _preflight_clamp_output_tokens shrinks that boost straight back to ~527.

Every round burns two more messages in an already-full window to produce output the continuation guard itself considers too small to write into.

The fix

Raise _OUTPUT_FIT_MIN_USABLE to 2048 and document the coupling at both ends. No import — conversation_loop imports model_metadata, not the other way round — so a test pins the two constants together instead.

Cost, measured

Over a 200,000-token window, the covered band shrinks at the top. None now starts at 85.8% fill rather than 86.6%, so that slice goes back to no clamp and the extra round-trip:

fill cap @512 cap @2048
81.8% 2,135 2,135
82.0% 1,655 10,877
85.0% 3,977 3,977
85.8% 2,137 2,137
86.0% 1,677 None
86.5% 527 None

That slice is exactly where the old floor was producing caps of 512–2047 — precisely the ones that would have fought the continuation guard — so the trade drops the cases that misbehave rather than trading good behaviour for good behaviour. Below 81.8% fill nothing changes at all; the tier switch moves from 82.5% to 81.8%, and the step size at the switch is unchanged (it is 0.05 * est + 1023 either way).

Verification

Brute-forced over every estimate in a 200,000-token window: zero safety-contract violations, zero sub-floor caps, and the largest contract head-room ever declined is 2,559 tokens — under the usable floor plus the window margin deliberately held back, so #302's no-cliff property still holds.

Tests gain a case pinning the invariant directly (no reported fit may fall below the continuation guard's minimum, swept in 500-token steps) and asserting the two constants agree. test_band_is_covered_without_a_hole now sweeps 83%–85%, the band that still clears the raised floor — it caught this change by failing at 86% fill, which is precisely the ~0.8% of coverage the higher floor gives up.

tests/test_output_fit_preflight_clamp.py    22 passed
tests/run_agent/test_run_agent.py          436 passed
tests/test_output_cap_parsing.py            20 passed
tests/test_ctx_halving_fix.py               27 passed

ruff check clean. Rebased onto current main (post-#302), so this PR is the single follow-up commit.

🤖 Generated with Claude Code

…uard

_OUTPUT_FIT_MIN_USABLE was 512, picked as "a short answer or a tool call is
still usable". conversation_loop already answers the same question with a
different number: _LENGTH_CONTINUE_MIN_HEADROOM_TOKENS = 2048, the point below
which the length-continuation path refuses to scaffold another round because
there is no room left for a useful completion.

Two guards, one question, two answers — and below 2048 they actively fight.
A clamped-to-527 request truncates, finish_reason="length" starts a
continuation round, the boost there sizes the next cap to the real context
head-room (~26,000 tokens at 86% fill, since the provider's true prompt count
is well under our 1.15x reservation), and _preflight_clamp_output_tokens
shrinks it straight back to ~527. Every round burns two more messages in an
already-full window for output the continuation guard itself considers too
small to write into.

Raise the floor to 2048 and document the coupling at both ends. No import —
conversation_loop imports model_metadata, not the other way round.

Cost, measured over a 200,000-token window: the covered band shrinks at the
top. None now starts at 85.8% fill rather than 86.6%, so the 85.8%-86.6% slice
goes back to no clamp and the extra round-trip. That slice is exactly where the
old floor was producing caps of 512-2047 — precisely the ones that would have
fought the continuation guard — so the trade is a strict improvement in kind,
not a compromise. Below 81.8% fill nothing changes; the tier switch moves from
82.5% to 81.8% and the step size at the switch is unchanged (it is
0.05 * est + 1023 either way).

Re-verified by brute force over every estimate in a 200,000-token window: zero
contract violations, zero sub-floor caps, and the largest contract head-room
ever declined is 2,559 tokens — under the usable floor plus the window margin,
so the no-cliff property still holds.

Tests: tests/test_output_fit_preflight_clamp.py gains a case pinning the
invariant directly (no reported fit may fall below the continuation guard's
minimum, swept in 500-token steps) and asserting the two constants agree.
test_band_is_covered_without_a_hole now sweeps 83%-85%, the band that still
clears the raised floor; it caught this change by failing at 86% fill, which is
the ~0.8% of coverage the higher floor gives up.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@OmarB97
OmarB97 force-pushed the fix/output-fit-graceful-degradation-fork-20260801 branch from 4d07514 to 4ce2148 Compare August 2, 2026 13:31
@OmarB97 OmarB97 changed the title fix(agent): degrade the output-fit reservation at high fill instead of giving up fix(agent): align the output-fit usable floor with the continuation guard Aug 2, 2026
@OmarB97
OmarB97 merged commit c23b899 into main Aug 2, 2026
35 checks passed
@OmarB97
OmarB97 deleted the fix/output-fit-graceful-degradation-fork-20260801 branch August 2, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant