fix(agent): grow output-cap retry margin exponentially to compensate for per-retry input-token drift (#61761)

[webtecnica]

Summary

When a request fails because input_tokens + max_tokens > context_length, the recovery path shrinks max_tokens and retries. A fixed 64-token margin was erased by ~65-token per-retry input drift, causing the loop to never converge and burn through all compression attempts.

Trace

Attempt 1: input 134,465, requested max_tokens 65,536, sum 200,001 (over by 1)
Attempt 2: input 134,530, requested max_tokens 65,471, sum 200,001 (over by 1)
Attempt 3: input 134,595, requested max_tokens 65,406, sum 200,001 (over by 1)
Attempt 4: input 134,660, requested max_tokens 65,341, sum 200,001 (over by 1) -> fail

Change

Replaced the fixed 64-token margin with an exponentially growing margin: 256 * (2 ** compression_attempts), producing margins of 256, 512, 1024, ... that outpace the ~65-token drift per retry.

safety_margin = 256 * (2 ** compression_attempts)
safe_out = max(1, available_out - safety_margin)

Verification

25 context/compression/retry tests pass.

[teknium1]

Thanks for isolating the fixed-margin convergence failure. Current main still has the exact available_out - 64 calculation at agent/conversation_loop.py:3451, so the proposed reserve addresses a real current-HEAD defect.

Problems

• The PR adds no regression test. The existing tests at tests/test_ctx_halving_fix.py:315-327 and :340-350 manually repeat the old 64-token calculation; they do not execute the recovery branch or cover the reported +65-token drift.
• Output-cap recovery still consumes compression_attempts and eventually reports compression exhaustion (agent/conversation_loop.py:3458-3477). A drift larger than the bounded reserve still receives a misleading compression result. The cross-referenced #62197 is relevant because it addresses this coupling separately.

Suggested changes

• Add a loop-level regression that simulates parseable output-cap failures with +65 input tokens between attempts and verifies recovery without context compression.
• Decide whether to retain this as a narrow margin fix or fold it into an output-cap-specific retry budget/terminal error path.

Automated hermes-sweeper review.

[teknium1]

Please add a regression test that drives this recovery path through successive provider errors with a +65-token input increase. The current tests manually reimplement the retired 64-token calculation, so they would not fail if this branch still used a non-converging reserve.

[quadseven]

Independent confirmation of the bug this fixes, from a different model and context size — I hit it before finding your PR and opened a duplicate (#72826, which I'll close in favour of this one).

Hermes 0.18.2, custom:vllm, 262,144-token window:

attempt 1: input 196,674  max_tokens 65,471  total 262,145  (over by 1)
attempt 2: input 196,739  max_tokens 65,406  total 262,145  (over by 1)
attempt 3: input 196,804  max_tokens 65,341  total 262,145  (over by 1)

Same 65-token per-retry drift, same constant overshoot of exactly 1, on a 262k window rather than #61761's 200k. Your diagnosis holds across both.

Worth stating explicitly because it shows raising max_attempts can't help: the overshoot is independent of the input size —

next_total = (input_n + drift) + [(ctx - input_n) - margin] = ctx + (drift - margin)

Two things I ran into while implementing the same fix, offered as review rather than as a competing opinion:

1. This also removes the local_available_out bound. The estimate_request_tokens_rough branch (-20 lines here) exists to catch the case where the provider's available_tokens is more generous than the real request shape — its comment cites API-only content that isn't in the persisted messages. Removing it makes the provider number the sole source. That might be deliberate, but it's a second behaviour change travelling with the margin fix, and it'd be easy to keep both: apply the escalating margin to the existing min(available_out, local_available_out) and leave the estimate in place.

2. 256 * (2 ** compression_attempts) is uncapped. By attempt 10 that's 262,144 — the entire window on the model I hit this on, which drives safe_out to the max(1, ...) floor. Unreachable at today's default of 3, but compression.max_attempts is config-driven, so anyone raising it walks into it. A min() on the shift makes it safe regardless of config.

On the base value: I tested 64 / 256 / 1024 against the observed 65-token drift. All converge, but with max_attempts=3 the margin of the budget matters —

base	converges (drift=65)	drift=1000
64	attempt 3 — the last one	never
256	attempt 2	attempt 3
1024	attempt 2	attempt 2

256 is comfortably fine for the observed drift; 1024 just buys more headroom if a provider drifts more than measured. Your call — the escalation is the part that matters.

I have 9 tests that pin the escalation property rather than any specific constant, so they'd pass against this PR's 256 unchanged. Happy to open them as a follow-up once this merges if that's useful.

[kshitijk4poor]

Closing as superseded, with thanks for the analysis on #61761. Main took a two-part path instead of margin growth: #55546 made the output-cap retry compress alongside the clamp, and #90563 (auto-merge armed) removes the drift's root cause — vLLM's "at least N input tokens" figure is back-computed from the constraint (window + 1 − requested), so the parser now detects that degenerate shape and halves the cap instead of chasing it. With the drift gone at the source, exponential margin growth no longer has a failure mode to compensate for.