fix(agent): converge vLLM output-cap retries and recover relay 'maximum output tokens' rejections - #90563
Merged
kshitijk4poor merged 4 commits intoAug 20, 2026
Conversation
Fixes the retry loop that spins forever when a vLLM server rejects a
request for having a max_tokens too big for what is left of the context
window.
The catch is that vLLM does not tell you how big your prompt actually is
in that situation. It works the number backwards from the constraint it
just failed, so you get:
"requested 65536 output tokens and your prompt contains at least
36865 input tokens, for a total of at least 102401 tokens"
That 36865 is just window + 1 - requested, and the total is always
exactly window + 1. Subtracting it from the window hands back
requested - 1 every single time, whatever the real prompt size is.
parse_available_output_tokens_from_error believed it and returned
requested - 1. conversation_loop then takes off its 64 token safety
margin and retries, which walks the cap down 65 tokens at a time while
the reported input walks up by the same 65:
65536 -> 65471 -> 65406 -> 65341
Three attempts is the default budget, so the session gives up with
"Context length exceeded" having closed 195 tokens of a roughly 28000
token gap. Compression cannot save it either, because the input was
never the problem, which is why the compressor keeps refusing with
"summary would have GROWN".
This is also what is behind the unexplained "input-token drift" in
issue NousResearch#61761. The input is not drifting. It is a derived number, and it
moves because we moved max_tokens.
So when that shape shows up (the "at least" wording, plus a budget that
works out to exactly requested - 1), halve the requested cap instead. It
is still guaranteed to sit under whatever was just rejected, and it
converges on the first retry: 65536 -> 32768, which next to a real 36865
token prompt comes to 69633 against a 102400 window.
Nothing else moves. A measured input is still trusted, and a genuine
input overflow still returns None so the caller falls through to
compression the way it always did.
The existing test asserted the bogus 65535, so it is updated. Added
tests for the measured input path, and for the retry actually
converging.
…errors Recognizes the DeepSeek/OpenAI-compatible relay wording max_tokens (98304) exceeds model's maximum output tokens (65536) in both parse_available_output_tokens_from_error (returns the cap) and is_output_cap_error (keeps the 400 out of the compression death-loop). Salvaged from PR NousResearch#72283; the conversation_loop early-clamp block was dropped in favor of routing through the existing output-cap handler (follow-up commit).
…andler Salvage follow-up for NousResearch#72283: instead of a second pre-retry clamp block (which bypassed the NousResearch#55546 clamp+compress path and broke its three regression tests), parse the output cap ONCE at classification time and: - exempt parseable wrapped output-cap 429s from the eager rate-limit provider fallback (a deterministic request-shape failure that failover cannot fix but the clamp fixes in one retry), and - widen is_context_length_error so they reach the SAME NousResearch#55546 clamp+compress recovery as plain output-cap 400s. Adds both NousResearch#72283 regression scenarios plus an ordering guard proving a NON-EMPTY fallback chain does not consume the wrapped 429 (fallback slot unspent, model unchanged). 119 fallback/rate-limit tests green.
kshitijk4poor
enabled auto-merge (rebase)
August 20, 2026 05:58
This was referenced Aug 20, 2026
alt-glitch
added
type/bug
Contributor
|
👍 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Output-cap retries now converge on vLLM and recover DeepSeek/relay "exceeds model's maximum output tokens" rejections — including when a relay wraps them as HTTP 429 — instead of crawling 65 tokens per retry or burning generic/rate-limit retries on a deterministic request-shape failure.
Salvages two contributor PRs onto current main with authorship preserved:
fix(model-metadata): converge output-cap retry on vLLM #89923 (@Dhruv7201) — when max_tokens is the binding constraint, vLLM back-computes a fake "at least N input tokens" lower bound (N = window + 1 − requested), so window − N is always requested − 1 and each retry shrank the cap by exactly the safety margin while the reported input walked up by the same amount. Detect the degenerate shape and halve the requested cap instead (converges in 1–2 retries; measured inputs stay trusted).
fix(agent): parse model output cap errors #72283 (@ekinnee) — parser +
is_output_cap_errorrecognition ofmax_tokens (98304) exceeds model's maximum output tokens (65536)with regression tests. The PR's second pre-retry clamp block was replaced (it bypassed the [Bug]: max_tokens over provider cap death-loops into context compression when the provider's 400 wording isn't recognized (e.g. DashScope/Qwen) #55546 clamp+compress path and broke its three regression tests); routing reworked in the follow-up commit below.Follow-up commit: parse the output cap ONCE at classification; exempt parseable wrapped output-cap 429s from the eager rate-limit provider fallback (failover can't fix a request-shape failure; the clamp fixes it in one retry) and widen
is_context_length_errorso they reach the SAME [Bug]: max_tokens over provider cap death-loops into context compression when the provider's 400 wording isn't recognized (e.g. DashScope/Qwen) #55546 clamp+compress recovery as plain output-cap 400s.Validation
28 targeted output-cap tests green including the 3 pre-existing main regression tests the original #72283 broke, both #72283 scenarios, and a new non-empty-fallback-chain ordering guard. 119 fallback/rate-limit tests green. E2E probes: parser on lowercased/wrapped forms, degenerate vs measured vLLM inputs, false-positive sweep.
Closes #89923. Closes #72283. Fixes #72281. Also removes the input-drift root cause behind #61761 (see that issue's competing-margin PRs #62197/#61846).
Credit
@Dhruv7201 (#89923) and @ekinnee (#72283) — commits cherry-picked with authorship preserved.
Infographic