fix(runtime): clamp output cap to fit context window (deepseek-v4-flash-w2 overflow) - #271
Merged
Merged
Conversation
…sh-w2 overflow)
Local sessions against deepseek-v4-flash-w2 (vLLM on taro, fronted by the
ko-nas ai-router) death-loop on "Context length exceeded: max compression
attempts (3) reached" once the conversation grows past ~half the window.
Root cause: the request carries an output cap (max_tokens) of ~65,536 — half
the 131,072 window — so once the prompt exceeds ~65K tokens, prompt + max_tokens
overflows the window and vLLM returns HTTP 400. vLLM reports the offending input
as a max_tokens-DEPENDENT LOWER BOUND ("prompt contains at least N input
tokens", where N = max_model_len + 1 - max_tokens) because it caps tokenization
at max_input_tokens + 1 for efficiency and never computes the true prompt length.
The reactive output-cap retry parses that number and shrinks max_tokens by the
reported delta, but the delta tracks max_tokens in lockstep, so it decrements by
~65 tokens per attempt and gives up after 3 — never converging.
Fix (client-side, provider-agnostic):
- New model_metadata.output_tokens_that_fit(): single source of truth for the
largest output cap that keeps prompt + output inside the window, using a
conservative (over-reserved) local estimate. The rough estimator assumes
~4 chars/token; dense tool/JSON transcripts tokenize closer to ~3.4, so the
input is over-reserved (1.2x + pad) to stay strictly in-window despite the gap.
- Pre-flight clamp in build_api_kwargs (all api_modes, via
_ephemeral_max_output_tokens): shrink an oversized cap before sending so the
400 never fires. Only ever shrinks an explicit cap; never sets one where the
caller intended the provider default; no-ops when the window is unknown.
- Anchor the reactive output-cap retry to the same estimate so it converges to a
fitting cap in one step even when the provider reports a lower bound.
Tests: tests/test_output_fit_preflight_clamp.py (9 cases: clamp fires on
overflow, leaves fitting caps and provider-defaults untouched, retry converges).
All existing output-cap / ctx-halving / provider-profile tests stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OmarB97
added a commit
that referenced
this pull request
Jul 20, 2026
…sh-w2 overflow) (#271) Local sessions against deepseek-v4-flash-w2 (vLLM on taro, fronted by the ko-nas ai-router) death-loop on "Context length exceeded: max compression attempts (3) reached" once the conversation grows past ~half the window. Root cause: the request carries an output cap (max_tokens) of ~65,536 — half the 131,072 window — so once the prompt exceeds ~65K tokens, prompt + max_tokens overflows the window and vLLM returns HTTP 400. vLLM reports the offending input as a max_tokens-DEPENDENT LOWER BOUND ("prompt contains at least N input tokens", where N = max_model_len + 1 - max_tokens) because it caps tokenization at max_input_tokens + 1 for efficiency and never computes the true prompt length. The reactive output-cap retry parses that number and shrinks max_tokens by the reported delta, but the delta tracks max_tokens in lockstep, so it decrements by ~65 tokens per attempt and gives up after 3 — never converging. Fix (client-side, provider-agnostic): - New model_metadata.output_tokens_that_fit(): single source of truth for the largest output cap that keeps prompt + output inside the window, using a conservative (over-reserved) local estimate. The rough estimator assumes ~4 chars/token; dense tool/JSON transcripts tokenize closer to ~3.4, so the input is over-reserved (1.2x + pad) to stay strictly in-window despite the gap. - Pre-flight clamp in build_api_kwargs (all api_modes, via _ephemeral_max_output_tokens): shrink an oversized cap before sending so the 400 never fires. Only ever shrinks an explicit cap; never sets one where the caller intended the provider default; no-ops when the window is unknown. - Anchor the reactive output-cap retry to the same estimate so it converges to a fitting cap in one step even when the provider reports a lower bound. Tests: tests/test_output_fit_preflight_clamp.py (9 cases: clamp fires on overflow, leaves fitting caps and provider-defaults untouched, retry converges). All existing output-cap / ctx-halving / provider-profile tests stay green. Co-authored-by: Omar B <omar@kostudios.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
13 tasks
OmarB97
added a commit
that referenced
this pull request
Aug 2, 2026
…text window (#300) output_tokens_that_fit() returned its ``min_output`` floor (1) when the reserved input filled the window, and both callers consumed that fabricated 1 as a real budget. The reservation is multiplicative — int(est * 1.2) + 1024 + 512 — so it exhausts the window once the rough estimate passes ~(ctx - 1536) / 1.2, about 82.7% fill. Past that point: * the pre-flight clamp in build_api_kwargs set max_tokens=1 on EVERY request, with no provider error involved at all; and * the reactive output-cap retry clamped a healthy provider-authoritative cap down to 1 — directly contradicting the branch immediately above it, which deliberately falls back to the provider's number precisely because "the rough local estimate can overshoot the real request size". Measured against a 200,000-token window: a ~170k-token prompt (85% fill) with the provider reporting available_tokens=25,000 produced a retry cap of 1 instead of 24,936. At 80% fill the cap is 6,455; at 83% it is 1 — a cliff, not a degradation. max_tokens=1 is the worst kind of failure here: the provider accepts it and returns a single truncated token, so the turn reports success while handing back output the user cannot use. Nothing fails loudly. Fix: report None ("no usable cap — leave max_tokens to your own budget logic") instead of fabricating a floor. Both call sites already guard None correctly, so neither needed a code change; the reactive one gains a comment so the floor is not reinstated. Above 82.7% fill the pre-flight clamp now no-ops and the provider reports its own authoritative budget on the reactive path. Introduced in #271; the vLLM/deepseek-v4-flash-w2 convergence property that motivated #271 is unaffected — that regime sits at ~44% fill, where the fit stays positive and the retry still converges in one step. Also reverts the test half of #295 for test_output_cap_retry_request_pressure_ lower_bound. That assertion was loosened to match the defect; the original arithmetic was right and now passes for the right reason. #295's other change (threading FailoverReason into the fallback switch) is correct and stands. Tests: 5 cases in tests/test_output_fit_preflight_clamp.py covering the 82.7%-100% fill band at both call sites. The sweep case holds every reported fit to the documented "server tokenizes ~15% denser" safety property, which a fabricated floor fails by construction. Verified they fail without the production change and pass with it. Co-authored-by: Omar Baradei <omar@kostudios.io> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
OmarB97
added a commit
that referenced
this pull request
Aug 2, 2026
…f giving up (#302) #300 stopped output_tokens_that_fit() from fabricating a max_tokens=1 floor and made it return None once the reserved input filled the window. That was right, but it left a gap: above ~82.7% fill the pre-flight clamp in build_api_kwargs stops firing entirely, so every turn in that regime sends the full configured max_tokens, eats a provider 400, and only then gets a usable cap from the reactive retry in conversation_loop. A guaranteed extra round-trip per turn. The 1.2x factor is an uncertainty allowance for servers that tokenize denser than the ~4 chars/token heuristic, not a real token cost — but near a full window that allowance (0.2 * est) exceeds the entire remaining headroom, so holding it reports "nothing fits" while thousands of output tokens genuinely do. The documented safety property is looser than the allowance: a server tokenizing ~15% denser must still leave the clamped request in-window. At est=166,008 in a 200,000-token window the 1.2x reservation admits nothing while the 1.15x contract admits ~9,000 tokens. Fix: reserve in two tiers. The preferred 1.2x + 1024 cushion is kept whenever the window can pay for it and still leave a usable cap — below ~82.5% fill the returned value is byte-identical to before, so nothing that works today gets looser. Past that point the reservation drops to exactly what the safety contract requires (23/20, kept rational so the ceiling is exact integer arithmetic) and hands back the rest, tapering from ~9,700 tokens at 82.5% fill down to the usable floor rather than falling off a cliff. The fixed pad is dropped in the degraded tier on purpose: it is a rounding cushion for small prompts, and there the 0.15 * est term is orders of magnitude larger. None is still the answer past the 1/1.15 ceiling (~87% fill), where the contract admits nothing usable — swept over every estimate in a 200,000-token window, the largest contract headroom ever declined is 1,024 tokens, exactly the usable floor plus the window margin we deliberately hold back. min_output now defaults to _OUTPUT_FIT_MIN_USABLE (512) rather than 1, so the "never hand back a cap too small to use" rule is the default rather than something each caller has to know. Neither call site needed a code change. The #271 vLLM/deepseek-v4-flash-w2 convergence property is strengthened, not weakened: in the 82.5%-87% band the local anchor used to be None, so the reactive retry fell back to whatever the provider's max_tokens-dependent lower bound reported. It now exists and pulls the retry straight to a fitting cap in one step from any starting value. Tests: 7 cases in tests/test_output_fit_preflight_clamp.py covering the 82.7%-100% fill band at both call sites — the reported fit where the preferred cushion admits none, band coverage without a hole, the no-cliff property (wherever None is still reported, the contract really does admit nothing usable, swept in 500-token steps), the no-unusable-cap property, a pin that low fill is unchanged, the pre-flight clamp firing in the band, and one-step reactive convergence in the band. Verified 6 of the 7 fail against the previous formula; the seventh is the low-fill pin, which must pass on both. Also re-runs green: tests/run_agent/test_run_agent.py, tests/test_output_cap_parsing.py, tests/test_ctx_halving_fix.py. Co-authored-by: Omar Baradei <omar@kostudios.io> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Local Hermes sessions against
deepseek-v4-flash-w2(vLLM on taro via the ko-nas ai-router) death-loop on "Context length exceeded: max compression attempts (3) reached" once the conversation grows past ~half the 131,072 window.Root cause
The request carries an output cap (
max_tokens) of ~65,536 — half the window — so once the prompt exceeds ~65K tokens,prompt + max_tokensoverflows and vLLM returns HTTP 400. vLLM reports the offending input as amax_tokens-dependent lower bound ("prompt contains at least N input tokens", whereN = max_model_len + 1 − max_tokens), because it caps tokenization atmax_input_tokens + 1for efficiency and never computes the true prompt length. The reactive output-cap retry shrinksmax_tokensby the reported delta — but that delta tracksmax_tokensin lockstep, so it decrements by ~65 tokens/attempt and gives up after 3, never converging.Fix (client-side, provider-agnostic)
model_metadata.output_tokens_that_fit()— single source of truth for the largest output cap that keepsprompt + outputinside the window, from a conservative (over-reserved) local estimate. The rough estimator assumes ~4 chars/token; dense tool/JSON transcripts tokenize closer to ~3.4, so the input is over-reserved (1.2× + pad) to stay strictly in-window despite the gap.build_api_kwargs(all api_modes, via_ephemeral_max_output_tokens): shrink an oversized cap before sending so the 400 never fires. Only ever shrinks an explicit cap; never sets one where the caller intended the provider default; no-ops when the window is unknown.Testing
tests/test_output_fit_preflight_clamp.py(9 cases): clamp fires on overflow, leaves fitting caps + provider-defaults untouched, reactive retry converges.test_output_cap_parsing.py,test_ctx_halving_fix.py, custom/deepseek provider-profile suites all stay green (122 related tests).Risk
Low. The clamp only reduces an already-oversized cap and no-ops when the window is unknown or the cap fits; behavior for every other provider is unchanged.