Skip to content

fix: distinguish vLLM 'at least' lower-bound from exact input token counts - #69045

Closed
jamesreader wants to merge 1 commit into
NousResearch:mainfrom
jamesreader:fix/vllm-at-least-input-token-handling
Closed

fix: distinguish vLLM 'at least' lower-bound from exact input token counts#69045
jamesreader wants to merge 1 commit into
NousResearch:mainfrom
jamesreader:fix/vllm-at-least-input-token-handling

Conversation

@jamesreader

Copy link
Copy Markdown

Problem

vLLM reports context window errors in two forms:

  1. Lower bound: your prompt contains at least 81921 input tokens — this is an early tokenizer cutoff, NOT the actual prompt size. vLLM stops counting once it determines N + requested_output exceeds the window.
  2. Exact count: your prompt contains 81921 input tokens — this is the actual token count.

The existing regex captured both indistinguishably:

r'prompt contains (?:at least )?(\d+)\s*input tokens'

This caused parse_available_output_tokens_from_error() to treat the lower bound as exact and compute a retry budget. On a retry, lowering max_tokens made the lower bound rise by the same amount, creating an infinite loop that always totals context_window + 1.

Fix

Use a negative lookahead to skip 'at least' qualified matches:

r'prompt contains (?!at least )(\d+)\s*input tokens'
  • Lower-bound errors (at least N) now return None from parse_available_output_tokens_from_error() and False from is_output_cap_error(), routing correctly to input compression.
  • Exact-count errors still provide a usable retry budget for output-cap reduction.

Additional changes

  • Add supports_tools field to ProviderProfile for providers without function calling support (avoids sending tools to models that deadlock on unparseable schemas).
  • Update test expectations to match new behavior.

Testing

Existing tests updated. test\_vllm\_token\_based\_lower\_bound\_routes\_to\_compression verifies that 'at least' messages route to compression, and test\_vllm\_exact\_count\_retry\_fits\_inside\_window confirms exact-count messages still provide a retry budget.

…ounts

vLLM reports 'at least N input tokens' as an early tokenizer cutoff, not
the actual prompt size. The existing regex captured both 'at least N' and
exact N indistinguishably, causing parse_available_output_tokens_from_error()
to treat the lower bound as exact and compute a retry budget. On a retry,
lowering max_tokens made the lower bound rise by the same amount, looping
forever at context_window + 1.

Fix:
- Use negative lookahead to skip 'at least' qualified matches in the vLLM
  input token regex — lower-bound errors now return None, routing to input
  compression instead of a futile output-cap reduction loop.
- Add supports_tools to ProviderProfile for providers without function
  calling support (avoids sending tools to models that can't parse them).
- Update test expectations: 'at least' messages route to compression,
  exact-count messages still provide a usable retry budget.

Closes infinite retry loop with local vLLM backends.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/openai OpenAI / Codex Responses API duplicate This issue or pull request already exists labels Jul 22, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #58869 for the exact lower-bound vLLM parser correction. #58869 already handles this wording and adds preflight context clamping; the separate tool-support hunk here does not make the overlapping fix distinct.

@jamesreader

Copy link
Copy Markdown
Author

Duplicate of #58869 which already covers the vLLM lower-bound fix with additional context clamping. Closing to avoid noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists provider/openai OpenAI / Codex Responses API type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants