Repository navigation
fix(sse): clamp out-of-range Gemini thinking_budget and learn the cap - #8726
Conversation
Unregistered Gemini models (any model id containing "gemini", on any provider) previously got no thinkingBudgetCap, so a high client budget (xhigh = 131072) was forwarded verbatim and the upstream returned 400 "thinking_budget must be in the range [-1, N]". Two layers: - Proactive: capThinkingBudget now falls back to the pro-tier ceiling (32768) for Gemini-family model ids with no registry cap, so the first request is already in range (no 400->retry round-trip). - Reactive: the executor detects the range 400, clamps the nested budget (native Gemini and Antigravity envelope) to the parsed upstream max, retries the same URL once, and records the surviving cap in an in-memory per-provider+model store so subsequent requests clamp proactively (the retry cost is paid once per process, not per chat session). Existing tests updated where they locked in the previous no-cap behavior (gemini-2.5-pro max/xhigh asserted the un-sendable 131072).
|
Thanks @fuko2935 — verified locally against the PR head: the four The two-layer design is exactly the right shape for this bug: A few small notes for the PR body archeology (no blocker, feel free
LGTM. Merge when ready. |
|
Validated in local merge-train /tmp/train1d-20260727-090022-suite.log on .113 @ 029cdf4215cf465f0e1716ac9f84a84692b1e881 (full unit suite green on CI-equivalent host) |
d78a836
into
cheaper-inference:release/v3.8.49
Problem
When a client sends a high reasoning budget (e.g. Claude Code's
xhigh/ultracode→ 131072 tokens) to a Gemini model that has no entry inMODEL_SPECS(e.g.gemini-2.5-pro, or any Gemini id served through another provider such as OpenRouter/Vertex), the budget was forwarded verbatim and the upstream rejected it:The proactive clamp (
capThinkingBudget) only knows caps for registered models, so unregistered Gemini models got no cap. And there was no reactive retry for this error:findOffendingFielddoesn't listthinking_budget(it's nested, not top-level, and must be clamped, not stripped), anddetectUnsupportedParamonly matches "unsupported parameter" phrasing — so the range 400 fell through every fallback and surfaced to the client as a failed request.Root cause
capThinkingBudgetreturnsMath.min(budget, cap)wherecap = resolved.thinkingBudgetCap ?? budget. For a model id with noMODEL_SPECSentry the cap isnull, so the requested budget passes through unchanged — a 131072xhighbudget sails straight to a 400.Fix — two layers, both keyed off the model id containing
gemini(provider-agnostic)1. Proactive fallback (avoids the double round-trip).
capThinkingBudgetnow falls back to the pro-tier ceiling32768when a model id containsgeminibut has no registry cap and no learned cap. Registered models keep their explicit caps (flash24576, pro32768), so this only fires for unregistered Gemini ids. The very first request is already in range — no 400→retry latency.2. Reactive clamp-and-retry with a learned-cap store (safety net). If a 400 still arrives (registry drift), the executor:
transformedBody— bothgenerationConfig.thinkingConfig.thinkingBudget(native Gemini / openai→gemini) andrequest.generationConfig...(Antigravity Cloud Code envelope) — only lowering, never raising,provider+modelstore (open-sse/services/learnedThinkingCaps.ts).Because the learned cap feeds back into layer 1 via
capThinkingBudget, the retry cost is paid once per process per provider+model — subsequent requests (any client, any chat session) clamp proactively and never re-hit the 400. In-memory only (per design): a restart re-learns at the cost of one 400.Files
open-sse/services/learnedThinkingCaps.tssrc/lib/modelCapabilities.tscapThinkingBudget: gemini-substring fallback + learned-cap consultation (lowest wins).open-sse/executors/base.tscontext_managementfallback; nested-budget read/clamp helpers; per-execute loop guard.tests/unit/translator-claude-to-gemini.test.tsgemini-2.5-promax/xhighpreviously expected131072, which locked in the old no-cap behavior the upstream never accepts. Now32768.Tests (TDD)
Three new suites — 32
test()cases / 61assert.*calls:tests/unit/learned-thinking-caps.test.ts— parser + step ladder + per-provider/model keying.tests/unit/cap-thinking-budget-gemini-fallback.test.ts— proactive fallback, provider-agnostic, registry cap precedence, learned-cap override, cross-provider isolation.tests/unit/gemini-thinking-budget-fallback.test.ts— executor clamp-retry (native + Antigravity envelope), guards (no budget / already-under-max / non-thinking 400 / loop prevention), and an end-to-end test of the reported scenario:reasoning_effort=xhighon unregisteredgemini-2.5-prosucceeds on the first request, no double round-trip.Verification