Skip to content

fix(sse): clamp out-of-range Gemini thinking_budget and learn the cap - #8726

Merged
diegosouzapw merged 1 commit into
cheaper-inference:release/v3.8.49from
fuko2935:fix/gemini-thinking-budget-fallback
Jul 27, 2026
Merged

diegosouzapw merged 1 commit into
cheaper-inference:release/v3.8.49from
fuko2935:fix/gemini-thinking-budget-fallback

Conversation

@fuko2935

@fuko2935 fuko2935 commented Jul 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

When a client sends a high reasoning budget (e.g. Claude Code's xhigh / ultracode → 131072 tokens) to a Gemini model that has no entry in MODEL_SPECS (e.g. gemini-2.5-pro, or any Gemini id served through another provider such as OpenRouter/Vertex), the budget was forwarded verbatim and the upstream rejected it:

400 GenerateContentRequest.generation_config.thinking_config.thinking_budget:
thinking_budget must be in the range [-1, 65535]

The proactive clamp (capThinkingBudget) only knows caps for registered models, so unregistered Gemini models got no cap. And there was no reactive retry for this error: findOffendingField doesn't list thinking_budget (it's nested, not top-level, and must be clamped, not stripped), and detectUnsupportedParam only matches "unsupported parameter" phrasing — so the range 400 fell through every fallback and surfaced to the client as a failed request.

Root cause

capThinkingBudget returns Math.min(budget, cap) where cap = resolved.thinkingBudgetCap ?? budget. For a model id with no MODEL_SPECS entry the cap is null, so the requested budget passes through unchanged — a 131072 xhigh budget sails straight to a 400.

Fix — two layers, both keyed off the model id containing gemini (provider-agnostic)

1. Proactive fallback (avoids the double round-trip). capThinkingBudget now falls back to the pro-tier ceiling 32768 when a model id contains gemini but has no registry cap and no learned cap. Registered models keep their explicit caps (flash 24576, pro 32768), so this only fires for unregistered Gemini ids. The very first request is already in range — no 400→retry latency.

2. Reactive clamp-and-retry with a learned-cap store (safety net). If a 400 still arrives (registry drift), the executor:

  • parses the upstream-advertised max from the error text,
  • clamps the nested budget in the live transformedBody — both generationConfig.thinkingConfig.thinkingBudget (native Gemini / openai→gemini) and request.generationConfig... (Antigravity Cloud Code envelope) — only lowering, never raising,
  • retries the same URL once (guarded so it can't loop),
  • records the surviving cap in an in-memory provider+model store (open-sse/services/learnedThinkingCaps.ts).

Because the learned cap feeds back into layer 1 via capThinkingBudget, the retry cost is paid once per process per provider+model — subsequent requests (any client, any chat session) clamp proactively and never re-hit the 400. In-memory only (per design): a restart re-learns at the cost of one 400.

Files

File Change
open-sse/services/learnedThinkingCaps.ts New. In-memory cap store, step ladder 32768→24576→8192, error-text parser.
src/lib/modelCapabilities.ts capThinkingBudget: gemini-substring fallback + learned-cap consultation (lowest wins).
open-sse/executors/base.ts Clamp-and-retry block after the context_management fallback; nested-budget read/clamp helpers; per-execute loop guard.
tests/unit/translator-claude-to-gemini.test.ts One assertion updated: gemini-2.5-pro max/xhigh previously expected 131072, which locked in the old no-cap behavior the upstream never accepts. Now 32768.

Tests (TDD)

Three new suites — 32 test() cases / 61 assert.* calls:

  • tests/unit/learned-thinking-caps.test.ts — parser + step ladder + per-provider/model keying.
  • tests/unit/cap-thinking-budget-gemini-fallback.test.ts — proactive fallback, provider-agnostic, registry cap precedence, learned-cap override, cross-provider isolation.
  • tests/unit/gemini-thinking-budget-fallback.test.ts — executor clamp-retry (native + Antigravity envelope), guards (no budget / already-under-max / non-thinking 400 / loop prevention), and an end-to-end test of the reported scenario: reasoning_effort=xhigh on unregistered gemini-2.5-pro succeeds on the first request, no double round-trip.

Verification

node --import tsx/esm --test tests/unit/<all suites>   → 209 pass, 0 fail
npm run typecheck:core                                  → clean
npm run lint                                            → clean (EXIT 0)
npm run check:cycles                                    → no cycles

Unregistered Gemini models (any model id containing "gemini", on any
provider) previously got no thinkingBudgetCap, so a high client budget
(xhigh = 131072) was forwarded verbatim and the upstream returned
400 "thinking_budget must be in the range [-1, N]".

Two layers:
- Proactive: capThinkingBudget now falls back to the pro-tier ceiling
  (32768) for Gemini-family model ids with no registry cap, so the
  first request is already in range (no 400->retry round-trip).
- Reactive: the executor detects the range 400, clamps the nested
  budget (native Gemini and Antigravity envelope) to the parsed
  upstream max, retries the same URL once, and records the surviving
  cap in an in-memory per-provider+model store so subsequent requests
  clamp proactively (the retry cost is paid once per process, not per
  chat session).

Existing tests updated where they locked in the previous no-cap
behavior (gemini-2.5-pro max/xhigh asserted the un-sendable 131072).
@fuko2935
fuko2935 requested a review from diegosouzapw as a code owner July 26, 2026 21:52
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks @fuko2935 — verified locally against the PR head: the four
directly-affected test suites go green
(learned-thinking-caps 14/14, cap-thinking-budget-gemini-fallback 10/10,
gemini-thinking-budget-fallback 8/8, translator-claude-to-gemini 18/18)
and npx tsc --noEmit -p tsconfig.typecheck-core.json is clean.

The two-layer design is exactly the right shape for this bug:
proactive gemini-substring fallback in capThinkingBudget plus a
reactive clamp-and-retry in BaseExecutor.execute() that records
into a process-local provider+model store consulted by layer 1.
The learned-cap feedback loop is the load-bearing piece — it means
the 400→retry round-trip is paid once per process per
provider+model, not once per chat session. E2E test confirms
xhigh on an unregistered gemini-2.5-pro succeeds on the first
fetch with no double round-trip.

A few small notes for the PR body archeology (no blocker, feel free
to amend on merge):

  • The "39 new assertions" line counts test() calls AND assert.*
    calls; the strict test() count is 32 (14+10+8) and the strict
    assert.* count is 61. Either is defensible; just want the
    number stable across reviewers.
  • #8494 (capability-filters-fail-closed) also touches
    src/lib/modelCapabilities.ts but on a different function
    (supportsTools at lines 497-499 vs your capThinkingBudget at
    666+); no merge conflict.
  • #8494 is a few days older and also merge-ready, so this can go
    in either order — if the maintainer prefers a clean per-PR merge,
    pick whichever base is currently the lower tip.

LGTM. Merge when ready.

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Validated in local merge-train /tmp/train1d-20260727-090022-suite.log on .113 @ 029cdf4215cf465f0e1716ac9f84a84692b1e881 (full unit suite green on CI-equivalent host)

@diegosouzapw
diegosouzapw merged commit d78a836 into cheaper-inference:release/v3.8.49 Jul 27, 2026
3 checks passed
@diegosouzapw diegosouzapw mentioned this pull request Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants