Skip to content

fix(gemini): forward OpenAI usage-only chunks and cached/thought token counts to /v1beta clients - #17842

Merged
diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.53from
AndresDiagoM:fix/gemini-usage-metadata-from-openai
Oct 26, 2026
Merged

diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.53from
AndresDiagoM:fix/gemini-usage-metadata-from-openai

Conversation

@AndresDiagoM

@AndresDiagoM AndresDiagoM commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ base-red inherited: #17219

Closes #17841.

Summary

  • Streaming. /v1beta/models/*:streamGenerateContent pipes handleChat's OpenAI SSE through transformOpenAISSEToGeminiSSE(). The default executor forces stream_options.include_usage: true, and stream.ts forwards the trailing choices: [] usage chunk on purpose (fix(sse): estimate usage in passthrough stream even with include_usage #12151). openAIChunkToGeminiChunk() returned null for any chunk without choices[0], so @google/genai clients never received usageMetadata from an OpenAI-style upstream. That chunk is now forwarded as its own trailing Gemini chunk, with an empty text part, usageMetadata and modelVersion, and no finishReason. It is skipped when usage already went out on the finish chunk, so usage is never counted twice.
  • Token detail. prompt_tokens_details.cached_tokens maps to cachedContentTokenCount on the streaming path and on both non-streaming paths. completion_tokens_details.reasoning_tokens maps to thoughtsTokenCount in the non-streaming Gemini-family envelope (convertOpenAINonStreamingToGeminiFamily); the streaming path already had it. Both fields appear only when non-zero, the same mapping as openai-to-antigravity.ts.
  • Structure. openAIChunkToGeminiChunk and convertOpenAIResponseToGemini already sit at cyclomatic 19 in complexity-baseline.json, so they get no new branches. The mapping and the usage-only-chunk handling live in the new leaf open-sse/translator/response/geminiUsageMetadata.ts. The two usage blocks that used to be duplicated in openai-to-gemini-sse.ts now call one helper, and that file shrinks by 22 lines. responseTranslator.ts gains one import and one spread.

Not changed on purpose

  • When usage rides the finish chunk, the output is unchanged except for the new cached and thought counts.
  • openai-to-antigravity.ts stashes a usage-only chunk but emits it only if another choice chunk follows, which doesn't happen with real OpenAI ordering. That is the same gap on another converter, so I left it for a separate PR to keep this one surgical.

Validation (TDD)

Two new streaming tests in tests/unit/translator-openai-to-gemini-sse.test.ts and the new tests/unit/translator-openai-to-gemini-nonstream-usage.test.ts (3 tests: Gemini, Antigravity, and no extra counts). Run without the fix, 4 of the 16 tests in the two files fail:

✖ translateNonStreamingResponse OpenAI -> gemini: maps reasoning and cached tokens
✖ translateNonStreamingResponse OpenAI -> antigravity: maps reasoning and cached tokens
✖ transformOpenAISSEToGeminiSSE: forwards a trailing usage-only chunk (choices: []) as usageMetadata
✖ transformOpenAISSEToGeminiSSE: usage on the finish chunk carries cachedContentTokenCount and is not duplicated by a later usage-only chunk
ℹ tests 16  pass 12  fail 4
# with the fix
ℹ tests 16  pass 16  fail 0

Regression: the 40 suites that touch openai-to-gemini, openai-to-antigravity, translateNonStreamingResponse or v1beta give 428/429. The one failure is cli/run-command.test.ts ("dry-run redacts sensitive passthrough flags"), which this branch doesn't touch and which fails the same way without it. typecheck:core, eslint on the changed files, check-file-size, and check-complexity-ratchets --base-ref origin/release/v3.8.53 (new-code mode, both OK) all pass.

…n counts to /v1beta clients

OpenAI upstreams with stream_options.include_usage send usage in a separate
final chunk with choices: [], which the OpenAI->Gemini SSE converter dropped,
so @google/genai clients never received usageMetadata. Emit it as its own
trailing chunk (skipped when usage already rode the finish chunk).

Also map prompt_tokens_details.cached_tokens to cachedContentTokenCount in
streaming and non-streaming Gemini responses, and reasoning_tokens to
thoughtsTokenCount in the non-streaming Gemini-family envelope, matching
openai-to-antigravity. The mapping lives in geminiUsageMetadata.ts.
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks, Andres. This is a very clean fix. The diagnosis in #17841 is right: the trailing choices: [] usage chunk was being dropped, so @google/genai clients never saw usageMetadata. Your approach is careful. You forward it as its own trailing chunk, guard against double counting with the usageSent flag, and map cached and thought tokens consistently with the antigravity converter. Moving the mapping into the geminiUsageMetadata.ts leaf also kept the two complexity-19 functions from growing and removed the duplicated usage blocks.

I ran your tests on my side. All 16 pass with your change. With only the two source files reverted to the base, exactly the 4 new tests fail, so the red-then-green proof holds. ESLint and Prettier are clean too.

The one thing missing is the changelog fragment (changelog.d/fixes/17842-<slug>.md). We'll add that in place so you don't need to do another round. The CI runs on the PR were cancelled and the release base is currently red from the inherited #17219, so don't worry about those. Nothing else is needed from you. Thanks also for flagging the antigravity usage-only gap as a separate follow-up, which seems like the right call.

@AndresDiagoM
AndresDiagoM force-pushed the fix/gemini-usage-metadata-from-openai branch from 1926801 to 0e47f11 Compare October 22, 2026 13:37
@AndresDiagoM

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough verification.

The changelog fragment was already on the branch (changelog.d/fixes/17842-gemini-usage-metadata-from-openai.md, pushed right after opening the PR), but GitHub never synced that push onto the PR, so the file list stayed at the fix commit. I re-pushed the fragment commit with a fresh SHA (0e47f11cc8) and the PR now shows it, so no need to add one on your side.

…ertOpenAIResponseToGemini

Carried over from cheaper-inference#17943: the non-streaming converter maps
prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens
to cachedContentTokenCount / thoughtsTokenCount.

Co-authored-by: skygunner <84759+skygunner@users.noreply.github.com>
@AndresDiagoM

Copy link
Copy Markdown
Contributor Author

@diegosouzapw I ported the non-streaming convertOpenAIResponseToGemini detail-counter test from #17943 myself in af1aa2a (Co-authored-by skygunner). It fails on the release base and passes with this branch, so there's nothing left to port.

@diegosouzapw
diegosouzapw merged commit 2cfce7c into cheaper-inference:release/v3.8.53 Oct 26, 2026
43 of 57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants