Repository navigation
fix(gemini): forward OpenAI usage-only chunks and cached/thought token counts to /v1beta clients - #17842
Conversation
…n counts to /v1beta clients OpenAI upstreams with stream_options.include_usage send usage in a separate final chunk with choices: [], which the OpenAI->Gemini SSE converter dropped, so @google/genai clients never received usageMetadata. Emit it as its own trailing chunk (skipped when usage already rode the finish chunk). Also map prompt_tokens_details.cached_tokens to cachedContentTokenCount in streaming and non-streaming Gemini responses, and reasoning_tokens to thoughtsTokenCount in the non-streaming Gemini-family envelope, matching openai-to-antigravity. The mapping lives in geminiUsageMetadata.ts.
|
Thanks, Andres. This is a very clean fix. The diagnosis in #17841 is right: the trailing I ran your tests on my side. All 16 pass with your change. With only the two source files reverted to the base, exactly the 4 new tests fail, so the red-then-green proof holds. ESLint and Prettier are clean too. The one thing missing is the changelog fragment ( |
1926801 to
0e47f11
Compare
|
Thanks for the thorough verification. The changelog fragment was already on the branch ( |
…ertOpenAIResponseToGemini Carried over from cheaper-inference#17943: the non-streaming converter maps prompt_tokens_details.cached_tokens and completion_tokens_details.reasoning_tokens to cachedContentTokenCount / thoughtsTokenCount. Co-authored-by: skygunner <84759+skygunner@users.noreply.github.com>
|
@diegosouzapw I ported the non-streaming |
2cfce7c
into
cheaper-inference:release/v3.8.53
Closes #17841.
Summary
/v1beta/models/*:streamGenerateContentpipeshandleChat's OpenAI SSE throughtransformOpenAISSEToGeminiSSE(). The default executor forcesstream_options.include_usage: true, andstream.tsforwards the trailingchoices: []usage chunk on purpose (fix(sse): estimate usage in passthrough stream even with include_usage #12151).openAIChunkToGeminiChunk()returnednullfor any chunk withoutchoices[0], so@google/genaiclients never receivedusageMetadatafrom an OpenAI-style upstream. That chunk is now forwarded as its own trailing Gemini chunk, with an empty text part,usageMetadataandmodelVersion, and nofinishReason. It is skipped when usage already went out on the finish chunk, so usage is never counted twice.prompt_tokens_details.cached_tokensmaps tocachedContentTokenCounton the streaming path and on both non-streaming paths.completion_tokens_details.reasoning_tokensmaps tothoughtsTokenCountin the non-streaming Gemini-family envelope (convertOpenAINonStreamingToGeminiFamily); the streaming path already had it. Both fields appear only when non-zero, the same mapping asopenai-to-antigravity.ts.openAIChunkToGeminiChunkandconvertOpenAIResponseToGeminialready sit at cyclomatic 19 incomplexity-baseline.json, so they get no new branches. The mapping and the usage-only-chunk handling live in the new leafopen-sse/translator/response/geminiUsageMetadata.ts. The two usage blocks that used to be duplicated inopenai-to-gemini-sse.tsnow call one helper, and that file shrinks by 22 lines.responseTranslator.tsgains one import and one spread.Not changed on purpose
openai-to-antigravity.tsstashes a usage-only chunk but emits it only if another choice chunk follows, which doesn't happen with real OpenAI ordering. That is the same gap on another converter, so I left it for a separate PR to keep this one surgical.Validation (TDD)
Two new streaming tests in
tests/unit/translator-openai-to-gemini-sse.test.tsand the newtests/unit/translator-openai-to-gemini-nonstream-usage.test.ts(3 tests: Gemini, Antigravity, and no extra counts). Run without the fix, 4 of the 16 tests in the two files fail:Regression: the 40 suites that touch
openai-to-gemini,openai-to-antigravity,translateNonStreamingResponseorv1betagive 428/429. The one failure iscli/run-command.test.ts("dry-run redacts sensitive passthrough flags"), which this branch doesn't touch and which fails the same way without it.typecheck:core, eslint on the changed files,check-file-size, andcheck-complexity-ratchets --base-ref origin/release/v3.8.53(new-code mode, both OK) all pass.