fix(vertex_ai): return a truncated choice for content-less Gemini candidates - #38301
yukimaru77 wants to merge 3 commits into
Conversation
… no content A content-less candidate (a thinking model that spends its whole token budget on reasoning returns finishReason MAX_TOKENS with no content) was skipped, leaving choices empty so callers hit IndexError on the first choice. Emit an empty-content choice, mapping finishReason through the same path the mainline uses; a missing finishReason is reported as length, never a false stop.
…endently Assert each content-less candidate keeps its index and maps its own finishReason (MAX_TOKENS -> length, SAFETY -> content_filter), so none is dropped when several arrive together.
Greptile SummaryThis PR preserves non-streaming Gemini candidates that omit Confidence Score: 4/5The PR appears safe to merge after considering the non-blocking semantic issue that reason-less candidates are labeled as token-limit truncations. The intended MAX_TOKENS and SAFETY mappings are preserved correctly, but the new fallback reports Files Needing Attention: litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py; tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py
|
| Filename | Overview |
|---|---|
| litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py | Adds the intended empty-choice rescue, but the missing-finish-reason fallback overstates an unknown termination cause as token truncation. |
| tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py | Adds focused regression coverage, including an assertion that currently codifies the unsupported length fallback for a reason-less candidate. |
Reviews (1): Last reviewed commit: "test(vertex_ai): cover multiple content-..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| mapped_finish_reason = ( | ||
| VertexGeminiConfig._check_finish_reason(None, finish_reason) | ||
| if finish_reason is not None | ||
| else "length" | ||
| ) | ||
| empty_message: ChatCompletionResponseMessage = { |
There was a problem hiding this comment.
A content-less candidate without finishReason is labeled length even though no provider signal establishes token exhaustion. This conflates unknown or metadata-only responses with truncation, leading clients to retry, continue generation, or display incorrect token-limit messaging.
Knowledge Base Used: Provider adapters and capabilities
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Yukito Nonaka seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
…ates A Gemini candidate with no content - an empty turn after a tool result (finishReason STOP), a thinking model that spent its budget (MAX_TOKENS), or a malformed function call - was skipped by the non-streaming transformation, leaving choices empty. That is not a valid OpenAI response: callers reading choices[0] hit IndexError on an HTTP 200 and lose the reason the turn was empty. Emit an empty-content choice carrying the mapped finish_reason, and keep the provider's own reason in provider_specific_fields, since the OpenAI enum has no equivalent for MALFORMED_FUNCTION_CALL. Streaming already recovers these in _apply_stream_candidates and is left untouched. Supersedes BerriAI#38301 and BerriAI#36870 (same approach, rebased with tests for the STOP-after-tool-result and MALFORMED_FUNCTION_CALL shapes). Fixes BerriAI#36881
…ates A Gemini candidate with no content - an empty turn after a tool result (finishReason STOP), a thinking model that spent its budget (MAX_TOKENS), or a malformed function call - was skipped by the non-streaming transformation, leaving choices empty. That is not a valid OpenAI response: callers reading choices[0] hit IndexError on an HTTP 200 and lose the reason the turn was empty. Emit an empty-content choice carrying the mapped finish_reason, and keep the provider's own reason in provider_specific_fields, since the OpenAI enum has no equivalent for MALFORMED_FUNCTION_CALL. Streaming already recovers these in _apply_stream_candidates and is left untouched. Supersedes BerriAI#38301 and BerriAI#36870 (same approach, rebased with tests for the STOP-after-tool-result and MALFORMED_FUNCTION_CALL shapes). Fixes BerriAI#36881
5 participants
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a developer receives no choice when Gemini exhausts its token budget during reasoning
POST https://litellm-domain/v1/chat/completionsto a Gemini thinking model with a smallmax_tokensfinishReason: "MAX_TOKENS"and nocontent"choices": [], so their application cannot inspect the first choiceAfter: the same request returns a choice that identifies token-limit truncation
POST https://litellm-domain/v1/chat/completionsto a Gemini thinking model with a smallmax_tokensfinishReason: "MAX_TOKENS"and nocontentfinish_reason: "length"Relevant issues
Fixes #36881
Supersedes #36870 by applying its two commits to the latest
litellm_internal_staging. Both commits retain the original author's attributionLinear ticket
Pre-Submission checklist
Screenshots / Proof of Fix
A stable real-provider Before/After run is not included. Repeated Vertex requests returned empty
contentobjects rather than omitting thecontentkey, so the reported upstream shape did not recurThe focused regression suite passes with 149 tests:
Type
🐛 Bug Fix
Caveats (if any)
Low
Final Attestation