fix(vertex_ai): only fall back to a placeholder thought signature on the first parallel function call

[ljogeiger]

TLDR

Problem this solves:

• Gemini signs only the first parallel function call
• LiteLLM fabricates a skip-validation placeholder on every sibling call
• Replayed history no longer matches what Gemini emitted

Google documents both halves of this. On the expected shape: "In cases of parallel function calls in a single response, only the first functionCall part will contain the thought_signature." On the placeholder: "You can set thought_signature to skip_thought_signature_validator, but, this should be a last resort as it will negatively impact model performance." Fabricating one per sibling call is the opposite of a last resort, and it costs quality on every parallel tool-calling turn

That cost is measurable, not theoretical. Internal testing at Google found that setting skip_thought_signature_validator on parallel function calls causes gemini-3.5 to return empty text responses. Following the documented shape, one real signature on the first call and nothing on the siblings, reduced the error rate

How it solves it:

• Placeholder fallback now applies to the first function call only
• Unsigned sibling calls are replayed with no signature
• Real signatures on any call are still preserved

User Flow

Before: a developer using parallel tool calling against a Gemini 3 model gets their conversation replayed with signatures the model never produced

1. They send POST https://litellm-domain/v1/chat/completions with a get_weather tool and ask for Paris, London and Tokyo in parallel
2. Three tool calls come back; only the first id carries a __thought__ suffix, the other two are bare
3. They append the assistant turn plus three tool results and send the same POST again
4. The turn is replayed upstream with the real signature on Paris and a skip_thought_signature_validator placeholder attached to London and Tokyo, so two of the three calls claim a signature the model never returned

After: the same replay carries exactly the signatures the model produced

1. They send the same POST https://litellm-domain/v1/chat/completions with the same tool and prompt
2. Three tool calls come back; only the first id carries a __thought__ suffix, the other two are bare
3. They append the assistant turn plus three tool results and send the same POST again
4. The turn is replayed upstream with the real signature on Paris and nothing on London or Tokyo, matching the original response, and the answer comes back covering all three cities

Relevant issues

Linear ticket

Resolves LIT-5888

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

E2e against real Vertex AI (gemini-3.7-flash, vertex_location: global). The observable that matters here is the exact payload Vertex receives on the replay turn, so the deployment's api_base points at a local recording relay that logs every request body and forwards it unchanged to the real https://aiplatform.googleapis.com. Proxy config:

model_list:
  - model_name: vertex_ai/gemini-3.7-flash
    litellm_params:
      model: vertex_ai/gemini-3.7-flash
      vertex_project: <project>
      vertex_location: global
      api_base: http://127.0.0.1:<relay port>
general_settings:
  master_key: sk-1234

Both legs run the same two turns against their own proxy, booted from the commit under test:

# Turn 1: three cities, one get_weather tool, parallel calls requested
curl -sS "http://127.0.0.1:<proxy port>/v1/chat/completions" \
  -H 'Content-Type: application/json' -H 'Authorization: Bearer sk-1234' \
  -d '{"model":"vertex_ai/gemini-3.7-flash","messages":[{"role":"user","content":"What is the weather in Paris, London and Tokyo? Call the get_weather tool once per city, all three in parallel."}],"tools":[{"type":"function","function":{"name":"get_weather","description":"get weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]}'

# Turn 2: turn 1's user message + the assistant tool_calls message + three {"role":"tool","tool_call_id":...,"content":"20C"} results
curl -sS "http://127.0.0.1:<proxy port>/v1/chat/completions" \
  -H 'Content-Type: application/json' -H 'Authorization: Bearer sk-1234' \
  -d @turn2_payload.json

Before (da7a10e, the merge base)

Turn 1 returns three parallel tool calls, a signature only on the first id:

{"city": "Paris"}  id=call_2117163__thought__AY89a18pdiowbEw4FF1FoNDFGqoXixvwugvVO...
{"city": "London"}  id=call_2117164
{"city": "Tokyo"}  id=call_2117165

Turn 2 answers normally (200, finish_reason: stop): "The current weather in Paris, London, and Tokyo is: Paris 20°C, London 20°C, Tokyo 20°C". The model parts of that request as the relay recorded them on their way to Vertex:

get_weather({"city":"Paris"})   thoughtSignature=AY89a18pdiowbEw4FF1FoNDFGqoXixvwugvVOvu4gYXn...  (the model's real signature, 468 chars)
get_weather({"city":"London"})  thoughtSignature=c2tpcF90aG91Z2h0X3NpZ25hdHVyZV92YWxpZGF0b3I=  (fabricated placeholder)
get_weather({"city":"Tokyo"})   thoughtSignature=c2tpcF90aG91Z2h0X3NpZ25hdHVyZV92YWxpZGF0b3I=  (fabricated placeholder)

c2tpcF90aG91Z2h0X3NpZ25hdHVyZV92YWxpZGF0b3I= is base64 for skip_thought_signature_validator: London and Tokyo were sent claiming a signature the model never produced

After (a5ad22b, the PR tip)

Turn 1 returns three parallel tool calls, same shape as before, a signature only on the first id:

{"city": "Paris"}  id=call_2351596__thought__AY89a1/YoawiNHzbcgAo4j41xnvh25ItNyHlm...
{"city": "London"}  id=call_2351597
{"city": "Tokyo"}  id=call_2351598

Turn 2 answers normally (200, finish_reason: stop):

The weather in Paris, London, and Tokyo is currently:

* **Paris:** 20°C
* **London:** 20°C
* **Tokyo:** 20°C

The model parts of that request as the relay recorded them on their way to Vertex:

get_weather({"city":"Paris"})   thoughtSignature=AY89a1/YoawiNHzbcgAo4j41xnvh25ItNyHlmafAJ8Ln...  (the model's real signature, 452 chars)
get_weather({"city":"London"})  no thoughtSignature field
get_weather({"city":"Tokyo"})   no thoughtSignature field

Paris keeps its real signature and the two unsigned siblings are replayed bare, matching what Gemini returned. No placeholder anywhere in the request

Notes from the runs, both pre-existing behavior this PR leaves alone:

• Real signature also rides inside the first tool_call id
• Model parts serialize snake_case (function_call); Vertex accepts them

Type

🐛 Bug Fix

Caveats (if any)

• Placeholder still applies when the whole turn is unsigned, since Vertex returns a 400 on a turn with no signature anywhere. That keeps history migrated off pre-gemini-3 models working
• The gemini-3 gate on the placeholder is pre-existing behavior from Add thought signature support to v1/messages api #16812 and is unchanged here, only narrowed to the first call
• Through the proxy, a signature replayed inside a tool call id does not reach this code, so provider_specific_fields.thought_signature is the carrier that works there. Pre-existing and unchanged by this PR: the first call still gets a placeholder and the siblings go bare, which Vertex accepts

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• a5ad22b passes /live-pr-risk

[greptile-apps]

Greptile Summary

The PR narrows Gemini placeholder thought signatures to the first parallel function call while preserving real signatures on any call

• Moves placeholder selection out of the signature-extraction helper and applies it while constructing Gemini parts
• Adds regression coverage for unsigned, signed, reordered, prefixed, and full-history parallel tool-call scenarios

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/prompt_templates/factory.py	Extracts only real signatures and limits Gemini-3 placeholder fallback to the first emitted function-call part
litellm/llms/vertex_ai/gemini/transformation.py	Updates signature-collection documentation to reflect that synthesized placeholders are no longer returned by the extraction helper
tests/test_litellm/llms/vertex_ai/gemini/test_vertex_ai_gemini_transformation.py	Adds focused coverage for parallel-call signature preservation and placeholder scoping across Gemini model variants

_{Reviews (2): Last reviewed commit: "test(vertex_ai): cover gemini-3.5-flash ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing ljogeiger:litellm_vertex_parallel_fc_thought_signatures (a5ad22b) with litellm_internal_staging (f22eeb2)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (629d768) during the generation of this report, so f22eeb2 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a5ad22b. Configure here.}

[mateo-berri]

LGTM. Thanks for the contribution!