fix(vertex_ai): strip LiteLLM-internal keys from extra_body before merging to Gemini request

[Sameerlite]

Problem

After upgrading from litellm proxy 1.79.3 to 1.81.12, Vertex AI Gemini models fail when using proxy cache with extra_body={"cache": {"use-cache": True, "ttl": 86400}}:

Invalid JSON payload received. Unknown name "cache": Cannot find field.

Fixes #22970

Root Cause

PR #20950 added extra_body forwarding to the Vertex AI Gemini completion transformation. Before that, extra_body was silently dropped—so cache never reached Vertex AI. After the change, all keys from extra_body are merged into the request body. Vertex AI enforces a strict JSON schema and rejects unknown fields like cache and tags.

Solution

• Add _LITELLM_INTERNAL_EXTRA_BODY_KEYS frozenset (cache, tags)
• Skip these keys in _pop_and_merge_extra_body before merging into the Vertex AI request
• cache is consumed by LiteLLM's proxy response caching layer (caching.md)
• tags is consumed by LiteLLM's logging/tracking (litellm_proxy.md)

Testing

• test_extra_body_cache_not_forwarded_to_vertex_ai — verifies cache is stripped
• test_extra_body_tags_not_forwarded_to_vertex_ai — verifies tags is stripped
• Both tests ensure legitimate extra_body keys still pass through

Made with Cursor

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Error		Mar 9, 2026 4:53am

[greptile-apps]

Greptile Summary

This PR fixes a regression introduced in #20950 where LiteLLM-internal extra_body keys (cache, tags) were being forwarded to the Vertex AI Gemini API, causing 400 errors due to Vertex AI's strict JSON schema validation. The fix adds a _LITELLM_INTERNAL_EXTRA_BODY_KEYS frozenset and filters those keys inside _pop_and_merge_extra_body before the request body is sent upstream.

Key changes:

• litellm/llms/vertex_ai/gemini/transformation.py: Adds _LITELLM_INTERNAL_EXTRA_BODY_KEYS = frozenset({"cache", "tags"}) and skips those keys during extra_body merging.
• tests/…/test_vertex_ai_gemini_transformation.py: Adds two mock-only regression tests covering both stripped keys.
• litellm/model_prices_and_context_window_backup.json: Unrelated addition of gemini/gemini-3.1-flash-image-preview pricing metadata.

Issues found:

• The comment on line 532 is truncated mid-sentence ("must never be forwarded to the").
• The frozenset type annotation should be frozenset[str] rather than the bare frozenset.
• The fix is scoped only to the Vertex AI/Gemini provider. Any other provider that merges extra_body (now or in the future) will not benefit from this filtering and will still forward cache/tags to the upstream API. A more robust fix would centralise this at the extra_body pre-processing layer (e.g. litellm_pre_call_utils.py) so all providers are protected. Other potential internal keys like metadata / litellm_metadata are also not covered.

Confidence Score: 3/5

• The fix is correct and safe for Vertex AI/Gemini, but its narrow scope leaves other providers unprotected and the hardcoded key list may be incomplete.
• The core logic is sound and the tests verify the intended behaviour. However, the fix is only applied in the Vertex AI transformation layer — other providers that merge extra_body remain unprotected. The list of internal keys (cache, tags) may also be incomplete (e.g. metadata, litellm_metadata are also consumed internally). The unrelated model_prices_and_context_window_backup.json addition is benign but should be in a separate PR for clarity.
• Pay close attention to litellm/llms/vertex_ai/gemini/transformation.py — the hardcoded frozenset and its narrow scope are the main concerns.

Important Files Changed

Filename	Overview
litellm/llms/vertex_ai/gemini/transformation.py	Adds _LITELLM_INTERNAL_EXTRA_BODY_KEYS frozenset and filters cache/tags from extra_body before merging into the Vertex AI request body. The comment on line 532 is truncated, the frozenset type annotation lacks a subscript, and the fix is scoped only to this provider — other providers merging extra_body are unprotected.
tests/test_litellm/llms/vertex_ai/gemini/test_vertex_ai_gemini_transformation.py	Adds two unit tests verifying that cache and tags are stripped from extra_body before forwarding to Vertex AI, while legitimate keys pass through. Tests are mock-only (no real network calls) and are self-contained.
litellm/model_prices_and_context_window_backup.json	Adds pricing and capability metadata for gemini/gemini-3.1-flash-image-preview. Unrelated to the stated fix but the entry looks well-formed and consistent with neighboring model entries.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["Client request with extra_body\n{cache: {...}, tags: [...], custom_param: ...}"] --> B["_transform_request_body()"]
    B --> C["_pop_and_merge_extra_body(data, optional_params)"]
    C --> D{"key in _LITELLM_INTERNAL_EXTRA_BODY_KEYS?"}
    D -- "yes (cache, tags)" --> E["skip — key is LiteLLM-internal"]
    D -- "no (custom_param, ...)" --> F{"key already in data\nand both are dicts?"}
    F -- "yes" --> G["deep-merge: data[k].update(v)"]
    F -- "no" --> H["shallow-set: data[k] = v"]
    G --> I["Final Vertex AI RequestBody\n(no cache, no tags)"]
    H --> I
    E --> I
    I --> J["POST /generateContent → Vertex AI"]

Loading

_{Last reviewed commit: 4dc277e}

[greptile-apps]

Truncated comment

The comment on line 532 is cut off mid-sentence — it currently reads # Keys that LiteLLM consumes internally and must never be forwarded to the with no completion. This should be finished to explain what the keys must not be forwarded to.

Suggested change

	# Keys that LiteLLM consumes internally and must never be forwarded to the
	# Keys that LiteLLM consumes internally and must never be forwarded to the Vertex AI request body.

[greptile-apps]

Hardcoded internal-key list is Vertex AI-only and incomplete

_LITELLM_INTERNAL_EXTRA_BODY_KEYS is defined and consumed only inside the Vertex AI/Gemini transformation. Any other provider that merges extra_body (e.g. future providers added via _pop_and_merge_extra_body or similar helpers) will not benefit from this filter and will silently forward cache/tags to the upstream API, causing the same 400 errors this PR is trying to fix.

Additionally, LiteLLM's proxy layer reads other internal keys from extra_body (e.g. metadata, litellm_metadata) that are not currently included in this frozenset. If those keys are ever added to a user's extra_body, they would still be forwarded to Vertex AI.

A more robust approach would be to centralise this filtering at the point where extra_body is first processed (e.g. in litellm_pre_call_utils.py or in the base provider transformation), so all providers are protected. At the very least, consider whether metadata and litellm_metadata should also be included here.

[greptile-apps]

Type annotation on frozenset is redundant

The type annotation frozenset on the right-hand side literal is redundant since Python infers the type automatically from the frozenset(...) constructor call. More importantly, using the bare frozenset without a subscript loses the element type. Either omit the annotation or make it explicit:

Suggested change

	_LITELLM_INTERNAL_EXTRA_BODY_KEYS: frozenset = frozenset({"cache", "tags"})
	_LITELLM_INTERNAL_EXTRA_BODY_KEYS: frozenset[str] = frozenset({"cache", "tags"})

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}