fix(vertex_ai): route Model Garden models through the generateContent adapter

[devin-ai-integration]

TLDR

Problem this solves:

• Vertex Model Garden xai/grok-* 404s on :generateContent and :streamGenerateContent
• The proxy called the native Google publisher path for a non-Google model
• :countTokens for the same model returned 500

How it solves it:

• The native Gemini config now serves only the Gemini Vertex routes
• Every other Vertex route goes through the chat completions adapter
• Streamed adapter calls ask for usage, so they bill like chat
• The adapter keeps upstream error codes and the router's model group
• :countTokens on non-Gemini routes falls back to local counting

Decisions worth a look. The native config allowlists the Gemini routes, so every other route, openai/<endpoint id> included, calls the endpoint's chat completions API, since an xai/ special case leaves the next publisher on the 404. Stream usage is asked for only where the deployment supports stream_options, the proxy's rule for streamed chat. :countTokens uses the local estimate, since Model Garden has no count endpoint

User Flow

Before: a developer pointing the Google GenAI SDK at a Vertex Grok deployment gets errors on every Gemini-style call, while the same deployment answers on chat completions

1. They configure model_name: grok-4.6 with litellm_params.model: vertex_ai/xai/grok-4.6, and POST https://litellm-domain/v1/chat/completions answers 200
2. The SDK sends POST https://litellm-domain/v1beta/models/grok-4.6:generateContent and gets a 404 reading litellm.NotFoundError: Vertex_aiException
3. :streamGenerateContent 404s the same way, and a request with maxOutputTokens: -5 also 404s instead of rejecting the bad value
4. POST https://litellm-domain/v1beta/models/grok-4.6:countTokens returns 500 Internal server error
5. GET https://litellm-domain/key/info shows $0 spend on their key, since every call failed

After: the same SDK calls succeed and bill like chat completions, and bad input gets a 400

1. They configure model_name: grok-4.6 with litellm_params.model: vertex_ai/xai/grok-4.6, and POST https://litellm-domain/v1/chat/completions answers 200
2. The SDK sends POST https://litellm-domain/v1beta/models/grok-4.6:generateContent and gets a 200 with candidates[0].content.parts[0].text filled in and finishReason: STOP
3. :streamGenerateContent streams the same reply, and maxOutputTokens: -5 now gets a 400 from Vertex
4. POST https://litellm-domain/v1beta/models/grok-4.6:countTokens returns 200 with totalTokens
5. GET https://litellm-domain/key/info shows the streamed call's real spend, and GET https://litellm-domain/spend/logs/ui lists it under model group grok-4.6

Relevant issues

Fixes #41835

Linear ticket

Resolves LIT-8120

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live run against real Vertex AI on 2026-10-15 with the reporter's client, the Google GenAI SDK (google-genai 2.31.0), plus curl for chat and spend. Our project has no grok-4.6 access, so the deployment is vertex_ai/xai/grok-4.20-non-reasoning on the same Model Garden route. Each leg runs one 2-worker proxy on a fresh Postgres database

Case	Before (0ececab)	After (371d5b6)
grok streamGenerateContent and its spend	404, key spend $0	200 144, key spend $0.0000915
grok generateContent	404	200, 144
grok generateContent with max_output_tokens=-5	404	400 from Vertex
grok countTokens	500	200, total_tokens=20
gemini generateContent	200, 144	200, 144
claude generateContent	200, 144	200, 144
gemini streamGenerateContent	200, 144	200, 144
grok chat completions (control)	200, 144	200, 144
spend log rows	grok rows $0	grok rows $0.0000915 with model group

Shared setup

1. config.yaml

model_list:
  - model_name: grok-4.20-non-reasoning
    litellm_params:
      model: vertex_ai/xai/grok-4.20-non-reasoning
      vertex_project: os.environ/VERTEXAI_PROJECT
      vertex_location: global
  - model_name: gemini-3.8-flash
    litellm_params:
      model: vertex_ai/gemini-3.8-flash
      vertex_project: os.environ/VERTEXAI_PROJECT
      vertex_location: global
  - model_name: claude-haiku-4-5
    litellm_params:
      model: vertex_ai/claude-haiku-4-5@20251001
      vertex_project: os.environ/VERTEXAI_PROJECT
      vertex_location: us-east5
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

2. Create a fresh Postgres database, prisma db push the proxy schema into it, and boot python -m litellm.proxy.proxy_cli --config config.yaml --host 127.0.0.1 --port $PORT --num_workers 2 --timeout_worker_healthcheck 900 with DATABASE_URL pointing at it and GOOGLE_APPLICATION_CREDENTIALS pointing at the service-account key, then wait for GET /health/liveliness to answer 200
3. Generate the virtual key the streamed call uses: VIRTUAL_KEY=$(curl -s $BASE/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"key_alias": "lit8120-<leg>-grok-stream"}' | jq -r .key)
4. Client genai_client.py BASE KEY call|stream|bad|count MODEL

import json
import sys

from google import genai
from google.genai import errors, types

BASE, KEY, MODE, MODEL = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]
client = genai.Client(api_key=KEY, http_options={"base_url": BASE, "api_version": "v1beta"})
PROMPT = "What is 12*12? Reply with only the number."

try:
    if MODE == "call":
        r = client.models.generate_content(model=MODEL, contents=PROMPT)
        print(f"200 text={r.text!r} finish_reason={r.candidates[0].finish_reason} usage={r.usage_metadata.total_token_count}")
    elif MODE == "count":
        r = client.models.count_tokens(model=MODEL, contents=PROMPT)
        print(f"200 total_tokens={r.total_tokens}")
    elif MODE == "bad":
        r = client.models.generate_content(
            model=MODEL, contents=PROMPT, config=types.GenerateContentConfig(max_output_tokens=-5)
        )
        print(f"200 text={r.text!r}")
    else:
        chunks = [c.text or "" for c in client.models.generate_content_stream(model=MODEL, contents=PROMPT)]
        print(f"200 chunks={len(chunks)} text={''.join(chunks)!r}")
except errors.APIError as e:
    print(f"HTTP {e.code}: {json.dumps(e.message)[:400]}")

Before (0ececab)

grok streamGenerateContent and its spend

1. python genai_client.py $BASE $VIRTUAL_KEY stream grok-4.20-non-reasoning

HTTP 404: "litellm.NotFoundError: Vertex_aiException - \n\nLiteLLM: model group 'grok-4.20-non-reasoning' failed with the error above. No fallback was attempted."

2. curl -s "$BASE/key/info?key=$VIRTUAL_KEY" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c "{key_alias: .info.key_alias, spend: .info.spend}"

{"key_alias":"lit8120-before-grok-stream","spend":0.0}

grok generateContent

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY call grok-4.20-non-reasoning

HTTP 404: "litellm.NotFoundError: Vertex_aiException - \n\nLiteLLM: model group 'grok-4.20-non-reasoning' failed with the error above. No fallback was attempted."

grok generateContent with max_output_tokens=-5

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY bad grok-4.20-non-reasoning

HTTP 404: "litellm.NotFoundError: Vertex_aiException - \n\nLiteLLM: model group 'grok-4.20-non-reasoning' failed with the error above. No fallback was attempted."

grok countTokens

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY count grok-4.20-non-reasoning

HTTP 500: "Internal server error"

gemini generateContent

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY call gemini-3.8-flash

200 text='144' finish_reason=FinishReason.STOP usage=89

claude generateContent

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY call claude-haiku-4-5

200 text='144' finish_reason=FinishReason.STOP usage=25

gemini streamGenerateContent

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY stream gemini-3.8-flash

200 chunks=2 text='144'

grok chat completions (control)

Steps

1. curl -s $BASE/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"model": "grok-4.20-non-reasoning", "messages": [{"role": "user", "content": "What is 12*12? Reply with only the number."}]}' | jq -c "{model: .model, content: .choices[0].message.content, usage: .usage.total_tokens}"

{"model":"grok-4.20-non-reasoning","content":"144","usage":341}

spend log rows

Steps

1. curl -s "$BASE/spend/logs/ui?start_date=2026-10-15%2000:00:00&end_date=2026-10-16%2000:00:00" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c ".data | sort_by(.startTime) | .[] | {call_type, model, model_group, spend, prompt_tokens, completion_tokens}"

{"call_type":"agenerate_content_stream","model":"vertex_ai/xai/grok-4.20-non-reasoning","model_group":"grok-4.20-non-reasoning","spend":0.0,"prompt_tokens":20,"completion_tokens":0}
{"call_type":"agenerate_content","model":"vertex_ai/xai/grok-4.20-non-reasoning","model_group":"grok-4.20-non-reasoning","spend":0.0,"prompt_tokens":20,"completion_tokens":0}
{"call_type":"agenerate_content","model":"vertex_ai/xai/grok-4.20-non-reasoning","model_group":"grok-4.20-non-reasoning","spend":0.0,"prompt_tokens":20,"completion_tokens":0}
{"call_type":"agenerate_content","model":"vertex_ai/gemini-3.8-flash","model_group":"gemini-3.8-flash","spend":0.000317625,"prompt_tokens":15,"completion_tokens":74}
{"call_type":"acompletion","model":"claude-haiku-4-5@20251001","model_group":"","spend":0.0000495,"prompt_tokens":20,"completion_tokens":5}
{"call_type":"agenerate_content_stream","model":"vertex_ai/gemini-3.8-flash","model_group":"gemini-3.8-flash","spend":0.0002887499999999999,"prompt_tokens":15,"completion_tokens":74}
{"call_type":"acompletion","model":"vertex_ai/xai/grok-4.20-non-reasoning","model_group":"grok-4.20-non-reasoning","spend":0.0000915,"prompt_tokens":340,"completion_tokens":1}

After (371d5b6)

grok streamGenerateContent and its spend

1. python genai_client.py $BASE $VIRTUAL_KEY stream grok-4.20-non-reasoning

200 chunks=2 text='144'

2. curl -s "$BASE/key/info?key=$VIRTUAL_KEY" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c "{key_alias: .info.key_alias, spend: .info.spend}"

{"key_alias":"lit8120-after-grok-stream","spend":0.0000915}

grok generateContent

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY call grok-4.20-non-reasoning

200 text='144' finish_reason=FinishReason.STOP usage=341

grok generateContent with max_output_tokens=-5

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY bad grok-4.20-non-reasoning

HTTP 400: "litellm.BadRequestError: Vertex_aiException BadRequestError - {\"code\":\"Client specified an invalid argument\",\"error\":\"Maximum number of tokens max_tokens must be positive but max_tokens = -5\"}\n\nLiteLLM: model group 'grok-4.20-non-reasoning' failed with the error above. No fallback was attempted."

grok countTokens

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY count grok-4.20-non-reasoning

200 total_tokens=20

gemini generateContent

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY call gemini-3.8-flash

200 text='144' finish_reason=FinishReason.STOP usage=91

claude generateContent

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY call claude-haiku-4-5

200 text='144' finish_reason=FinishReason.STOP usage=25

gemini streamGenerateContent

Steps

1. python genai_client.py $BASE $LITELLM_MASTER_KEY stream gemini-3.8-flash

200 chunks=2 text='144'

grok chat completions (control)

Steps

1. curl -s $BASE/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"model": "grok-4.20-non-reasoning", "messages": [{"role": "user", "content": "What is 12*12? Reply with only the number."}]}' | jq -c "{model: .model, content: .choices[0].message.content, usage: .usage.total_tokens}"

{"model":"grok-4.20-non-reasoning","content":"144","usage":341}

spend log rows

Steps

1. curl -s "$BASE/spend/logs/ui?start_date=2026-10-15%2000:00:00&end_date=2026-10-16%2000:00:00" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c ".data | sort_by(.startTime) | .[] | {call_type, model, model_group, spend, prompt_tokens, completion_tokens}"

{"call_type":"acompletion","model":"vertex_ai/xai/grok-4.20-non-reasoning","model_group":"grok-4.20-non-reasoning","spend":0.0000915,"prompt_tokens":340,"completion_tokens":1}
{"call_type":"acompletion","model":"vertex_ai/xai/grok-4.20-non-reasoning","model_group":"grok-4.20-non-reasoning","spend":0.0000915,"prompt_tokens":340,"completion_tokens":1}
{"call_type":"agenerate_content","model":"grok-4.20-non-reasoning","model_group":"","spend":0.0,"prompt_tokens":0,"completion_tokens":0}
{"call_type":"agenerate_content","model":"vertex_ai/gemini-3.8-flash","model_group":"gemini-3.8-flash","spend":0.000325875,"prompt_tokens":15,"completion_tokens":76}
{"call_type":"acompletion","model":"vertex_ai/claude-haiku-4-5@20251001","model_group":"claude-haiku-4-5","spend":0.0000495,"prompt_tokens":20,"completion_tokens":5}
{"call_type":"agenerate_content_stream","model":"vertex_ai/gemini-3.8-flash","model_group":"gemini-3.8-flash","spend":0.0002849999999999999,"prompt_tokens":15,"completion_tokens":73}
{"call_type":"acompletion","model":"vertex_ai/xai/grok-4.20-non-reasoning","model_group":"grok-4.20-non-reasoning","spend":0.0000915,"prompt_tokens":340,"completion_tokens":1}

Observations from the run

• Streamed grok spend now equals the chat control's spend
• The failed 400 call logs a $0 row with no model group, new on this path
• countTokens says 20, while Vertex bills 340 prompt tokens, caused by this PR's local estimate
• Claude rows now carry their model group, which this PR fixes

Type

🐛 Bug Fix

Caveats (if any)

Low

• gemma/* and agent_engine/* deployments now use the adapter, not live-tested here
• Streamed generateContent usageMetadata reads 0 to the client, while billing is correct
  • Fixing it means buffering every chat-backed stream, net-negative now
• :countTokens on non-Gemini routes is a local estimate and skips server-side system prompts

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/22a1bdcb2f9c4c59bf930c39cf178e80
Open in Devin Desktop: https://app.devin.ai/desktop/session/22a1bdcb2f9c4c59bf930c39cf178e80?variant=devin

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed]

Merging this PR will not alter performance

⚠️ 1 benchmark spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• test_completion_streaming

✅ 31 untouched benchmarks

---

_{Comparing litellm_vertex_grok_generatecontent_route (e9270ba) with main (4d52918)¹}

Footnotes

1. No successful run was found on main (c70dd4e) during the generation of this report, so 4d52918 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge, with the previous live-provider proof requirement satisfied

Summary

Model Garden requests now use the chat completions adapter for generateContent and streaming. Native Gemini requests keep their existing config

• Vertex Model Garden requests use the chat adapter on Google’s generateContent endpoint.

_{Reviews (6) · Last reviewed commit: "fix(vertex_ai): type litellm_params in g..." · Reviewed by Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[devin-ai-integration]

@greptileai please re-review the merge commit da29aa8 that resolved the conflict with main in tests/test_litellm/test_utils.py

[devin-ai-integration]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[greptile-apps]

Automatic reviews are disabled for this repository.

[mateo-berri]

@litereview code-review

[litereview]

✅ code-review

5/5 · Ready for human review

Scope: full · Full PR review.

No correctness findings.

Checks and execution details

• Passed · Real testing evidence (required): Two changed files fall under the scoped paths.

• litellm/llms/databricks/streaming_utils.py now carries prompt_tokens_details and completion_tokens_details on streamed usage chunks. This is a usage translation change, and Vertex Model Garden streaming reaches it through OpenAILikeChatHandler.

• litellm/llms/vertex_ai/common_utils.py changes route selection and count-tokens routing.

The PR description includes inspectable manual E2E evidence: real Vertex AI on 2026-10-15, before and after SHAs 0ececab7a1 and 371d5b6fdb, a 2-worker proxy with Postgres, the google-genai 2.31.0 client script, the config, and the exact commands with outputs.

The streamed grok spend is $0.0000915 for 340 prompt and 1 completion tokens. I checked that figure against the cost map: 20×1.25e-6 + 320×2e-7 (cached) + 1×2.5e-6 = 9.15e-5. The streaming path therefore preserved the 320 cached tokens. Its spend also matches the chat-completions control row.

The countTokens leg returns 200 after the change, where it returned 500 before. The gemini and claude regression legs still return 200.

• Passed · Targeted unit tests at head (optional): I ran uv run --frozen pytest tests/unit/google_genai tests/unit/google_genai/test_google_genai_handler.py tests/unit/llms/databricks/test_databricks_streaming_utils.py tests/unit/llms/vertex_ai/test_vertex_ai_common_utils.py tests/unit/test_cost_calculator.py tests/unit/test_utils.py::test_vertex_generate_content_config_follows_model_route -n 4 after uv sync --frozen --all-extras --all-groups. Result: 572 passed.
• Passed · Cost-calculator blast radius (old vs new model_without_prefix) (optional): I ran completion_cost over all 1349 chat or completion cost-map entries whose provider-relative name contains /, once with the base logic and once with the head logic. Only 7 vertex_ai/xai/grok-* entries changed, and each went from ModelNotMappedError to a priced result. No other model's cost changed.
• Passed · Routing and caller trace (optional): I traced the routing from google_genai/main.py through get_provider_google_genai_generate_content_config to uses_vertex_gemini_api. Only the GEMINI and NON_GEMINI routes keep VertexAIGoogleGenAIConfig. Numeric endpoint IDs still route to NON_GEMINI and the native path, so their behavior is unchanged.

For :countTokens, VertexAITokenCounter.count_tokens now returns None for routes that are neither partner nor Gemini. Both callers, proxy_server._try_provider_token_count and main.py, then fall back to local counting.

The adapter handler no longer wraps errors in ValueError. Exceptions now reach litellm.exception_type in google_genai/main.py, which keeps the upstream status, such as the 400.

---

View review · Reviewed 371d5b6fdb5e

[mateo-berri]

@litereview code-review