fix(vertex_ai): return create_vertex_url result directly for openai-path partner models with custom api_base by mateo-berri · Pull Request #32380 · BerriAI/litellm

mateo-berri · 2026-07-07T20:59:43Z

Relevant issues

Linear ticket

Resolves LIT-4240

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live LiteLLM proxy on localhost (.venv/bin/python litellm/proxy/proxy_cli.py --config config.yaml --port <port> --detailed_debug), making real paid Vertex AI MiniMax MaaS calls (vertex_ai/minimaxai/minimax-m2-maas, location global); no mocks anywhere. The config has two custom api_base models plus a control with no api_base

model_list:
  - model_name: minimax-psc
    litellm_params:
      model: vertex_ai/minimaxai/minimax-m2-maas
      vertex_project: vertex-check-481318
      vertex_location: global
      api_base: https://aiplatform.googleapis.com
  - model_name: minimax-psc-path
    litellm_params:
      model: vertex_ai/minimaxai/minimax-m2-maas
      vertex_project: vertex-check-481318
      vertex_location: global
      api_base: http://127.0.0.1:53159/vertex
  - model_name: minimax-control
    litellm_params:
      model: vertex_ai/minimaxai/minimax-m2-maas
      vertex_project: vertex-check-481318
      vertex_location: global

minimax-psc-path exercises a path-bearing api_base the way a PSC or gateway host in front of Vertex sees it: a local nginx listener on 127.0.0.1:53159 forwards /vertex/ to the real https://aiplatform.googleapis.com/, so every request that reaches it hits the real Google API and returns real completions

location /vertex/ {
    proxy_pass https://aiplatform.googleapis.com/;
    proxy_set_header Host aiplatform.googleapis.com;
    proxy_ssl_server_name on;
}

Before (litellm_internal_staging, commit `b8248a2`)

The path-bearing api_base fails with a 404, non-streaming and streaming, because the URL builder splits the already-complete URL on : and grafts everything after the scheme back onto the api_base

$ curl -sS http://localhost:53007/v1/chat/completions \
    -H 'Authorization: Bearer sk-qa-1234' -H 'Content-Type: application/json' \
    -d '{"model": "minimax-psc-path", "messages": [{"role": "user", "content": "Reply with exactly: PSC OK"}], "max_tokens": 4096}'
{"error":{"message":"litellm.NotFoundError: Vertex_aiException - <html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.31.2</center>\r\n</body>\r\n</html>\r\n. Received Model Group=minimax-psc-path\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"404"}}

$ curl -sS http://localhost:53007/v1/chat/completions \
    -H 'Authorization: Bearer sk-qa-1234' -H 'Content-Type: application/json' \
    -d '{"model": "minimax-psc-path", "messages": [{"role": "user", "content": "Reply with exactly: PSC OK"}], "stream": true, "max_tokens": 4096}'
{"error":{"message":"litellm.NotFoundError: Vertex_aiException - <html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.31.2</center>\r\n</body>\r\n</html>\r\n. Received Model Group=minimax-psc-path\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"404"}}

The proxy log (proxy-before.log) shows the malformed URLs LiteLLM actually called, with the port and path duplicated after a stray colon, plus the ?alt=sse suffix on the streaming call

POST Request Sent from LiteLLM:
curl -X POST \
http://127.0.0.1:53159/vertex:53159/vertex/v1/projects/vertex-check-481318/locations/global/endpoints/openapi/chat/completions \
-H 'Authorization: Be****06' -H 'Content-Type: application/json' \

POST Request Sent from LiteLLM:
curl -X POST \
http://127.0.0.1:53159/vertex:53159/vertex/v1/projects/vertex-check-481318/locations/global/endpoints/openapi/chat/completions?alt=sse \
-H 'Authorization: Be****06' -H 'Content-Type: application/json' \

To be fully transparent about the bare-host model: minimax-psc succeeds on the base branch for both non-streaming and streaming, because #32367 (already on the base branch) grafts the default path onto bare-host api_bases and because the Google endpoint happens to tolerate the extra ?alt=sse query param that the custom-api_base streaming path appends. The streaming URL on base still diverges from what the default no-api_base path sends

https://aiplatform.googleapis.com/v1/projects/vertex-check-481318/locations/global/endpoints/openapi/chat/completions?alt=sse \

After (litellm_fix_vertex_psc_partner_api_base, commit `64c4cfe`)

Same config and the exact same curls against the fix branch (proxy restarted on port 53279). The previously failing path-bearing calls now return real completions

$ curl -sS http://localhost:53279/v1/chat/completions \
    -H 'Authorization: Bearer sk-qa-1234' -H 'Content-Type: application/json' \
    -d '{"model": "minimax-psc-path", "messages": [{"role": "user", "content": "Reply with exactly: PSC OK"}], "max_tokens": 4096}'
{"id":"qGtNasyZEbnN998P9YbJmAc","created":1783458729,"model":"minimax-psc-path","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"<think>The user asks: \"Reply with exactly: PSC OK\". [... thinking content truncated ...]\n</think>\n\nPSC OK","role":"assistant"}}],"usage":{"completion_tokens":208,"prompt_tokens":28,"total_tokens":236,"prompt_tokens_details":{"cached_tokens":26},"extra_properties":{"google":{"traffic_type":"ON_DEMAND"}}}}

$ curl -sS http://localhost:53279/v1/chat/completions \
    -H 'Authorization: Bearer sk-qa-1234' -H 'Content-Type: application/json' \
    -d '{"model": "minimax-psc-path", "messages": [{"role": "user", "content": "Reply with exactly: PSC OK"}], "stream": true, "max_tokens": 4096}'
data: {"id":"smtNap20IPOg9LsP1ffp0A8","created":1783458739,"model":"minimax-psc-path","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"provider_specific_fields":{},"content":"","role":"assistant"}}]}

data: {"id":"smtNap20IPOg9LsP1ffp0A8","object":"chat.completion.chunk","created":1783458739,"model":"minimax-psc-path","choices":[{"index":0,"delta":{"content":"<think>We"}}]}

[... SSE chunks truncated ...]

data: {"id":"smtNap20IPOg9LsP1ffp0A8","object":"chat.completion.chunk","created":1783458739,"model":"minimax-psc-path","choices":[{"index":0,"delta":{"content":"PSC OK"}}]}

data: {"id":"smtNap20IPOg9LsP1ffp0A8","object":"chat.completion.chunk","created":1783458739,"model":"minimax-psc-path","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

The proxy log (proxy-after.log) now shows a single clean host and path for every call, with no ?alt=sse divergence on streaming; these are the only outbound URL shapes across the whole after run

http://127.0.0.1:53159/vertex/v1/projects/vertex-check-481318/locations/global/endpoints/openapi/chat/completions \
https://aiplatform.googleapis.com/v1/projects/vertex-check-481318/locations/global/endpoints/openapi/chat/completions \

The bare-host minimax-psc also succeeded non-streaming and streaming on the fix branch, and the minimax-control model with no api_base succeeded in both the before and the after runs

Type

🐛 Bug Fix

Changes

Users calling Vertex AI partner models served through the OpenAI-compatible route (Llama, DeepSeek, Qwen, MiniMax, Moonshot, ZAI, GPT-OSS, Gemma MaaS) with a custom api_base, for example a GCP Private Service Connect host like https://aiplatform-myendpoint.p.googleapis.com, got a malformed request URL with the host duplicated (https://HOST://HOST/v1/projects/PROJ/locations/LOC/endpoints/openapi/chat/completions) and the call failed with a 404

Root cause: VertexBase.get_complete_vertex_url first builds the complete, correct URL via create_vertex_url (which already honors the custom api_base), then splits that URL on : to extract a :rawPredict style verb and re-prefixes it onto the custom api_base through _check_custom_proxy. The OpenAI-compatible path has no :verb suffix, so the split grabs everything after the scheme and the host gets duplicated. Streaming requests through this path also picked up a ?alt=sse suffix that the default no-custom-api_base path never sends to the openapi chat completions endpoint

The fix returns the create_vertex_url result directly for this partner path, mirroring what litellm/llms/vertex_ai/vertex_model_garden/main.py already does for the same reason. The rawPredict style partners (mistral, ai21, claude) are untouched and keep the existing {api_base}:{endpoint} custom proxy behavior

Regression tests in tests/test_litellm/llms/vertex_ai/test_vertex_llm_base.py assert the exact URL for a PSC style bare host, a path-bearing api_base, streaming parity (no ?alt=sse), the default no-api_base path, and the unchanged mistral {api_base}:rawPredict format

…ath partner models with custom api_base

greptile-apps · 2026-07-07T21:04:32Z

Greptile Summary

Fixes a malformed URL bug for Vertex AI OpenAI-compatible partner models (Llama, DeepSeek, MiniMax, etc.) when a custom api_base is provided: get_complete_vertex_url now returns the result of create_vertex_url directly for the llama partner type, bypassing the :split reconstruction that was doubling the host and incorrectly appending ?alt=sse to streaming requests.

vertex_llm_base.py: 2-line early return for VertexPartnerProvider.llama before the :split logic; mistralai, ai21, and claude partners (which use :rawPredict/:streamRawPredict suffixes) continue through the unchanged _check_custom_proxy path.
test_vertex_llm_base.py: Five new unit tests covering PSC bare-host, path-bearing api_base, streaming parity, default (no api_base), and a mistralai regression guard — all mock-only.

Confidence Score: 5/5

Safe to merge — the change is a 2-line targeted early return that removes a URL-reconstruction branch for exactly the partner type that lacked a :verb suffix, while leaving all three rawPredict-style partners untouched.

The root cause analysis in the PR description is accurate: create_vertex_url already builds the correct full URL from api_base, so re-splitting on : to extract a pseudo-verb was inherently lossy for the OpenAI-compatible path. The early return mirrors the pre-existing pattern in vertex_model_garden/main.py. All edge cases (bare host, path-bearing host, streaming, default with no api_base, mistralai regression) are covered by the new tests. No other partner providers are affected — ai21 and claude both carry :rawPredict/:streamRawPredict suffixes that make the split logic work correctly for them.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/llms/vertex_ai/vertex_llm_base.py	Adds a 2-line early return in get_complete_vertex_url for the llama (OpenAI-compatible path) partner, bypassing the :split logic that duplicated the host when a custom api_base was provided
tests/test_litellm/llms/vertex_ai/test_vertex_llm_base.py	Adds five parametrized tests covering: PSC bare-host api_base, PSC bare-host streaming, path-bearing api_base, default (no api_base), and mistralai rawPredict regression; all tests are mock-only with no network calls

_{Reviews (1): Last reviewed commit: "fix(vertex_ai): return create_vertex_url..." | Re-trigger Greptile}

codecov · 2026-07-07T21:05:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

codspeed-hq · 2026-07-07T21:20:28Z

Merging this PR will not alter performance

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 27 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`test_completion_multi_turn`	3.1 ms	4.2 ms	-25.3%
❌	`test_completion_simple_message`	4.2 ms	4.8 ms	-12.9%
⚡	`test_completion_with_tools`	4.2 ms	3.2 ms	+31.39%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing litellm_fix_vertex_psc_partner_api_base (64c4cfe) with litellm_internal_staging (b8248a2)}

…d of model refusal wording (#32388) test_text_message_blocked_by_guardrail_no_ai_response classified the model's reply against a safe_markers keyword list to decide whether the guardrail had blocked the message. gpt-realtime words its refusal of the guardrail's "say exactly" voice prompt nondeterministically, so any new phrasing outside the list turned CI red on unrelated PRs; the list had already been extended in #28191, #28200 and #29477, and drifted again to "Sorry, I can't comply with that request" (11 of the 13 failed realtime_translation_testing runs since 2026-06-24, e.g. CircleCI job 2009316 on #32380). Record every frame the proxy sends to the backend through a RecordingBackendWebSocket wrapper and assert the invariant the product actually guarantees: the blocked phrase never reaches OpenAI, only the guardrail's own conversation.item.create and response.create are forwarded (the client's reflexive response.create is dropped), and the blocked phrase never appears in AI output. Replace the fixed 0.3s/3.0s sleeps with an event-driven wait for response.done; client frames are processed sequentially so no inter-message sleep is needed. Verified by mutation: disabling the response.create drop fails the response.create count assertion, and disabling the guardrail fails the guardrail_violation assertion.

fix(vertex_ai): return create_vertex_url result directly for openai-p…

64c4cfe

…ath partner models with custom api_base

tin-berri approved these changes Jul 7, 2026

View reviewed changes

mateo-berri merged commit 46d9742 into litellm_internal_staging Jul 7, 2026
126 of 127 checks passed

mateo-berri deleted the litellm_fix_vertex_psc_partner_api_base branch July 7, 2026 21:49

mateo-berri mentioned this pull request Jul 7, 2026

test(realtime): assert guardrail block on backend wire traffic instead of model refusal wording #32388

Merged

4 tasks

Navigation Menu

Uh oh!

Uh oh!

fix(vertex_ai): return create_vertex_url result directly for openai-path partner models with custom api_base - #32380

fix(vertex_ai): return create_vertex_url result directly for openai-path partner models with custom api_base#32380
mateo-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_fix_vertex_psc_partner_api_base

mateo-berri commented Jul 7, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 7, 2026

Important Files Changed

Uh oh!

codecov Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

mateo-berri commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (litellm_internal_staging, commit b8248a2)

After (litellm_fix_vertex_psc_partner_api_base, commit 64c4cfe)

Type

Changes

Uh oh!

greptile-apps Bot commented Jul 7, 2026

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

codecov Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq Bot commented Jul 7, 2026

Merging this PR will not alter performance

Performance Changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mateo-berri commented Jul 7, 2026 •

edited

Loading

Before (litellm_internal_staging, commit `b8248a2`)

After (litellm_fix_vertex_psc_partner_api_base, commit `64c4cfe`)

codecov Bot commented Jul 7, 2026 •

edited

Loading