feat(vertex_ai): add Google Cloud Speech-to-Text Chirp 3 transcription support

[mateo-berri]

Relevant issues

Linear ticket

Resolves LIT-3682

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live before/after demonstration on a local proxy hitting the real Google Speech-to-Text v2 API with application default credentials, LITELLM_LOCAL_MODEL_COST_MAP=True, and the 17.6 second tests/gettysburg.wav clip. The proxy config maps chirp-3 to vertex_ai/chirp_3 with only vertex_project set, so the us location default is what gets exercised

Before commit: b4a10fb134ac0de14a642efc7a5a5adde95c7b6e (merge base, feature absent). After commit: 2b8706776f4b25f5745e819232166b8baf04a3cf (branch head). The transcription and security command outputs below were captured at aaf8b1f0abaa36cde7afb511c2661aa7af9178fb, the security-fix commit; the head commit only adds response_format validation on top and leaves these results unchanged

model_list:
  - model_name: chirp-3
    litellm_params:
      model: vertex_ai/chirp_3
      vertex_project: os.environ/VERTEXAI_PROJECT

Before, on the merge base, the request fails with an unmapped provider error:

$ git rev-parse --short HEAD
b4a10fb134
$ .venv/bin/python litellm/proxy/proxy_cli.py --config qa_config.yaml --port 61687 --detailed_debug
$ curl -sS http://localhost:61687/v1/audio/transcriptions \
    -H "Authorization: Bearer sk-qa-1234" \
    -F model="chirp-3" \
    -F file=@tests/gettysburg.wav
{"error":{"message":"litellm.APIConnectionError: Unmapped provider passed in. Unable to get the response.\n[...traceback trimmed...]\nValueError: Unmapped provider passed in. Unable to get the response.\n. Received Model Group=chirp-3\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}}

After, on this branch, the identical request returns HTTP 200 with the transcript in the OpenAI response shape:

$ git rev-parse --short HEAD
aaf8b1f0ab
$ .venv/bin/python litellm/proxy/proxy_cli.py --config qa_config.yaml --port 61687 --detailed_debug
$ curl -sS http://localhost:61687/v1/audio/transcriptions \
    -H "Authorization: Bearer sk-qa-1234" \
    -F model="chirp-3" \
    -F file=@tests/gettysburg.wav
{"text":"Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived and so dedicated, can long endure.","usage":null,"task":"transcribe","language":"en","duration":18.0}

An explicit OpenAI style bare ISO-639-1 language also works and gets normalized to the BCP-47 code Google requires, so language="en" returns 200 with the language echoed back as en-US:

$ curl -sS http://localhost:61687/v1/audio/transcriptions \
    -H "Authorization: Bearer sk-qa-1234" \
    -F model="chirp-3" \
    -F file=@tests/gettysburg.wav \
    -F language="en"
{"text":"Four score and seven years ago our fathers brought forth on this continent a new nation, [...] can long endure.","usage":null,"task":"transcribe","language":"en-US","duration":18.0}

Cost tracking prices the 18 billed seconds at Google's $0.016 per minute rate; both successful requests in the run logged the same figure:

$ grep -o "response_cost[': ]*[0-9.e-]*" proxy.log | sort | uniq -c
   2 response_cost: 0.00480006

Security: client-controlled location is validated

vertex_location is client-controllable on the proxy and lands in the request host, so a crafted value could otherwise redirect the outbound call, along with the admin-minted Google bearer token, to an attacker host. On this branch every non-conforming location is rejected with a 400 before any request is built, and the token never leaves the proxy. Three malicious values, a hostname with a trailing slash, one with a fragment, and one pointing at a local listener, all return the same 400:

$ for loc in "attacker.example/" "evil.com#" "127.0.0.1:61688/"; do
    curl -sS -w " [HTTP %{http_code}]\n" http://localhost:61687/v1/audio/transcriptions \
      -H "Authorization: Bearer sk-qa-1234" \
      -F model="chirp-3" -F file=@tests/gettysburg.wav \
      -F vertex_location="$loc"
  done
{"error":{"message":"litellm.BadRequestError: Vertex_aiException BadRequestError - Invalid vertex_location format. Received Model Group=chirp-3\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"400"}} [HTTP 400]
{"error":{"message":"litellm.BadRequestError: Vertex_aiException BadRequestError - Invalid vertex_location format. Received Model Group=chirp-3\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"400"}} [HTTP 400]
{"error":{"message":"litellm.BadRequestError: Vertex_aiException BadRequestError - Invalid vertex_location format. Received Model Group=chirp-3\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"400"}} [HTTP 400]

To prove no egress, a throwaway listener ran on 127.0.0.1:61688 and its log was truncated to empty before firing the 127.0.0.1:61688/ case above. After the malicious requests the listener log is still empty, so the token never reached it:

$ python3 -m http.server 61688 --bind 127.0.0.1 > listener.log 2>&1 &   # started earlier, log truncated to empty
$ wc -l < listener.log
       0

The proxy debug log confirms the only outbound Speech-to-Text hosts were Google's own regional endpoints; the attacker strings appear solely as echoes of the rejected input, never in an outbound URL:

$ grep -oE "https://[a-z0-9.-]*speech.googleapis.com[^ \"']*" proxy.log | sort | uniq -c
   3 https://us-central1-speech.googleapis.com/v2/projects/<project>/locations/us-central1/recognizers/_:recognize
   2 https://us-speech.googleapis.com/v2/projects/<project>/locations/us/recognizers/_:recognize

A well-formed location is still honored: vertex_location="us-central1" builds a real us-central1-speech.googleapis.com request, which Google answers with a benign 400 because chirp_3 is not served from that region, not an SSRF:

$ curl -sS -w "\nHTTP_STATUS:%{http_code}\n" http://localhost:61687/v1/audio/transcriptions \
    -H "Authorization: Bearer sk-qa-1234" \
    -F model="chirp-3" -F file=@tests/gettysburg.wav \
    -F vertex_location="us-central1"
{"error":{"message":"litellm.BadRequestError: Vertex_aiException BadRequestError - {\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"The model \\\"chirp_3\\\" does not exist in the location named \\\"us-central1\\\"..\",\n    \"status\": \"INVALID_ARGUMENT\"\n  }\n}\n. Received Model Group=chirp-3\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"400"}}
HTTP_STATUS:400

Type

🆕 New Feature

Changes

Adds Google Cloud Speech-to-Text support for Chirp 3, so vertex_ai/chirp_3 works on /v1/audio/transcriptions through both litellm.transcription() and litellm.atranscription() (the async path is what the proxy uses). The new VertexAIAudioTranscriptionConfig is wired through ProviderConfigManager and calls the Speech-to-Text v2 recognize endpoint. Authentication reuses the standard Vertex AI credential resolution (vertex_project, vertex_location, vertex_credentials, or application default credentials), so users configure it exactly like any other vertex_ai model

The location defaults to the us multi-region because Google serves chirp_3 only from the us and eu multi-regions, and non-global locations are routed to the regional <location>-speech.googleapis.com host. The OpenAI language param maps to languageCodes, defaulting to auto (Chirp 3's language-agnostic transcription). Because OpenAI clients send bare ISO-639-1 codes like en while Google rejects anything that is not region-qualified BCP-47, a shared normalize_transcription_language_to_bcp47 helper in litellm_core_utils/audio_utils maps common bare codes (en becomes en-US) and passes region-qualified codes through unchanged; the NVIDIA Riva transcription config had already hand-rolled the same mapping privately, so the shared helper gives future providers one place to reuse. The response joins the top alternative of every result into the transcript, and duration comes from the totalBilledDuration Google returns, so cost tracking uses Google's own billing number

A vertex_ai/chirp_3 cost map entry is added at Google's published $0.016 per minute for V2 standard recognition. The entry deliberately omits output_cost_per_second: the shared cost_per_second calculator prefers output_cost_per_second whenever it is not None, so a 0.0 there (the shape several existing STT entries use) silently prices every request at $0.00. A regression test computes the chirp_3 transcription cost from model info so that exact failure cannot recur

Because vertex_location is client-controllable on the proxy and lands in the request host, get_complete_url validates it before use so a caller cannot steer the request (and its admin-minted Google bearer token) at an attacker host. The ^[a-z][a-z0-9-]*$ plus global check that the rest of vertex_ai already applied through get_vertex_base_url is factored into a shared validate_vertex_location helper in common_utils.py and called from both the chat host builder and the new speech host builder, so there is one validation, not two copies. An invalid location raises a 400 VertexAIError rather than falling back silently, and vertex_project (which lands in the URL path) is likewise rejected if it carries path-structural characters. Tests assert on the parsed netloc so the security property is pinned, not just the string

Like every other vertex_ai flow, authentication needs google-auth at request time, which is not part of the base install; bare proxy installs should use the google extra (pip install "litellm[google]"), the same requirement vertex_ai chat and TTS models already have

Wiring this end to end surfaced a latent bug in the shared audio transcription HTTP handler: dict request bodies were passed to httpx's data= parameter, which form-encodes them and silently ignores the json= argument the handler also passed, so the JSON body path had never actually worked (no existing provider on the generic path returned dict data). The handler now sends JSON when a provider returns dict data without files, in both the sync and async paths; providers using multipart or raw bytes are unaffected. Regression tests pin the JSON body behavior at the handler level and the full request/response transformation (regional URL building, language mapping, transcript joining, billed duration parsing, provider routing) at the Vertex config level

Example usage:

curl http://localhost:4000/v1/audio/transcriptions \
  -H "Authorization: Bearer sk-1234" \
  -F model="vertex_ai/chirp_3" \
  -F file="@audio.wav"

---

Note

Medium Risk
Touches proxy-facing Vertex URL construction and OAuth-backed outbound calls, with deliberate SSRF guards; the shared transcription HTTP change affects all JSON-body providers but is covered by regression tests.

Overview
Adds vertex_ai/chirp_3 on /v1/audio/transcriptions via a new VertexAIAudioTranscriptionConfig that calls Google Speech-to-Text v2 recognize (base64 JSON body, OAuth headers, regional *-speech.googleapis.com URLs). OpenAI language maps to languageCodes (default auto); bare ISO codes are normalized through shared normalize_transcription_language_to_bcp47. Responses are shaped as OpenAI transcripts with optional language and billed duration.

Security: validate_vertex_location is centralized in common_utils (used by chat base URLs and speech URLs) so client-controlled vertex_location / vertex_project cannot inject hosts or path segments before outbound Google calls.

HTTP fix: Sync/async transcription handlers now send dict payloads with json= only (not data= alongside json=), fixing JSON-body providers.

Pricing: vertex_ai/chirp_3 is registered with input_cost_per_second for transcription billing; tests cover cost from duration.

^{Reviewed by Cursor Bugbot for commit 2b87067. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mateo-berri]

@greptileai

[greptile-apps]

Greptile Summary

This PR adds Google Cloud Speech-to-Text Chirp 3 transcription support via vertex_ai/chirp_3, wired through the existing ProviderConfigManager and BaseAudioTranscriptionConfig pattern. It also fixes a latent bug in the shared audio transcription HTTP handler where dict bodies were passed to both data= and json= simultaneously, causing httpx to silently form-encode the body instead of sending JSON.

• New VertexAIAudioTranscriptionConfig (litellm/llms/vertex_ai/audio_transcription/transformation.py): builds the Speech-to-Text v2 recognize URL, base64-encodes audio, maps OpenAI language param to BCP-47 languageCodes, validates vertex_location and vertex_project before interpolating them into the host/path, and parses totalBilledDuration for cost tracking. A new normalize_transcription_language_to_bcp47 helper is shared in audio_utils.
• HTTP handler fix (litellm/llms/custom_httpx/llm_http_handler.py): dict request data is now routed exclusively through json= (never data=) when no files are present, fixing a previously broken JSON-body code path that affected all providers using that pattern.
• Cost map entry adds vertex_ai/chirp_3 at Google's published $0.016/min rate using only input_cost_per_second (no output_cost_per_second: 0.0) to avoid the silent zero-cost regression documented in the PR.

Confidence Score: 5/5

Safe to merge — the changes are well-isolated, all new code paths are covered by mocked tests, and the security guards on client-controllable inputs are validated by parametrized injection tests.

The new Chirp 3 integration follows the established BaseAudioTranscriptionConfig pattern exactly, the httpx dict-body fix is mechanically correct and symmetrically applied to sync and async paths, and the cost map entry is structured to avoid the silent zero-cost trap. No pre-existing behavior is altered for other providers.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/llms/vertex_ai/audio_transcription/transformation.py	New VertexAIAudioTranscriptionConfig implementing the full request/response transformation, URL construction, and security validation for Chirp 3 STT. Logic is sound: location and project validated before URL interpolation, language normalization via shared helper, billed duration parsed for cost tracking.
litellm/llms/custom_httpx/llm_http_handler.py	Fixes latent bug where dict bodies were passed to both data= and json= in httpx calls; now routes dict data exclusively through json= when no files are present. Both sync and async paths corrected symmetrically.
litellm/llms/vertex_ai/common_utils.py	Extracts validate_vertex_location as a shared helper called by both the chat URL builder (get_vertex_base_url) and the new speech URL builder, eliminating duplicated validation logic. Behavior preserved; empty string and None are rejected as before.
litellm/litellm_core_utils/audio_utils/utils.py	Adds normalize_transcription_language_to_bcp47 shared helper and BARE_ISO_639_1_TO_BCP47 mapping table; unknown bare codes pass through unchanged so the downstream provider returns a clear error.
litellm/types/llms/vertex_ai_speech_to_text.py	New TypedDicts and Pydantic models for the Speech-to-Text v2 recognize request/response schema. Optional fields with defaults handle missing keys in API responses gracefully.
litellm/utils.py	Routes VERTEX_AI provider to the new VertexAIAudioTranscriptionConfig in ProviderConfigManager.get_provider_audio_transcription_config; lazy import keeps google-auth optional.
model_prices_and_context_window.json	Adds vertex_ai/chirp_3 entry with input_cost_per_second only (no output_cost_per_second: 0.0), correctly avoiding the silent zero-cost bug in the shared cost_per_second calculator.
tests/test_litellm/llms/vertex_ai/audio_transcription/test_vertex_ai_audio_transcription_transformation.py	Comprehensive mock-based test suite covering URL construction, security validation, request body shape, language mapping, response parsing, provider routing, and cost map integrity. No network calls.
tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py	Adds regression tests for the dict-body-as-JSON fix using httpx.MockTransport in both sync and async paths; correctly asserts content-type and parsed body.
tests/test_litellm/test_cost_calculator.py	Adds regression test for chirp_3 cost calculation that would have caught the output_cost_per_second:0.0 zero-cost bug; uses LITELLM_LOCAL_MODEL_COST_MAP to avoid network access.

_{Reviews (3): Last reviewed commit: "fix(vertex_ai): reject unsupported trans..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

Adds Vertex AI Chirp 3 speech-to-text support via the Google Speech-to-Text v2 recognize endpoint, wired through ProviderConfigManager and reusing the standard Vertex AI credential and location resolution. It also fixes a latent bug in the shared audio transcription HTTP handler where dict request bodies were form-encoded instead of JSON-encoded.

• New VertexAIAudioTranscriptionConfig: builds the regional Speech-to-Text v2 URL, base64-encodes the audio, maps language → languageCodes, and parses totalBilledDuration for cost tracking. The default location is us (Chirp 3's required multi-region).
• HTTP handler fix: data=dict was passed alongside json=dict in both sync and async post() calls; httpx form-encodes data=dict and silently ignores json=, so JSON-body providers were never working on the generic path. The fix passes the dict exclusively to json= when no files are present.
• response_format is declared in get_supported_openai_params but never read in the request transformation or applied to the response — non-JSON formats will silently return plain transcript text.

Confidence Score: 4/5

Safe to merge; the new provider integration is well-isolated, reuses existing Vertex auth infrastructure, and is covered by thorough mock tests. The HTTP handler change is a genuine bug fix with regression tests.

The core implementation is solid and the httpx bug fix is correct. The main gap is that response_format is listed as a supported parameter and passes validation, but is never consumed — callers passing srt, vtt, or verbose_json get plain text with no error. The missing __init__.py is inconsistent with all other audio-transcription provider directories in the repo.

litellm/llms/vertex_ai/audio_transcription/transformation.py — response_format handling and missing __init__.py

Important Files Changed

Filename	Overview
litellm/llms/vertex_ai/audio_transcription/transformation.py	New Vertex AI Speech-to-Text Chirp 3 transformation config; response_format is advertised as supported in get_supported_openai_params but is never read in the request or response transformation
litellm/llms/custom_httpx/llm_http_handler.py	Fixes latent bug where dict request body was sent to httpx's data= (form-encoding it) instead of json=; both sync and async paths corrected symmetrically
litellm/types/llms/vertex_ai_speech_to_text.py	New Pydantic/TypedDict types for Google Speech-to-Text v2 request/response; fields have appropriate Optional defaults for graceful handling of partial responses
litellm/utils.py	Wires VertexAIAudioTranscriptionConfig into ProviderConfigManager.get_provider_audio_transcription_config alongside existing providers
tests/test_litellm/llms/vertex_ai/audio_transcription/test_vertex_ai_audio_transcription_transformation.py	Comprehensive mock-only unit tests covering URL construction, request body shaping, response transformation, auth header population, provider routing, and cost-map entries
tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py	Adds regression tests for the JSON body fix; both sync and async paths are covered with a mock transport that captures content-type and body
model_prices_and_context_window.json	Adds vertex_ai/chirp_3 cost entry at Google's published $0.016/min, correctly represented as per-second pricing

_{Reviews (2): Last reviewed commit: "feat(vertex_ai): add Google Cloud Speech..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 94.73684% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ms/vertex_ai/audio_transcription/transformation.py	91.86%	7 Missing ⚠️

📢 Thoughts on this report? Let us know!

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[mateo-berri]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Vertex transcription routes all models
  • Gated the VERTEX_AI branch of get_provider_audio_transcription_config on model_cost_entry.get("mode") == "audio_transcription" so only entries marked as transcription models (e.g. vertex_ai/chirp_3) are routed to Google Speech-to-Text.

_{You can send follow-ups to the cloud agent here.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 2b87067. Configure here.}

[cursor]

Vertex transcription routes all models

Medium Severity

get_provider_audio_transcription_config returns VertexAIAudioTranscriptionConfig for every vertex_ai transcription call, without checking that the model is an audio-transcription entry (for example mode in the cost map). Before this change those calls failed as unmapped; now they are sent to Google Speech-to-Text with the chat model id in the payload.

 

^{Reviewed by Cursor Bugbot for commit 2b87067. Configure here.}

[mateo-berri]

every provider branch there routes unconditionally (OpenAI sends any model to whisper), nothing working is shadowed, and misuse now gets Google's clear 400 instead of "Unmapped provider"

[CLAassistant]

All committers have signed the CLA.

[yucheng-berri]

api_base on the new STT endpoint is client-controllable and wholesale replaces the validated host — so the SSRF fix that constrained vertex_location is bypassable by sending api_base="http://attacker.example". Proxy would POST the audio + its Google bearer + x-goog-user-project header to the attacker.

[mateo-berri]

api_base on the new STT endpoint is client-controllable and wholesale replaces the validated host — so the SSRF fix that constrained vertex_location is bypassable by sending api_base="http://attacker.example". Proxy would POST the audio + its Google bearer + x-goog-user-project header to the attacker.

api_base from /audio/transcriptions form data is rejected by the proxy request-body safety check unless explicitly enabled by admin configuration

[javimp2003uma]

hi @mateo-berri , how is it going? from what i know, google lets developer to build input streaming STT pipeline with https://docs.cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v1.types.StreamingRecognitionConfig

is it available in LiteLLM?