Optional labels field in Vertex AI request

[vvidovic]

If the client sets the labels field in the request to the LiteLLM:

• pass the labels field to the Vertex AI backend

If the client sets the metadata field in the request to the LiteLLM:

• if the labels field is not set, fill it with metadata key/value pairs for all string values

Title

Optional labels field in Vertex AI request

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• I have added a screenshot of my new test passing locally
• My PR passes all unit tests on (make test-unit)[https://docs.litellm.ai/docs/extras/contributing_code]
• My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🆕 New Feature
✅ Test

Changes

• send the labels field to the Vertex AI backend (use the labels or metadata field from the client request)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
litellm	Ready	Preview	Comment	Sep 9, 2025 2:09am

[ghost]

Hi @vvidovic i'm confused were we not passing the labels field before?

Any non-openai param is passed along directly to the provider

[vvidovic]

Hi @vvidovic i'm confused were we not passing the labels field before?

Any non-openai param is passed along directly to the provider

No, unfortunately, non-openai params are not passed directly to the provider.
I did a few tests using the latest stable model (ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.63.2-stable) and used the mitmproxy to intercept all calls to Vertex AI and local Ollama models.

Here is the configuration excerpt for the two models from the example calls below:

model_list:
  - model_name: granite3-dense
    litellm_params:
      model: ollama/granite3-dense:2b
      api_base: http://host.docker.internal:11435
      input_cost_per_token: 0.00005
      output_cost_per_token: 0.00005
  - model_name: gemini-pro
    litellm_params:
      model: vertex_ai/gemini-1.5-pro
      api_base: http://host.docker.internal:11435/
      vertex_project: eu-p-ustd-its-genaigw-7144
      # vertex_location: 
      vertex_credentials: /app/eu-p-ustd-its-genaigw-7144-ee0bb89f05bf.json
      input_cost_per_token: 0.00005
      output_cost_per_token: 0.00005

Here is the curl call for the Vertex AI configured gemini-pro model:

curl -s http://0.0.0.0:4000/chat/completions -H "Authorization: Bearer $ut2" -H 'Content-Type: application/json' -d '{ "model": "gemini-pro", "messages": [ { "role": "user", "content": "respond in 20 words. who are you?" } ], "user": "vedran", "metadata": { "md_key": "md_value" }, "labels": {"client": "my label"}, "seed": 42, "new_key1": "new value 1", "new_key2": "new value 2", "new_map": {"mk1": "mv1", "mk2": "mv2"} }'

The request sent to the backend, as visible in the mitmproxy is:

Flow Details
2025-03-13 07:34:35 POST http://localhost:11434/:generateContent                                                                                                                                                                                                                                               
                         ← 404 Not Found text/plain 18b 5ms
                                               Request                                                                                               Response                                                                                               Detail
Host:             localhost:11434                                                                                                                                                                                                                                                                              
Accept:           */*                                                                                                                                                                                                                                                                                          
Accept-Encoding:  gzip, deflate                                                                                                                                                                                                                                                                                
Connection:       keep-alive                                                                                                                                                                                                                                                                                   
User-Agent:       litellm/1.63.2                                                                                                                                                                                                                                                                               
Content-Type:     application/json                                                                                                                                                                                                                                                                             
Authorization:    Bearer ***                                                                                                          
Content-Length:   111                                                                                                                                                                                                                                                                                          
JSON                                                                                                                                                                                                                                                                                                     
{
    "contents": [
        {
            "parts": [
                {
                    "text": "Why is the sky blue?"
                }
            ],
            "role": "user"
        }
    ],
    "generationConfig": {
        "seed": 42
    }
}

Note: I didn't pass this request to the real Google Vertex AI during this test but configured the backend for both, Ollama and Vertex AI to be a local Ollama for debugging purposes.

Here is the curl call for the Ollama configured granite3-dense model:

curl -s http://0.0.0.0:4000/chat/completions -H "Authorization: Bearer $ut2" -H 'Content-Type: application/json' -d '{ "model": "granite3-dense", "messages": [ { "role": "user", "content": "respond in 20 words. who are you?" } ], "user": "vedran", "metadata": { "md_key": "md_value" }, "labels": {"client": "my label"}, "seed": 42, "new_key1": "new value 1", "new_key2": "new value 2", "new_map": {"mk1": "mv1", "mk2": "mv2"} }'

In the Ollama request sent to the backend:

• unrecognized keys are just passed to the backend within the options field
• metadata fields are not passed to the backend

Flow Details
2025-03-13 07:29:37 POST http://localhost:11434/api/generate                                                                                                                                                                                                                                                   
                         ← 200 OK application/json 1.05k 6.45s
                                               Request                                                                                               Response                                                                                               Detail
Host:             localhost:11434                                                                                                                                                                                                                                                                              
Accept:           */*                                                                                                                                                                                                                                                                                          
Accept-Encoding:  gzip, deflate                                                                                                                                                                                                                                                                                
Connection:       keep-alive                                                                                                                                                                                                                                                                                   
User-Agent:       litellm/1.63.2                                                                                                                                                                                                                                                                               
Content-Length:   250                                                                                                                                                                                                                                                                                          
Raw                                                                                                                                                                                                                                                                                                      
{"model": "granite3-dense:2b", "prompt": "### User:\\nWhy is the sky blue?\\n\\n", "options": {"seed": 42, "labels": {"client": "my label"}, "new_key1": "new value 1", "new_key2": "new value 2", "new_map": {"mk1": "mv1", "mk2": "mv2"}}, "stream": false}

[vvidovic]

@krrishdholakia - here are the lines where the request is built for the Gemini calls (function _transform_request_body):

• litellm/litellm/llms/vertex_ai/gemini/transformation.py

  Line 338 in be35c9a

  data = RequestBody(contents=content)

Only a subset of params from the optional_params is copied, without explicitly adding labels (or any other unrecognized parameter) to the request sent to the provider API.

[vvidovic]

@krrishdholakia - please let me know if there is anything else I can do about this PR (more examples or tests).

[CLAassistant]

All committers have signed the CLA.

[vvidovic]

@ishaan-jaff - did you have time to check this PR?

[vvidovic]

Hi, this one is important for Vertex AI users.

Unfortunately, the Google costs can't be simply calculated from token usage. They do apply some custom logic to bill customers, based on variable prices agreed for customers and depending on the number of requests and tokens (progressively cheaper prices per token when the number of calls/tokens increases).
Labels are a mechanism that can be used to distinguish between different costs. For example, Lite LLM can send a label api_key_name, team_name or similar. That way, we can separate costs for different users and teams on the Google billing side.

Can you please check this small PR and see if it can be merged to the main branch so this functionality becomes available in some of the next releases?

[ghost]

Hi @vvidovic there seems to be some provider specific filtering here - which makes sense. This (hardcoding specific non-llm provider k, v pair) is probably not the pattern we want to adopt.

We can either:

• fix the provider specific filtering to pass along any unmapped param as is
• expand support for an explicit list of variables you want to send along - e.g. set via extra_body

Which would make more sense to you as a user?

[vvidovic]

Hi @vvidovic there seems to be some provider specific filtering here - which makes sense. This (hardcoding specific non-llm provider k, v pair) is probably not the pattern we want to adopt.

We can either:

• fix the provider specific filtering to pass along any unmapped param as is
• expand support for an explicit list of variables you want to send along - e.g. set via extra_body

Which would make more sense to you as a user?

To me, the labels field in the Vertex AI API looks as equivalent of the metadata field in Open AI API.

However, if you think it is too much to find and add such mappings for all providers/vendors, I guess the extra_body would make more sense to me.
That way, you can enable clients to add any kind of complex data to the request that LiteLLM prepares for the backend, maybe even to override some values if necessary (within the end-user requests or with a custom callback implemented by the LiteLLM service provider). You could even add config param allow_extra_body to switch on/off this feature.
That way, we can avoid sending some fields to the provider by mistake.

[fillassuncao]

Hello @vvidovic and @krrishdholakia,

Are there any updates / blockers to this PR? I was just looking into implementing this myself, and then realised there was already an open PR.

Agree with @vvidovic reasoning that is needed to track costs.

[vvidovic]

Hello @vvidovic and @krrishdholakia,

Are there any updates / blockers to this PR? I was just looking into implementing this myself, and then realised there was already an open PR.

Agree with @vvidovic reasoning that is needed to track costs.

I don't have much to add to my last comments.
Maybe @krrishdholakia have some news for us.

[bhanu97]

@krrishdholakia & @ishaan-jaff Please let us know if there is any update/ blocker for this PR. This is a must have feature for cost observability in Vertex-AI

[520444555]

@krrishdholakia I am also looking at this feature in order to understand the consumption of llm based on labels used, if you are looking for something else to be part of this PR. please do let us know ,happy to contribute

[ghost]

@vvidovic can you update the vertex ai doc, telling people this exists? i'm sure people would find this very useful

[vvidovic]

@vvidovic can you update the vertex ai doc, telling people this exists? i'm sure people would find this very useful

@krrishdholakia here is the PR for vertex ai doc update: #14448