Skip to content
Merged
Show file tree
Hide file tree
Changes from 100 commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
dbbc57e
fix(vertex_ai): support pluggable (executable) credential_source for …
michelligabriele Apr 3, 2026
0873145
feat(logging): add component and logger fields to JSON logs for 3rd p…
J-Byron Apr 3, 2026
0f48f79
Feat - Add organization into the metrics metadata for org_id & org_al…
J-Byron Apr 3, 2026
0033ac0
fix: 429s from batch creation being converted to 500 (#24703)
J-Byron Apr 3, 2026
19a6e60
add us gov models (#24660)
mubashir1osmani Apr 3, 2026
30dfff0
Litellm dev 04 02 2026 p1 (#25052)
krrish-berri-2 Apr 3, 2026
c6c8087
fix(anthropic): pass logging_obj to client.post for litellm_overhead_…
milan-berri Apr 3, 2026
5211e7a
sap - add additional parameters for grounding
alpkom Feb 12, 2026
d0bb619
sap - fix models
alpkom Feb 12, 2026
f6e331a
(sap) add filtering, masking, translation SAP GEN AI Hub modules
vasilisazayka Feb 17, 2026
f7cfda4
(sap) add tests and docs for new SAP modules
vasilisazayka Feb 18, 2026
91c4471
(sap) add support of multiple modules config
vasilisazayka Feb 27, 2026
2ea74ec
(sap) code refactoring
vasilisazayka Mar 2, 2026
78eb67a
(sap) rename file
vasilisazayka Mar 4, 2026
00adbca
test(): add safeguard tests
I507220 Mar 4, 2026
fe390e0
(sap) update tests
vasilisazayka Mar 5, 2026
41d5072
(sap) update docs, solve merge conflict in transformation.py
vasilisazayka Mar 5, 2026
210ca20
(sap) linter fix
vasilisazayka Mar 5, 2026
834ed13
(sap) Align embedding request transformation with current API
vasilisazayka Mar 9, 2026
338379b
(sap) fix after bot review
vasilisazayka Mar 11, 2026
889861f
(sap) fix after bot review
vasilisazayka Mar 11, 2026
8f7018d
(sap) fix after bot review
vasilisazayka Mar 11, 2026
dc8e9d6
(sap) fix after bot review
vasilisazayka Mar 11, 2026
bde544c
(sap) fix after bot review
vasilisazayka Mar 11, 2026
4630ef9
(sap) fix after bot review
vasilisazayka Mar 11, 2026
3a9610f
(sap) fix after bot review
vasilisazayka Mar 11, 2026
1bd5b63
(sap) fix after bot review
vasilisazayka Mar 11, 2026
9bdf6e0
(sap) fix after bot review
vasilisazayka Mar 11, 2026
315f175
(sap) fix after bot review
vasilisazayka Mar 11, 2026
6123651
(sap) fix after bot review
vasilisazayka Mar 11, 2026
1132a79
(sap) fix after bot review
vasilisazayka Mar 11, 2026
f2dd120
(sap) mock commit
vasilisazayka Mar 11, 2026
341b559
(sap) run black formater
vasilisazayka Mar 13, 2026
a099c28
(sap) add literals to models, add negative tests, fix test for tool t…
vasilisazayka Mar 23, 2026
13d9cf5
(sap) fix formating
vasilisazayka Mar 23, 2026
19fe0cf
(sap) fix models
vasilisazayka Mar 23, 2026
8eb81b5
(sap) fix after bot review
vasilisazayka Mar 23, 2026
470aee5
(sap) fix after bot review
vasilisazayka Mar 23, 2026
178ce61
(sap) fix after bot review
vasilisazayka Mar 23, 2026
9e5d3d4
(sap) fix after bot review
vasilisazayka Mar 23, 2026
43d84b1
(sap) fix after bot review
vasilisazayka Mar 23, 2026
9415f58
(sap) fix after bot review
vasilisazayka Mar 23, 2026
6e8c993
(sap) commit for rerun bot review
vasilisazayka Mar 23, 2026
58bea41
(sap) minor improve
vasilisazayka Mar 24, 2026
a0994fe
(sap) fix after bot review
vasilisazayka Mar 24, 2026
4f018c7
(sap) lint fix
vasilisazayka Mar 26, 2026
5c02af8
docs(sap): update documentation
vasilisazayka Dec 17, 2025
881619a
fix(sap): change creds priority
vasilisazayka Dec 18, 2025
76292bc
fix(sap): change creds priority
vasilisazayka Dec 19, 2025
ff5ed4c
fix(sap): fix sap creds unit test
vasilisazayka Dec 19, 2025
afe3cc6
fix(sap): linter fix
vasilisazayka Dec 23, 2025
5ac885b
fix(sap): linter fix
vasilisazayka Dec 24, 2025
a473061
linter fix
vasilisazayka Jan 12, 2026
70e64c5
(sap) update logic of fetching creds, add additional tests
vasilisazayka Feb 3, 2026
3736812
(sap) clean up code
vasilisazayka Feb 4, 2026
85d9569
(sap) fix after review
vasilisazayka Feb 4, 2026
6bbc35d
(sap) fix after bot review
vasilisazayka Mar 5, 2026
b38bd57
(sap) fix after bot review
vasilisazayka Mar 5, 2026
1005461
(sap) fix after bot review
vasilisazayka Mar 5, 2026
26bdc87
(sap) fix after bot review
vasilisazayka Mar 5, 2026
7f5db20
(sap) fix after bot review
vasilisazayka Mar 5, 2026
4764583
(sap) fix after bot review
vasilisazayka Mar 5, 2026
4696848
(sap) fix after bot review
vasilisazayka Mar 5, 2026
2723c69
(sap) fix after bot review
vasilisazayka Mar 5, 2026
033a6eb
(sap) fix after bot review
vasilisazayka Mar 5, 2026
9b26087
(sap) fix after bot review
vasilisazayka Mar 5, 2026
a7d23d6
(sap) fix after bot review
vasilisazayka Mar 11, 2026
61a3bd1
(sap) add a possibility to put the service key by both variants
vasilisazayka Mar 11, 2026
0fde223
(sap) fix after bot review
vasilisazayka Mar 11, 2026
1f0bdb1
(sap) fix after bot review
vasilisazayka Mar 11, 2026
5ec3f3e
(sap) fix after bot review
vasilisazayka Mar 11, 2026
a5df768
(sap) update test
vasilisazayka Mar 11, 2026
35ab4d4
(sap) update service key resolve function
vasilisazayka Mar 12, 2026
439368e
(sap) run black formater
vasilisazayka Mar 13, 2026
aabbe78
(sap) fix validate credentials, add negative tests for credential fet…
vasilisazayka Mar 23, 2026
f3c1777
(sap) fix validate credentials, add negative tests for credential fet…
vasilisazayka Mar 23, 2026
ffeff44
(sap) fix after bot review
vasilisazayka Mar 23, 2026
f903a64
(sap) fix after bot review
vasilisazayka Mar 23, 2026
00878a2
(sap) fix after bot review
vasilisazayka Mar 24, 2026
3a55e59
(sap) fix after bot review
vasilisazayka Mar 24, 2026
870db5d
(sap) lint fix
vasilisazayka Mar 26, 2026
885fea2
(sap) lint fix
vasilisazayka Mar 26, 2026
25b266f
feat: support service_tier in gemini
markmcd Mar 24, 2026
7617d4e
chore: add a service_tier field mapping from openai to gemini
markmcd Mar 24, 2026
75901cc
fix: use x-gemini-service-tier header in response
markmcd Mar 26, 2026
48af865
docs: add service_tier to gemini docs
markmcd Mar 26, 2026
e04af74
chore: add defaut/standard mapping, and some tests
markmcd Mar 30, 2026
af74e80
chore: tidying up some case insensitivity
markmcd Mar 30, 2026
34e3899
chore: remove unnecessary guard
markmcd Apr 1, 2026
9bbbf72
fix: remove redundant test file
markmcd Apr 1, 2026
c2d2df2
fix: handle 'auto' case-insensitively
markmcd Apr 1, 2026
4c923f6
fix: return service_tier on final steamed chunk
markmcd Apr 1, 2026
5afce93
chore: black
markmcd Apr 1, 2026
a6aaa93
feat: enable supports_service_tier to gemini models
markmcd Apr 2, 2026
2008a59
Fix get_standard_logging_metadata tests
Sameerlite Apr 7, 2026
2232e97
Fix test_get_model_info_bedrock_models
Sameerlite Apr 7, 2026
76c24e1
Fix test_get_model_info_bedrock_models
Sameerlite Apr 7, 2026
58c3c52
Fix remaining tests
Sameerlite Apr 7, 2026
d0a6e6d
Fix mypy issues
Sameerlite Apr 7, 2026
3fe23fa
Fix tests
Sameerlite Apr 7, 2026
1ef7286
Fix merge conflicts
Sameerlite Apr 8, 2026
8795198
Fix code qa
Sameerlite Apr 8, 2026
d7c77fc
Fix code qa
Sameerlite Apr 8, 2026
881421f
Fix code qa
Sameerlite Apr 8, 2026
a076dab
Fix greptile review
Sameerlite Apr 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions docs/my-website/docs/providers/gemini.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,14 +65,13 @@ response = completion(
- modalities
- reasoning_content
- audio (for TTS models only)
- service_tier

**Anthropic Params**
- thinking (used to set max budget tokens across anthropic/gemini models)

[**See Updated List**](https://github.com/BerriAI/litellm/blob/main/litellm/llms/gemini/chat/transformation.py#L70)



## Usage - Thinking / `reasoning_content`

LiteLLM translates OpenAI's `reasoning_effort` to Gemini's `thinking` parameter. [Code](https://github.com/BerriAI/litellm/blob/620664921902d7a9bfb29897a7b27c1a7ef4ddfb/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py#L362)
Expand Down Expand Up @@ -298,6 +297,19 @@ curl http://0.0.0.0:4000/v1/chat/completions \



## Usage - `service_tier`

LiteLLM propagates OpenAI's `service_tier` parameter to Gemini, and also extracts it from the response headers (`x-gemini-service-tier`) into `model_response.service_tier`.

| OpenAI `service_tier` | Gemini `service_tier` | Notes |
| --------------------- | --------------------- | ----- |
| `"auto"` | `"priority"` | LiteLLM maps OpenAI's `"auto"` to Gemini's `"priority"` tier, as `priority` will fall back on Gemini. |
| `"flex"` | `"flex"` | Direct mapping. |
| `"priority"` | `"priority"` | Direct mapping. |
| `"default"` | `"standard"` | LiteLLM maps `"default"` to `"standard"`. |
| Any other value | Passed as-is (lowercased) | Values are case-insensitive and normalized to lowercase. |

On the response, LiteLLM maps `"standard"` back to `"default"` for the Gemini API.


## Text-to-Speech (TTS) Audio Output
Expand Down
267 changes: 261 additions & 6 deletions docs/my-website/docs/providers/sap.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,24 +55,33 @@ pip install litellm
```

### Step 2: Set Your Credentials

Choose **one** of these authentication methods:

> **Breaking change**: credential resolution is "first-source-wins"
>
> Credential resolution no longer merges individual fields across sources.
>
> Resolution order is:
`kwargs` → `service key` → `env (AICORE_*)` → `config` → `VCAP service`
>
> **Important behavior:** once LiteLLM finds *any* credential value in a source, it takes **all** credentials from that source exclusively (except `resource_group`, which may still be resolved separately).

Choose **one** of these authentication methods:
<Tabs>
<TabItem value="service-key" label="Service Key JSON (Recommended)">

<Tabs>
<TabItem value="service-key" label="Service Key JSON (Recommended)">
The simplest approach - paste your entire service key as a single environment variable.

The simplest approach - paste your entire service key as a single environment variable. The service key must be wrapped in a `credentials` object:
> **Note:** the service key no more needs to be wrapped in a "credentials" key.

```bash
export AICORE_SERVICE_KEY='{
"credentials": {
"clientid": "your-client-id",
"clientsecret": "your-client-secret",
"url": "https://<your-instance>.authentication.sap.hana.ondemand.com",
"serviceurls": {
"AI_API_URL": "https://api.ai.<your-region>.aws.ml.hana.ondemand.com"
}
}
}'
export AICORE_RESOURCE_GROUP="default"
```
Expand Down Expand Up @@ -220,6 +229,17 @@ model="sap/gemini-2.5-pro"
# Incorrect - missing prefix
model="gpt-4o" # ❌ Won't work
```
3. **Environment variables** - Set the following list of credentials in .env file
<pre>
AICORE_AUTH_URL = "https://* * * .authentication.sap.hana.ondemand.com/oauth/token",
AICORE_CLIENT_ID = " *** ",
AICORE_CLIENT_SECRET = " *** ",
AICORE_RESOURCE_GROUP = " *** ",
AICORE_BASE_URL = "https://api.ai.***.cfapps.sap.hana.ondemand.com/v2"
</pre>

Other credential configuration options are also available. For more information, see the [SAP AI Core Documentation](https://help.sap.com/doc/generative-ai-hub-sdk/CLOUD/en-US/_reference/README_sphynx.html#configuration).
## Usage - LiteLLM Python SDK

### Proxy Usage

Expand Down Expand Up @@ -506,6 +526,241 @@ response = embedding(
print(response.data[0]["embedding"]) # Vector representation
```

### Additional Modules
The SAP Gen AI Hub includes additional modules for advanced use cases:
- [Grounding](https://help.sap.com/docs/sap-ai-core/generative-ai/grounding-035c455a5a424697b60f4a24b6d791fe?locale=en-US)
- [Translation](https://help.sap.com/docs/sap-ai-core/generative-ai/translation?locale=en-US)
- [Data Masking](https://help.sap.com/docs/sap-ai-core/generative-ai/data-masking-d9a54d9ca54b40beacbd24e1663ec3b4?locale=en-US)
- [Content Filtering](https://help.sap.com/docs/sap-ai-core/generative-ai/content-filtering?locale=en-US)

#### Grounding
Grounding is a service designed to handle data-related tasks, such as grounding and retrieval, using vector databases. It provides specialized data retrieval through these databases, grounding the retrieval process with your own external and context-relevant data. Grounding combines generative AI capabilities with the ability to use real-time, precise data to improve decision-making and business operations for specific AI-driven business solutions.
##### Prerequisites
To use the Grounding module in the orchestration pipeline, you need to prepare the knowledge base in advance.

Generative AI hub offers multiple options for users to provide data (prepare a knowledge base):
- For Option 1: Upload the documents to a supported data repository and run the data pipeline to vectorize the documents.
- For Option 2: Provide the chunks of document via Vector API directly.

To use grounding, choose from one of the following options.

Usage example:
```python showLineNumbers title="Grounding Example"
from litellm import completion

grounding_config = {
'type': 'document_grounding_service',
'config': {
'filters': [
{'id': 's3-docs',
'data_repository_type': 'vector',
'search_config': {'max_chunk_count': 2},
'data_repositories': ['012345-6789-0123-4567-890123456789']
}
],
'placeholders': {'input': ['user_query'], 'output': 'grounding_response'},
'metadata_params': ['source', 'webUrl', 'title', 'mimeType', 'fileSuffix']
}
}

response = completion(model="sap/gpt-4o",
messages=[
{"content":"""Facility Solutions Company provides services to luxury residential complexes,
apartments, individual homes, and commercial properties such as office buildings, retail
spaces, industrial facilities, and educational institutions. Customers are encouraged to
reach out with maintenance requests, service deficiencies, follow-ups, or any issues they
need by email.""", "role": "system"},
{"content":"""You are a helpful assistant for any queries for answering questions.
Answer the request by providing relevant answers that fit to the request.
Request: {{ ?user_query }}
Context:{{ ?grounding_response }}""", "role": "user"}
],
placeholder_values={"user_query": "Is there a complaint?"},
grounding=grounding_config
)
print(response.choices[0].message.content)
```
For more information about all available grounding configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/using-grounding-module-e1c4dd100dfb42ab890e1d95f3516187?locale=en-US).

#### Translation
The translation module allows you to translate LLM text prompts into a chosen target language.

```python showLineNumbers title="Translation Example"
from litellm import completion

translation_config = {
'input':
{'type': 'sap_document_translation',
'config':
{'source_language': 'en-US',
'target_language': 'de-DE'}
},
'output':
{'type': 'sap_document_translation',
'config':
{'source_language': 'de-DE',
'target_language': 'fr-FR'}
}
}

response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "Hello world!"}],
translation=translation_config)

print(response.choices[0].message.content)
```
For more information about all available translation configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/translation?locale=en-US)

#### Data Masking
The data masking module serves to anonymize or pseudonymize personally identifiable information from the input for selected entities.

```python showLineNumbers title="Data Masking Example"
from litellm import completion, embedding
masking_config = {
'providers':
[
{
'type': 'sap_data_privacy_integration',
'method': 'anonymization',
'entities': [
{'type': 'profile-address'},
{'type': 'profile-email'},
{'type': 'profile-phone'},
{'type': 'profile-person'},
{'type': 'profile-location'}
]
}
]
}

mock_cv = "some text with personal information"

response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "Give a one sentence summary of the CV. CV: {{?cv}}?"}],
placeholder_values={"cv": mock_cv},
masking=masking_config)
print(response.choices[0].message.content)

# Data masking module also available for embedding
response = embedding(model="sap/text-embedding-3-small",
input=mock_cv,
masking=masking_config)
print(response.data[0])
```
For more information about all available data masking configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/enhancing-model-consumption-with-data-masking-66ad6f469afc4c2cbaa91a27a33f7b21?locale=en-US)





#### Content Filtering
The content filtering module allows you to filter input and output based on content safety criteria.

The module supports two services:
* Azure Content Safety
* Llama Guard 3

```python showLineNumbers title="Content Filtering Example"
from litellm import completion

filtering_config_azure = {
'input':
{
'filters':
[
{'type': 'azure_content_safety',
'config':
{'hate': 0,
'sexual': 0,
'violence': 0,
'self_harm': 0
}
}
]
},
'output':
{
'filters':
[
{'type': 'azure_content_safety',
'config': {'hate': 0,
'sexual': 0,
'violence': 0,
'self_harm': 0
}
}
]
}
}

response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "Hello world!"}],
filtering=filtering_config_azure)
print(response.choices[0].message.content)
# The model responds normally because the content does not violate any safety rules.

try:
response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "I hate you"}],
filtering=filtering_config_azure)
except Exception as e:
print(e)
# The service raises an error:
# "Input Filter: Content filtered due to safety violations. Please modify the prompt and try again."
```
For more information about all available content filtering configurations, see the [documentation](https://help.sap.com/docs/sap-ai-core/generative-ai/content-filtering?locale=en-US)

#### List of modules configuration for fallback
SAP GEN AI Hub supports a fallback mechanism for handling errors. This mechanism allows you to specify a list of fallback modules to use in case of errors. The fallback modules should contain all parameters that are required for configuring the request.

Required parameters:
- `model`
- `messages`

Optional parameters:
- `filtering`
- `grounding`
- `translation`
- `masking`
- `tools`

- and any of model's specific parameters.


```python showLineNumbers title="Fallback Example"
from litellm import completion

translation_config = {
'input':
{'type': 'sap_document_translation',
'config':
{'source_language': 'en-US',
'target_language': 'de-DE'}
},
'output':
{'type': 'sap_document_translation',
'config':
{'source_language': 'de-DE',
'target_language': 'fr-FR'}
}
}

response = completion(model="sap/gpt-4o",
messages=[{"role": "user", "content": "Hello world!"}],
translation=translation_config,
fallback_sap_modules=[{
"model":"sap/gemini-2.5-flash",
"messages":[{"role": "user", "content": "Hello world!"}],
"translation":translation_config
}])

# In case of error with the first configuration (model gpt-4o), the fallback module is used.

print(response.choices[0].message.content)

```


## Reference

### Supported Parameters
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ async def async_log_failure_event(self, kwargs, response_obj, start_time, end_ti
user_api_key_max_budget=_meta.get("user_api_key_max_budget"),
user_api_key_budget_reset_at=_meta.get("user_api_key_budget_reset_at"),
user_api_key_org_id=_meta.get("user_api_key_org_id"),
user_api_key_org_alias=_meta.get("user_api_key_org_alias"),
user_api_key_team_id=_meta.get("user_api_key_team_id"),
user_api_key_project_id=_meta.get("user_api_key_project_id"),
user_api_key_project_alias=_meta.get("user_api_key_project_alias"),
Expand Down Expand Up @@ -196,6 +197,7 @@ async def hanging_response_handler(
else None
),
user_api_key_org_id=user_api_key_dict.org_id,
user_api_key_org_alias=user_api_key_dict.organization_alias,
user_api_key_team_id=user_api_key_dict.team_id,
user_api_key_project_id=user_api_key_dict.project_id,
user_api_key_project_alias=user_api_key_dict.project_alias,
Expand Down
6 changes: 6 additions & 0 deletions litellm/_logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,12 @@ def format(self, record):
if key not in _STANDARD_RECORD_ATTRS and key not in json_record:
json_record[key] = value

# Set component/logger only if not already supplied via extra={...}
if "component" not in json_record:
json_record["component"] = record.name
if "logger" not in json_record:
json_record["logger"] = f"{record.filename}:{record.lineno}"

if record.exc_info:
json_record["stacktrace"] = record.exc_text or self.formatException(
record.exc_info
Expand Down
6 changes: 3 additions & 3 deletions litellm/a2a_protocol/providers/pydantic_ai_agents/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ async def handle_non_streaming(
request_id: str,
params: Dict[str, Any],
api_base: Optional[str] = None,
**kwargs,
**kwargs: Any,
) -> Dict[str, Any]:
"""Handle non-streaming request to Pydantic AI agent."""
if not api_base:
raise ValueError("api_base is required for Pydantic AI agents")
if api_base is None:
raise ValueError("api_base is required for PydanticAIProviderConfig")
return await PydanticAIHandler.handle_non_streaming(
request_id=request_id,
params=params,
Expand Down
Loading
Loading