Skip to content
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,36 @@ That's it! The provider is now available.
}
```

## Responses API Support

If your provider also supports the OpenAI Responses API (`/v1/responses`), add `supported_endpoints`:

```json
{
"your_provider": {
"base_url": "https://api.yourprovider.com/v1",
"api_key_env": "YOUR_PROVIDER_API_KEY",
"supported_endpoints": ["/v1/chat/completions", "/v1/responses"]
}
}
```

This enables `litellm.responses()` with zero additional code:

```python
import litellm

response = litellm.responses(
model="your_provider/model-name",
input="Hello, what can you do?",
)
print(response.output)
```

If `supported_endpoints` is omitted, it defaults to `[]`. Chat completions is always enabled for JSON providers regardless of this field.

The provider inherits all request/response handling from OpenAI's Responses API — streaming, tools, and all standard parameters work out of the box.

## Usage

```python
Expand All @@ -89,11 +119,17 @@ import os
# Set your API key
os.environ["YOUR_PROVIDER_API_KEY"] = "your-key-here"

# Use the provider
# Chat completions
response = litellm.completion(
model="your_provider/model-name",
messages=[{"role": "user", "content": "Hello"}],
)

# Responses API (if supported_endpoints includes "/v1/responses")
response = litellm.responses(
model="your_provider/model-name",
input="Hello",
)
```

## When to Use Python Instead
Expand All @@ -105,7 +141,9 @@ Use a Python config class if you need:
- Provider-specific streaming logic
- Advanced tool calling modifications

For these cases, create a config class in `litellm/llms/your_provider/chat/transformation.py` that inherits from `OpenAIGPTConfig` or `OpenAILikeChatConfig`.
For chat completions, create a config class in `litellm/llms/your_provider/chat/transformation.py` that inherits from `OpenAIGPTConfig` or `OpenAILikeChatConfig`.

For responses API with small overrides, inherit from `OpenAIResponsesAPIConfig` and override only what's needed. See `litellm/llms/perplexity/responses/transformation.py` for a minimal example (~40 lines vs 400+).

## Testing

Expand Down
38 changes: 35 additions & 3 deletions litellm/llms/openai_like/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@ Instead of creating a full Python module for simple OpenAI-compatible providers,

- `providers.json` - Configuration file for all JSON-based providers
- `json_loader.py` - Loads and parses the JSON configuration
- `dynamic_config.py` - Generates Python config classes from JSON
- `chat/` - Existing OpenAI-like chat completion handlers
- `dynamic_config.py` - Generates Python config classes from JSON (chat + responses)
- `chat/` - OpenAI-like chat completion handlers
- `responses/` - OpenAI-like Responses API handlers

## Adding a New Provider

Expand Down Expand Up @@ -96,6 +97,32 @@ response = litellm.completion(
)
```

## Responses API Support

Providers that support the OpenAI Responses API (`/v1/responses`) can declare it via `supported_endpoints`:

```json
{
"your_provider": {
"base_url": "https://api.yourprovider.com/v1",
"api_key_env": "YOUR_PROVIDER_API_KEY",
"supported_endpoints": ["/v1/chat/completions", "/v1/responses"]
}
}
```

This enables `litellm.responses(model="your_provider/model-name", ...)` with zero Python code.
The provider inherits all request/response handling from OpenAI's Responses API config.

If `supported_endpoints` is omitted, it defaults to `[]` (only chat completions, which is always enabled for JSON providers).

### How It Works

1. `json_loader.py` checks `supported_endpoints` for `/v1/responses`
2. `dynamic_config.py` generates a responses config class (inherits from `OpenAIResponsesAPIConfig`)
3. `ProviderConfigManager.get_provider_responses_api_config()` returns the generated config
4. Request/response transformation is inherited from OpenAI — no custom code needed

## Benefits

- **Simple**: 2-5 lines of JSON vs 100+ lines of Python
Expand All @@ -112,6 +139,10 @@ Use a Python config class if you need:
- Provider-specific streaming logic
- Advanced tool calling transformations

For providers that are *mostly* OpenAI-compatible but need small overrides (e.g. preset model handling),
you can inherit from `OpenAIResponsesAPIConfig` and override only what's needed — see
`litellm/llms/perplexity/responses/transformation.py` for a minimal example (~40 lines).

## Implementation Details

### How It Works
Expand All @@ -125,5 +156,6 @@ Use a Python config class if you need:

The JSON system is integrated at:
- `litellm/litellm_core_utils/get_llm_provider_logic.py` - Provider resolution
- `litellm/utils.py` - ProviderConfigManager
- `litellm/utils.py` - ProviderConfigManager (chat + responses)
- `litellm/responses/main.py` - Responses API routing
- `litellm/constants.py` - openai_compatible_providers list
60 changes: 60 additions & 0 deletions litellm/llms/openai_like/dynamic_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -166,3 +166,63 @@ def custom_llm_provider(self) -> Optional[str]:
return provider.slug

return JSONProviderConfig


_responses_config_cache: dict = {}


def create_responses_config_class(provider: SimpleProviderConfig):
"""Generate a Responses API config class dynamically from JSON configuration.

Parallel to create_config_class() but for /v1/responses endpoints.
Classes are cached per provider slug to avoid regeneration on every request.
"""
if provider.slug in _responses_config_cache:
return _responses_config_cache[provider.slug]
Comment thread
Chesars marked this conversation as resolved.
Resolved

from litellm.llms.openai_like.responses.transformation import (
OpenAILikeResponsesConfig,
)
from litellm.types.router import GenericLiteLLMParams

class JSONProviderResponsesConfig(OpenAILikeResponsesConfig):
@property
def custom_llm_provider(self): # type: ignore[override]
return provider.slug

def validate_environment(
self,
headers: dict,
model: str,
litellm_params: Optional[GenericLiteLLMParams],
) -> dict:
litellm_params = litellm_params or GenericLiteLLMParams()
api_key = (
litellm_params.api_key
or get_secret_str(provider.api_key_env)
)
if api_key:
Comment thread
Chesars marked this conversation as resolved.
Resolved
headers["Authorization"] = f"Bearer {api_key}"
return headers
Comment thread
Chesars marked this conversation as resolved.
Resolved

def get_complete_url(
self,
api_base: Optional[str],
litellm_params: dict,
) -> str:
if not api_base:
if provider.api_base_env:
api_base = get_secret_str(provider.api_base_env)
if not api_base:
api_base = provider.base_url

if api_base is None:
raise ValueError(
f"api_base is required for provider {provider.slug}"
)

api_base = api_base.rstrip("/")
return f"{api_base}/responses"
Comment on lines +208 to +225

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generated URL appends only /responses to api_base. This silently produces an incorrect endpoint for any provider whose base_url does NOT already end with /v1 (e.g., "https://api.provider.com"). The documentation examples show base_url ending in /v1 (e.g., "https://api.yourprovider.com/v1"), but nothing in the code validates or enforces this convention.

If a provider sets base_url: "https://api.provider.com" (without /v1) and supported_endpoints: ["/v1/responses"], the generated URL becomes https://api.provider.com/responses — which fails at runtime with a confusing 404.

Consider adding a validation check to enforce the convention and provide a clear error message:

Suggested change
def get_complete_url(
self,
api_base: Optional[str],
litellm_params: dict,
) -> str:
if not api_base:
if provider.api_base_env:
api_base = get_secret_str(provider.api_base_env)
if not api_base:
api_base = provider.base_url
if api_base is None:
raise ValueError(
f"api_base is required for provider {provider.slug}"
)
api_base = api_base.rstrip("/")
return f"{api_base}/responses"
if api_base is None:
raise ValueError(
f"api_base is required for provider {provider.slug}"
)
api_base = api_base.rstrip("/")
# Validate that base_url ends with /v1 for responses endpoint
if not api_base.endswith("/v1"):
raise ValueError(
f"base_url for provider '{provider.slug}' must end with '/v1' to use the Responses API "
f"(got '{api_base}'). Example: 'https://api.yourprovider.com/v1'"
)
return f"{api_base}/responses"


_responses_config_cache[provider.slug] = JSONProviderResponsesConfig
return JSONProviderResponsesConfig
9 changes: 9 additions & 0 deletions litellm/llms/openai_like/json_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ def __init__(self, slug: str, data: dict):
self.param_mappings = data.get("param_mappings", {})
self.constraints = data.get("constraints", {})
self.special_handling = data.get("special_handling", {})
self.supported_endpoints = data.get("supported_endpoints", [])


class JSONProviderRegistry:
Expand Down Expand Up @@ -64,6 +65,14 @@ def exists(cls, slug: str) -> bool:
"""Check if a provider is defined via JSON"""
return slug in cls._providers

@classmethod
def supports_responses_api(cls, slug: str) -> bool:
"""Check if a JSON provider supports the Responses API"""
provider = cls._providers.get(slug)
if provider is None:
return False
return "/v1/responses" in provider.supported_endpoints

@classmethod
def list_providers(cls) -> list:
"""List all registered provider slugs"""
Expand Down
5 changes: 5 additions & 0 deletions litellm/llms/openai_like/responses/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
from litellm.llms.openai_like.responses.transformation import (
OpenAILikeResponsesConfig,
)

__all__ = ["OpenAILikeResponsesConfig"]
51 changes: 51 additions & 0 deletions litellm/llms/openai_like/responses/transformation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
"""
OpenAI-like Responses API transformation.

Base class for JSON-declared providers that support the /v1/responses endpoint.
Inherits everything from OpenAIResponsesAPIConfig; subclasses only override
provider-specific resolution (slug, API key env var, base URL).
"""

from typing import Optional, Union

from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import LlmProviders


class OpenAILikeResponsesConfig(OpenAIResponsesAPIConfig):
"""
Responses API config for OpenAI-compatible providers declared via JSON.

Concrete per-provider classes are generated dynamically in dynamic_config.py.
This base provides the three overridable hooks that the dynamic generator
fills in: custom_llm_provider, validate_environment, get_complete_url.
"""

@property
def custom_llm_provider(self) -> Union[str, LlmProviders]: # type: ignore[override]
return "openai_like"

def validate_environment(
self,
headers: dict,
model: str,
litellm_params: Optional[GenericLiteLLMParams],
) -> dict:
litellm_params = litellm_params or GenericLiteLLMParams()
api_key = litellm_params.api_key or get_secret_str("OPENAI_LIKE_API_KEY")
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
return headers

def get_complete_url(
self,
api_base: Optional[str],
litellm_params: dict,
) -> str:
api_base = api_base or get_secret_str("OPENAI_LIKE_API_BASE")
if not api_base:
raise ValueError("api_base is required for openai_like provider")
api_base = api_base.rstrip("/")
return f"{api_base}/responses"
Loading
Loading