Skip to content
Closed
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ nebius = []
neosantara = []
openai = []
openrouter = []
portkey = []
portkey = ["portkey-ai"]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] portkey-ai has no lower bound. The code needs portkey_ai.AsyncPortkey and the response shapes of a recent release, so a resolver that picks an old version would fail in a confusing way. Add a floor such as portkey-ai>=2.3.4, as other extras here do.

qiniu = []
requesty = []
sambanova = []
Expand Down
21 changes: 21 additions & 0 deletions src/any_llm/providers/portkey/portkey.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,16 @@
from any_llm.types.completion import ChatCompletion, ChatCompletionChunk, CompletionParams
from any_llm.utils.structured_output import get_json_schema, is_structured_output_type

try:
import portkey_ai # type: ignore[import-not-found]

AsyncPortkey: Any = portkey_ai.AsyncPortkey
except ImportError as e: # pragma: no cover - exercised by the package guard
AsyncPortkey = None
MISSING_PACKAGES_ERROR: ImportError | None = e
else:
MISSING_PACKAGES_ERROR = None


class PortkeyProvider(XMLReasoningOpenAIProvider):
"""Portkey provider for accessing 200+ LLMs through Portkey's AI Gateway."""
Expand All @@ -30,6 +40,17 @@ class PortkeyProvider(XMLReasoningOpenAIProvider):
SUPPORTS_LIST_MODELS = True

_DEFAULT_REASONING_EFFORT = None
MISSING_PACKAGES_ERROR = MISSING_PACKAGES_ERROR
client: Any

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] client: Any (with AsyncPortkey: Any above) removes all type checking from every self.client... call in the base class. Import the class directly inside the try block and declare client: AsyncPortkey behind TYPE_CHECKING, the way xai and mistral do, so mypy keeps checking those calls.


@override
def _init_client(self, api_key: str | None = None, api_base: str | None = None, **kwargs: Any) -> None:
"""Initialize Portkey's native async client."""
self.client = AsyncPortkey(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Requests now go out with no timeout at all. Portkey's SDK pops timeout out of the call kwargs and passes timeout=None to its inner OpenAI client, and None means "no limit" in httpx. The old AsyncOpenAI client sent connect=5.0, read=600; AsyncPortkey sends connect=None, read=None, write=None, pool=None. A stalled gateway call now blocks forever instead of failing after 10 minutes. Client options do not help: AsyncPortkey(request_timeout=60) still yields an all-None timeout, because the per-call timeout=None wins. Set a default timeout in _convert_completion_params (and for the models.list call) to keep the old behaviour.

api_key=api_key,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] The request on the wire changed. Portkey's client sends Authorization: Bearer OPENAI_API_KEY (a literal placeholder) and puts the real key in the x-portkey-api-key header, where the old client sent the key as the bearer token. That is correct for api.portkey.ai, but a user who points PORTKEY_API_BASE at a gateway expecting a bearer token loses authentication. Please confirm this against a real endpoint.

base_url=api_base or self.API_BASE,
**kwargs,
)
Comment on lines +62 to +66
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

@staticmethod
@override
Expand Down
17 changes: 17 additions & 0 deletions tests/unit/providers/test_portkey_provider.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,26 @@
from dataclasses import dataclass
from typing import Any

import pytest

from any_llm.providers.portkey.portkey import PortkeyProvider
from any_llm.types.completion import CompletionParams


def test_portkey_uses_native_async_client(monkeypatch: pytest.MonkeyPatch) -> None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] This change has no CI signal. Every workflow run on the branch is action_required, so unit tests, lint, and mypy never ran, and the description says pre-commit did not complete. Portkey is in the integration matrix and the client that talks to the gateway changed, so please apply the run-integration-tests label before merge.

"""The provider initializes Portkey's native SDK instead of OpenAI's client."""
calls: dict[str, Any] = {}

class FakeAsyncPortkey:
def __init__(self, **kwargs: Any) -> None:
calls.update(kwargs)

monkeypatch.setattr("any_llm.providers.portkey.portkey.AsyncPortkey", FakeAsyncPortkey)
PortkeyProvider(api_key="test-key", api_base="https://example.test/v1")

assert calls == {"api_key": "test-key", "base_url": "https://example.test/v1"}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved


def test_convert_completion_params_with_dataclass_response_format() -> None:
"""Test that dataclass response_format is converted to JSON schema format."""

Expand Down