feat(provider): Add Hanzo provider as OpenAI Compatible - #30197
Closed
abhicris wants to merge 5 commits into
Closed
Conversation
…re) (BerriAI#29775) * Add gpt-realtime-whisper Realtime transcription support (OpenAI + Azure) Adds first-class support for the gpt-realtime-whisper streaming speech-to-text model, which uses the Realtime transcription session API rather than the file-based /audio/transcriptions path. Model registration: registers gpt-realtime-whisper and azure/gpt-realtime-whisper with audio-duration pricing (input_cost_per_second = 0.017/60, matching the published $0.017/minute input audio rate). REST endpoint: implements POST /v1/realtime/transcription_sessions (plus /realtime and /openai/v1 aliases) to mint an ephemeral transcription session for the WebRTC flow. Adds request/response types, OpenAI and Azure URL builders, a shared base handler (refactored from the client_secrets handler), the acreate_realtime_transcription_session SDK function, and route registration. The proxy encrypts the ephemeral key returned under client_secret.value and records the session type in the token so the follow-up /realtime/calls replays type=transcription rather than type=realtime. WebSocket: forwards intent=transcription through to the Azure handler (OpenAI already received it) with URL-encoding, so gpt-realtime-whisper opens a transcription session. Transcription-only sessions no longer trigger an erroneous response.create. Cost tracking: transcription sessions emit no response.done events; their usage arrives on conversation.item.input_audio_transcription.completed as {type: duration, seconds}. That usage is captured out-of-band (usage only, no transcript duplication) and billed by input_cost_per_second, with a token-billed fallback for token-priced transcription models. Adds tests for pricing math, URL builders, request/response types, the proxy route and SDK function, WebSocket intent forwarding, transcription-session streaming behavior, and the /realtime/calls session-type replay. * Address PR review: URL-encode all Azure WS query params; forward query_params through provider_config branch * Address PR review: session_type validation, model auth fix, cost perf, billing fallback, detail/docs cleanup * Improve test coverage: detection from backend, error paths, unknown usage type, resolved_model None * Backport realtime transcription websocket fixes * Enforce authorized realtime transcription model * Enforce realtime transcription model access * Enforce realtime resolved model scopes * Enforce WebRTC transcription model scope
* Pass through debug lazy logging * fix(proxy): convert remaining eager pass-through debug logs to lazy formatting
…nt (BerriAI#30157) * fix(parallel_ai): migrate search integration from v1beta to v1 endpoint The Parallel Search API moved from /v1beta/search (processor: base/pro, parallel-beta header) to /v1/search (mode: turbo/basic/advanced, no beta header). Request fields moved too: max_results, source_policy, and excerpt settings are now nested under advanced_settings, and source_policy uses include_domains/exclude_domains. The v1 response returns publish_date per result, which now maps to SearchResult.date instead of being hardcoded to None. The legacy processor param is mapped to the equivalent mode so existing callers keep working. * fix(parallel_ai): default mode to basic and simplify param handling The v1 API defaults to advanced mode when mode is omitted, while v1beta defaulted to the base processor. Without an explicit default, callers who pass no mode would be silently upgraded to a tier costing 2.25x more while litellm's cost map reports the basic-tier price. Sending mode=basic preserves the v1beta default and keeps cost tracking accurate. Also replaces the handled_params set with pop-as-consumed param handling so mapped params no longer need to be tracked in two places, and extends the tests to pin the default mode, processor=base mapping, mode-over-processor precedence, and top-level v1 param passthrough. * fix(parallel_ai): avoid double /v1 when api_base is already versioned A PARALLEL_AI_API_BASE like https://api.parallel.ai/v1 previously produced .../v1/v1/search. Strip a trailing /v1 before appending the search path and cover the api_base variants with a parametrized test. --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Adds `hanzo/` provider routing to api.hanzo.ai. Mirrors the OpenAI v1 chat-completions interface via the existing JSON-configured provider path (same shape as Neosantara / Tensormesh / Parasail). Auth via HANZO_API_KEY env var; api_base overridable via HANZO_API_BASE. Includes: - providers.json entry with base_url, api_key_env, api_base_env, max_completion_tokens -> max_tokens param mapping, and /v1/chat/completions supported endpoint - LlmProviders.HANZO enum entry - constants.py wiring (openai_compatible_endpoints, provider_list) - provider_endpoints_support.json + backup entries - Model cost map entries for hanzo/zen4 (\$1.50/\$4.50 per MTok) and hanzo/zen4-max (\$3/\$12 per MTok, 1M context) - Docs page at docs/my-website/docs/providers/hanzo.md - Provider + cost-map unit tests under tests/test_litellm/llms/openai_like/test_hanzo_provider.py Hanzo Cloud (hanzo.ai) is an OpenAI-compatible AI gateway that also fronts ~444 models from ~56 underlying providers via a unified /v1/chat/completions surface. This PR covers the Hanzo native Zen family; other upstream models are reachable by passing the upstream model id through Hanzo's gateway under the hanzo/ prefix.
Sameerlite
force-pushed
the
litellm_oss_110626
branch
from
June 11, 2026 10:16
160409f to
516a4e7
Compare
|
|
Contributor
Greptile SummaryAdds Hanzo (
Confidence Score: 4/5The provider integration itself is correct and follows established patterns; the only issue is a documentation file that should not be committed to this repository. The core provider wiring — JSON config, enum registration, constants, cost map — is clean and mirrors the neosantara/tensormesh shape faithfully. The test suite uses only in-process, mock-based assertions with no real network calls. The one problem is docs/my-website/docs/providers/hanzo.md — this file should be moved to the litellm-docs repository.
|
| Filename | Overview |
|---|---|
| docs/my-website/docs/providers/hanzo.md | New provider docs page added — violates the project rule that docs belong in the litellm-docs repo, not in this repository. |
| litellm/llms/openai_like/providers.json | Adds hanzo entry with correct base URL, key/base env vars, param mappings, and supported endpoint — matches the pattern of neosantara/tensormesh peers. |
| litellm/constants.py | Adds https://api.hanzo.ai/v1 to openai_compatible_endpoints and "hanzo" to openai_compatible_providers — consistent with other JSON-configured providers. |
| litellm/types/utils.py | Adds HANZO = "hanzo" enum entry in the correct position among other JSON-configured providers. |
| model_prices_and_context_window.json | Adds hanzo/zen4 ($1.50/$4.50 per MTok, 200k ctx) and hanzo/zen4-max ($3/$12 per MTok, 1M ctx) with capability flags stored in JSON per the model-flags rule. |
| provider_endpoints_support.json | Adds hanzo capability entry (chat_completions only) consistent with other providers in the same JSON. |
| litellm/provider_endpoints_support_backup.json | Backup mirror of provider_endpoints_support.json — hanzo entry is identical to the main file, no issues. |
| tests/test_litellm/llms/openai_like/test_hanzo_provider.py | All tests use mocked data (monkeypatched cost map, in-process registry lookups, Router with a fake key) with zero real network calls — satisfies the no-real-network-calls rule. |
Reviews (1): Last reviewed commit: "feat(provider): add Hanzo as an OpenAI-c..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @@ -0,0 +1,176 @@ | |||
| import Tabs from '@theme/Tabs'; | |||
Contributor
There was a problem hiding this comment.
litellm-docs repo
Per the project's contribution rules, provider documentation pages must not be added to this repository — they need to live in the litellm-docs repo. Please remove this file from the PR and open a corresponding PR in litellm-docs instead.
Rule Used: Prevent documentation from being added - needs to ... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
|
Thanks for adding Hanzo support, @abhicris! A few things to address before this is ready:
Thanks for the contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Hanzo (
hanzo/) as a JSON-configured OpenAI-compatible provider. Mirrors the same shape as the recently-merged Neosantara (#29646) and Tensormesh (#29063) entries.hanzo/https://api.hanzo.ai/v1HANZO_API_KEY(Bearer token, OpenAI-style).HANZO_API_BASEoverrides the default base URL./v1/chat/completionsmax_completion_tokens->max_tokensparam mapping.Hanzo Cloud (hanzo.ai) is an OpenAI-compatible AI gateway. This PR covers Hanzo's native Zen model family; other upstream models routed through Hanzo's gateway can be reached by passing the upstream id under the
hanzo/prefix.Files
litellm/llms/openai_like/providers.json—hanzoentrylitellm/types/utils.py—LlmProviders.HANZOlitellm/constants.py— base URL,provider_listentryprovider_endpoints_support.json+litellm/provider_endpoints_support_backup.json—hanzocapability entrymodel_prices_and_context_window.json—hanzo/zen4($1.50 / $4.50 per MTok),hanzo/zen4-max($3 / $12 per MTok, 1M context)docs/my-website/docs/providers/hanzo.md— provider docs pagetests/test_litellm/llms/openai_like/test_hanzo_provider.py— provider config + cost map unit testsTest plan
LlmProviders.HANZOresolves and is inlitellm.provider_listJSONProviderRegistry.exists("hanzo")and config fields matchget_llm_provider("hanzo/zen4")returns base URLhttps://api.hanzo.ai/v1litellm.cost_per_tokenforhanzo/zen4andhanzo/zen4-maxreturns the expected valuesapi_base/api_keyoverrides the default