chore: add soniox provider with support for async stt-async-v4 model - #26885
Closed
dan2k3k4 wants to merge 475 commits into
Closed
chore: add soniox provider with support for async stt-async-v4 model#26885dan2k3k4 wants to merge 475 commits into
dan2k3k4 wants to merge 475 commits into
Conversation
|
Michael Riad Zaky seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Contributor
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
Greptile SummaryThis PR adds a new
Confidence Score: 3/5Not safe to merge as-is — the missing One P1 (extra_headers silently dropped in main.py dispatch) lowers the ceiling to 4, and the combination with two P2s (zero cost price, unused max_retries parameter) and the in-place mutation of optional_params brings the score to 3.
|
| Filename | Overview |
|---|---|
| litellm/llms/soniox/audio_transcription/handler.py | Core orchestration handler for the multi-step Soniox async transcription flow (upload → create → poll → fetch → cleanup); max_retries is accepted but unused, and _prepare() mutates the caller's optional_params dict in-place. |
| litellm/llms/soniox/audio_transcription/transformation.py | Parameter mapping and response transformation for Soniox; well-structured with clear passthrough lists and diarization/language-ID rendering. No significant issues found. |
| litellm/llms/soniox/common_utils.py | Shared utilities (exception class, API key/base resolution, token rendering); clean implementation with sensible defaults. |
| litellm/main.py | Adds Soniox dispatch in transcription(); extra_headers extracted from kwargs but not forwarded to the Soniox handler, silently dropping caller-supplied headers. |
| model_prices_and_context_window.json | Adds soniox/stt-async-v4 entry; input_cost_per_second is hardcoded to 0.0, which will cause cost tracking to always report $0 for Soniox transcriptions. |
| litellm/litellm_core_utils/get_llm_provider_logic.py | Adds Soniox to _get_openai_compatible_provider_info solely for key/base resolution; works correctly for the transcription use-case. |
| litellm/init.py | Registers soniox_api_key, soniox_models, and SonioxAudioTranscriptionConfig lazy import; follows existing provider registration pattern correctly. |
| litellm/utils.py | Adds SonioxAudioTranscriptionConfig lookup in ProviderConfigManager.get_provider_audio_transcription_config; straightforward addition following existing pattern. |
| tests/test_litellm/llms/soniox/audio_transcription/test_soniox_audio_transcription_handler.py | Comprehensive mock-based tests covering sync/async flows, multi-step polling, error states, cleanup behavior, and missing-input validation — all without real network calls. |
| tests/test_litellm/llms/soniox/audio_transcription/test_soniox_audio_transcription_transformation.py | Good coverage of parameter mapping, env validation, URL construction, request/response transformation, and error class instantiation. |
Comments Outside Diff (1)
-
litellm/main.py, line 6598-6630 (link)extra_headerssilently dropped for Sonioxextra_headersis extracted from kwargs at line 6457 but never forwarded toSonioxAudioTranscriptionHandler().audio_transcriptions(). Any headers a caller passes viaextra_headers={"X-Custom": "..."}will be silently ignored. Compare Azure's handling at line 6546 which merges them first:optional_params["extra_headers"] = extra_headers
The fix is to either inject
extra_headersintooptional_paramsbefore the handler call (so_preparepicks them up and merges them viavalidate_environment), or pass them directly asheaders=extra_headers or {}.
Reviews (1): Last reviewed commit: "chore: add soniox provider with support ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
29047
to
+29054
| "mode": "chat" | ||
| }, | ||
| "soniox/stt-async-v4": { | ||
| "input_cost_per_second": 0.0, | ||
| "litellm_provider": "soniox", | ||
| "mode": "audio_transcription", | ||
| "source": "https://soniox.com/pricing", | ||
| "supported_endpoints": ["/v1/audio/transcriptions"] |
Contributor
There was a problem hiding this comment.
input_cost_per_second: 0.0 will break cost tracking
Setting the cost to 0.0 means all Soniox transcriptions will be reported as free regardless of actual usage. If Soniox has not published per-second pricing yet the field should be omitted (or set to null) rather than hardcoded to zero, so that litellm's cost tracking either skips the calculation or treats it as unknown — the same way other providers handle unpublished pricing.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+8505
to
+8507
| from litellm.llms.soniox.audio_transcription.transformation import ( | ||
| SonioxAudioTranscriptionConfig, | ||
| ) |
| response["duration"] = ( | ||
| float(transcription_meta["audio_duration_ms"]) / 1000.0 | ||
| ) | ||
| except (TypeError, ValueError): |
| AllMessageValues, | ||
| OpenAIAudioTranscriptionOptionalParams, | ||
| ) | ||
| from litellm.types.utils import FileTypes, TranscriptionResponse |
|
|
||
| from typing import Any, Dict, List, Optional | ||
|
|
||
| from litellm.llms.base_llm.chat.transformation import BaseLLMException |
| """Resolve the Soniox API key from arg or env var.""" | ||
| # Local import to avoid a circular import: litellm.secret_managers.main | ||
| # imports from litellm at top-level. | ||
| from litellm.secret_managers.main import get_secret_str |
|
|
||
| def get_soniox_api_base(api_base: Optional[str] = None) -> str: | ||
| """Resolve the Soniox API base URL (defaults to public API).""" | ||
| from litellm.secret_managers.main import get_secret_str |
…transport vcrpy's aiohttp stub captures response bodies via 'await response.read()', which drains aiohttp's StreamReader. Downstream consumers of the same ClientResponse (litellm's AiohttpResponseStream, which iterates response.content.iter_chunked) then see an empty body and surface as JSON 'Expecting value: line 1 column 1 (char 0)' errors on every record-path call. The previous workaround set litellm.disable_aiohttp_transport=True for the whole VCR-active session, which made the tests exercise pure httpx instead of the production aiohttp transport. That hid the production transport from coverage and surfaced its own bugs (e.g. the Azure DELETE-with-empty-body case fixed in upstream staging). Replace the workaround with a targeted monkey-patch that re-feeds the captured body into the StreamReader via unread_data after vcrpy records it. Tests now run through the same transport customers do, both on first record and on replay, for both unary and streaming endpoints. Verified locally against api.anthropic.com with the production LiteLLMAiohttpTransport: record path passes (real network, 4.2s), replay path passes (Redis cache, 1.8s).
The batch rate limiter (`_check_and_increment_batch_counters`) and the dynamic rate limiter (`_check_rate_limits`) implemented rate limiting in two disjoint awaits: a `should_rate_limit(read_only=True)` check followed by a separate increment. Concurrent requests could all observe the same pre-increment state, all pass enforcement, and all then increment — multiplying the effective quota by the concurrency level. Demonstrated bypass (see new test): - Batch: 5 concurrent batches of 40 tokens each against TPM=100 consumed 200 tokens (100% over). - Dynamic: 5 concurrent priority="high" requests against RPM=2 all passed Phase 1 + Phase 3. Wrap both critical sections in a per-instance asyncio.Lock so the read and increment execute atomically within a process. Multi-replica deployments still rely on Redis Lua atomicity for cross-process safety; that is a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/litellm into litellm_auth_bypass_tag_based_routing
…afety The previous fix for the TOCTOU bypass relied on a per-instance asyncio.Lock, which closed the window only within a single proxy worker. Multi-replica deployments still raced across processes — A and B both read counter=99, both passed validation, both incremented to 100/100 → effective limit doubled. Add `CHECK_AND_INCREMENT_BY_N_SCRIPT` Lua script that processes any number of (window_key, counter_key, limit, increment, ttl) descriptors atomically with all-or-nothing semantics: if any descriptor would exceed its limit, no counter is modified and the script returns OVER_LIMIT with the offending descriptor's state. When Redis isn't configured, the in-memory fallback uses the existing asyncio.Lock for single-process atomicity. Expose this as `_PROXY_MaxParallelRequestsHandler_v3.atomic_check_and_increment_by_n` and rewire both call sites: - batch_rate_limiter._check_and_increment_batch_counters: replace the read_only=True check + separate async_increment_tokens_with_ttl_preservation with a single atomic call passing the batch's (request_count, total_tokens) as the increment. - dynamic_rate_limiter_v3._check_rate_limits: bundle model_saturation_check (always enforced) and priority_model (enforced only when saturated) into one atomic call. When priority is unenforced, increment its counter via the existing should_rate_limit(read_only=False) path for tracking only. Update structural regression tests to assert the new atomic path is used rather than the legacy two-phase pattern. Tests: 4/4 TOCTOU tests pass, 59 existing rate-limiter tests pass, no regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply organization object_permission as a ceiling on allowed MCP servers and tool permissions, consistent with vector store org checks. Includes unit tests for org ceiling, intersection, and tool filtering. Made-with: Cursor
…orcement - Cache org object_permission in user_api_key_cache to avoid a DB hit on every MCP request (was: raw find_unique on every call). - Expand org mcp_servers list via expand_permission_list() so name-based entries resolve to canonical IDs, consistent with key/team/end-user path. - Expand org mcp_tool_permissions via expand_tool_permissions() in both _get_allowed_mcp_servers_for_org and get_allowed_tools_for_server, closing the silent name-vs-ID mismatch that could let restricted tools through. - The second _get_org_object_permission call in get_allowed_tools_for_server now hits the cache (warm from the earlier server-list check), resolving the double DB round-trip without changing the call structure. Made-with: Cursor
…equest DB hits Orgs with no MCP permissions configured (the common default) previously returned None without writing to cache, meaning every subsequent MCP request triggered a fresh find_unique against litellm_organizationtable. Cache a sentinel string on the negative path so the DB is queried at most once per cache TTL per org, regardless of whether the org has an object_permission or not. Made-with: Cursor
…ions feat(mcp): enforce org-level MCP server and toolset permissions
Newer Cloudflare Workers AI models (e.g. Nemotron) emit 'response_text' instead of 'response' on streamed chunks. The non-streaming path was already updated to fall back to 'response_text' (BerriAI#26385), but the streaming chunk parser still only read 'response', which caused streaming requests against those models to silently produce empty content. Mirror the non-streaming fallback in CloudflareChatResponseIterator.chunk_parser and add a streaming test for the response_text shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…itellm_oss_staging_04_25_2026 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…ses-stream-passthrough fix(guardrails): preserve responses event streams in presidio output masking
…25_2026 chore(staging): roll oss_staging_04_25_2026 into internal staging (output_config fix + 4 upstream sync fixes)
…m_id - Remove unused _encode_gcp_label_value / _decode_gcp_label_value singular helpers; only the _chunks variants are actually called. - Use 'is not None' check for custom_id so empty-string custom_ids are still labeled and round-trip through batch outputs. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…itellm_vertex-batch-output-transformation Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
…_output_to_openai The method was overwriting logging_obj.optional_params, logging_obj.model, and logging_obj.start_time on the caller's Logging instance. When invoked from llm_http_handler.py's generic framework path, the framework's own logging_obj (which already went through pre_call) had its properties clobbered, causing model and start_time to reflect the last batch line's values rather than the original call context. Fix: create a fresh local Logging instance for the per-line transformation instead of mutating the incoming logging_obj. The caller's object is now left entirely untouched regardless of whether a logging_obj was passed in or not. Regression tests added to verify model, start_time, and optional_params are not mutated on the caller's logging_obj. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
Adds litellm.disable_vertex_batch_output_transformation (default False). When True, afile_content returns raw Vertex predictions.jsonl untouched so users that parse candidates/modelVersion directly are not broken.
…fort="none"
Setting reasoning_effort="none" on Anthropic chat models (direct, Bedrock
Invoke, Bedrock Converse, Vertex AI Anthropic, Azure AI Anthropic) crashed
LiteLLM with:
litellm.APIConnectionError: 'NoneType' object has no attribute 'get'
Both the Anthropic chat transformation and Bedrock Converse called
``AnthropicConfig._map_reasoning_effort`` and assigned the ``None`` it returns
for ``"none"`` directly to ``optional_params["thinking"]``. Downstream
``is_thinking_enabled`` then did ``optional_params["thinking"].get("type")``
and crashed.
Pop ``thinking`` (and on Claude 4.6/4.7, ``output_config``) instead of
assigning ``None``, restoring the documented contract that
``reasoning_effort="none"`` means "do not enable thinking". This also
prevents downstream Anthropic 400s ("thinking: Input should be an object",
"output_config.effort: Input should be ...") if the bug were ever masked.
Verified end-to-end against the live Anthropic API and Bedrock Converse
on claude-opus-4-{5,6,7} and claude-sonnet-4-6, plus Bedrock Invoke for
Claude 4.5/4.6. Vertex AI Anthropic and Azure AI Anthropic inherit the
fixed ``map_openai_params`` from ``AnthropicConfig`` and need no further
changes.
…tput-transformation feat(vertex-ai): transform batch prediction outputs to OpenAI format
The Vertex batch output transformer was emitting both a populated 'response' and 'error' for failed batch entries. The OpenAI Batch output spec defines them as mutually exclusive: on error 'response' MUST be null. This broke any consumer using 'result["response"] is None' to detect failures.
…itellm_fix_reasoning_effort_none_anthropic Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…ffort_none_anthropic fix(anthropic,bedrock): omit thinking/output_config when reasoning_effort="none"
…ror-response-null-46dd fix(vertex-ai): set response=null on batch error entries per OpenAI spec
dan2k3k4
force-pushed
the
add-soniox-provider
branch
from
May 4, 2026 09:24
3a83f3f to
a00a53d
Compare
| {canModify && ( | ||
| <AddFallbacks | ||
| models={modelData?.data ? modelData.data.map((data: any) => data.model_name) : []} | ||
| accessToken={accessToken || ""} |
Contributor
Author
|
Restarted the branch and PR against the litellm_internal_staging branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-Submission checklist
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewCI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🆕 New Feature
Changes