Litellm oss staging 04 21 2026 - #26216
Conversation
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Error
GitHub Actions
Inactive
Greptile SummaryThis staging PR bundles several independent features: a new Reducto OCR integration, a Rubrik guardrail/batch-logging plugin, an async single-flight credential refresh for Vertex AI, selective Confidence Score: 4/5Safe to merge with awareness of the two pre-existing Rubrik/Vertex issues tracked in thread comments. The only new finding in this pass is a P2 KeyError in the Reducto upload helpers. The two P1-level issues (asyncio.create_task in init and STALE+malformed token skip) were identified in the prior review cycle and are already tracked in thread comments, so the score ceiling is 4/5. litellm/integrations/rubrik.py and litellm/proxy/guardrails/guardrail_hooks/rubrik/init.py (asyncio.create_task + event_hook=None bypass); litellm/llms/vertex_ai/vertex_llm_base.py (STALE+malformed token path)
|
| Filename | Overview |
|---|---|
| litellm/integrations/rubrik.py | New Rubrik guardrail/batch-logging plugin; two known issues (asyncio.create_task in init, setdefault bypass when mode=None) are tracked in previous threads and not yet resolved. |
| litellm/llms/reducto/ocr/transformation.py | New Reducto OCR transformation; sync and async upload paths look correct but file_id access lacks graceful error handling. |
| litellm/llms/reducto/common.py | Reducto upload helpers and page builder; upload functions use unguarded dict key access that raises a bare KeyError on unexpected API responses. |
| litellm/llms/vertex_ai/vertex_llm_base.py | New async token refresh with single-flight locking and STALE/INVALID state machine; two issues (STALE+malformed token silently skips refresh, _background_refresh_tasks unbounded growth) remain from previous threads. |
| litellm/llms/fireworks_ai/chat/transformation.py | get_provider_info now reads supports_function_calling/reasoning from model_cost map with a default True fallback for unmapped models; parallel_tool_calls added when tool_choice is supported. |
| litellm/llms/bedrock/chat/converse_transformation.py | output_config now conditionally forwarded via additionalModelRequestFields for models with supports_output_config=true (Claude 4.6+); correct and well-tested. |
| litellm/llms/chatgpt/responses/transformation.py | Refactored SSE parsing into helper methods and added output_item.done tracking to fill in missing output when response.completed payload is empty. |
| litellm/router.py | Prevents shared model key mode from being downgraded when a deployment alias with a different mode registers (e.g. chat alias can't downgrade a responses backend). |
| litellm/proxy/guardrails/guardrail_hooks/rubrik/init.py | Rubrik guardrail initializer; passes event_hook=litellm_params.mode explicitly, bypassing the setdefault fallback when mode is None (tracked in previous threads). |
| litellm/cost_calculator.py | Added credit-based pricing support (cost_per_credit) for Reducto and changed missing pages_processed from hard error to silent zero-cost return. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Vertex _ensure_access_token_async] --> B{Token in cache?}
B -- FRESH --> C[Return cached token]
B -- Not cached / STALE / INVALID --> D[Acquire per-key asyncio.Lock]
D --> E{Double-check after lock}
E -- FRESH --> C
E -- Not cached --> F[load_auth in thread pool]
F --> G{token_state?}
G -- STALE + valid token --> H[Schedule background refresh task]
H --> I[Return current token immediately]
G -- STALE + malformed token --> J[pass — fall-through]
J --> K{token_state == INVALID?}
K -- No, still STALE --> L[Final validation raises ValueError ⚠️]
G -- INVALID --> M[Block on asyncify refresh_auth]
M --> N[Return refreshed token]
Reviews (11): Last reviewed commit: "Fix failing tests" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| ] = {} | ||
| self.project_id: Optional[str] = None | ||
| self.async_handler: Optional[AsyncHTTPHandler] = None | ||
| # Per-credential-key asyncio.Lock for single-flight async refresh. | ||
| # Prevents thundering herd when token expires under high concurrency. | ||
| self._async_refresh_locks: Dict[tuple, asyncio.Lock] = {} | ||
| # Tracks in-flight background refresh tasks to avoid duplicate refreshes. | ||
| self._background_refresh_tasks: Dict[tuple, asyncio.Task] = {} |
There was a problem hiding this comment.
_background_refresh_tasks grows unboundedly
_background_refresh_tasks replaces completed tasks for the same credential key when the STALE path is re-entered, but completed tasks for keys that never go STALE again are never pruned. In long-running processes with many distinct credential configurations, this dict accumulates asyncio.Task references indefinitely.
A lightweight cleanup: before storing a new task, also sweep out done tasks that are no longer needed — or periodically prune the dict.
| if token_state == TokenState.STALE: | ||
| resolved_project = project_id | ||
| if resolved_project is None: | ||
| raise ValueError("Could not resolve project_id") | ||
| current_token = _credentials.token | ||
| if current_token is None or not isinstance(current_token, str): | ||
| # Token is malformed despite STALE state — fall through | ||
| # to INVALID path which will block on a full refresh. | ||
| pass | ||
| else: | ||
| # Schedule a single background refresh — skip if one is | ||
| # already in flight for this credential key. | ||
| existing = self._background_refresh_tasks.get(credential_cache_key) | ||
| if existing is None or existing.done(): | ||
| task = asyncio.create_task( | ||
| self._background_refresh_credentials( | ||
| _credentials, | ||
| credential_cache_key, | ||
| credential_project_id, | ||
| ) | ||
| ) | ||
| self._background_refresh_tasks[credential_cache_key] = task | ||
| return current_token, resolved_project | ||
|
|
||
| if token_state == TokenState.INVALID: | ||
| # Token is expired or missing — must block until refresh completes. | ||
| try: | ||
| verbose_logger.debug("Credentials expired, refreshing") | ||
| await asyncify(self.refresh_auth)(_credentials) | ||
| self._credentials_project_mapping[credential_cache_key] = ( | ||
| _credentials, | ||
| credential_project_id, | ||
| ) | ||
| except Exception as e: | ||
| if "Reauthentication is needed" in str(e): | ||
| verbose_logger.debug( | ||
| "Reauthentication needed, clearing cache and retrying" | ||
| ) | ||
| if credential_cache_key in self._credentials_project_mapping: | ||
| del self._credentials_project_mapping[credential_cache_key] | ||
| return await asyncify(self._handle_reauthentication)( | ||
| credentials=credentials, | ||
| project_id=project_id, | ||
| credential_cache_key=credential_cache_key, | ||
| error=e, | ||
| ) | ||
| raise |
There was a problem hiding this comment.
When token_state == TokenState.STALE but current_token is None (or non-string), the code hits pass and does not return. The next block (if token_state == TokenState.INVALID:) is then skipped because token_state is still STALE — not INVALID — so no refresh happens. Execution falls through to "Final validation" which raises a bare ValueError without ever refreshing, directly contradicting the comment that says it will "fall through to INVALID path which will block on a full refresh."
The fix is to track whether a blocking refresh is needed with a boolean flag (needs_sync_refresh = False), set it to True in the malformed-token branch of the STALE block, and change the INVALID guard to if token_state == TokenState.INVALID or needs_sync_refresh:.
GitHub Actions
Inactive
|
|
GitHub Actions
Error
GitHub Actions
Inactive
GitHub Actions
Inactive
| "supports_vision": True, # via document inlining | ||
| } | ||
|
|
||
| if supports_function_calling_value is not None: | ||
| provider_specific_model_info["supports_function_calling"] = ( | ||
| supports_function_calling_value | ||
| ) | ||
|
|
||
| # Only include supports_reasoning if True | ||
| if supports_reasoning_value: | ||
| provider_specific_model_info["supports_reasoning"] = True | ||
| if supports_reasoning_value is not None: | ||
| provider_specific_model_info["supports_reasoning"] = ( | ||
| supports_reasoning_value | ||
| ) | ||
|
|
||
| return provider_specific_model_info | ||
|
|
There was a problem hiding this comment.
supports_function_calling on unmapped models
The old get_provider_info always seeded "supports_function_calling": True in the base dict. The new version only adds the key when _get_model_cost_capability returns a non-None value. For any model not present in litellm.model_cost, the key is absent entirely.
The companion test test_unmapped_model_fallback_function_calling asserts info["supports_function_calling"] is True, but because ProviderSpecificModelInfo is a plain TypedDict(total=False) with no runtime defaults, this will raise KeyError for any unmapped model — making the test itself a failing test that ships with the regression.
Callers that rely on get_provider_info to decide whether to include tool-call params will silently skip function calling for any new Fireworks model that hasn't been added to the model cost map yet.
| "supports_vision": True, # via document inlining | |
| } | |
| if supports_function_calling_value is not None: | |
| provider_specific_model_info["supports_function_calling"] = ( | |
| supports_function_calling_value | |
| ) | |
| # Only include supports_reasoning if True | |
| if supports_reasoning_value: | |
| provider_specific_model_info["supports_reasoning"] = True | |
| if supports_reasoning_value is not None: | |
| provider_specific_model_info["supports_reasoning"] = ( | |
| supports_reasoning_value | |
| ) | |
| return provider_specific_model_info | |
| provider_specific_model_info: ProviderSpecificModelInfo = { | |
| "supports_function_calling": True, # default: Fireworks supports FC | |
| "supports_parallel_function_calling": True, | |
| "supports_vision": True, # via document inlining | |
| } | |
| if supports_function_calling_value is not None: | |
| provider_specific_model_info["supports_function_calling"] = ( | |
| supports_function_calling_value | |
| ) |
GitHub Actions
Inactive
GitHub Actions
Error
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Error
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Error
| if system_prompt_msg_list: | ||
| system_scaffold = { | ||
| "role": "system", | ||
| "content": system_prompt_msg_list, |
There was a problem hiding this comment.
asyncio.create_task called from synchronous __init__
asyncio.create_task(self.periodic_flush()) requires a running event loop. When RubrikLogger is instantiated during synchronous proxy startup (e.g., before the ASGI lifespan is active), this raises RuntimeError: no running event loop in Python 3.10+. The tests already confirm this is broken — every test fixture wraps construction with patch("asyncio.create_task", Mock()) to suppress the error rather than expose it.
A safer pattern is to defer the task to the first async call, or use a startup hook:
self._periodic_flush_task: Optional[asyncio.Task] = None
async def _ensure_periodic_flush_started(self) -> None:
if self._periodic_flush_task is None or self._periodic_flush_task.done():
self._periodic_flush_task = asyncio.create_task(self.periodic_flush())Then call await self._ensure_periodic_flush_started() at the top of the async logging/guardrail hooks.
GitHub Actions
Inactive
GitHub Actions
Error
GitHub Actions
Inactive
GitHub Actions
Error
…21_2026' into fix/bedrock-invoke-output-config-effort-4-6" This reverts commit d10ef78.
The partner/gemma/model-garden handlers now call self._ensure_access_token (inherited from VertexBase) instead of instantiating VertexLLM, so tests must patch VertexBase._ensure_access_token for the mock to take effect.
The gate at three call sites was calling _supports_factory with custom_llm_provider=None, which relies on get_llm_provider inferring the provider from the model string. For the invoke path, the model still carries an 'invoke/' routing prefix (e.g. 'invoke/us.anthropic.claude-opus-4-6-v1') that is not a known provider, so inference raises BadRequestError, _supports_factory swallows it and returns False, and the user's output_config.effort gets silently dropped before the Bedrock request. Strip the routing prefix with the existing strip_bedrock_routing_prefix helper and pass custom_llm_provider='bedrock' explicitly so the declarative 'supports_output_config' flag in model_prices_and_context_window.json is the actual source of truth. Also adds a regression test that exercises the full 'invoke/us.anthropic.claude-opus-4-6-v1' path and asserts output_config survives.
* feat(ocr): add Reducto parse OCR support * fix(reducto): address OCR review feedback * chore: refresh uv lockfile * Revert "chore: refresh uv lockfile" This reverts commit 47200c0.
be7b0ff to
91c355d
Compare
Low: No security issues foundThis PR adds a Rubrik guardrail integration (tool blocking + batch logging), a Reducto OCR provider, Vertex AI credential caching optimizations, Bedrock The Rubrik integration follows existing guardrail patterns — it uses operator-configured endpoints, deep-copies logging payloads, and forwards Status: 0 open Posted by Veria AI · 2026-04-24T11:26:27.697Z |
Low: No security issues foundThis PR adds a Rubrik guardrail integration, a Reducto OCR provider, Vertex AI credential caching improvements, Bedrock output_config feature gating, and dependency version range loosening. The Rubrik integration follows established guardrail patterns (operator-configured external service). The Reducto provider validates input properly, rejecting plain HTTP URLs and only accepting reducto:// IDs or base64 data URIs. The Vertex AI changes improve credential refresh with proper per-key locking. No exploitable vulnerabilities identified. Status: 0 open Posted by Veria AI · 2026-04-24T11:42:13.883Z |
Low: No significant security issues foundThis PR adds a Rubrik guardrail integration (tool blocking + batch logging), a Reducto OCR provider, improves Vertex AI credential caching with async single-flight refresh, and updates Bedrock/Fireworks model capabilities. The new integrations follow established patterns for operator-configured external services and provider API calls. No injection, auth bypass, or privilege escalation vectors identified. Status: 0 open Posted by Veria AI · 2026-04-24T11:46:54.889Z |
Low: No security issues foundThis PR adds a Reducto OCR provider integration, a Rubrik guardrail/logging integration, Vertex AI credential caching improvements, Bedrock output_config feature flags, and dependency range loosening. The new Rubrik integration sends data to an operator-configured external service using the same patterns as existing guardrail integrations (cleaned headers, standard logging payloads). The Reducto integration properly validates input URLs and rejects plain http(s) URLs to prevent SSRF. No exploitable vulnerabilities identified. Status: 0 open Posted by Veria AI · 2026-04-24T11:50:33.293Z |
Low: No security issues foundThis PR adds a new Rubrik guardrail integration (operator-configured tool blocking and batch logging), a new Reducto OCR provider, Vertex AI credential caching improvements with proper async locking, Bedrock output_config support for Claude 4.6+ models, SSE response recovery logic, model pricing updates, and dependency version loosening. All new external service integrations (Rubrik, Reducto) are operator-configured and auth-gated through the standard proxy authentication middleware. The Vertex AI credential refactor correctly uses per-key async locks and double-checked locking to prevent thundering herd issues. Status: 0 open Posted by Veria AI · 2026-04-24T11:51:58.799Z |
Low: No security issues foundThis PR adds a Rubrik guardrail integration, Reducto OCR provider, Vertex AI credential caching improvements, Bedrock output_config support, ChatGPT SSE output recovery, and dependency range relaxation. All new integrations follow established patterns — the Rubrik guardrail sends data to an operator-configured endpoint (consistent with other guardrails), and the Reducto provider properly validates input URLs and rejects plain HTTP URLs. The dependency version ranges in pyproject.toml are loosened from exact pins to minimums for the core SDK, while proxy extras retain exact pins. Status: 0 open Posted by Veria AI · 2026-04-24T11:58:14.122Z |
Low: No security issues foundThis PR adds a Rubrik guardrail integration (tool blocking + batch logging), a Reducto OCR provider, async credential refresh for Vertex AI, Bedrock output_config support, ChatGPT SSE parsing improvements, and model pricing updates. The Rubrik integration sends request context to an operator-configured external service (consistent with other guardrail integrations). The Reducto integration properly rejects plain HTTP URLs and validates base64 input. Dependency version ranges were relaxed for core SDK but proxy extras retain exact pins. No exploitable vulnerabilities identified. Status: 0 open Posted by Veria AI · 2026-04-24T11:59:48.415Z |
Low: No security issues foundThis PR adds a Reducto OCR provider integration, a Rubrik guardrail/logging integration, Vertex AI credential caching improvements, Bedrock output_config support, and loosens core SDK dependency pins from exact to range specifiers. All new integrations follow existing security patterns: OCR endpoints are auth-protected, Reducto rejects plain HTTP URLs (preventing SSRF), and the Rubrik webhook is operator-configured. The dependency loosening applies only to the core SDK (not proxy extras which remain pinned). Status: 0 open Posted by Veria AI · 2026-04-24T12:07:05.575Z |
The 12 core `[project.dependencies]` entries in pyproject.toml were exact `==` pins, a side effect of the Poetry → uv migration. This forces every downstream package that lists litellm as a dependency to downgrade common runtime libraries (openai, pydantic, aiohttp, click, jsonschema, ...) to the exact versions we ship. Customers have flagged this as a coexistence blocker. Switch to lower-bounded ranges with upper bounds where the upstream package is pre-1.0 or has a known breaking-major-version policy. Reproducibility for our Docker proxy and CI continues to come from `uv.lock`, which is regenerated here as a metadata-only diff (no resolved versions or hashes change). Inspired by #26157 (which got stranded on `litellm_oss_staging_04_21_2026` when the forward-merge to internal staging in #26216 was closed). Floors in this PR are tighter than #26157's: they were validated by installing litellm at `--resolution=lowest-direct` and importing the openai-namespace symbols the codebase actually uses. Floor highlights vs #26157: - openai >= 2.20 (was 2.0) — Responses API symbols + `Omit` need a 2.x mid-range floor - httpx >= 0.28, < 1.0 (was no upper) — pre-1.0 - importlib-metadata >= 8.0 (was 6.0) — stay in tested major - tokenizers >= 0.20, < 1.0 (was 0.19, no upper) — pre-1.0 - aiohttp >= 3.10, < 4.0 (was no upper) — bound major - pydantic >= 2.5, < 3.0 — kept - All other floors: keep tested major, add upper bound Adds a `check-dependency-floors.yml` GitHub Actions workflow that installs litellm at `--resolution=lowest-direct` on Python 3.10 and 3.13 and import-checks every openai symbol the codebase uses, so a future floor regression fails fast in CI rather than silently in the field.
The 12 core `[project.dependencies]` entries in pyproject.toml were exact `==` pins, a side effect of the Poetry → uv migration. This forces every downstream package that lists litellm as a dependency to downgrade common runtime libraries (openai, pydantic, aiohttp, click, jsonschema, ...) to the exact versions we ship. Customers have flagged this as a coexistence blocker. Switch to lower-bounded ranges with upper bounds where the upstream package is pre-1.0 or has a known breaking-major-version policy. Reproducibility for our Docker proxy and CI continues to come from `uv.lock`, which is regenerated here as a metadata-only diff (no resolved versions or hashes change). Inspired by BerriAI#26157 (which got stranded on `litellm_oss_staging_04_21_2026` when the forward-merge to internal staging in BerriAI#26216 was closed). Floors in this PR are tighter than BerriAI#26157's: they were validated by installing litellm at `--resolution=lowest-direct` and importing the openai-namespace symbols the codebase actually uses. Floor highlights vs BerriAI#26157: - openai >= 2.20 (was 2.0) — Responses API symbols + `Omit` need a 2.x mid-range floor - httpx >= 0.28, < 1.0 (was no upper) — pre-1.0 - importlib-metadata >= 8.0 (was 6.0) — stay in tested major - tokenizers >= 0.20, < 1.0 (was 0.19, no upper) — pre-1.0 - aiohttp >= 3.10, < 4.0 (was no upper) — bound major - pydantic >= 2.5, < 3.0 — kept - All other floors: keep tested major, add upper bound Adds a `check-dependency-floors.yml` GitHub Actions workflow that installs litellm at `--resolution=lowest-direct` on Python 3.10 and 3.13 and import-checks every openai symbol the codebase uses, so a future floor regression fails fast in CI rather than silently in the field.
12 participants
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes