🔄 Upstream Sync: LiteLLM v1.81.0-stable - #83
Closed
Cartofante wants to merge 7463 commits into
Closed
Conversation
* feat: add clientip and user agent in metrics * fix: lint errors * Add model id and other req labels --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* add timeout to onyx guardrail * add tests
Automatic sync from upstream BerriAI/litellm Preparing for v1.81.0-stable release Strategy: Merge with history preservation (main mirrors upstream)
…s_hook (BerriAI#19670) * fix(proxy): use return value from CustomLogger.async_post_call_success_hook Previously the return value was ignored for CustomLogger callbacks, preventing users from modifying responses. Now the return value is captured and used to replace the response (if not None), consistent with CustomGuardrail and streaming iterator hook behavior. Fixes issue with custom_callbacks not being able to inject data into LLM responses. * fix(proxy): also fix async_post_call_streaming_hook to use return value Previously the streaming hook only used return values that started with "data: " (SSE format). Now any non-None return value is used, consistent with async_post_call_success_hook and streaming iterator hook behavior. Added tests for streaming hook transformation. --------- Co-authored-by: Gabriele Michelli <michelligabriele0@gmail.com>
Adds support for Anthropic-style 'thinking' parameter in hosted_vllm, converting it to OpenAI-style 'reasoning_effort' since vLLM is OpenAI-compatible. This enables users to use Claude Code CLI with hosted vLLM models like GLM-4.6/4.7 through the /v1/messages endpoint. Mapping (same as Anthropic adapter): - budget_tokens >= 10000 -> "high" - budget_tokens >= 5000 -> "medium" - budget_tokens >= 2000 -> "low" - budget_tokens < 2000 -> "minimal" Fixes BerriAI#19761
- Added a Pydantic validator to convert empty string inputs for max_budget to None, preventing float parsing errors from the frontend. - Modified the internal user update logic to explicitly allow max_budget to be None, ensuring the value isn't filtered out and can be reset to unlimited in the database. - Added unit tests for validation and logic. Closes BerriAI#19781
…t user (BerriAI#19795) - Create a test user with auto_create_key=False to ensure known starting state - Filter get_users by user_ids to target only the test user - Verify initial key count is 0 before creating a key - Clean up test user after test completes - This ensures consistent behavior across CI and local environments
…BerriAI#19797) - Add test_get_valid_args in test_router_helper_utils.py to cover get_valid_args - Use encoding='utf-8' in router_code_coverage.py for cross-platform file reads
…tion error (BerriAI#19801) Mock _create_mcp_client to avoid network calls in health checks. This prevents asyncio.CancelledError when the test teardown closes the event loop while health checks are still pending. The test focuses on conversion logic (access_groups, description) not health check functionality, so mocking the network call is appropriate.
…19803) * fix: make HTTPHandler mockable in OIDC secret manager tests - Add _get_oidc_http_handler() factory function to make HTTPHandler easily mockable in tests - Update test_oidc_github_success to patch factory function instead of HTTPHandler directly - Update Google OIDC tests for consistency - Fixes test_oidc_github_success failure where mock was bypassed This change allows tests to properly mock HTTPHandler instances used for OIDC token requests, fixing the test failure where the mock was not being used. * fix: patch base_llm_http_handler method directly in container tests - Use patch.object to patch container_create_handler method directly on the base_llm_http_handler instance instead of patching the module - Fixes test_provider_support[openai] failure where mock wasn't applied - Also fixes test_error_handling_integration with same approach The issue was that patching 'litellm.containers.main.base_llm_http_handler' didn't work because the module imports it with 'from litellm.main import', creating a local reference. Using patch.object patches the method on the actual object instance, which works regardless of import style. * fix: resolve flaky test_openai_env_base by clearing cache - Add cache clearing at start of test_openai_env_base to prevent cache pollution - Ensures no cached clients from previous tests interfere with respx mocks - Fixes intermittent failures where aiohttp transport was used instead of httpx - Test-only change with low risk, no production code modifications Resolves flaky test marked with @pytest.mark.flaky(retries=3, delay=1) Both parametrized versions (OPENAI_API_BASE and OPENAI_BASE_URL) now pass consistently * test: add explicit mock verification in test_provider_support - Capture mock handler with 'as mock_handler' for explicit validation - Add assert_called_once() to verify mock was actually used - Ensures test verifies no real API calls are made - Follows same pattern as test_openai_env_base validation
* cache control for user messages and system messages * add cache createion tokens in reponse * cache controls in tool calls and assistant turns * refactor with _should_preserve_cache_control * add cache control unit tests * use simpler cache creation token count logic * use helper function * remove unused function * fix unit tests
* enable progress notifications for MCP tool calls * adjust mcp test
… variable (BerriAI#19780) * fix: add CLI_JWT_EXPIRATION_HOURS * docs: CLI_JWT_EXPIRATION_HOURS * fix: get_cli_jwt_auth_token * test_get_cli_jwt_auth_token_custom_expiration
fix(ui): prevent clearing content filter patterns when editing guardrail
[Infra] CI/CD - Fixing Flaky Tests in OIDC and Email
Fix(BerriAI#19781): Unable to reset user max budget to unlimited
…iAI#19826) * Fix PLR0915: Extract system message handling to reduce statement count * fix mypy * fix: add host_progress_callback parameter to mock_call_tool in test The test_call_tool_without_broken_pipe_error was failing because the mock function did not accept the host_progress_callback keyword argument that the actual implementation passes to client.call_tool(). Updated the mock to accept this parameter to match the real implementation signature. * fixing flaky tests around oidc and email * Add documentation comment to test file * add retry * add dependency * increase retry --------- Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Collaborator
Author
✅ CI Fix CompleteFixes applied and pushed to this PR.
Note CI will re-run automatically. Monitor check results below. Next Steps
🔧 Workflow Details (click to expand)Workflow Run: https://github.com/CartoDB/litellm/actions/runs/21500309767 |
Collaborator
Author
🔧 CI Auto-Fix StartedStatus: ⏳ In progress... Claude Code (Opus 4.5) is analyzing and fixing CI failures. Failed Checks:
Note This may take 15-30 minutes. Fixes will be pushed directly to this PR. |
- Added missing reconstruct_model_name function to core_helpers.py - Fixed CredentialsPanel TypeScript error (component now uses hooks internally) CARTO customizations preserved: All Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Collaborator
Author
✅ CI Failures Fixed (CARTO-first strategy)CARTO Customizations Status: ✅ All preserved Fixes Applied:
Root Cause:
Verification:
Commit: 🤖 Generated with Claude Code |
Collaborator
Author
✅ CI Fix CompleteFixes applied and pushed to this PR.
Note CI will re-run automatically. Monitor check results below. Next Steps
🔧 Workflow Details (click to expand)Workflow Run: https://github.com/CartoDB/litellm/actions/runs/21500750640 |
Collaborator
Author
🔧 CI Auto-Fix StartedStatus: ⏳ In progress... Claude Code (Opus 4.5) is analyzing and fixing CI failures. Failed Checks:
Note This may take 15-30 minutes. Fixes will be pushed directly to this PR. |
Root cause: Circular import error during test initialization.
The error was: "ImportError: cannot import name 'get_llm_provider' from
partially initialized module 'litellm' (most likely due to a circular import)"
Fixes applied:
1. Restored truncated files from upstream that were incomplete after merge:
- litellm/images/main.py (+267 lines)
- litellm/litellm_core_utils/prompt_templates/common_utils.py (+177 lines)
- litellm/litellm_core_utils/prompt_templates/factory.py (+771 lines)
- litellm/llms/custom_httpx/http_handler.py (+325 lines)
- litellm/realtime_api/main.py (restored from upstream)
- litellm/types/router.py (+35 lines)
- litellm/vector_stores/vector_store_registry.py (+369 lines)
2. Fixed circular imports:
- litellm/llms/azure/azure.py: Changed litellm.ImageResponse to ImageResponse
- litellm/llms/bedrock/image_generation/amazon_titan_transformation.py:
Import get_model_info from litellm.utils instead of litellm
- litellm/llms/ovhcloud/chat/transformation.py: Direct module imports
The upstream files use proper import patterns (getattr for lazy loading,
direct module imports to avoid circular dependencies) which resolve the
circular import chain that was blocking test initialization.
CARTO Customizations Status: All preserved (no CARTO-specific code was reverted)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Collaborator
Author
✅ CI Failures Fixed (CARTO-first strategy)CARTO Customizations Status: ✅ All preserved (no CARTO-specific code was reverted) Root CauseThe test run was failing with a circular import error: This occurred during pytest's DiagnosisThe circular import chain was:
Additionally, several files were truncated during the merge:
Fixes Applied
Verification
Commit
CI should re-run automatically. The test initialization should now succeed. 🤖 Generated with Claude Code |
Collaborator
Author
✅ CI Fix CompleteFixes applied and pushed to this PR.
Note CI will re-run automatically. Monitor check results below. Next Steps
🔧 Workflow Details (click to expand)Workflow Run: https://github.com/CartoDB/litellm/actions/runs/21500858867 |
Collaborator
Author
🔧 CI Auto-Fix StartedStatus: ⏳ In progress... Claude Code (Opus 4.5) is analyzing and fixing CI failures. Failed Checks:
Note This may take 15-30 minutes. Fixes will be pushed directly to this PR. |
- Restored callback_utils.py from upstream (was truncated in merge): - Added missing normalize_callback_names function - Added missing process_callback function - Added missing add_policy_to_applied_policies_header function - Added missing add_guardrail_response_to_standard_logging_object function - Added new callback handlers (websearch_interception, datadog_cost_management) - Updated initialize_callbacks_on_proxy with LoggingCallbackManager integration - Fixed TypeScript type error in PriceDataManagementTab.tsx: - Changed modelCostMap(accessToken) to modelCostMap() (function takes 0 args) CARTO customizations preserved: - No CARTO-specific customizations in callback_utils.py (pure upstream code) - No CARTO changes affected in UI components Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Collaborator
Author
✅ CI Failures Fixed (CARTO-first strategy)CARTO Customizations Status: ✅ All preserved (no CARTO-specific code was reverted) Root CauseTest failures: Multiple tests were failing with Docker build failure: TypeScript type error - Fixes Applied
Functions Restored in callback_utils.py
Verification
Commit: CI should re-run automatically. All test imports should now resolve correctly. 🤖 Generated with Claude Code |
Collaborator
Author
❌ CI Fix FailedThe automated fix encountered an error.
Warning Manual intervention required. Check the workflow logs below. 🔧 Troubleshooting (click to expand)Common issues:
Manual fix steps:
|
Collaborator
Author
🔧 CI Auto-Fix StartedStatus: ⏳ In progress... Claude Code (Opus 4.5) is analyzing and fixing CI failures. Failed Checks:
Note This may take 15-30 minutes. Fixes will be pushed directly to this PR. |
The custom_guardrail.py file was truncated during the merge (599 lines vs 799 in upstream), missing the ModifyResponseException class which is imported by multiple test files. Restored complete file from upstream v1.81.0-stable which includes: - ModifyResponseException class (lines 38-74) - render_violation_message method - raise_passthrough_exception helper - violation_message_template parameter - GenericGuardrailAPIInputs type usage - TYPE_CHECKING imports This file had no CARTO-specific customizations, so using upstream version directly. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Collaborator
Author
✅ CI Failures Fixed (CARTO-first strategy)CARTO Customizations Status: ✅ All preserved (no CARTO code was reverted) Root CauseThe test run was failing with multiple This affected 10+ test files including:
AnalysisThe
Fix AppliedRestored the complete file from upstream v1.81.0-stable. This file had no CARTO-specific customizations - it was simply an incomplete merge artifact. Verification
Commit: 🤖 Generated with Claude Code |
Collaborator
Author
✅ CI Fix CompleteFixes applied and pushed to this PR.
Note CI will re-run automatically. Monitor check results below. Next Steps
🔧 Workflow Details (click to expand)Workflow Run: https://github.com/CartoDB/litellm/actions/runs/21501234601 |
Collaborator
Author
🔧 CI Auto-Fix StartedStatus: ⏳ In progress... Claude Code (Opus 4.5) is analyzing and fixing CI failures. Failed Checks:
Note This may take 15-30 minutes. Fixes will be pushed directly to this PR. |
… props
Error fixed 1: ImportError: cannot import name 'convert_upload_files_to_file_data'
File: litellm/proxy/common_utils/http_parsing_utils.py
Change: Added missing convert_upload_files_to_file_data function from upstream
Error fixed 2: Type error: '{ accessToken: any; ... proxySettings: ProxySettings | undefined; }' is not assignable to type 'ChatUIProps'
File: ui/litellm-dashboard/src/app/(dashboard)/playground/page.tsx
Change: Removed proxySettings prop that ChatUI doesn't accept
Previous attempts: 5+ commits already tried
This fix addresses: Missing Python function and TypeScript prop mismatch
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Collaborator
Author
🔧 CI Fix AppliedError Fixed 1 (Mock Tests): Fix Applied:
Error Fixed 2 (Docker Build): Fix Applied:
Verification: Pushed to branch, CI will re-run automatically. Fix attempt #6 - If this fails, I'll analyze the new errors and try again. |
Collaborator
Author
✅ CI Fix CompleteFixes applied and pushed to this PR.
Note CI will re-run automatically. Monitor check results below. Next Steps
🔧 Workflow Details (click to expand)Workflow Run: https://github.com/CartoDB/litellm/actions/runs/21501361595 |
Collaborator
Author
🔧 CI Auto-Fix StartedStatus: ⏳ In progress... Claude Code (Opus 4.5) is analyzing and fixing CI failures. Failed Checks:
Note This may take 15-30 minutes. Fixes will be pushed directly to this PR. |
Error fixed: ImportError: cannot import name 'create_response' from 'litellm.proxy.common_request_processing' File: litellm/proxy/common_request_processing.py Change: Added create_response and _extract_error_from_sse_chunk functions, and JSONResponse import These functions were missing after the upstream merge. They are required by proxy_server.py and are used for streaming response error handling. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Collaborator
Author
❌ CI Fix FailedThe automated fix encountered an error.
Warning Manual intervention required. Check the workflow logs below. 🔧 Troubleshooting (click to expand)Common issues:
Manual fix steps:
|
Collaborator
Author
🔧 CI Auto-Fix StartedStatus: ⏳ In progress... Claude Code (Opus 4.5) is analyzing and fixing CI failures. Failed Checks:
Note This may take 15-30 minutes. Fixes will be pushed directly to this PR. |
Collaborator
Author
❌ CI Fix FailedThe automated fix encountered an error.
Warning Manual intervention required. Check the workflow logs below. 🔧 Troubleshooting (click to expand)Common issues:
Manual fix steps:
|
Collaborator
|
Closing due to messy commit history from CI Fixer iterations. Will re-trigger with improved CI Fixer that reads full logs and understands the feedback loop better. |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔄 Upstream Sync: LiteLLM v1.81.0-stable
Syncs CARTO's LiteLLM fork with upstream stable release v1.81.0-stable.
1.79.1→v1.81.0-stableCaution
Use "Create a merge commit" only. Squashing destroys upstream history and breaks future syncs.
🧪 Pre-Merge Checklist
pyproject.tomlversion matches upstream📊 Release Information (click to expand)
v1.81.0-stable1.79.1🔀 Branch Flow (click to expand)
BerriAI/litellm:mainmerged intoCartoDB/litellm:mainupstream-sync/v1.81.0-stableupstream-sync/v1.81.0-stable→carto/main📝 CARTO-Specific File Guidelines (click to expand)
When reviewing or resolving conflicts:
✅ Keep CARTO Versions (Ours)
.github/workflows/carto_*.yaml- CARTO workflows.github/workflows/carto-*.yml- CARTO workflowsCARTO_*.md,docs/CARTO_*.md- CARTO documentation🔄 Accept Upstream (Theirs)
pyproject.toml- Version fieldlitellm/- Core library codetests/- Upstream testsrequirements.txt- DependenciesDockerfile,docker/Dockerfile.non_root- CARTO customizationsMakefile- Check# CARTO:sections🔧 Conflict Resolution (click to expand)
If this PR has conflicts:
Option 1: Automated (Recommended)
The carto-upstream-sync-resolver workflow triggers automatically.
What it does:
carto/main→ ✏️ Resolves conflicts → 🧪 Runs tests → 📌 Pushes to this PRYou just need to: Wait for resolution commits, verify CARTO customizations, merge.
Option 2: Manual Resolution
📚 Documentation Links (click to expand)
🤖 This PR was automatically created by the carto-upstream-sync workflow.