Repository navigation
[Feat] Add support for 0 cost models - #20249
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile OverviewGreptile SummaryThis PR adds support for zero-cost models that bypass budget checks when both Key changes:
Implementation approach: Issues found:
Confidence Score: 4/5
|
| Filename | Overview |
|---|---|
| docs/my-website/docs/proxy/custom_pricing.md | Added comprehensive documentation for zero-cost models feature, explaining configuration and behavior |
| litellm/proxy/auth/auth_checks.py | Adds _is_model_cost_zero helper and wraps budget checks in conditional block based on skip_budget_checks flag |
| litellm/proxy/auth/user_api_key_auth.py | Integrates zero-cost check in two authentication paths with duplicated code blocks; wraps budget checks conditionally |
| tests/proxy_unit_tests/test_zero_cost_model_budget_bypass.py | Comprehensive test suite covering zero-cost model scenarios across user, team, end-user budgets, and edge cases |
Sequence Diagram
sequenceDiagram
participant C as Client
participant U as user_api_key_auth
participant A as auth_checks
participant R as Router
participant CC as common_checks
C->>U: Request
U->>U: Extract model
U->>A: Check cost
A->>R: Get pricing
R-->>A: Pricing data
alt Zero cost
A-->>U: Bypass enabled
else Has cost
A-->>U: Normal flow
end
U->>CC: Common checks
alt Normal flow
CC->>CC: All budget checks
else Bypass
CC->>CC: Skip budget checks
end
CC-->>U: Result
U-->>C: Response
| # Check if model has zero cost - if so, skip all budget checks | ||
| model = get_model_from_request(request_data, route) | ||
| skip_budget_checks = False | ||
| if model is not None and llm_router is not None: | ||
| from litellm.proxy.auth.auth_checks import _is_model_cost_zero | ||
|
|
||
| skip_budget_checks = _is_model_cost_zero( | ||
| model=model, llm_router=llm_router | ||
| ) | ||
| if skip_budget_checks: | ||
| verbose_proxy_logger.info( | ||
| f"Skipping all budget checks for zero-cost model: {model}" | ||
| ) |
There was a problem hiding this comment.
This code block for checking zero-cost models is duplicated at lines 1027-1039. Consider extracting this logic into a helper function to reduce duplication and maintain consistency.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Prompt To Fix With AI
This is a comment left during a code review.
Path: litellm/proxy/auth/user_api_key_auth.py
Line: 608:620
Comment:
This code block for checking zero-cost models is duplicated at lines 1027-1039. Consider extracting this logic into a helper function to reduce duplication and maintain consistency.
<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>
How can I resolve this? If you propose a fix, please make it concise.| for model_name in model_list: | ||
| try: | ||
| # Use router's get_model_group_info method directly for better reliability | ||
| model_group_info = llm_router.get_model_group_info(model_group=model_name) |
There was a problem hiding this comment.
Verify that llm_router.get_model_group_info() uses caching and doesn't cause performance degradation in the request path, as per the custom rule about avoiding expensive operations.
Context Used: Rule from dashboard - What: Avoid creating new database requests or Router objects in the critical request path.
Why: Cre... (source)
Prompt To Fix With AI
This is a comment left during a code review.
Path: litellm/proxy/auth/auth_checks.py
Line: 103:103
Comment:
Verify that `llm_router.get_model_group_info()` uses caching and doesn't cause performance degradation in the request path, as per the custom rule about avoiding expensive operations.
**Context Used:** Rule from `dashboard` - What: Avoid creating new database requests or Router objects in the critical request path.
Why: Cre... ([source](https://app.greptile.com/review/custom-context?memory=0c2a17ad-5f29-423f-a48b-371852ac4169))
How can I resolve this? If you propose a fix, please make it concise.| return False | ||
|
|
||
| # If either cost is non-zero, return False | ||
| if input_cost > 0 or output_cost > 0: |
There was a problem hiding this comment.
The condition input_cost > 0 or output_cost > 0 doesn't handle negative costs (though unlikely in practice). Consider using input_cost != 0 or output_cost != 0 for completeness, or add validation that costs cannot be negative.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Prompt To Fix With AI
This is a comment left during a code review.
Path: litellm/proxy/auth/auth_checks.py
Line: 126:126
Comment:
The condition `input_cost > 0 or output_cost > 0` doesn't handle negative costs (though unlikely in practice). Consider using `input_cost != 0 or output_cost != 0` for completeness, or add validation that costs cannot be negative.
<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>
How can I resolve this? If you propose a fix, please make it concise.…del_bypass2 [Feat] Add support for 0 cost models
Relevant issues
Adding back reverted #19027
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitCI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes