Repository navigation
(feat) add usage / cost tracking for Anthropic passthrough routes - #6835
Conversation
|
Hey there, my name is Korbit. I've joined the team as your AI code reviewer. I'll automatically scan your PRs and raise any issues I find. I can also help you write a PR description. As you work with me, I'll learn about your priorities and codebase and get smarter over time. My goal is to help improve your code quality and save time for the human reviewers on your team. Let's get started! |
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
![korbit-ai[bot]](https://avatars.githubusercontent.com/in/322216?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Summary by Korbit AI
Code Health Comments
- Log exception details with a logging library before raising AnthropicError to assist in debugging and monitoring.
Files scanned
| File Path | Reviewed |
|---|---|
| litellm/proxy/pass_through_endpoints/success_handler.py | ✅ |
| litellm/proxy/pass_through_endpoints/pass_through_endpoints.py | ✅ |
| litellm/llms/anthropic/chat/transformation.py | ✅ |
| litellm/llms/anthropic/chat/handler.py | ✅ |
Explore our documentation to understand the languages and file types we support and the files we ignore.
Need a new review? Comment
/korbit-reviewon this PR and I'll review your latest changes.Korbit Guide: Usage and Customization
Interacting with Korbit
- You can manually ask Korbit to review your PR using the
/korbit-reviewcommand in a comment at the root of your PR.- You can ask Korbit to generate a new PR description using the
/korbit-generate-pr-descriptioncommand in any comment on your PR.- Too many Korbit comments? I can resolve all my comment threads if you use the
/korbit-resolvecommand in any comment on your PR.- Chat with Korbit on issues we post by tagging @korbit-ai in your reply.
- Help train Korbit to improve your reviews by giving a 👍 or 👎 on the comments Korbit posts.
Customizing Korbit
- Check out our docs on how you can make Korbit work best for you and your team.
- Customize Korbit for your organization through the Korbit Console.
Current Korbit Configuration
General Settings
Setting Value Review Schedule Automatic excluding drafts Max Issue Count 10 Automatic PR Descriptions ✅ Issue Categories
Category Enabled Naming ✅ Database Operations ✅ Documentation ✅ Logging ✅ Error Handling ✅ Systems and Environment ✅ Objects and Data Structures ✅ Readability and Maintainability ✅ Asynchronous Processing ✅ Design Patterns ✅ Third-Party Libraries ✅ Performance ✅ Security ✅ Functionality ✅ Feedback and Support
| except Exception as e: | ||
| response_headers = getattr(response, "headers", None) | ||
| raise AnthropicError( | ||
| message="Unable to get json response - {}, Original Response: {}".format( | ||
| str(e), response.text | ||
| ), | ||
| status_code=response.status_code, | ||
| headers=response_headers, | ||
| ) |
There was a problem hiding this comment.
Log exception details before raising AnthropicError.
Tell me more
Consider logging the exception details before raising the AnthropicError. This can help with debugging and monitoring. You can use a logging library or your existing logging_obj to log the error message, status code, and original response.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| def get_endpoint_type(url: str) -> EndpointType: | ||
| if ("generateContent") in url or ("streamGenerateContent") in url: | ||
| return EndpointType.VERTEX_AI | ||
| elif ("api.anthropic.com") in url: |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI almost 2 years ago
To fix the problem, we need to ensure that the URL is parsed and the hostname is checked correctly. Instead of using a simple substring check, we should use the urlparse function from the urllib.parse module to extract the hostname from the URL and then check if it matches the expected hostname.
The best way to fix the problem without changing existing functionality is to modify the get_endpoint_type function to parse the URL and check the hostname. We will import the urlparse function and use it to extract the hostname from the URL, then check if the hostname matches "api.anthropic.com".
| @@ -7,3 +7,3 @@ | ||
| from typing import AsyncIterable, List, Optional | ||
|
|
||
| from urllib.parse import urlparse | ||
| import httpx | ||
| @@ -303,5 +303,6 @@ | ||
| def get_endpoint_type(url: str) -> EndpointType: | ||
| if ("generateContent") in url or ("streamGenerateContent") in url: | ||
| parsed_url = urlparse(url) | ||
| if ("generateContent") in parsed_url.path or ("streamGenerateContent") in parsed_url.path: | ||
| return EndpointType.VERTEX_AI | ||
| elif ("api.anthropic.com") in url: | ||
| elif parsed_url.hostname == "api.anthropic.com": | ||
| return EndpointType.ANTHROPIC |
…rriAI#6835) * move _process_response in transformation * fix AnthropicConfig test * add AnthropicConfig * fix anthropic_passthrough_handler * fix get_response_body * fix check for streaming response * use 1 helper to return stream_response on passthrough
This branch was successfully deployed
2 participants
Title
Relevant issues
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes
[REQUIRED] Testing - Attach a screenshot of any new tests passing locall
If UI changes, send a screenshot/GIF of working UI fixes
Description by Korbit AI
What change is being made?
Refactor and enhance the Anthropic pass-through routes to include usage and cost tracking, moving response processing logic into a new
AnthropicConfigclass, and updating logging behavior for improved consistency.Why are these changes being made?
This change aims to improve the system's clarity and efficiency by modularizing the code and enhancing it with usage/cost tracking capabilities. It addresses the need for consistent transforming and logging of responses from Anthropic routes and adds missing features to support cost management, aligning with existing OpenAI response formats.