[Bedrock] Fix Anthropic file_id support - async path + document URL→base64 + beta header filtering - #25047
Conversation
…ase64 + beta header filtering
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR fixes three issues in Bedrock's Anthropic (Claude 3+) integration: (1) async calls via the Invoke API now correctly route to a dedicated Key observations:
Confidence Score: 4/5Safe to merge with minor risk — the core logic is sound, but the missing tests and a potential silent failure for edge-case PDF URLs warrant attention before merging The three main bug fixes (async routing, URL→base64 conversion, beta header filtering) are logically correct and address real Bedrock failures. Score is 4 rather than 5 because: (1) no tests were added despite a hard project requirement and the checklist being unchecked, (2) the document URL conversion reuses image-only download infrastructure that will raise a misleading 'Unsupported image format' error for PDFs served without a Content-Type header, and (3) user-supplied beta headers bypass the Bedrock filter, so the fix is incomplete for explicit header passthrough. litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py — document URL conversion and beta filter logic; litellm/llms/bedrock/chat/invoke_handler.py — sync/async format inconsistency for is_claude_messages_api_model
|
| Filename | Overview |
|---|---|
| litellm/llms/bedrock/chat/invoke_handler.py | Adds _async_anthropic_messages_completion to correctly route async Claude 3+ Bedrock Invoke calls to the Messages API transformation; sync path for the same models still uses a different (older) request format, creating a sync/async inconsistency |
| litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py | Refactors transform_request into base + beta header helpers, adds async_transform_request, and URL→base64 document conversion; beta filter correctly excludes unsupported auto-betas but leaves user-supplied headers unfiltered; document conversion reuses image-only download infrastructure which can fail with a misleading error for PDFs served without Content-Type headers |
| litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py | Applies the same auto-only beta header filtering pattern as the invoke path; user-supplied betas still bypass the Bedrock filter, which is the same minor concern |
Sequence Diagram
sequenceDiagram
participant Caller
participant BedrockLLM
participant AmazonAnthropicClaudeConfig
participant BetaFilter as filter_and_transform_beta_headers
participant URLFetcher as convert_url_to_base64
Caller->>BedrockLLM: acompletion(model=claude-3+, messages)
BedrockLLM->>BedrockLLM: Build endpoint_url (/invoke or /invoke-with-response-stream)
BedrockLLM->>BedrockLLM: is_claude_messages_api_model? → true
BedrockLLM->>BedrockLLM: _async_anthropic_messages_completion()
BedrockLLM->>AmazonAnthropicClaudeConfig: async_transform_request()
AmazonAnthropicClaudeConfig->>AmazonAnthropicClaudeConfig: _build_bedrock_anthropic_request_base()
AmazonAnthropicClaudeConfig->>URLFetcher: _async_convert_document_url_sources_to_base64()
URLFetcher-->>AmazonAnthropicClaudeConfig: base64-encoded document
AmazonAnthropicClaudeConfig->>BetaFilter: _compute_bedrock_invoke_beta_headers() [auto betas only]
BetaFilter-->>AmazonAnthropicClaudeConfig: filtered beta list (files-api-2025-04-14 removed)
AmazonAnthropicClaudeConfig-->>BedrockLLM: transformed request dict
BedrockLLM->>BedrockLLM: sign request headers (AWS SigV4)
BedrockLLM->>BedrockLLM: async_streaming / async_completion
BedrockLLM-->>Caller: ModelResponse / CustomStreamWrapper
Reviews (1): Last reviewed commit: "[Bedrock] Fix Anthropic file_id support ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| base64_url = convert_url_to_base64(url=source_url) | ||
| image_chunk = convert_to_anthropic_image_obj( | ||
| openai_image_url=base64_url, | ||
| format=inferred_format, | ||
| ) | ||
| block["source"] = { | ||
| "type": "base64", | ||
| "media_type": image_chunk["media_type"], | ||
| "data": image_chunk["data"], | ||
| } |
There was a problem hiding this comment.
convert_url_to_base64 and async_convert_url_to_base64 are backed by _process_image_response, which has a fallback MIME-type lookup that only covers image extensions (jpg, jpeg, png, gif, webp). When the HTTP server doesn't return a Content-Type header and the URL path doesn't end with one of those extensions (e.g. https://example.com/report.pdf from an S3 bucket without explicit Content-Type), the function raises:
"Error: Unsupported image format. Format=None. Supported types = ['image/jpeg', 'image/png', 'image/gif', 'image/webp']"
This is a confusing, misleading error for a document-handling code path. Any PDF URL served from storage without an explicit Content-Type: application/pdf response header will fail here. The inferred_format = "application/pdf" variable set above is never reached in that failure case — it is only used as an argument to convert_to_anthropic_image_obj after convert_url_to_base64 already returns.
Consider wrapping the call with error translation, or using httpx directly to fetch + encode the document instead of delegating to the image utility. The same issue applies to _async_convert_document_url_sources_to_base64 (line 285).
| auto_beta_list = filter_and_transform_beta_headers( | ||
| beta_headers=list(beta_set - user_beta_set), | ||
| provider="bedrock", | ||
| ) | ||
| return sorted(user_beta_set.union(set(auto_beta_list))) |
There was a problem hiding this comment.
The filter is applied only to auto-generated betas (beta_set - user_beta_set). Any beta header the caller explicitly supplies (via extra_headers) is passed through to Bedrock unchanged:
auto_beta_list = filter_and_transform_beta_headers(
beta_headers=list(beta_set - user_beta_set), # ← user betas never filtered
provider="bedrock",
)
return sorted(user_beta_set.union(set(auto_beta_list)))If a user explicitly passes anthropic-beta: files-api-2025-04-14 (or any other header unsupported by Bedrock) in extra_headers, it will still be forwarded and cause Bedrock to reject the request with the same error this PR set out to fix. The same pattern exists in bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py at line ~469.
If the intent is "always respect user-supplied headers", that should be documented. Otherwise both user_beta_set and auto_betas should be filtered.
| if acompletion and provider == "anthropic" and self.is_claude_messages_api_model( | ||
| model | ||
| ): | ||
| if isinstance(client, HTTPHandler): | ||
| client = None | ||
| return self._async_anthropic_messages_completion( | ||
| model=model, | ||
| messages=messages, | ||
| endpoint_url=endpoint_url, | ||
| proxy_endpoint_url=proxy_endpoint_url, | ||
| credentials=credentials, | ||
| aws_region_name=aws_region_name, | ||
| model_response=model_response, | ||
| print_verbose=print_verbose, | ||
| encoding=encoding, | ||
| logging_obj=logging_obj, | ||
| optional_params=optional_params, | ||
| stream=stream, | ||
| litellm_params=litellm_params, | ||
| logger_fn=logger_fn, | ||
| extra_headers=extra_headers, | ||
| timeout=timeout, | ||
| client=client, | ||
| stream_chunk_size=stream_chunk_size, | ||
| ) # type: ignore[return-value] |
There was a problem hiding this comment.
is_claude_messages_api_model paths use different request formats
The new async branch calls AmazonAnthropicClaudeConfig().async_transform_request, which produces a proper Anthropic Messages API request (JSON format via AnthropicConfig.transform_request).
However, the sync branch at line 916 for the same is_claude_messages_api_model(model) == True case still uses the older XML-based path:
messages = prompt_factory(model=model, messages=messages, custom_llm_provider="anthropic_xml")
data = json.dumps({"messages": messages, **inference_params})This means an identical call to the same Claude 3+ Bedrock model would produce a different request payload depending on whether it's sync or async. The async path is likely the more correct one since it uses the full AnthropicConfig transformation logic. A follow-up to align the sync path (or at minimum a comment noting the discrepancy) would be valuable to avoid subtle behavior differences.
| def _convert_document_url_sources_to_base64(self, anthropic_request: dict) -> None: | ||
| """ | ||
| Bedrock Invoke does not accept document URL sources. Convert to base64 payloads. | ||
| """ | ||
| messages = anthropic_request.get("messages") | ||
| if not isinstance(messages, list): | ||
| return | ||
|
|
||
| for message in messages: | ||
| if not isinstance(message, dict): | ||
| continue | ||
| content = message.get("content") | ||
| if not isinstance(content, list): | ||
| continue | ||
|
|
||
| for block in content: | ||
| if not isinstance(block, dict) or block.get("type") != "document": | ||
| continue | ||
| source = block.get("source") | ||
| if not isinstance(source, dict) or source.get("type") != "url": | ||
| continue | ||
| source_url = source.get("url") | ||
| if not isinstance(source_url, str): | ||
| continue | ||
|
|
||
| inferred_format: Optional[str] = None | ||
| if source_url.lower().endswith(".pdf"): | ||
| inferred_format = "application/pdf" | ||
| base64_url = convert_url_to_base64(url=source_url) | ||
| image_chunk = convert_to_anthropic_image_obj( | ||
| openai_image_url=base64_url, | ||
| format=inferred_format, | ||
| ) | ||
| block["source"] = { | ||
| "type": "base64", | ||
| "media_type": image_chunk["media_type"], | ||
| "data": image_chunk["data"], | ||
| } | ||
|
|
||
| async def _async_convert_document_url_sources_to_base64( | ||
| self, anthropic_request: dict | ||
| ) -> None: | ||
| """ | ||
| Async version of document URL conversion for async completion paths. | ||
| """ | ||
| messages = anthropic_request.get("messages") | ||
| if not isinstance(messages, list): | ||
| return | ||
|
|
||
| for message in messages: | ||
| if not isinstance(message, dict): | ||
| continue | ||
| content = message.get("content") | ||
| if not isinstance(content, list): | ||
| continue | ||
|
|
||
| for block in content: | ||
| if not isinstance(block, dict) or block.get("type") != "document": | ||
| continue | ||
| source = block.get("source") | ||
| if not isinstance(source, dict) or source.get("type") != "url": | ||
| continue | ||
| source_url = source.get("url") | ||
| if not isinstance(source_url, str): | ||
| continue | ||
|
|
||
| inferred_format: Optional[str] = None | ||
| if source_url.lower().endswith(".pdf"): | ||
| inferred_format = "application/pdf" | ||
| base64_url = await async_convert_url_to_base64(url=source_url) | ||
| image_chunk = convert_to_anthropic_image_obj( | ||
| openai_image_url=base64_url, | ||
| format=inferred_format, | ||
| ) | ||
| block["source"] = { | ||
| "type": "base64", | ||
| "media_type": image_chunk["media_type"], | ||
| "data": image_chunk["data"], | ||
| } |
There was a problem hiding this comment.
_convert_document_url_sources_to_base64 and _async_convert_document_url_sources_to_base64 share identical structure — the only difference is await async_convert_url_to_base64(...) vs convert_url_to_base64(...). Any future bug fix or format extension (e.g. supporting .txt or .docx) would need to be applied to both methods.
Consider extracting the inner conversion into a small sync helper, then having the async method call await async_convert_url_to_base64 only at the I/O boundary:
def _build_base64_source(self, source_url: str) -> dict:
inferred_format = "application/pdf" if source_url.lower().endswith(".pdf") else None
base64_url = convert_url_to_base64(url=source_url)
chunk = convert_to_anthropic_image_obj(openai_image_url=base64_url, format=inferred_format)
return {"type": "base64", "media_type": chunk["media_type"], "data": chunk["data"]}Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
957b654
into
litellm_internal_staging_04_02_2026
…ase64 + beta header filtering (BerriAI#25047) (BerriAI#25050) Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
2 participants
Relevant issues
Pre-Submission checklist
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewCI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🐛 Bug Fix
Changes
Three fixes for Anthropic file_id / document support on Bedrock:
1. Async Claude Messages API routing (
invoke_handler.py)Async calls to Claude models that use the Messages API were falling through to the sync path. Added
_async_anthropic_messages_completionand route async calls there whenacompletion=Trueand the model uses the Messages API.2. Document URL → base64 conversion (
bedrock/chat/.../anthropic_claude3_transformation.py)Bedrock Invoke doesn't accept
source.type = "url"for document blocks. Added_convert_document_url_sources_to_base64(sync) and_async_convert_document_url_sources_to_base64(async) to convert URL-sourced documents to base64 before sending. Also addedasync_transform_requestso the async path gets the same transformation logic as the sync path.3. Beta header filtering (
bedrock/chat/andbedrock/messages/)Both transformation classes now use
filter_and_transform_beta_headers(provider="bedrock")to strip beta headers that Bedrock doesn't support (e.g.files-api-2025-04-14) before settinganthropic_beta. Previously these were passed through unfiltered, causing Bedrock to reject requests with file_id content.4. Test URL fix (
base_llm_unit_tests.py)The file_id test was using a Wikimedia URL that 403s in CI. Switched to a fixture PDF in the repo.