fix(bedrock): resolve the region prefix the same way on every Bedrock surface - #39865
Open
mateo-berri wants to merge 5 commits into
Open
mateo-berri wants to merge 5 commits into
mateo-berri wants to merge 5 commits into
Conversation
bedrock/<region>/<embedding model> failed every embedding request: Titan answered 500 "Unable to map Bedrock request to provider" and Nova or Cohere sent the prefixed id to AWS and got a 400 back, while Bedrock chat models already accept the same shape. Split the region off the model id with one shared helper, use the bare id for provider detection, the request body, and the invoke URL, let the region in the id set the request region unless aws_region_name is passed, and keep the prefixed id on the response so cost lookup hits the region-specific row. The chat handler now uses the same helper instead of its inline strip
Contributor
Contributor
Greptile SummaryThis PR standardizes parsing and resolution of region-prefixed Bedrock model identifiers across Bedrock request surfaces.
Confidence Score: 4/5The behavioral fix appears sound, but the outstanding repository requirement to keep provider-specific logic under The unresolved previous thread remains applicable because Files Needing Attention: litellm/proxy/hooks/model_max_budget_limiter.py, litellm/proxy/health_check.py
|
| Filename | Overview |
|---|---|
| litellm/llms/bedrock/common_utils.py | Centralizes region-prefix parsing and model lookup candidates while expanding recognized Bedrock regions. |
| litellm/llms/bedrock/base_aws_llm.py | Resolves explicit, ARN, model-prefix, and environment regions consistently and removes prefixes from AWS model IDs. |
| litellm/llms/bedrock/embed/embedding.py | Uses bare model IDs for provider detection and requests while preserving prefixed IDs for response pricing. |
| litellm/llms/bedrock/chat/converse_handler.py | Replaces inline region parsing with the shared Bedrock helper. |
| litellm/proxy/health_check.py | Carries model-path regions into health-check parameters, but remains part of the outstanding provider-boundary rule finding. |
| litellm/proxy/hooks/model_max_budget_limiter.py | Adds Bedrock-specific lookup behavior outside llms/, which remains an outstanding repository-rule violation. |
| litellm/utils.py | Removes Bedrock region prefixes when constructing model-name lookup candidates. |
| tests/test_litellm/llms/bedrock/test_bedrock_common_utils.py | Adds regression coverage for region parsing, profile lookup candidates, routing, and region-list completeness. |
Reviews (6): Last reviewed commit: "fix(bedrock): keep region-prefixed profi..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…alth checks The region prefix helper only knew 23 Bedrock regions, so ids like bedrock/ap-southeast-3/<model> never split and still failed. Add the 12 regions the cost map already prices and a test that keeps the list in step with the cost map. The combined list is now an immutable tuple. The proxy health check stripped the region from the model id and then probed the env or default region. It now carries that region into aws_region_name unless the deployment already sets one. Drive the converse region tests through litellm.completion so a wiring regression in the shared helper fails them, and drop a stray comment.
Contributor
Author
The region prefix fix covered converse and embeddings only. Invoke chat (bedrock/us-east-1/mistral..., bedrock/invoke/us-east-1/...), image generation, and rerank still posted model/us-east-1/<id>/invoke to the env region, which AWS rejects, while /health, which strips the prefix, reported those deployments healthy. BaseAWSLLM._get_aws_region_name now reads the region out of the model id ahead of ARN and env fallbacks, get_bedrock_model_id and get_bedrock_invoke_provider drop the prefix, and rerank sends the bare ARN as modelArn, so every Bedrock surface that resolves a region through the shared base sees the prefix The converse handler now also lets the id's region stand in for an empty aws_region_name, matching embeddings, instead of signing for "" Type the region-prefix test helpers (Greptile P2): an optional aws_region_name and a Mapping for the canned response
Contributor
Author
… surface One region precedence for every call: aws_region_name, then the ARN region (model_id or model), then the region prefix in the model id, then env, with an empty aws_region_name treated as unset. The converse handler stops injecting the prefix region itself, the invoke config inherits the BaseAWSLLM provider resolver and transforms with the bare model, and count_tokens, batch jobs, realtime sessions, and the OpenAI-compatible invoke route send the bare id to AWS. get_bedrock_base_model strips the routing prefix, the region prefix, the ARN, the throughput suffix, and the cross-region prefix in one pass, and the cost lookup keeps the us./eu. profile row for a region-prefixed profile id. mx-central-1 leaves the region list because Bedrock does not run there.
Contributor
Author
mateo-berri
changed the title
…oute Route detection, cache points, parallel tool use, Claude 4.5 detection, and the model budget limiter now look a Bedrock id up as the full id, the routing-free id, the region-free profile id, and the base in that order, so bedrock/<region>/us.<model> whose only cost-map row is the profile row keeps its converse route and its capability row instead of falling to invoke
Contributor
Author
Comment on lines
+162
to
+174
| base_model: Final = next( | ||
| (candidate for candidate in reversed(bedrock_model_lookup_candidates(model)) if _priced_as_bedrock(candidate)), | ||
| None, | ||
| ) | ||
| if base_model is None: | ||
| return () | ||
| _, _, without_vendor = base_model.partition(".") | ||
| return (base_model, without_vendor) if without_vendor else (base_model,) | ||
|
|
||
|
|
||
| def _priced_as_bedrock(model: str) -> bool: | ||
| cost_entry: Final = litellm.model_cost.get(model) | ||
| return isinstance(cost_entry, dict) and str(cost_entry.get("litellm_provider", "")).startswith("bedrock") |
Contributor
There was a problem hiding this comment.
This proxy hook now resolves and identifies Bedrock models directly. The same pattern appears in health_check.py, which parses Bedrock regions and sets Bedrock request parameters. This violates the repository directive to keep provider-specific code under llms/, so the requirement must be satisfied before merging
Rule Used: What: Avoid writing provider-specific code outside... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
bedrock/<region>/<embedding model>fails every embedding requestap-southeast-3never split off/healthdropped the id's region and probed the env or default region insteadHow it solves it:
BaseAWSLLMresolves the region out of the model id ahead of the ARN and env fallbacks and drops it from the model id it signs and posts, so invoke chat, image generation, and rerank (bare ARN asmodelArn) followaws_region_nameis set; an emptyaws_region_nameno longer wins over the id in the converse handler eitheraws_region_nameunless the deployment already sets oneBaseAWSLLM(chat, embeddings, images, rerank, count_tokens, batches, realtime):aws_region_name, then the ARN region, then the id's prefix, then env; an emptyaws_region_namecounts as unset there. Vector stores, Claude platform, search, and SageMaker keep reading the value as isget_bedrock_base_modelstrips the region before the cross-region prefix, and cost lookup keeps theus./eu.profile row for a region-prefixed profile idbedrock/<region>/us.deepseek.r1-v1:0keeps its converse route and its profile rowUser Flow
Before: a proxy admin who deploys the region-prefixed Bedrock ids the cost map documents (
bedrock/us-gov-west-1/amazon.titan-embed-text-v2:0,bedrock/us-east-1/mistral.mistral-7b-instruct-v0:2, and so on) gets embeddings, invoke-route chat, streaming, image editing, and rerank requests refused, Anthropic Messages calls answered with an AWS error body, token counts from a local fallback, image generation silently empty, and/healthreporting the deployments unhealthy, because every request goes to the proxy's default region with the region still glued to the model idgovcloud-titanwithmodel: bedrock/us-gov-west-1/amazon.titan-embed-text-v2:0and GovCloud keys, plusmistral-east(bedrock/us-east-1/mistral.mistral-7b-instruct-v0:2),haiku-invoke-east(bedrock/invoke/us-east-1/us.anthropic.claude-haiku-4-5-20251001-v1:0),haiku-east(bedrock/us-east-1/us.anthropic.claude-haiku-4-5-20251001-v1:0),canvas-east(bedrock/us-east-1/amazon.nova-canvas-v1:0),rerank-east(bedrock/us-east-1/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0), anddeepseek-jakarta(bedrock/ap-southeast-3/deepseek.v3.2) with commercial credentials, leavesaws_region_nameunset everywhere except one copy ofhaiku-eastwhere it is cleared to"", and starts the proxy in an environment whose default region isAWS_REGION_NAME=us-west-2{"model": "govcloud-titan", "input": "region prefix"}and gets HTTP 500litellm.APIConnectionError: Unable to map Bedrock request to providerwithx-litellm-response-cost: 0; the same shape withamazon.nova-2-multimodal-embeddings-v1:0orcohere.embed-english-v3behind the prefix gets HTTP 400BedrockException - {"message":"The provided model identifier is invalid."}{"model": "mistral-east", "messages": [{"role": "user", "content": "Reply with the single word ok"}], "max_tokens": 5}and gets HTTP 422BedrockException - Error processing={"Output":{"__type":"com.amazon.coral.service#UnknownOperationException"},"Version":"1.0"}; the same call tohaiku-invoke-eastgets HTTP 500litellm.APIConnectionError: 'content', and with"stream": trueit gets HTTP 500litellm.APIConnectionError: Checksum mismatchwith no SSE chunkshaiku-eastcopy whoseaws_region_nameis""and gets HTTP 500litellm.APIConnectionError: Invalid AWS region format: ''. Region names must contain only lowercase letters, digits, and hyphens., and todeepseek-jakartaand gets HTTP 404BedrockException - Bedrock Invoke HTTPX: Unknown provider=None, model=ap-southeast-3/deepseek.v3.2{"model": "haiku-invoke-east", "max_tokens": 5, "messages": [{"role": "user", "content": "Reply with the single word ok"}]}and gets HTTP 200 whose body is{"Output": {"__type": "com.amazon.coral.service#UnknownOperationException"}, "Version": "1.0"}, nocontent, nousage, andx-litellm-response-cost-input: 0.0; POST https://litellm-domain/v1/messages/count_tokens with the same messages forhaiku-eastgets HTTP 200{"input_tokens": 13}, a number from the proxy's local tokenizer because the provider count was requested in us-west-2, where the deployment's credentials are not allowed{"model": "canvas-east", "prompt": "a red circle on white", "n": 1, "size": "320x320"}and gets HTTP 200 with"data": [], no image and no cost header; POST https://litellm-domain/v1/images/edits withmodel=canvas-east, a PNGimage, andprompt=make the circle bluegets HTTP 500BedrockException - Nova Canvas image edit returned no images{"model": "rerank-east", "query": "capital of the United States", "documents": [...], "top_n": 2}and gets HTTP 4001 validation error detected: Value 'us-east-1/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0' at 'rerankingConfiguration.bedrockRerankingConfiguration.modelConfiguration.modelArn' failed to satisfy constraintgovcloud-titanunderunhealthy_endpointswithBedrockException Invalid Authentication - {"message":"The security token included in the request is invalid."},canvas-eastwithThe provided model identifier is invalid., the""copy ofhaiku-eastwithInvalid AWS region format: '', anddeepseek-jakartawithUnknown provider=None, model=ap-southeast-3/deepseek.v3.2, because every probe went to us-west-2 or nowhereaws_region_nameper deployment; the requests now succeed, but the titan embedding prices at6e-08for 3 tokens, the commercial rate, not the GovCloud rate the cost map carries under the prefixed idAfter: the same region-prefixed deployments embed, chat, stream, count tokens, draw, edit, and rerank against the region named in the id even though the environment defaults to us-west-2, price from that region's row, treat an empty
aws_region_nameas unset, accept ids prefixed with newer regions, and show healthygovcloud-titanwithmodel: bedrock/us-gov-west-1/amazon.titan-embed-text-v2:0and GovCloud keys, plusmistral-east(bedrock/us-east-1/mistral.mistral-7b-instruct-v0:2),haiku-invoke-east(bedrock/invoke/us-east-1/us.anthropic.claude-haiku-4-5-20251001-v1:0),haiku-east(bedrock/us-east-1/us.anthropic.claude-haiku-4-5-20251001-v1:0),canvas-east(bedrock/us-east-1/amazon.nova-canvas-v1:0),rerank-east(bedrock/us-east-1/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0), anddeepseek-jakarta(bedrock/ap-southeast-3/deepseek.v3.2) with commercial credentials, leavesaws_region_nameunset everywhere except one copy ofhaiku-eastwhere it is cleared to"", and starts the proxy in an environment whose default region isAWS_REGION_NAME=us-west-2{"model": "govcloud-titan", "input": "region prefix"}and gets HTTP 200 with a 1024-dimension embedding, 3 prompt tokens, andx-litellm-response-cost: 6e-07(the GovCloud rate; a plainbedrock/amazon.titan-embed-text-v2:0deployment still prices at6e-08); the nova and cohere shapes answer 200 with 3072- and 1024-dimension embeddingscontent: "\nOk."frommistral-eastandcontent: "ok"fromhaiku-invoke-east, each with a non-zerox-litellm-response-cost, and with"stream": truegets HTTP 200text/event-streamchunks that spellokand end withfinish_reason: "stop"""copy ofhaiku-eastand gets HTTP 200content: "ok"withx-litellm-response-cost: 3.63e-05, and todeepseek-jakartaand gets HTTP 200content: "ok"withx-litellm-response-cost: 1.184e-05haiku-invoke-eastand gets HTTP 200 withcontent: [{"type": "text", "text": "ok"}],stop_reason: "end_turn",usage.input_tokens: 13, andx-litellm-response-cost: 3.3e-05; the same POST https://litellm-domain/v1/messages/count_tokens forhaiku-eastgets HTTP 200{"input_tokens": 29}, the count Bedrock itself returns from us-east-1b64_jsonimage indataandx-litellm-response-cost: 0.06; the same POST https://litellm-domain/v1/images/edits gets HTTP 200 with one editedb64_jsonimage andx-litellm-response-cost: 0.06results: [{"index": 0, "relevance_score": 0.875}, {"index": 1, "relevance_score": 0.182}]andx-litellm-response-cost: 0.002healthy_endpointswithaws_region_namefilled in from the id (us-gov-west-1for titan,ap-southeast-3for deepseek,us-east-1for the rest, including the""copy)deepseek-jakartawithlitellm_provider: bedrock,max_input_tokens: 163840, andinput_cost_per_token: 7.4e-07from the cost mapRelevant issues
Linear ticket
Resolves LIT-6969
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Live proxy against the real Bedrock GovCloud account (us-gov-west-1, SigV4 keys) and the real commercial account (us-east-1, Bedrock API key as
api_key; SigV4 keys for rerank, which takes no bearer token), each leg booted as one proxy instance with two uvicorn workers (AWS_REGION_NAME=us-west-2 python litellm/proxy/proxy_cli.py --config config.yaml --port <port> --num_workers 2 --detailed_debug) andLITELLM_LOCAL_MODEL_COST_MAP=True, so the cost headers come from that commit's cost map.AWS_REGION_NAME=us-west-2in the proxy env is the trap: every*-prefixeddeployment exceptgovcloud-titan-prefixedhas noaws_region_name, so only the region inside the model id can send it to the right region. Same config, same eleven deployments, same requests on both sides; only the proxy commit differsBefore (78ad88f)
govcloud-titan-prefixed (POST /v1/embeddings)
Run
Observed
govcloud-titan-prefixed-noregion (POST /v1/embeddings)
Run
Observed
govcloud-titan-plain (POST /v1/embeddings)
Run
Observed
govcloud-nova-prefixed (POST /v1/embeddings)
Run
Observed
govcloud-nova-plain (POST /v1/embeddings)
Run
Observed
commercial-cohere-embed-prefixed (POST /v1/embeddings)
Run
Observed
commercial-mistral-invoke-prefixed (POST /v1/chat/completions)
Run
Observed
commercial-haiku-invoke-prefixed (POST /v1/chat/completions)
Run
Observed
commercial-haiku-converse-prefixed (POST /v1/chat/completions)
Run
Observed
commercial-nova-canvas-prefixed (POST /v1/images/generations)
Run
Observed
commercial-rerank-prefixed (POST /v1/rerank)
Run
Observed
GET /health
Run
Observed
After (18a8c6b)
govcloud-titan-prefixed (POST /v1/embeddings)
Run
Observed
govcloud-titan-prefixed-noregion (POST /v1/embeddings)
Run
Observed
govcloud-titan-plain (POST /v1/embeddings)
Run
Observed
govcloud-nova-prefixed (POST /v1/embeddings)
Run
Observed
govcloud-nova-plain (POST /v1/embeddings)
Run
Observed
commercial-cohere-embed-prefixed (POST /v1/embeddings)
Run
Observed
commercial-mistral-invoke-prefixed (POST /v1/chat/completions)
Run
Observed
commercial-haiku-invoke-prefixed (POST /v1/chat/completions)
Run
Observed
commercial-haiku-converse-prefixed (POST /v1/chat/completions)
Run
Observed
commercial-nova-canvas-prefixed (POST /v1/images/generations)
Run
Observed
commercial-rerank-prefixed (POST /v1/rerank)
Run
Observed
GET /health
Run
Observed
Cost sanity check: the cost map prices
amazon.titan-embed-text-v2:0at 2e-08 per token andbedrock/us-gov-west-1/amazon.titan-embed-text-v2:0at 2e-07 per token, so 3 tokens cost 6e-08 on the plain deployment and 6e-07 on the region-prefixed onesQA notes:
Every other Bedrock surface, same Before and After commits
The live-pr-risk rig boots the same way (
AWS_REGION_NAME=us-west-2,LITELLM_LOCAL_MODEL_COST_MAP=True, two workers) with nineteen deployments covering what the embeddings proof above does not:/v1/messageson invoke and converse ids,/v1/messages/count_tokenswith SigV4 keys, streaming invoke chat, anaws_region_name: ""copy,bedrock/ap-southeast-3/deepseek.v3.2,/v1/images/edits,/v1/rerankon a bare prefixed id and on a plain ARN, two region-prefixed ids whose only cost-map row is theus.profile row (us.deepseek.r1-v1:0,us.openai.gpt-5.6-luna),/v1/model/info, and/health. Every request hits real Bedrock. Payloads: chat and messages sendReply with the single word okwithmax_tokens5 (60 for the profile-only ids, plusreasoning_effort: lowon the gpt-5.6 one), rerank sends the two-document capital query withtop_n2, image edits post a 320x320 circle PNG withmake the circle blue. The output below is the rig's log verbatim: thecurl -s -D -status and cost headers, the parsed body, then the upstream URLs the proxy log shows for count_tokens and for the profile-only idsBefore (78ad88f)
After (18a8c6b)
Type
🐛 Bug Fix
Caveats (if any)
Four adversarial caveat rounds ran on this PR (78ad88f-era tip, pre-4de94a0460, 4de94a0, dfac628). Round 4 still raised a High and a Medium, so the loop stopped at its four-round cap without converging; both are fixed at 18a8c6b and stay listed here with the evidence
High
us./eu./global.profile row lost their converse route (raised at dfac628, fixed at 18a8c6b)get_bedrock_base_modelstrips the region and then the cross-region prefix, sobedrock/us-east-1/us.deepseek.r1-v1:0resolved todeepseek.r1-v1:0, which has no row; route detection, cache points, parallel tool use, Claude 4.5 detection, and the model budget limiter all missed and the call fell to the invoke route (Unknown provider=None) or sentcachePointblocks to a model whose row says nobedrock_model_lookup_candidateswalks the full id, the routing-free id, the region-free profile id, and the base in that order, and every one of those lookups plus_bedrock_candidatesin the budget limiter walks it too. Regressions intest_bedrock_common_utils.pyandtest_unit_test_max_model_budget_limiter.pyfail at dfac628 (5 failed) and pass at 18a8c6b; the rig below runsbedrock/us-east-1/us.deepseek.r1-v1:0andbedrock/us-east-1/us.openai.gpt-5.6-lunaon the/converseURL at both commitsMedium
/v1/messages, count_tokens, theaws_region_name: ""copy,bedrock/ap-southeast-3/deepseek.v3.2,/v1/images/edits, and/v1/model/infowere claimed from the live-pr-risk run without pasting it. The rig's Before and After output is now pasted verbatim under "Every other Bedrock surface"Low
aws_region_namenames a different region than the id, the call goes toaws_region_namebutresponse.modelkeeps the id's prefix, so pricing comes from the id's row. Same design as chat; aus-gov-west-1id sent to commercialus-east-1bills the 10x GovCloud ratebedrock/<region>/<model>shape for embeddingsbedrock.mdandbedrock_embedding.mdin litellm-docs have no region-prefixed exampleAmazonBedrockGlobalConfiglearns it, the same ruleget_bedrock_base_modeland the health check already follow; a pattern-based split is a separate change with its own false-positive risk (us/...style ids)split_bedrock_region_prefixreturns a tuple, not a named shapeaws_region_name, then the ARN region (model_idor the model), then the prefix, then env. At base the converse handler let the prefix beat amodel_idARN; a config that names two different regions there now signs for the ARN's region, the one AWS acceptseusc-de-east-1is left out of the region listmx-central-1was dropped because Bedrock does not run therebedrock/<region>/openai/<model>still sendsopenai/in the model id_get_openai_model_idstripsbedrock/, thenopenai/, then the region, so the region has to come afteropenai/(bedrock/openai/us-east-1/<model>works). Same at base;openai/is a routing prefix that sits right afterbedrock/in every documented shape, so reordering that stripper is its own changebedrock/mantle/<region>/<model>posts<region>/<model>, same as base. Mantle calls a user-supplied endpoint, so a region prefix has no meaning there and no documented shape carries oneFinal Attestation