fix(bedrock): resolve the managed-batch output bucket on every path that reads it by mateo-berri · Pull Request #37047 · BerriAI/litellm

mateo-berri · 2026-08-15T18:46:47Z

TLDR

Problem this solves:

Completed Bedrock batch output cannot be downloaded
500 says to set a bucket that is already set
Batch cost goes unrecorded for the same reason
Three call sites never build the trusted credential snapshot
The retrieve's cost read dropped the deployment's AWS keys

How it solves it:

Resolve the deployment's credentials on the model-routed retrieve
Pass the same snapshot from the batch cost poller
Carry it into the retrieve's own cost accounting
Forward the deployment's AWS credentials on that same read

User Flow

Before: a developer whose batch job finished cannot download its results, and the error tells them to configure a bucket their admin already configured

The developer uploads a batch JSONL with POST https://litellm-domain/v1/files (purpose=batch) and gets a file id back
They create the job with POST https://litellm-domain/v1/batches and get a batch id back
They poll GET https://litellm-domain/v1/batches/{batch_id} until it reports "status": "completed" with an output_file_id
They fetch the results with GET https://litellm-domain/v1/files/{output_file_id}/content and get HTTP 500: S3 bucket_name is required. Set 's3_bucket_name' in proxy config or AWS_S3_BUCKET_NAME for Bedrock file content retrieval
The bucket is already set on the deployment, so the message is misleading and there is no way to read the results through the proxy. The only remaining option is to bypass the gateway and read the storage bucket directly, which most callers have no access to

After: the same fetch returns the batch records

Same upload with POST https://litellm-domain/v1/files
Same create with POST https://litellm-domain/v1/batches
Same poll until GET https://litellm-domain/v1/batches/{batch_id} reports "status": "completed"
GET https://litellm-domain/v1/files/{output_file_id}/content returns HTTP 200 and the JSONL results
No global environment variable is needed; the bucket the admin set on the deployment is the one used, and the completed retrieve also writes the batch's usage to GET https://litellm-domain/spend/logs

Relevant issues

Supersedes #36634 and carries its commits unchanged, plus one import-order fix and one follow-on fix. That PR's fork is organization-owned, so maintainers cannot push the CI fix to it

The follow-on fix (1524880): once the retrieve path could resolve the bucket, its cost read still failed with Missing Dependency: ... botocore[crt] because _extract_file_access_credentials only forwarded a fixed key list and dropped the deployment's aws_* credentials, so the S3 signer fell through to environment auth. The keys in AWS_CREDENTIAL_KWARGS_KEYS are now forwarded

#37050 (which supersedes #36877) decides which component accounts for a batch's cost, and routes that work to the retrieve path whenever the cost poller cannot be relied on. This PR is what makes that path able to read the output file at all, so the two are complementary

Linear ticket

Resolves LIT-5599

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Two live proxies from two worktrees, each with a fresh database, against real Bedrock (bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0, cross-region inference profile in us-west-2) with a real 100-record batch job per leg. The deployment carries aws_access_key_id / aws_secret_access_key / aws_session_token, s3_bucket_name, s3_region_name, s3_encryption_key_id, and aws_batch_role_arn in its litellm_params. No AWS_* variable is set in the proxy's environment, so the per-deployment values are the only source

Config used on both legs:

model_list:
  - model_name: bedrock-batch-claude
    litellm_params:
      model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
      aws_access_key_id: os.environ/QA_AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/QA_AWS_SECRET_ACCESS_KEY
      aws_session_token: os.environ/QA_AWS_SESSION_TOKEN
      aws_region_name: us-west-2
      s3_bucket_name: <bucket>
      s3_region_name: us-west-2
      s3_encryption_key_id: <kms key arn>
      aws_batch_role_arn: <batch role arn>
    model_info:
      mode: batch
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Same commands on both legs ($PORT and $KEY differ per leg):

$ curl -sS -w "HTTP %{http_code}\n" http://127.0.0.1:$PORT/v1/files -H "Authorization: Bearer $KEY" \
    -F purpose=batch -F file=@batch_input.jsonl -F target_model_names=bedrock-batch-claude
HTTP 200

$ curl -sS -w "HTTP %{http_code}\n" http://127.0.0.1:$PORT/v1/batches -H "Authorization: Bearer $KEY" \
    -H "Content-Type: application/json" \
    -d '{"input_file_id": "'"$FILE_ID"'", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
HTTP 200

$ curl -sS -w "HTTP %{http_code}\n" http://127.0.0.1:$PORT/v1/batches/$BATCH_ID -H "Authorization: Bearer $KEY"
{"status": "completed", "output_file_id": "bGl0ZWxsbV9wcm94eT...", ...}
HTTP 200

Before, at the merge base 6704a10:

$ curl -sS -w "HTTP %{http_code}  bytes=%{size_download}\n" http://127.0.0.1:$PORT/v1/files/$OUTPUT_FILE_ID/content \
    -H "Authorization: Bearer $KEY" -H "custom-llm-provider: bedrock" -o output_before.jsonl
HTTP 500  bytes=188
$ head -c 400 output_before.jsonl
{"error":{"message":"S3 bucket_name is required. Set 's3_bucket_name' in proxy config or AWS_S3_BUCKET_NAME for Bedrock file content retrieval.","type":"None","param":"None","code":"500"}}

$ curl -sS http://127.0.0.1:$PORT/spend/logs -H "Authorization: Bearer $KEY" | jq -c '.[] | {call_type, model, spend, prompt_tokens, completion_tokens, total_tokens}'
{"call_type": "acreate_batch", "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "spend": 0.0, "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
{"call_type": "acreate_file", "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "spend": 0.0, "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
  rows: 7

grep -c "S3 bucket_name is required" proxy_before.log returns 8 across the endpoint hit's traceback and the logging worker's _handle_completed_batch failure (LoggingWorker error: S3 bucket_name is required ...), so no batch usage row was ever written on that leg (the other 5 rows are retrieve and status rows carrying no usage)

After, at f930980:

$ curl -sS -w "HTTP %{http_code}  bytes=%{size_download}\n" http://127.0.0.1:$PORT/v1/files/$OUTPUT_FILE_ID/content \
    -H "Authorization: Bearer $KEY" -H "custom-llm-provider: bedrock" -o output_after.jsonl
HTTP 200  bytes=63372
$ head -c 400 output_after.jsonl
{"modelInput":{"messages":[{"role":"user","content":[{"type":"text","text":"Reply with exactly one word, the color \"purple\", and nothing else."}]}],"max_tokens":5,"anthropic_version":"bedrock-2023-05-31"},"modelOutput":{"model":"claude-haiku-4-5-20251001","id":"msg_bdrk_01JHTSpXP2F6GEkiCrPrszXr","
  records returned: 100
  records with modelOutput: 100
  first record text: Purple

$ curl -sS http://127.0.0.1:$PORT/spend/logs -H "Authorization: Bearer $KEY" | jq -c '.[] | {call_type, model, spend, prompt_tokens, completion_tokens, total_tokens}'
{"call_type": "afile_content", "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "spend": 0.0, "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
{"call_type": "aretrieve_batch", "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "spend": 0.0, "prompt_tokens": 2200, "completion_tokens": 400, "total_tokens": 2600}
{"call_type": "acreate_batch", "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "spend": 0.0, "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
{"call_type": "acreate_file", "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "spend": 0.0, "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
  rows: 4

grep -c "S3 bucket_name is required" proxy_after.log returns 0. The aretrieve_batch row is the batch usage the retrieve path computed from the output file (100 records, 2200 input and 400 output tokens, matching Bedrock's own manifest), written by the logging worker without any environment credentials

The intermediate commit 2a75381 (this branch before 1524880) served the same HTTP 200 with 100 records, but its logging worker failed on that read with LoggingWorker error: ... _handle_completed_batch ... _auth_with_env_vars ... Missing Dependency: ... botocore[crt] and wrote no usage row, which is what 1524880 fixes

Before: content read 500s, no batch usage row
After: content read 200s, usage row written
No AWS variable in the proxy environment on either leg

Type

🐛 Bug Fix

Caveats (if any)

Resolving the snapshot is best effort so an unroutable batch still serves
One existing test asserted the unified path resolved no credentials
The read inherits the existing managed-id authorization model, unchanged here
The usage row above prices at 0.0 because of a separate, pre-existing gap

On the last point: _batch_cost_calculator prices each record by the model name Bedrock writes into modelOutput.model (claude-haiku-4-5-20251001), and bedrock/claude-haiku-4-5-20251001 is not a cost-map key (the Bedrock keys are the anthropic./us.anthropic. ids), so the poller and the retrieve path both record tokens with spend 0.0 for this deployment. That is independent of reading the file and is left for a follow-up

On the authorization point: which callers may read a managed file id is decided before this code runs, by validate_managed_id_requirement and litellm.require_managed_files, and this change neither adds nor removes a check. What it does do is bring Bedrock to parity with the providers whose reads already worked, so the same model now applies to one more provider rather than to one fewer. The id itself stays bounded by validate_managed_cloud_file_id, which rejects any file id outside the configured bucket and its managed object prefixes. Making ownership validation unconditional is a worthwhile discussion and a breaking one, since it would reject raw provider ids on every file and batch route for every provider; it belongs in its own change rather than riding along with a bucket-resolution fix

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

^{Cursor Bugbot is generating a summary for commit 2a75381. Configure here.}

…uted and cost-poller paths get_configured_s3_bucket_name accepts the output bucket only from the immutable _litellm_internal_model_credentials snapshot or AWS_S3_BUCKET_NAME. That refusal to read litellm_params is deliberate: the bucket is what validate_managed_cloud_file_id checks a file id against, so trusting a request-supplied value would let a caller redirect reads to a bucket of their choosing Two live entry points reach the Bedrock file-content transformation without ever building that snapshot. The managed-files pre-call hook sets data["model"] for any id carrying llm_output_file_id, which is every batch output, so get_file_content always takes the model-routed branch; that branch called llm_router.afile_content directly, and managed_files_obj.afile_content, the only caller that built the snapshot, is therefore unreachable for batch output. CheckBatchCost spread the deployment credentials as plain kwargs, and get_litellm_params does not carry s3_bucket_name across (gcs_bucket_name is listed for exactly this reason, its S3 counterpart is not), so the poller lost the bucket the same way The result was that every completed Bedrock managed batch failed files.content with "S3 bucket_name is required" and never had its cost tracked, leaving the row to be re-polled every cycle. Both paths now resolve the deployment credentials and pass the same MappingProxyType snapshot the managed-files hook already builds

The mock merged every call into one shared dict, so a second routed retrieval would overwrite the first and the assertions would still pass. Keep one frozen snapshot per call and assert exactly one call, which also makes an unintended second retrieval a failure rather than something the merge hides

…ccounting path too A third path reads a completed batch's output file, and it could not resolve the bucket either. When cost is accounted from the retrieve itself rather than from the poller, the batch success handler calls _handle_completed_batch, which fetches the output file through _extract_file_access_credentials. That helper forwarded a whitelist covering Azure and Vertex, gcs_bucket_name included, but nothing for Bedrock, and retrieve_batch built its litellm_params through get_litellm_params, whose fixed signature drops the trusted credential snapshot. So the snapshot never reached the file read and it failed with "S3 bucket_name is required" for a bucket the deployment had configured, leaving the batch's cost unrecorded. Adding s3_bucket_name to that whitelist would not have worked. The Bedrock file config deliberately resolves the bucket only from the immutable server-side snapshot or the environment, never from a request param, because the bucket is what managed file ids are validated against. The snapshot is therefore what has to flow, exactly as it already does for the model-routed and cost-poller paths. retrieve_batch now re-adds the snapshot after get_litellm_params, the same way the file operations already do, the whitelist forwards it, and the proxy attaches it for router-routed managed batches from the deployment behind the unified id. Verified against a live proxy reading a real completed Bedrock batch: the cost row appears within seconds of the retrieve carrying the batch's real spend and usage, where before the read raised and no row was written. Resolving those credentials is best effort. A batch whose deployment no longer resolves, which happens when a model group is removed while batches are in flight, still serves its status instead of failing the request on the lookup. This matters for the OSS and polling-disabled configurations, where the retrieve path is the only thing that accounts for a batch at all.

…all paths The helper that carries the credential snapshot into litellm_params lived private in files/main.py, and the batch retrieve needed it too. It now sits beside get_litellm_params, which is what it augments, so neither caller reaches into the other's private surface. Typed as Mapping/MutableMapping of object rather than Any, which the strict import rules ban. The file-content route builds the snapshot through the same helper as the batch route instead of assembling a conditional mapping inline, which drops two mutable constructions and leaves one way to attach it. Its name loses the batch suffix now that both routes use it.

greptile-apps · 2026-08-15T18:49:41Z

Greptile Summary

The PR propagates immutable deployment credential snapshots through Bedrock managed-batch retrieval and cost-accounting paths

Resolves deployment credentials for model-routed file and batch retrieval
Preserves trusted bucket configuration and AWS credentials through completed-batch output reads
Adds regression coverage for proxy retrieval and both cost-accounting paths

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/proxy/openai_files_endpoints/common_utils.py	Adds best-effort construction of an immutable deployment credential snapshot for routed operations
litellm/proxy/openai_files_endpoints/files_endpoints.py	Attaches deployment credentials before model-routed managed-file content retrieval
litellm/proxy/batches_endpoints/endpoints.py	Resolves the deployment snapshot before retrieving a unified batch through the router
litellm/batches/main.py	Preserves trusted credentials in retrieve-batch logging parameters for completed-batch accounting
litellm/batches/batch_utils.py	Forwards the trusted snapshot and deployment AWS credentials when reading batch output
enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py	Supplies the immutable credential snapshot to the enterprise batch-cost poller's output read
litellm/litellm_core_utils/get_litellm_params.py	Centralizes validation and propagation of immutable server-side credential snapshots
litellm/files/main.py	Reuses the centralized trusted-credential propagation helper across file operations

_{Reviews (3): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

mateo-berri · 2026-08-15T18:50:05Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2a75381. Configure here.}

codecov · 2026-08-15T18:56:06Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…s AWS credentials

mateo-berri · 2026-08-15T19:10:10Z

@greptileai

mateo-berri · 2026-08-15T19:10:11Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1524880. Configure here.}

…itellm_do_36634 # Conflicts: # litellm/batches/batch_utils.py

mateo-berri · 2026-08-15T19:13:59Z

@greptileai

mateo-berri · 2026-08-15T19:14:00Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit f930980. Configure here.}

codspeed-hq · 2026-08-15T19:53:48Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_bedrock_batch_output_bucket (f930980) with litellm_internal_staging (87abb87)¹}

No successful run was found on litellm_internal_staging (3ac2fbe) during the generation of this report, so 87abb87 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

marty-sullivan and others added 5 commits August 14, 2026 01:17

style(batches): sort the common_utils import block

2a75381

mateo-berri mentioned this pull request Aug 15, 2026

fix(bedrock): resolve the managed-batch output bucket on every path that reads it #36634

Merged

5 tasks

cursor Bot reviewed Aug 15, 2026

View reviewed changes

fix(batches): sign the retrieve-path output read with the deployment'…

1524880

…s AWS credentials

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Merge remote-tracking branch 'origin/litellm_internal_staging' into l…

f930980

…itellm_do_36634 # Conflicts: # litellm/batches/batch_utils.py

cursor Bot reviewed Aug 15, 2026

View reviewed changes

mateo-berri enabled auto-merge August 15, 2026 19:52

yucheng-berri approved these changes Aug 15, 2026

View reviewed changes

mateo-berri merged commit 5e9adb9 into litellm_internal_staging Aug 15, 2026
73 checks passed

mateo-berri deleted the litellm_bedrock_batch_output_bucket branch August 15, 2026 21:37

mateo-berri mentioned this pull request Aug 15, 2026

fix(batches): account a managed batch's cost exactly once #37050

Merged

5 tasks

Navigation Menu

Uh oh!

Uh oh!

fix(bedrock): resolve the managed-batch output bucket on every path that reads it - #37047

fix(bedrock): resolve the managed-batch output bucket on every path that reads it#37047
mateo-berri merged 7 commits into
litellm_internal_stagingfrom
litellm_bedrock_batch_output_bucket

mateo-berri commented Aug 15, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Aug 15, 2026 •

edited

Loading

Important Files Changed

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

codecov Bot commented Aug 15, 2026 •

edited

Loading

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

codspeed-hq Bot commented Aug 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

mateo-berri commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Type

Caveats (if any)

Final Attestation

Uh oh!

greptile-apps Bot commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

mateo-berri commented Aug 15, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

codspeed-hq Bot commented Aug 15, 2026

Merging this PR will not alter performance

Footnotes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mateo-berri commented Aug 15, 2026 •

edited

Loading

greptile-apps Bot commented Aug 15, 2026 •

edited

Loading

codecov Bot commented Aug 15, 2026 •

edited

Loading