feat(batches): store each batch line's request and response in S3 and GCS

[devin-ai-integration]

TLDR

Problem this solves:

• A finished batch logs one total cost row, not its individual requests
• Providers delete batch files, so each request and answer is lost

How it solves it:

• New opt-in setting batch_line_item_callbacks: ["s3_v2", "gcs_bucket"]
• After billing a managed batch, the proxy writes one record per line to the bucket
• Each record has the request, the answer or error, tokens, cost and the batch's creator
• Request and answer are left out when the key, team or proxy turned message logging off
• Records are written in the background, so a slow or broken bucket never delays billing

User Flow

Before: an operator who keeps batch traffic in S3 for audits only ever sees the batch total

1. They POST https://litellm-domain/v1/files with 100 chat requests and target_model_names, then POST https://litellm-domain/v1/batches with that file id
2. Once the batch finishes, https://litellm-domain/ui/?page=logs shows one Batch row with the total
3. Their bucket has nothing for request 42, and the provider later deletes the batch files

After: the same operator finds every request and answer of the batch in their bucket

1. They add batch_line_item_callbacks: ["s3_v2"] under litellm_settings and restart the proxy
2. They upload the file and create the batch the same way
3. Once the batch finishes, the Logs page shows the same Batch row with the same total
4. s3://<bucket>/<s3_path>/batch_line_items/<day>/<batch id>/part-00000.jsonl holds 100 lines, each with request, answer or error, tokens and cost
5. If the key or team turned message logging off, the same file arrives with request and answer left out

Linear ticket

Resolves LIT-3395

Docs: BerriAI/litellm-docs#2318

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Before is the merge base f101347 and After is the head 315d705, the current tip

Setup: proxies at 2 workers with real OpenAI, Mistral, Bedrock and Vertex AI, the real S3 bucket litellm-logs-test2 and the real GCS bucket litellm_bucket-16. A recorder in front of OpenAI and Mistral logs every request litellm sends them. The merge base and a head leg with the setting off run without the batch_line_item_callbacks line. One flag-on proxy sets BATCH_LINE_ITEMS_MAX_PART_BYTES=20000 to force several parts, and another accepts Google OAuth2 tokens (enable_oauth2_auth: true)

litellm_settings:
  batch_line_item_callbacks: ["s3_v2", "gcs_bucket"]
  s3_callback_params: {s3_bucket_name: litellm-logs-test2, s3_region_name: us-east-2, s3_path: pr41691-hook/<leg>}
# env: GCS_BUCKET_NAME=litellm_bucket-16, GCS_PATH_SERVICE_ACCOUNT=<service account json>

The batches are all managed, and each has one line the provider fails on purpose: OpenAI (3 lines), OpenAI from a key with message logging off (3), Mistral (3), Bedrock (101), Vertex AI (3), Vertex AI with one line named unknown and one with no custom_id (3), and OpenAI created with a real Google OAuth2 token instead of a key (2)

Before (f101347)

1. Each batch finishes and gets one spend row, shown as one Batch row on the Logs page

2. batch_line_items/ in S3 is empty (KeyCount 0), and GCS has nothing for these batches either

After (315d705)

Every line lands in S3 and GCS

1. Each batch gets the same single spend row and Batch row as on the merge base

2. Parts appear under batch_line_items/2026-10-15/<batch id>/, with Bedrock's 101 lines split into 4 parts by the 20000-byte limit

3. Read back, each batch's parts hold every line exactly once, and the line costs add up to the batch's spend row

batch            parts  lines  ok  failed  line costs    spend row     same in GCS
openai               1      3   2       1  $0.000052375  $0.000052375  yes
openai, logs off     1      3   2       1  $0.000059675  $0.000059675  yes
mistral              1      3   2       1  $0.00000189   $0.00000189   yes
bedrock              4    101 100       1  $0.002145     $0.002145     yes
vertex               1      3   3       0  $0.00007315   $0.00007315   yes
vertex unknown       1      3   3       0  $0.00006815   $0.00006815   yes
openai via oauth2    1      2   2       0  $0.00005655   $0.000057     yes

The OAuth2 batch's spend row is read off the Logs page, which rounds to six decimals

4. Each record pairs the uploaded line with the provider's answer or error, the same in S3 and GCS

5. Vertex AI lines get their own requests, and the two lines Vertex reports as unknown get request: null instead of another line's request

A key with message logging off

1. Its lines keep custom_id, status, usage, cost and creator, with request and response null
2. litellm never reads that batch's input file, so the prompts are not even fetched

A batch created with an OAuth2 token

1. A real Google OAuth2 token uploads the file and creates the batch, with no virtual key involved
2. Its lines keep request and answer, since no team or proxy setting turned message logging off
3. The creator field holds a SHA-256 hash, and the token itself appears nowhere in S3 or GCS

What litellm sent the providers

1. Setting off: the head sends OpenAI and Mistral exactly the merge base's requests, bills each batch once, and writes nothing to either bucket
2. Setting on: the only extra provider calls are one input file read per batch (skipped when message logging is off), and one more output and error file read when its lines are written

Audit cells

All cells ran at 315d705

1. Integration cells in tests/integration/observability/test_batch_line_items_cold_storage.py (real proxy at 2 workers, real Postgres and Redis, a scripted provider, local S3 and GCS): 13 passed in two head runs, and 12 failed at the merge base, where only the null-list cell passes

cell                                                  base  head a  head b
every line with its request and answer in S3 and GCS  FAIL  PASS    PASS
key turned off message logging                        FAIL  PASS    PASS
team turned off message logging                       FAIL  PASS    PASS
creating key deleted                                  FAIL  PASS    PASS
created with a JWT and no key                         FAIL  PASS    PASS
created with an OAuth2 token                          FAIL  PASS    PASS
bigger than one part                                  FAIL  PASS    PASS
every line failed                                     FAIL  PASS    PASS
failed batch on the 2nd deployment of its group       FAIL  PASS    PASS
failing bucket: batch billed, other bucket written    FAIL  PASS    PASS
hung bucket: next batch still billed                  FAIL  PASS    PASS
unsupported name next to s3_v2                        FAIL  PASS    PASS
null list: billed, nothing written                    PASS  PASS    PASS

2. Neighbouring suites match the merge base: batch accounting 7 passed, s3_v2 extensions 89, batch management 140
3. A bucket that rejects every part leaves nothing behind in memory, and writing a 100,000-line batch peaks at 130 MB instead of growing with the batch
4. Unit mutations caught by the tests: storing the raw credential, re-hashing a key hash, skipping the output re-read, reading output for an unbilled batch, closing a part late, never splitting parts, and staying silent when the batch's model is gone
5. Billing does the same work with the setting on as off: five batches of up to 50,000 lines show the same garbage-collector counts and about 4.3 s of CPU on both

Behavior changes

Removed / renamed

• _parse_batch_output_line is now parse_batch_output_line, and two other helpers gain public aliases next to their old names

Silent regressions to watch

• None with the setting unset, which is the default
• With it set, each batch's files are read once more, and request logging callbacks log those reads
• With s3_v2 listed, line-item writes share the request log's S3 failure metric, retry warning and upload limit

Migration for existing customers

• None required

Regression tests

• tests/integration/observability/test_batch_line_items_cold_storage.py pins the 13 cells above
• test_check_batch_cost.py, test_batch_line_items.py, test_s3_v2.py and test_gcs_bucket.py pin the order, background writing, parts, redaction, hashing and object names

Type

🆕 New Feature

Caveats (if any)

Medium

• Only managed batches, uploaded with target_model_names, get records
• A failed part is never queued again
  • A failed write, an unreadable output file or a model removed before the write loses those lines, with an error logged
  • Lines still waiting at shutdown are lost
  • Past 50 waiting batches on a worker, new batches' lines are dropped with an error
• Pricing lines and encrypting parts for HTTPS use the worker's CPU between polls

Low

• Batch files are read twice, and the extra reads show up in request logging callbacks
• Batches made from an Admin UI session or through custom auth always drop request and answer
  • litellm can't find their key or team, so it plays safe
• An OAuth2 token of exactly 64 hex characters looks like a key hash and is stored as is
• Only s3_v2 and gcs_bucket work from config, and custom loggers are a follow-up
• Presidio, excluded fields and the S3 prefix options don't apply to these files
• request is null for Anthropic batches and for Vertex AI lines reported as unknown
• On the old schema without batch_processed, every worker writes the lines again

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Live A/B at 315d705 against the merge base f101347 with real providers, a real OAuth2 token and real buckets: with the setting off the head matches the merge base request for request, and with it on every line lands once with costs that add up to the spend row. ran /live-pr-risk and found no regressions/backward incompatible risks

REVIEWER MUST KNOW BEFORE APPROVING

Nothing changes unless batch_line_item_callbacks is set. With it set, these changes are all opt-in and were agreed in the LIT-3395 design sessions:

• Each batch line goes to the listed buckets in part-NNNNN.jsonl files of up to 16 MB, set by BATCH_LINE_ITEMS_MAX_PART_BYTES
• The proxy reads each batch's input file, and reads its output and error files once more when writing
• Lines are written in the background after billing, one batch at a time per worker, so bucket trouble never delays billing
• A part gets the S3 logger's usual in-call retries, and a part that still fails is not queued again
• s3_v2 line items reuse the request log's S3 logger
  • Their failures count in its S3Logger failure metric, and their retries in its per-flush retry warning
  • With s3_adaptive_concurrency on, a throttled line-item write lowers the shared upload limit, never below s3_max_concurrent_uploads
  • A line-item write does not wait for that limit, so a worker can have one PUT above it
• Request and answer are dropped when the creator's key or team turned message logging off or no longer exists
• The creator is recorded only as a SHA-256 hash
• Batches made with a JWT, an OAuth2 token or the master key follow their team's and the proxy's settings

Not covered live: Azure (no Azure batch deployment here) and Anthropic (batch create returns 400, so nothing reaches the poller)

Link to Devin session: https://app.devin.ai/sessions/5a08a43ae7f140acb90639c83d8b450c
Open in Devin Desktop: https://app.devin.ai/desktop/session/5a08a43ae7f140acb90639c83d8b450c?variant=devin
Requested by: @yucheng-berri

---

Note

Medium Risk
Touches batch billing poller timing, reads batch files again for export, and handles creator credentials/logging—failures are best-effort but can drop line items without affecting batch cost.

Overview
Adds opt-in batch_line_item_callbacks so finished managed batches emit per-line records (request, response/error, usage, cost, creator metadata) to s3_v2 and gcs_bucket, not just a single batch spend row.

After the batch-cost poller bills or retires a batch, CheckBatchCost builds line items in bounded part-NNNNN.jsonl chunks (via BATCH_LINE_ITEMS_MAX_PART_BYTES) and dispatches them in the background—one batch at a time between poll cycles—so slow or failing bucket writes do not block billing. Request/response bodies are omitted when the creating key, team, or proxy has message logging off (or the creator cannot be resolved); credentials are stored only as SHA-256 hashes.

New litellm/batches/batch_line_items.py pairs JSONL lines across providers (including Vertex/Bedrock edge cases), prices each line, and exposes CustomLogger.async_log_batch_line_items. S3/GCS upload NDJSON under batch_line_items/<day>/<batch id>/.

^{Reviewed by Cursor Bugbot for commit 315d705. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge with no new blocking findings

Summary

Adds opt-in callbacks for batch JSONL lines while keeping aggregate billing separate

• Completed batches can send callbacks one paired event per JSONL line.
• Batch line callbacks leave accounting to the aggregate and keep the line marker internal.
• Request fields cannot mark ordinary calls as batch line events.

_{Reviews (38) · Last reviewed commit: "fix(batches): let running line fan-outs ..." · Reviewed by Greptile}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_batch_jsonl_line_item_callbacks (c75d257) with main (1f041a3)}

[codecov]

Codecov Report

❌ Patch coverage is 98.29932% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
..._enterprise/proxy/common_utils/check_batch_cost.py	96.39%	4 Missing ⚠️
litellm/batches/batch_line_items.py	99.03%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai please re-review, the credentials leak is fixed and the unrelated bedrock diff is gone

[mateo-berri]

bugbot run

[mateo-berri]

@greptileai please re-review at the latest head

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@litereview code-review

[yucheng-berri]

@litereview code-review

[yucheng-berri]

bugbot run

[yucheng-berri]

@litereview full

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@litereview code-review

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@litereview full

[greptile-apps]

Automatic reviews are disabled for this repository.

[yucheng-berri]

@litereview code-review

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@litereview code-review

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 315d705. Configure here.}

[yucheng-berri]

@litereview full

[litereview]

Low · Custom loggers named in batch_line_item_callbacks proxy config are never resolved, so the new hook cannot be reached from config

This is still unfixed at head 315d705be7 (previously upheld; the author deferred it to a follow-up).

CustomLogger.async_log_batch_line_items is added as a general extension point. _resolve_logger, however, accepts only a CustomLogger instance or the literal strings "s3_v2" and "gcs_bucket".

On the proxy, litellm_settings.batch_line_item_callbacks has no dedicated branch in proxy_server.py. It falls through to the generic setattr(litellm, key, value), so dotted paths are never turned into instances by get_instance_fn. The callbacks and audit_log_callbacks settings do get that step (proxy_server.py:6744-6762).

Failure scenario:

1. An operator sets batch_line_item_callbacks: ["custom_callbacks.proxy_handler_instance"].
2. The CustomLogger subclass behind that path overrides async_log_batch_line_items.
3. A managed batch completes.
4. _deliver logs cannot store batch line items, part N of batch ... was not stored and drops every part. The override is never called.

Nothing rejects the entry at startup, so the operator only finds out from runtime error logs after the batch's lines are already lost.

Fix: either add an elif key == "batch_line_item_callbacks" branch that resolves dotted entries with get_instance_fn, as audit_log_callbacks does, or reject unsupported names when the config loads.

Evidence

litellm/batches/batch_line_items.py:236-243: the match statement handles CustomLogger() and "s3_v2" | "gcs_bucket"; every other value hits case _: return None.

litellm/batches/batch_line_items.py _deliver: when logger is None, it logs an error and returns, so that part is never stored.

grep batch_line_item_callbacks litellm/proxy returns nothing: the proxy has no resolution step for this setting, so it reaches the generic setattr at proxy_server.py:~6853.

Latest summary by full · View this review · Reviewed 315d705be711

[yucheng-berri]

Still a follow-up, same as the earlier thread. LIT-3395 asks only for S3 and GCS cold storage of batch lines, so this PR ships s3_v2 and gcs_bucket as the sinks, and the PR body lists module-path custom loggers under the Low caveats. Resolving dotted entries also means adding batch_line_item_callbacks to the DB config overlay's remote-module scrub next to audit_log_callbacks, so it belongs in its own PR with its own tests

[litereview]

Finding still applies

At head 315d705be7, _resolve_logger still returns None for any callback other than a CustomLogger instance or the strings "s3_v2" / "gcs_bucket". The proxy has no branch that resolves dotted paths in batch_line_item_callbacks, so custom loggers configured by module path can't be reached and their parts are dropped at runtime. The author is deferring the fix to a follow-up for scope reasons. That is a scheduling decision, not evidence that the defect is gone.

Evidence

litellm/batches/batch_line_items.py:236-243: the match handles CustomLogger() and "s3_v2" | "gcs_bucket"; every other value hits case _: return None.

grep -rn batch_line_item_callbacks litellm/proxy returns no matches at head, so nothing resolves dotted entries with get_instance_fn.

The author's reply confirms the fix is deferred to a separate PR and does not claim it was fixed.

Reviewed 315d705be711 · View follow-up