feat(proxy): cap batch file records, daily batch uploads, and per-file downloads

[devin-ai-integration]

TLDR

Problem this solves:

• A caller can upload any number of batch files per day
• One batch file can hold any number of request lines
• One file's content can be downloaded in a tight loop

How it solves it:

• New max_batch_file_records rejects a bigger batch file with a 413
• New max_batch_file_uploads_per_day returns 429 once the day's count is used
• New max_file_downloads_per_minute returns 429 per file with Retry-After
• All three are off unless set; general_settings sets a per-key default
• Key or team metadata overrides it, and only a proxy admin can set that

Intentional product change: the Router Settings page (General tab) and GET /config/list show the three new settings, and a user who is not a proxy admin can no longer put any batch limit, batch_enqueued_token_limit included, on the key that POST /user/new or POST /management/v1/users/bulk creates. Before, those two routes accepted any value there, so an org admin could hand a new user a raised limit

User Flow

Before: a team running batch jobs through the gateway has no way to cap how much one caller pushes through the batch routes

1. The caller sends POST https://litellm-domain/v1/files with purpose=batch six times in one day, and every upload returns 200 with a file-... id
2. They upload a 2,000-line batch file to the same route, and it returns 200 and goes to the provider
3. They call GET https://litellm-domain/v1/files/{output_file_id}/content twelve times in a minute, and all twelve return 200
4. The admin opens https://litellm-domain/ui/?page=router-settings, clicks General, and finds only a batch file size cap in MB

After: the admin sets the three limits, and a caller who goes past one gets a clear 413 or 429 that says which limit it hit

1. The admin sets the three limits on the same General tab (or in general_settings, or in key or team metadata)
2. The sixth upload of the day returns 429 "Batch file upload limit reached ..." with a Retry-After counting down to 00:00 UTC
3. The 2,000-line upload returns 413 "Batch input file has more than 1000 records ..." and nothing reaches the provider
4. The eleventh download of one file in a minute returns 429 with Retry-After, while other files still download
5. A non-admin who raises a limit on their own key (POST /key/update) or a new user's key (POST /user/new) gets a 403

Router Settings, General tab, before (50190134c3): only the MB size cap exists for batch files

After (c74eb7d1c9): the three new settings, here read from the config file

Docs for the three settings and the admin-only rule are in BerriAI/litellm-docs#1910

Linear ticket

Resolves LIT-5275

Design decisions

These are the calls made on purpose, each with the alternative it beat:

• Records per file counts non-blank request lines, both OpenAI and native Vertex passthrough rows. It is checked in the same pass as the existing line validation, so an invalid line before the limit still gets the line error first. A 413 matches the sibling max_batch_file_size_mb
• Uploads per day counts accepted batch uploads in a UTC calendar day (fixed window, resets at 00:00 UTC) rather than a rolling 24 hours or the number of files currently stored. A calendar day is easy to explain in the error and in Retry-After; counting stored files would need a DB query per upload and would ignore deleted-then-reuploaded files
• An upload that fails validation (bad JSON, too many records, too big, a malformed expires_after) does not use up a slot, since the count happens after validation. One that a pre-call hook or batch guardrail rejects, or that fails at the provider, does use one: the slot is taken before the guardrail scan so a caller over the limit cannot keep triggering scans, and counting only after success would let concurrent uploads go over the limit
• Downloads per minute is counted per caller and per file id in a fixed one-minute window, on every file id the caller downloads (input, output, and error files alike), not only batch output files. Telling an output file apart would need a provider lookup before the throttle
• Each counter's key carries the start of its window, so a new day or minute always starts from zero. The counter lives for one full window length, so it never expires before its window ends
• When a key and its team both set a limit, each keeps its own counter and a request must fit both. A rejection on one gives back the slot it already took on the other, so a team limit never silently eats the key's quota
• The general_settings value is a default for every key, admin keys included. A key's own metadata value replaces the default for that key, and a team's metadata value always applies on top
• A caller without a key (a JWT caller) is counted per user id, so the general_settings default applies to it too. A JWT caller with neither a key nor a user id gets only its team's limit
• Only a proxy admin can set, change, or clear the three keys in key or team metadata, the same rule batch_enqueued_token_limit already had (the existing guard now covers all four keys). A non-admin resending the stored value unchanged still works, so the UI edit form keeps saving. The same guard now also runs on the key that POST /user/new and POST /management/v1/users/bulk create for a new user, since both copy the request metadata onto that key and an org admin could otherwise hand a new user any limit
• /config/update and /config/field/update reject a value that is not a positive integer. One that still reaches key or team metadata or the config file is ignored with a warning rather than failing requests
• Counters live in their own store, separate from the bounded in-memory cache the rate limiters use, so downloading many different files cannot push rate-limit windows out of that cache. With Redis configured the store uses it and counts are shared across every pod and worker; without it each worker process keeps its own count, so a caller can get up to the limit times the number of workers, and the counts reset on restart. This matches how the existing request rate limits behave without Redis. The store tracks up to 20,000 live counters per worker (FILE_USAGE_MAX_TRACKED_COUNTERS, the same bound the login throttle uses) instead of the in-memory cache's default of 200, which let a caller get a fresh allowance on one file by downloading about 200 other file ids in the same minute
• Scope is the key and its team (the user id for a JWT caller with no key), not the end user. The ticket left the scope open, and the customer described its batch callers as internal teams, which the proxy models as teams and their keys. A per-end-user override would need a new column on the end-user table plus a migration, since that table has no metadata. Counting the general_settings default per end-user id as a third counter was the other option, but it only binds callers who choose to send an end-user id, so it would not bound what a team pushes through. Either can be added later without changing what these limits mean
• Provider pass-through routes (/openai/..., /vertex_ai/..., /azure/..., /bedrock/...) are not counted, the same boundary max_batch_file_size_mb and the batch enqueued-token limit already have. Those routes forward each provider's own file API as is, so counting them means parsing every provider's upload and download shapes, which is its own change. An admin who needs the caps airtight gives the key allowed_routes: ["openai_routes"], which keeps /v1/files and /v1/batches open and returns 403 on the pass-through file routes
• These are self-contained counters instead of going through the parallel request limiter, because which limiter runs depends on LEGACY_MULTI_INSTANCE_RATE_LIMITING and these limits should not change behavior with it

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Last updated for c74eb7d1c9. Both legs ran the same scenario script against real OpenAI (gpt-5.4-nano, Files and Batches API) with curl as the client, the way the customer's batch callers hit /v1/files. Each leg is two proxy processes on two random ports (B1, B2), each with --num_workers 2, sharing one Postgres database and one Redis, and requests alternate between them, so a count taken on one instance has to be seen by the other

The config sets max_batch_file_records: 1000, max_batch_file_uploads_per_day: 5, and max_file_downloads_per_minute: 10 in general_settings, turns on JWT auth against a local JWKS, and sets LITELLM_LICENSE so team and org admins can be assigned. The callers are:

• KEY, a team key
• RUN_KEY, the team key that ran the two-request batch whose output file is downloaded
• DEV_KEY, a team admin's own key
• ORG_ADMIN_KEY, an org admin's key in org $ORG
• XKEY, a fresh team key whose metadata sets max_batch_file_uploads_per_day: 1
• JWT, a signed token with no virtual key behind it

Before (5019013)

B1=http://127.0.0.1:28045, B2=http://127.0.0.1:38829

1. A 2,000-line batch file (max_batch_file_records: 1000)

1. Upload it, and it is accepted and forwarded

   $ curl -s -w "\nHTTP %{http_code}\n" $B1/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@records_2000.jsonl
   {"id":"file-NUmN1vXSLUnCA8yoGG4Jgv","bytes":441786,"created_at":1791057215,"filename":"records_2000.jsonl","object":"file","purpose":"batch","status":"processed","expires_at":1793649215,"status_details":null}
   HTTP 200
   

2. Six batch uploads in one day across both instances (max_batch_file_uploads_per_day: 5)

1. Five uploads, alternating instances

   $ for i in 1 2 3 4 5; do B=$([ $((i % 2)) = 1 ] && echo $B1 || echo $B2); curl -s -o upload_$i.json -w "%{http_code} " $B/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@two_records.jsonl; done; echo
   200 200 200 200 200
   

2. The sixth, at 2026-10-03T19:53:37Z, is accepted too

   $ curl -s -i $B2/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@two_records.jsonl | grep -iE "^HTTP|^retry-after|^\{"
   HTTP/1.1 200 OK
   {"id":"file-VxnX11Nz5JP9868QYHe9XT","bytes":432,"created_at":1791057218,"filename":"two_records.jsonl","object":"file","purpose":"batch","status":"processed","expires_at":1793649218,"status_details":null}
   

3. Twelve downloads of one batch output file in a minute across both instances (max_file_downloads_per_minute: 10)

1. Eleven downloads, alternating instances

   $ for i in $(seq 1 11); do B=$([ $((i % 2)) = 1 ] && echo $B1 || echo $B2); curl -s -o /dev/null -w "%{http_code} " $B/v1/files/file-98GiY6B9Ehw9uqtRYvkvei/content -H "Authorization: Bearer $RUN_KEY"; done; echo
   200 200 200 200 200 200 200 200 200 200 200
   

2. The twelfth, at 2026-10-03T19:56:04Z, returns the batch output again

   $ curl -s -i $B2/v1/files/file-98GiY6B9Ehw9uqtRYvkvei/content -H "Authorization: Bearer $RUN_KEY" | grep -iE "^HTTP|^retry-after|^\{" | cut -c1-330
   HTTP/1.1 200 OK
   {"id": "batch_req_6ac15dbda3d081909e5c98e18462db67", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "e88581f7-1a25-4f29-b18d-73a93b840540", "body": {"id": "chatcmpl-EV06YDVly0dduOhDfAnXujENVazwY", "object": "chat.completion", "created": 1791057278, "model": "gpt-5.4-nano-2026-03-17", "choices": [{"index": 0
   {"id": "batch_req_6ac15dbd81b481908decaa4592dd326e", "custom_id": "req-2", "response": {"status_code": 200, "request_id": "1441a2fa-edc1-4cd3-9b1f-beb13b615cbe", "body": {"id": "chatcmpl-EV07XO0GHmW0zaZV2cyMSly43QpNV", "object": "chat.completion", "created": 1791057339, "model": "gpt-5.4-nano-2026-03-17", "choices": [{"index": 0
   

3. The same batch's input file

   $ curl -s -o /dev/null -w "%{http_code}\n" $B1/v1/files/file-1QQP3dYvx6xN1FSBBEoeGs/content -H "Authorization: Bearer $RUN_KEY"
   200
   

4. A proxy admin sets a download limit of 2 in the key's metadata

1. The limit lands in the key's metadata

   $ curl -s -o /dev/null -w "HTTP %{http_code}\n" $B1/key/update -H "Authorization: Bearer $MASTER_KEY" -H "Content-Type: application/json" -d "{\"key\":\"$KEY\",\"metadata\":{\"max_file_downloads_per_minute\":2}}"
   HTTP 200
   

2. Two downloads after a 61 s wait for the other instance's key cache, from the top of a minute

   $ for i in 1 2; do B=$([ $i = 1 ] && echo $B1 || echo $B2); curl -s -o /dev/null -w "%{http_code} " $B/v1/files/file-3HgHjDZRtHUtzLa2NVgSr3/content -H "Authorization: Bearer $KEY"; done; echo
   200 200
   

3. The third download is served too

   $ curl -s -i $B1/v1/files/file-3HgHjDZRtHUtzLa2NVgSr3/content -H "Authorization: Bearer $KEY" | grep -iE "^HTTP|^retry-after|^\{" | cut -c1-330
   HTTP/1.1 200 OK
   {"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-5.4-nano", "messages": [{"role": "user", "content": "Reply with the single word OK (1)"}], "max_completion_tokens": 16}}
   {"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-5.4-nano", "messages": [{"role": "user", "content": "Reply with the single word OK (2)"}], "max_completion_tokens": 16}}
   

5. A team admin, not a proxy admin, raises their own key's upload limit

1. The team admin raises it on their own key

   $ curl -s -o key_update.json -w "HTTP %{http_code}\n" $B2/key/update -H "Authorization: Bearer $DEV_KEY" -H "Content-Type: application/json" -d "{\"key\":\"$DEV_KEY\",\"metadata\":{\"max_batch_file_uploads_per_day\":1000}}"; jq -c "if has(\"metadata\") then {metadata} else . end" key_update.json
   HTTP 200
   {"metadata":{"max_batch_file_uploads_per_day":1000}}
   

6. An org admin, not a proxy admin, creates a user with a download limit in metadata

1. The user is created

   $ curl -s -o user_new_1.json -w "HTTP %{http_code}\n" $B2/user/new -H "Authorization: Bearer $ORG_ADMIN_KEY" -H "Content-Type: application/json" -d "{\"user_email\":\"new-hire-$SUFFIX@example.com\",\"organizations\":[\"$ORG\"],\"metadata\":{\"max_file_downloads_per_minute\":1000}}"; jq -c "if has(\"key\") then {user_id, key} else . end" user_new_1.json
   HTTP 200
   {"user_id":"bddb0daa-9257-4a51-b375-39a8846526a7","key":"sk-xh..."}
   

2. Its key carries the limit the org admin asked for

   $ curl -s "$B1/key/info?key=$(jq -r .key user_new_1.json)" -H "Authorization: Bearer $MASTER_KEY" | jq -c .info.metadata
   {"max_file_downloads_per_minute":1000}
   

3. A user created without a limit

   $ curl -s -o user_new_2.json -w "HTTP %{http_code}\n" $B1/user/new -H "Authorization: Bearer $ORG_ADMIN_KEY" -H "Content-Type: application/json" -d "{\"user_email\":\"new-hire-2-$SUFFIX@example.com\",\"organizations\":[\"$ORG\"]}"; jq -c "if has(\"key\") then {user_id, key} else . end" user_new_2.json
   HTTP 200
   {"user_id":"1bdbe671-1876-4972-8be7-b910b9bd4141","key":"sk-gq..."}
   

4. Its key's metadata is empty

   $ curl -s "$B2/key/info?key=$(jq -r .key user_new_2.json)" -H "Authorization: Bearer $MASTER_KEY" | jq -c .info.metadata
   {}
   

7. A JWT caller with no virtual key uploads six batch files in one day

1. Five uploads, alternating instances

   $ for i in 1 2 3 4 5; do B=$([ $((i % 2)) = 1 ] && echo $B1 || echo $B2); curl -s -o /dev/null -w "%{http_code} " $B/v1/files -H "Authorization: Bearer $JWT" -F purpose=batch -F file=@two_records.jsonl; done; echo
   200 200 200 200 200
   

2. The sixth is accepted too

   $ curl -s -i $B2/v1/files -H "Authorization: Bearer $JWT" -F purpose=batch -F file=@two_records.jsonl | grep -iE "^HTTP|^retry-after|^\{"
   HTTP/1.1 200 OK
   {"id":"file-HjNNxyqembS7Qej4E1jUWU","bytes":432,"created_at":1791057221,"filename":"two_records.jsonl","object":"file","purpose":"batch","status":"processed","expires_at":1793649221,"status_details":null}
   

8. A key limited to one upload a day sends a malformed expires_after, then two valid uploads

1. The malformed upload is a 400

   $ curl -s -w "\nHTTP %{http_code}\n" $B1/v1/files -H "Authorization: Bearer $XKEY" -F purpose=batch -F file=@two_records.jsonl -F "expires_after[anchor]=created_at" -F "expires_after[seconds]=soon"
   {"error":{"message":"{'error': \"expires_after[seconds] must be a valid integer, got 'soon': invalid literal for int() with base 10: 'soon'\"}","type":"invalid_request_error","param":null,"code":"400"}}
   HTTP 400
   

2. Both valid uploads go through

   $ for i in 1 2; do B=$([ $i = 1 ] && echo $B2 || echo $B1); curl -s -o /dev/null -w "%{http_code} " $B/v1/files -H "Authorization: Bearer $XKEY" -F purpose=batch -F file=@two_records.jsonl; done; echo
   200 200
   

9. A proxy admin sets a limit of 0 through the settings API

1. /config/field/update does not know the field

   $ curl -s -w "\nHTTP %{http_code}\n" $B1/config/field/update -H "Authorization: Bearer $MASTER_KEY" -H "Content-Type: application/json" -d "{\"field_name\":\"max_file_downloads_per_minute\",\"field_value\":0,\"config_type\":\"general_settings\"}"
   {"detail":{"error":"Invalid field=max_file_downloads_per_minute passed in."}}
   HTTP 400
   

2. /config/update accepts the 0 and says it saved it (the stored row stays {})

   $ curl -s -w "\nHTTP %{http_code}\n" $B2/config/update -H "Authorization: Bearer $MASTER_KEY" -H "Content-Type: application/json" -d "{\"general_settings\":{\"max_batch_file_uploads_per_day\":0}}"
   {"message":"Config updated successfully"}
   HTTP 200
   

10. The settings a proxy admin sees

1. Only the MB size cap is listed

   $ curl -s "$B1/config/list?config_type=general_settings" -H "Authorization: Bearer $MASTER_KEY" | jq -c ".[] | select(.field_name | test(\"^max_(batch_file|file_downloads)\")) | {field_name, field_type, field_value}"
   {"field_name":"max_batch_file_size_mb","field_type":"Integer","field_value":null}
   

After (c74eb7d)

B1=http://127.0.0.1:59122, B2=http://127.0.0.1:45191

1. A 2,000-line batch file (max_batch_file_records: 1000)

1. Upload it, and it is refused with a 413 and never reaches the provider

   $ curl -s -w "\nHTTP %{http_code}\n" $B1/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@records_2000.jsonl
   {"error":{"message":"Batch input file has more than 1000 records, which exceeds the max_batch_file_records of 1000 set in general_settings. The file was not forwarded to the provider.","type":"invalid_request_error","param":"file","code":"413"}}
   HTTP 413
   

2. Six batch uploads in one day across both instances (max_batch_file_uploads_per_day: 5)

1. Five uploads, alternating instances

   $ for i in 1 2 3 4 5; do B=$([ $((i % 2)) = 1 ] && echo $B1 || echo $B2); curl -s -o upload_$i.json -w "%{http_code} " $B/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@two_records.jsonl; done; echo
   200 200 200 200 200
   

2. The sixth, at 2026-10-03T19:53:53Z, is a 429 whose retry-after counts down to 00:00 UTC

   $ curl -s -i $B2/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@two_records.jsonl | grep -iE "^HTTP|^retry-after|^\{"
   HTTP/1.1 429 Too Many Requests
   retry-after: 14767
   {"error":{"message":"Batch file upload limit reached, the file was not forwarded to the provider: max_batch_file_uploads_per_day is 5 for this key (set in general_settings). The count resets at 00:00 UTC, in 14767 seconds.","type":"rate_limit_error","param":null,"code":"429"}}
   

3. Twelve downloads of one batch output file in a minute across both instances (max_file_downloads_per_minute: 10)

1. Ten downloads go through and the eleventh is refused, alternating instances

   $ for i in $(seq 1 11); do B=$([ $((i % 2)) = 1 ] && echo $B1 || echo $B2); curl -s -o /dev/null -w "%{http_code} " $B/v1/files/file-3aZAdRrys53FEKswm4hMUS/content -H "Authorization: Bearer $RUN_KEY"; done; echo
   200 200 200 200 200 200 200 200 200 200 429
   

2. The twelfth, at 2026-10-03T19:56:03Z, is a 429 with retry-after

   $ curl -s -i $B2/v1/files/file-3aZAdRrys53FEKswm4hMUS/content -H "Authorization: Bearer $RUN_KEY" | grep -iE "^HTTP|^retry-after|^\{" | cut -c1-330
   HTTP/1.1 429 Too Many Requests
   retry-after: 57
   {"error":{"message":"Download limit reached for file file-3aZAdRrys53FEKswm4hMUS: max_file_downloads_per_minute is 10 for this key (set in general_settings). Retry in 57 seconds.","type":"rate_limit_error","param":null,"code":"429"}}
   

3. The same batch's input file still downloads, since the limit is per file

   $ curl -s -o /dev/null -w "%{http_code}\n" $B1/v1/files/file-SNuhYGNSCuAGwYuuoXmyFN/content -H "Authorization: Bearer $RUN_KEY"
   200
   

4. A proxy admin sets a download limit of 2 in the key's metadata

1. The limit lands in the key's metadata

   $ curl -s -o /dev/null -w "HTTP %{http_code}\n" $B1/key/update -H "Authorization: Bearer $MASTER_KEY" -H "Content-Type: application/json" -d "{\"key\":\"$KEY\",\"metadata\":{\"max_file_downloads_per_minute\":2}}"
   HTTP 200
   

2. Two downloads after a 61 s wait for the other instance's key cache, from the top of a minute

   $ for i in 1 2; do B=$([ $i = 1 ] && echo $B1 || echo $B2); curl -s -o /dev/null -w "%{http_code} " $B/v1/files/file-JFukripzGSb5sEueEtq4aL/content -H "Authorization: Bearer $KEY"; done; echo
   200 200
   

3. The third download is a 429 that names the key's own limit

   $ curl -s -i $B1/v1/files/file-JFukripzGSb5sEueEtq4aL/content -H "Authorization: Bearer $KEY" | grep -iE "^HTTP|^retry-after|^\{" | cut -c1-330
   HTTP/1.1 429 Too Many Requests
   retry-after: 57
   {"error":{"message":"Download limit reached for file file-JFukripzGSb5sEueEtq4aL: max_file_downloads_per_minute is 2 for this key (set in this key's metadata). Retry in 57 seconds.","type":"rate_limit_error","param":null,"code":"429"}}
   

5. A team admin, not a proxy admin, raises their own key's upload limit

1. The team admin gets a 403

   $ curl -s -o key_update.json -w "HTTP %{http_code}\n" $B2/key/update -H "Authorization: Bearer $DEV_KEY" -H "Content-Type: application/json" -d "{\"key\":\"$DEV_KEY\",\"metadata\":{\"max_batch_file_uploads_per_day\":1000}}"; jq -c "if has(\"metadata\") then {metadata} else . end" key_update.json
   HTTP 403
   {"error":{"message":"{'error': 'Only proxy admins can set max_batch_file_uploads_per_day on a key. It limits what the holder can do with batches, so the holder cannot raise it.'}","type":"auth_error","param":"None","code":"403"}}
   

6. An org admin, not a proxy admin, creates a user with a download limit in metadata

1. The org admin gets a 403 and no user is created

   $ curl -s -o user_new_1.json -w "HTTP %{http_code}\n" $B2/user/new -H "Authorization: Bearer $ORG_ADMIN_KEY" -H "Content-Type: application/json" -d "{\"user_email\":\"new-hire-$SUFFIX@example.com\",\"organizations\":[\"$ORG\"],\"metadata\":{\"max_file_downloads_per_minute\":1000}}"; jq -c "if has(\"key\") then {user_id, key} else . end" user_new_1.json
   HTTP 403
   {"error":{"message":"{'error': 'Only proxy admins can set max_file_downloads_per_minute on a key. It limits what the holder can do with batches, so the holder cannot raise it.'}","type":"internal_server_error","param":null,"code":"403"}}
   

2. A user created without a limit

   $ curl -s -o user_new_2.json -w "HTTP %{http_code}\n" $B1/user/new -H "Authorization: Bearer $ORG_ADMIN_KEY" -H "Content-Type: application/json" -d "{\"user_email\":\"new-hire-2-$SUFFIX@example.com\",\"organizations\":[\"$ORG\"]}"; jq -c "if has(\"key\") then {user_id, key} else . end" user_new_2.json
   HTTP 200
   {"user_id":"40723bc3-18d9-4464-ba16-489920488afb","key":"sk-nR..."}
   

3. Its key's metadata is empty

   $ curl -s "$B2/key/info?key=$(jq -r .key user_new_2.json)" -H "Authorization: Bearer $MASTER_KEY" | jq -c .info.metadata
   {}
   

7. A JWT caller with no virtual key uploads six batch files in one day

1. Five uploads, alternating instances

   $ for i in 1 2 3 4 5; do B=$([ $((i % 2)) = 1 ] && echo $B1 || echo $B2); curl -s -o /dev/null -w "%{http_code} " $B/v1/files -H "Authorization: Bearer $JWT" -F purpose=batch -F file=@two_records.jsonl; done; echo
   200 200 200 200 200
   

2. The sixth is a 429 counted for the JWT user id

   $ curl -s -i $B2/v1/files -H "Authorization: Bearer $JWT" -F purpose=batch -F file=@two_records.jsonl | grep -iE "^HTTP|^retry-after|^\{"
   HTTP/1.1 429 Too Many Requests
   retry-after: 14764
   {"error":{"message":"Batch file upload limit reached, the file was not forwarded to the provider: max_batch_file_uploads_per_day is 5 for user lit5275-jwt-after_1791057226 (set in general_settings). The count resets at 00:00 UTC, in 14764 seconds.","type":"rate_limit_error","param":null,"code":"429"}}
   

8. A key limited to one upload a day sends a malformed expires_after, then two valid uploads

1. The malformed upload is a 400 and uses no slot

   $ curl -s -w "\nHTTP %{http_code}\n" $B1/v1/files -H "Authorization: Bearer $XKEY" -F purpose=batch -F file=@two_records.jsonl -F "expires_after[anchor]=created_at" -F "expires_after[seconds]=soon"
   {"error":{"message":"{'error': \"expires_after[seconds] must be a valid integer, got 'soon': invalid literal for int() with base 10: 'soon'\"}","type":"invalid_request_error","param":null,"code":"400"}}
   HTTP 400
   

2. The first valid upload goes through and the second is a 429

   $ for i in 1 2; do B=$([ $i = 1 ] && echo $B2 || echo $B1); curl -s -o /dev/null -w "%{http_code} " $B/v1/files -H "Authorization: Bearer $XKEY" -F purpose=batch -F file=@two_records.jsonl; done; echo
   200 429
   

9. A proxy admin sets a limit of 0 through the settings API

1. /config/field/update refuses the 0

   $ curl -s -w "\nHTTP %{http_code}\n" $B1/config/field/update -H "Authorization: Bearer $MASTER_KEY" -H "Content-Type: application/json" -d "{\"field_name\":\"max_file_downloads_per_minute\",\"field_value\":0,\"config_type\":\"general_settings\"}"
   {"detail":{"error":"Invalid type of field value=<class 'int'> passed in."}}
   HTTP 400
   

2. /config/update refuses the 0 with a 422

   $ curl -s -w "\nHTTP %{http_code}\n" $B2/config/update -H "Authorization: Bearer $MASTER_KEY" -H "Content-Type: application/json" -d "{\"general_settings\":{\"max_batch_file_uploads_per_day\":0}}"
   {"detail":[{"type":"greater_than","loc":["body","general_settings","max_batch_file_uploads_per_day"],"msg":"Input should be greater than 0"}]}
   HTTP 422
   

10. The settings a proxy admin sees

1. All four batch file settings are listed with their values

   $ curl -s "$B1/config/list?config_type=general_settings" -H "Authorization: Bearer $MASTER_KEY" | jq -c ".[] | select(.field_name | test(\"^max_(batch_file|file_downloads)\")) | {field_name, field_type, field_value}"
   {"field_name":"max_batch_file_size_mb","field_type":"Integer","field_value":null}
   {"field_name":"max_batch_file_records","field_type":"Integer","field_value":1000}
   {"field_name":"max_batch_file_uploads_per_day","field_type":"Integer","field_value":5}
   {"field_name":"max_file_downloads_per_minute","field_type":"Integer","field_value":10}
   

What changed between the legs: the 2,000-line file gets a 413 and is not forwarded, the sixth upload of the day gets a 429 whose retry-after counts down to 00:00 UTC (14767 s at 19:53:53Z is exactly the time left to midnight), the eleventh download of one file in a minute gets a 429 while the batch's input file still downloads, a key's own metadata limit of 2 wins over the default of 10, a JWT caller is counted per user, a 400 upload uses no slot, a non-admin can no longer raise a limit on their own key or a new user's key, a 0 is refused by both settings routes instead of being saved as nothing, and /config/list shows the three settings. The five uploads that fit were split 3 and 2 across the two instances and the sixth was refused on the second one, so the count is shared through Redis

Also checked at the tip

The PR merged into main

Main has moved six commits past the merge base 50190134c3 (traces and lens, cost map rows, a Bedrock streaming timeout, a Makefile target), none touching a file this PR changes, and the merge is clean. The core cases ran again on that merge (ebda28890b, main f445e466b4 plus this tip) with one proxy with --num_workers 2 and Redis

1. A 2,000-line batch file is a 413

   $ curl -s -w "\nHTTP %{http_code}\n" $B/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@records_2000.jsonl
   {"error":{"message":"Batch input file has more than 1000 records, which exceeds the max_batch_file_records of 1000 set in general_settings. The file was not forwarded to the provider.","type":"invalid_request_error","param":"file","code":"413"}}
   HTTP 413
   

2. Five uploads go through and the sixth, at 2026-10-03T20:15:26Z, is a 429

   $ for i in 1 2 3 4 5; do curl -s -o upload_$i.json -w "%{http_code} " $B/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@two_records.jsonl; done
   200 200 200 200 200
   $ curl -s -i $B/v1/files -H "Authorization: Bearer $KEY" -F purpose=batch -F file=@two_records.jsonl | grep -iE "^HTTP|^retry-after|^\{"
   HTTP/1.1 429 Too Many Requests
   retry-after: 13474
   {"error":{"message":"Batch file upload limit reached, the file was not forwarded to the provider: max_batch_file_uploads_per_day is 5 for this key (set in general_settings). The count resets at 00:00 UTC, in 13474 seconds.","type":"rate_limit_error","param":null,"code":"429"}}
   

3. Ten downloads of the first upload go through and the eleventh is a 429

   $ for i in $(seq 1 10); do curl -s -o /dev/null -w "%{http_code} " $B/v1/files/file-Adpvj6aNVn4KAMPbuTqFsN/content -H "Authorization: Bearer $KEY"; done
   200 200 200 200 200 200 200 200 200 200
   $ curl -s -i $B/v1/files/file-Adpvj6aNVn4KAMPbuTqFsN/content -H "Authorization: Bearer $KEY" | grep -iE "^HTTP|^retry-after|^\{" | cut -c1-330
   HTTP/1.1 429 Too Many Requests
   retry-after: 54
   {"error":{"message":"Download limit reached for file file-Adpvj6aNVn4KAMPbuTqFsN: max_file_downloads_per_minute is 10 for this key (set in general_settings). Retry in 54 seconds.","type":"rate_limit_error","param":null,"code":"429"}}
   

4. The settings list shows all four batch file settings

   $ curl -s "$B/config/list?config_type=general_settings" -H "Authorization: Bearer $MASTER_KEY" | jq -c ".[] | select(.field_name | test(\"^max_(batch_file|file_downloads)\")) | {field_name, field_value}"
   {"field_name":"max_batch_file_size_mb","field_value":null}
   {"field_name":"max_batch_file_records","field_value":1000}
   {"field_name":"max_batch_file_uploads_per_day","field_value":5}
   {"field_name":"max_file_downloads_per_minute","field_value":10}
   

The counter's Redis span in a trace

The same merged proxy with callbacks: ["otel"], service_callback: ["otel"], and OTEL_EXPORTER=otlp_http pointed at a local OTLP receiver that decodes the protobuf. With LITELLM_OTEL_V2=true, one batch upload exports the counter's Redis increment as redis.incr rate_limits, nested under the request span, with litellm.service.target = rate_limits and the caller chain _increment_all_or_none <- consume_file_usage; without the rate_limits declaration the span name would be the bare redis.incr (read from litellm/integrations/otel/model/spans.py, not observed). With the default (V1) logger, no Redis service span is exported for this request at all, not for these counters and not for the parallel request limiter that runs in the same request, so nothing changes there

'postgres.select LiteLLM_VerificationToken'   parent='auth /v1/files'   target=None
'auth /v1/files'                              parent='POST /v1/files'   target=None
'redis.incr rate_limits'                      parent='POST /v1/files'   target='rate_limits'  caller='_increment_all_or_none <- consume_file_usage'
'POST /v1/files'                              parent=ROOT               target=None

A file's count survives many other file ids, no Redis

One proxy process with --num_workers 1 and no Redis, max_file_downloads_per_minute: 3. The caller downloads file X to its limit, then 205 other file ids, then file X again. At 5583f60542 (this PR before the counter store got its own bound) the download counters shared the rate limiters' bounded in-memory cache, so the 205 other ids pushed file X's count out and it got a fresh allowance; at c74eb7d1c9 the counters keep their own store of 20,000 entries and file X stays refused

1. At 5583f60542, three downloads then a 429, 205 other ids, then file X is served again

   $ for i in 1 2 3 4; do curl -s -o /dev/null -w "%{http_code} " $B/v1/files/file-Lfqi1crcaRsZuA5aAFev6K/content -H "Authorization: Bearer $KEY"; done
   200 200 200 429
   $ seq 1 205 | xargs -P 8 -I{} curl -s -o /dev/null -w "%{http_code}\n" $B/v1/files/file-lit5275-other-{}/content -H "Authorization: Bearer $KEY" | sort | uniq -c
    205 404
   $ curl -s -i $B/v1/files/file-Lfqi1crcaRsZuA5aAFev6K/content -H "Authorization: Bearer $KEY" | grep -iE "^HTTP|^retry-after|^\{" | cut -c1-330
   HTTP/1.1 200 OK
   {"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-5.4-nano", "messages": [{"role": "user", "content": "Reply with the single word OK (1)"}], "max_completion_tokens": 16}}
   {"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-5.4-nano", "messages": [{"role": "user", "content": "Reply with the single word OK (2)"}], "max_completion_tokens": 16}}
   

2. At c74eb7d1c9, the same sequence in one minute window, and file X stays refused

   $ for i in 1 2 3 4; do curl -s -o /dev/null -w "%{http_code} " $B/v1/files/file-Y5DbzY46eFSYdXQNLfAZ8d/content -H "Authorization: Bearer $KEY"; done
   200 200 200 429
   $ seq 1 205 | xargs -P 8 -I{} curl -s -o /dev/null -w "%{http_code}\n" $B/v1/files/file-lit5275-other-{}/content -H "Authorization: Bearer $KEY" | sort | uniq -c
    205 404
   $ curl -s -i $B/v1/files/file-Y5DbzY46eFSYdXQNLfAZ8d/content -H "Authorization: Bearer $KEY" | grep -iE "^HTTP|^retry-after|^\{" | cut -c1-330
   HTTP/1.1 429 Too Many Requests
   retry-after: 53
   {"error":{"message":"Download limit reached for file file-Y5DbzY46eFSYdXQNLfAZ8d: max_file_downloads_per_minute is 3 for this key (set in general_settings). Retry in 53 seconds.","type":"rate_limit_error","param":null,"code":"429"}}
   

Surprises from the run:

• /key/update errors nest a stringified Python dict; pre-existing
• /user/new 403 carries type internal_server_error, a pre-existing mapping
• /user/new 403 says "on a key", meaning the auto-created one
• /config/field/update answers a 0 with "Invalid type"
• The expires_after 400 leaks the int() error text; pre-existing
• Each cap rejection logs a traceback, like other rate-limit 429s
• The before leg's /config/update saved the 0 as {}

Audit round 1 of 3

Merge base 50190134c3, product tip c74eb7d1c9, tests commits 2085d71244 and 4ee2d9a9de. Every cell is a checked-in tests/integration/ test the audit wrote: the proxy (two workers), Postgres, and Redis run for real, the provider's Files API is the owned scripted upstream, and the span cell posts to a local OTLP sink. Nothing inside the proxy is mocked. The selection is the 134 cases (48 test functions) in the three management/ files and the one observability/ file, run as node ids with tests/integration/run.py management ... --seed 43632 --order-seed 43632 (the span cell through the extensions group), once on the merge base and twice on the head with the same seeds: identical collected and passed selections both times, no skips, no retries

Two runs went into the matrix. The whole selection ran at 2085d71244 on 2026-10-05 and 2026-10-06 UTC (base 23:46:28Z, head 1 00:12:34Z, head 2 00:47:35Z; local result dirs 2085d71244/{base,head1,head2}_{management,extensions}). The merge base fails 127 of the 134 cases and passes the 7 controls, and the head passes all 134 both times. An earlier full run at the tests commit b7851f201e was thrown out: make check had copied the worktree's .env back in before it, so every proxy in it booted with that file loaded (the Environment variables loaded from ../../.env log line), and in this run none of the 31 proxies did. That discarded run also showed two chaos cells failing once each (test_a_download_cap_stored_through_the_config_api_reaches_every_worker_and_survives_a_restart, test_downloads_keep_answering_while_redis_hangs), both root-caused to the tests, not the product: the restart check ran three sequential downloads with no guard against straddling a UTC minute boundary, and the exact-cap probe ran while both workers' Redis circuit breakers were still open from the previous cell's outage (60 s default recovery), so it counted in per-process memory and the two workers split the cap. 2085d71244 fixes both with window-guarded bursts, a 2 s breaker recovery in the chaos fixture, and a shared-count check that drives a fresh key to its cap through a one-worker sibling proxy and then expects the two-worker candidate to refuse everything

4ee2d9a9de is a tests-only commit on top of it (it releases the held upload handlers when the killed-worker chaos cell fails early, after a Greptile P2 at 2085d71244), so per the rerun rule only the cell it changed, management/test_batch_file_usage_caps_chaos.py::test_a_killed_worker_keeps_its_upload_slots_used_and_the_sibling_serving, ran again at 4ee2d9a9de (base 01:43:26Z, head 1 02:06:23Z, head 2 02:26:16Z on 2026-10-06; result dirs 4ee2d9a9de/{base,head1,head2}_management): red on the merge base, green on both head runs. Every other row keeps its 2085d71244 result, and the product code under every row is still c74eb7d1c9

Each row is one test function; cases counts its parametrized cases, and a column shows how many of them passed or failed on that leg

cell	cases	base	head run 1	head run 2	last ran at	verdict
management/test_batch_file_usage_caps.py::test_a_file_id_with_a_slash_never_reaches_the_file_route_or_a_counter	2	2 pass	2 pass	2 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_a_limit_changed_during_a_download_burst_only_answers_200_or_429	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_a_lowered_limit_reaches_the_other_proxy_process	2	2 fail	2 pass	2 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_async_openai_sdk_caller_sees_each_limit_as_a_status_error	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_callers_without_a_valid_key_are_rejected_before_anything_is_counted	1	pass	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_concurrent_requests_across_processes_never_exceed_a_limit	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_download_the_provider_cannot_find_still_uses_a_slot	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_downloads_are_unlimited_when_no_limit_is_set	1	pass	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_every_download_route_shares_one_count_per_file	3	3 fail	3 pass	3 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_every_upload_route_shares_one_daily_count	3	3 fail	3 pass	3 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_hostile_file_ids_are_counted_as_the_provider_receives_them	2	2 fail	2 pass	2 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_jwt_callers_are_counted_per_user_across_tokens	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_key_daily_upload_limit_replaces_the_yaml_limit	2	2 fail	2 pass	2 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_key_download_limit_counts_one_file_per_key_across_proxy_processes	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_limited_key_still_reaches_chat_completions_and_file_metadata	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_malformed_limits_are_ignored_and_the_yaml_limits_still_apply	20	20 fail	20 pass	20 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_managed_files_are_capped_and_counted_under_their_unified_id	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_model_routed_files_are_counted_under_the_id_the_caller_uses	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_numeric_string_limits_are_honored	2	2 fail	2 pass	2 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_only_batch_purpose_uploads_are_limited_and_counted	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_openai_sdk_caller_sees_each_limit_as_a_status_error	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_record_limit_counts_request_lines_not_blank_lines_or_line_endings	4	2 fail, 2 pass	4 pass	4 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_record_limit_is_the_lower_of_the_key_level_and_team_limits	6	6 fail	6 pass	6 pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_rejected_uploads_do_not_use_a_daily_upload_slot	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_team_download_limit_is_shared_by_every_key_on_the_team	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_upload_rejected_by_the_key_limit_does_not_use_a_team_slot	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_upload_the_provider_rejects_still_uses_a_daily_upload_slot	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_yaml_daily_upload_limit_counts_one_key_across_proxy_processes	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps.py::test_yaml_record_limit_rejects_a_longer_batch_file_before_it_reaches_the_provider	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps_chaos.py::test_a_download_cap_stored_through_the_config_api_reaches_every_worker_and_survives_a_restart	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps_chaos.py::test_a_killed_worker_keeps_its_upload_slots_used_and_the_sibling_serving	1	fail	pass	pass	4ee2d9a9de	PASS
management/test_batch_file_usage_caps_chaos.py::test_downloads_keep_answering_while_redis_hangs	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps_chaos.py::test_the_daily_count_survives_a_proxy_restart	1	fail	pass	pass	2085d71244	PASS
management/test_batch_file_usage_caps_chaos.py::test_uploads_fall_back_to_per_process_counts_while_redis_is_down	1	fail	pass	pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_bulk_user_creation_refuses_only_the_row_that_puts_a_batch_limit_on_a_key	4	4 fail	4 pass	4 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_config_field_update_refuses_a_malformed_file_limit	18	18 fail	18 pass	18 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_config_list_shows_each_file_limit_as_an_unset_integer	3	3 fail	3 pass	3 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_config_update_refuses_a_zero_file_limit	3	3 fail	3 pass	3 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_org_admin_cannot_put_a_batch_limit_on_a_new_team	4	4 fail	4 pass	4 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_org_admin_cannot_put_a_batch_limit_on_a_new_users_key	4	4 fail	4 pass	4 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_org_admin_cannot_put_a_batch_limit_on_an_existing_team	4	4 fail	4 pass	4 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_org_admin_may_resend_a_stored_team_limit_but_not_change_it	5	5 fail	5 pass	5 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_proxy_admin_sets_every_batch_limit_on_every_route	1	pass	pass	pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_team_admin_cannot_put_a_batch_limit_on_a_new_key	4	4 fail	4 pass	4 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_team_admin_cannot_put_a_batch_limit_on_a_regenerated_key	4	4 fail	4 pass	4 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_team_admin_cannot_put_a_batch_limit_on_an_existing_key	4	4 fail	4 pass	4 pass	2085d71244	PASS
management/test_batch_limit_metadata_admin_only.py::test_team_admin_may_resend_a_stored_key_limit_but_not_change_it	5	5 fail	5 pass	5 pass	2085d71244	PASS
observability/test_file_usage_counter_span.py::test_a_counted_upload_exports_its_counter_increment_as_a_rate_limits_span	1	fail	pass	pass	2085d71244	PASS

Inventory walk: every row of the surface inventory (records per file, daily uploads, downloads per minute, malformed and hostile values, limit changes under traffic, concurrency across workers, who may set the limits, the config endpoints, Redis stopped and Redis hanging mid-burst, a killed worker, a proxy restart, and the counter's OTLP span) has a cell above that ran at the head and matched its expected outcome, so no row is unverified. The seven cases that pass on the merge base are the controls whose behavior the PR does not change: an unauthenticated caller, no limit set anywhere, a slash in the file id (two routes), blank lines and CRLF around a file within the record limit (two shapes), and a proxy admin setting every limit

• 4ee2d9a passes /audit

Type

🆕 New Feature

Caveats (if any)

Medium

• A user who is not a proxy admin now gets a 403 setting a batch limit in metadata on POST /user/new (when it creates a key) and on bulk user creation
  • Both accepted batch_enqueued_token_limit before, so a provisioning script that set it as a non-admin breaks; key and team routes already refused it
  • Workaround: a proxy admin sets the value
  • Kept, since allowing it reopens the way around the admin-only rule
• Without Redis, each worker process keeps its own count
  • A caller can get the limit times the number of workers, and the counts reset when the proxy restarts, like the request rate limits
  • Each worker tracks 20,000 live counters, then drops the one expiring soonest
  • While Redis is down the counts fall back to each instance's memory; fail_closed_rate_limit_enforcement covers the request rate limiter only, the same as the batch enqueued-token limiter
  • After Redis comes back, each worker keeps counting in its own memory until its Redis circuit breaker closes (up to REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT, 60 s by default), and the counts taken in memory during the outage are not carried back to Redis
  • Workaround: run Redis, which is how the counts are shared across pods
  • Kept, since a shared count without Redis needs a new table and a write per call, and failing closed would block every batch upload and download
• Pass-through routes (/openai/..., /azure/..., /vertex_ai/..., /bedrock/...) are not capped
  • A key allowed on them can upload and download around the caps, the same boundary max_batch_file_size_mb and the enqueued-token limit have
  • Workaround: allowed_routes: ["openai_routes"] on the key keeps /v1/files and /v1/batches open and returns 403 on those routes
  • Kept, since capping them means parsing each provider's own file API

Low

• No per-end-user limit: the scope is the key, its team, and the user id for a JWT caller with no key
  • The ticket left the scope open among key, team, and end user; see Design decisions for why the end-user table cannot carry it yet
• The download cap counts every file's content, not only batch output files
  • Kept, since telling an output file apart needs a provider lookup before the throttle
• Uploads per day is a UTC calendar day, so a caller can use two days' allowance in the hours around 00:00 UTC
  • Kept, since a calendar day is what the error and Retry-After can state plainly
• A JWT caller with no key and no user id gets only its team's limit
  • Kept, since counting the key default per team makes it team-wide
• The 403 text for batch_enqueued_token_limit changed wording
  • Status and type are unchanged; the old text fit only that one key
• A download that 404s at the provider still uses a slot
  • Kept, since counting after the call lets a loop reach the provider
• A request cancelled mid-count may keep one slot until the window ends
  • Kept, since it is at most one slot and clears with the window
• Hook, guardrail, or provider failures still use an upload slot
  • On purpose, see Design decisions
• A 0 or non-integer in metadata or the config file is ignored with a warning
  • Kept, since failing requests on a bad value would stop batch traffic
• /config/update with a 0 for a new setting is now a 422, not 200, and /config/field/update rejects a 0 with an unclear "Invalid type" message
  • Intended, since a 0 would have done nothing; the text comes from the validator every setting shares
• The /user/new 403 says internal_server_error, a pre-existing mapping
  • Kept, since the route maps every error that way today
• Each rejected upload or download logs a traceback at error level
  • Same as every other rejection on these routes today

Live PR risk (c74eb7d)

Run at c74eb7d1c9 over the two product commits since the last pass at c13b47730d (the counter store's own 20,000-entry bound, and the rate_limits service target on the counter increments), plus the merge of main at 5583f60542; the earlier findings below were re-checked and stand

Breaking

• A user who is not a proxy admin setting batch_enqueued_token_limit in metadata on POST /user/new (with a key created) or POST /management/v1/users/bulk got a 200 and the value on the new key before, and now gets a 403. Read from the diff: neither route called the admin-only check at the merge base; the same shape was observed live for max_file_downloads_per_minute in case 6 (200 before, 403 after). Intended, since it closes the way around the rule the key and team routes already had. Shipping approved by: mateo-berri, in the approving review on this PR

Backward incompatible

• The 403 message for batch_enqueued_token_limit on key and team routes changed from "It replaces the standard rate limit checks for batch submissions." to "It limits what the holder can do with batches, so the holder cannot raise it." Status and type are unchanged. Shipping approved by: mateo-berri, in the approving review on this PR
• enforce_batch_enqueued_token_limit_is_admin_only and its BatchEnqueuedTokenLimitRequest protocol are renamed to enforce_batch_limits_are_admin_only and BatchLimitRequest. Every caller in litellm/ and enterprise/ is updated; code outside the repo importing the old name would break. Shipping approved by: mateo-berri, in the approving review on this PR
• The two late commits change no contract: FILE_USAGE_MAX_TRACKED_COUNTERS is a new constant with no config key or env var behind it, and with_service_target only names the counter's Redis calls for the service spans

Regression risk

• POST /management/v1/users/bulk was not driven live. Its unit tests cover the new 403, and it shares the check /user/new ran live in case 6
• Team metadata limits were not driven live across two instances. The unit tests cover the team counter and the rollback when the key's counter refuses

Dependency graph

• POST /v1/files (record and upload caps): unit tests plus live, cases 1, 2, 7, 8, and again on the merged tree
• GET /v1/files/{id}/content (download cap): unit tests plus live, cases 3 and 4, and again on the merged tree
• The batch file line validation shared by /v1/files and managed files: unit tests, the Buildkite files and batches e2e, and tests/pass_through_unit_tests/test_passthrough_managed_ids.py run locally
• Key and team create or update routes (the admin-only check): unit tests plus live, case 5
• POST /user/new with a created key: unit tests plus live, case 6
• POST /management/v1/users/bulk: unit tests only
• The admin-only check's rename: every caller updated, tests/local_testing/test_auth_utils.py run locally
• The batch enqueued-token limiter next to the new counters: tests/batches_tests/test_batch_rate_limits.py run locally
• /config/update, /config/field/update, /config/list, and the Router Settings page: live, cases 9 and 10 plus the User Flow screenshots
• The dashboard's schema.d.ts: regenerated, and the sync check passes
• ProxyLogging.file_usage_cache and its 20,000-entry bound: update_values is its only other reader (it attaches Redis), no setting resizes it (update_in_memory_max_size is applied to user_api_key_cache only), unit test through ProxyLogging, and live in the no-Redis check above (a count survives 205 other file ids at the tip and did not at 5583f60542)
• Readers of the rate_limits service target: the service logger's ServiceLoggerPayload.target (OTel span name and litellm.service.target, verified live under the V2 logger on the merged tree), the Redis pipeline's family label (not reached, the counters use single increments, unit tests in tests/unit/caching/test_redis_batch.py), and the response-cache phase check in litellm/caching/caching.py (compares to llm_response, unaffected); tests/unit/test_internal_context.py gates the declaration
• Pass-through file routes: not reached by the caps (Low caveat and Design decisions)

Not verified

• The tests/openai_endpoints_tests files and batches suites need a live proxy and were not run. They set no limit, so they take the path that returns early, which the Buildkite files and batches e2e covered at this tip
• The counter's redis.incr rate_limits span was read from a local OTLP receiver, not from a vendor's trace UI; the main checkout's .env carries no OTel backend credential
• Veria posted no review on this PR
• Pass-through routes were not driven

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes rate limiting and admin-gated metadata on /v1/files and key/team/user provisioning; behavior depends on Redis for cross-worker accuracy, but the change set is heavily tested.

Overview
Adds three optional proxy caps on batch file traffic: max_batch_file_records (413 on oversized .jsonl), max_batch_file_uploads_per_day (429 per UTC day for purpose=batch), and max_file_downloads_per_minute (429 per file with Retry-After). Limits come from general_settings, with key and team metadata overrides; record caps use the lower of key vs team.

Enforcement is wired into /v1/files upload validation and a new file_usage_caps layer that increments Redis-backed counters via a dedicated file_usage_cache (separate from RPM/TPM caches). enforce_batch_enqueued_token_limit_is_admin_only is generalized to enforce_batch_limits_are_admin_only for all batch-limit metadata keys, including on POST /user/new and bulk user creation when auto_create_key is true.

Config types, Router Settings / schema.d.ts, and broad unit plus integration tests cover precedence, JWT-per-user counting, multi-worker Redis, and admin-only rules.

^{Reviewed by Cursor Bugbot for commit 4ee2d9a. Bugbot is set up for automated code reviews on this repo. Configure here.}

• c74eb7d passes /live-pr-risk

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[codecov]

Codecov Report

❌ Patch coverage is 98.35165% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...lm/proxy/openai_files_endpoints/file_usage_caps.py	98.36%	2 Missing ⚠️
litellm/proxy/utils.py	66.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds file usage rate-limiting to the proxy server.

The PR appears safe to merge; no new actionable issue or outstanding previous finding remains.

Findings

1.  Global record cap is overridden ▶
2.  Interrupted checks retain quota charges ▶

Summary

Adds optional record, daily-upload, and per-file download caps for proxy file routes, with admin-only metadata overrides.

• Adds dedicated usage counters and exposes the settings through proxy configuration.
• Adds unit and integration coverage for limits, authorization, Redis behavior, and tracing.

_{Reviews (7) · Last reviewed commit: "test(files): release the held uploads wh..."}

[greptile-apps]

Global record cap is overridden When general_settings.max_batch_file_records is 10 and a key has an admin-set value of 100, _key_limit drops the general setting before the code chooses the lowest limit. An 11-record batch is accepted even though the new setting says the lowest applicable value wins.

[mateo-berri]

By design, a key's metadata replaces the general_settings default for that key, and only a proxy admin can set that value

[greptile-apps]

You're right — the key metadata is intentionally an override of the general_settings default for that key, rather than an additional limit. _key_limit implements that precedence, while the team metadata remains an independent limit that is applied alongside it. My finding was incorrect; no change is needed.

[greptile-apps]

Interrupted checks retain quota charges If a request is cancelled or errors after the key counter is incremented but before the team counter is checked or the charge is reversed, the key increment remains. Valid uploads or downloads can then receive 429 responses for the rest of the counter's lifetime.

Knowledge Base Used: Spend, budgets, and rate limits

[mateo-berri]

A cancel between increment and rollback leaks at most one slot for the rest of that window, which we accept over adding a lock

[greptile-apps]

Understood. Given the explicit tradeoff of accepting at most one leaked slot per window to avoid introducing a lock, I’ll withdraw this as a blocking finding. The behavior is documented and bounded by the counter window.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_batch_file_usage_caps (4ee2d9a) with main (6f5ca84)¹}

Footnotes

1. No successful run was found on main (d232b00) during the generation of this report, so 6f5ca84 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4ee2d9a. Configure here.}

[mateo-berri]

LGTM