test(e2e): cover key rpm/tpm rate limiting, window reset, and pacing headers

[mateo-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

This PR is stacked on #32928, which introduces the quota_management registry module these tests claim cells in; merge that first

Everything below was run on 2026-07-11 with the test code at commit 36dff0f, against proxies built from source at exact refs, with real Anthropic, OpenAI, and Gemini traffic. Outputs are pasted verbatim; following the same steps reproduces them end to end

1. Build a proxy image from the ref under test (shown: e7f4144, the litellm_internal_staging tip at capture time):

   $ git worktree add /tmp/wt-staging e7f41442d0
   $ docker build -t litellm-local:stg-e7f41442d0 /tmp/wt-staging
   

2. Point the tests/e2e compose stack at that image on a free host port via an override file:

   $ cat > /tmp/override-stg.yml <<'EOF'
   services:
     litellm:
       image: litellm-local:stg-e7f41442d0
       ports: !override
         - "4521:4000"
   EOF
   

3. Make sure tests/e2e/.env carries the provider keys the compose proxy loads (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY) plus LITELLM_LICENSE; the stack reads env only from that file

4. Bring the stack up under a unique compose project name and confirm it answers:

   $ cd tests/e2e && docker compose -p e2e_stg_4521 -f docker-compose.yml -f /tmp/override-stg.yml up -d
   $ curl -fs http://localhost:4521/health/liveliness
   "I'm alive!"
   

5. Run the whole quota_management family against it:

   $ LITELLM_PROXY_URL=http://localhost:4521 uv run pytest quota_management/ -v
   FAILED quota_management/budgets/test_budget_fallback_e2e.py::test_budget_fallback_reroutes_anthropic_messages_to_openai
   FAILED quota_management/budgets/test_model_max_budget_e2e.py::test_model_max_budget_isolates_per_model
   ============= 2 failed, 73 passed, 2 skipped in 625.81s (0:10:25) ==============
   

   All four tests this PR adds pass against the staging-tip proxy. The two failures are license-gated budget tests from the base branch: at capture time tests/e2e/.env had no LITELLM_LICENSE yet, so both 500 on /key/generate with "You must have an enterprise license to set model_max_budget". After adding the license line and recreating the proxy container (same up -d command plus --force-recreate litellm) they pass too, so with a properly populated .env every non-skipped test in the family passes on this proxy:

   $ LITELLM_PROXY_URL=http://localhost:4521 uv run pytest \
       "quota_management/budgets/test_model_max_budget_e2e.py::test_model_max_budget_isolates_per_model" \
       "quota_management/budgets/test_budget_fallback_e2e.py::test_budget_fallback_reroutes_anthropic_messages_to_openai" -q
   2 passed in 16.67s
   

6. Repeat steps 1-5 at ref cfac552 (head of fix: atomic TPM rate limit #27001, the May 2026 atomic-TPM-limit fix; image tag litellm-local:pr27001-cfac552c1e, port 4522, project e2e_pr27001_4522) to see the new tests fail on code whose limiter genuinely misbehaves, rather than rubber-stamping whatever proxy they hit:

   $ LITELLM_PROXY_URL=http://localhost:4522 uv run pytest quota_management/ -v
   FAILED quota_management/budgets/test_budget_fallback_e2e.py::test_budget_fallback_reroutes_anthropic_messages_to_openai
   FAILED quota_management/budgets/test_budget_reset_advances_e2e.py::test_team_member_budget_reset_at_advances
   FAILED quota_management/budgets/test_model_max_budget_e2e.py::test_model_max_budget_isolates_per_model
   FAILED quota_management/budgets/test_team_member_budget_reset_e2e.py::test_team_member_budget_reset_keeps_advancing
   FAILED quota_management/budgets/test_team_multi_window_budget_e2e.py::test_team_short_window_blocks_then_resets
   FAILED quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_tpm_limit_blocks_over_limit
   FAILED quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_headers_report_limit_and_remaining
   FAILED quota_management/spend_tracking/test_spend_tracking_e2e.py::test_request_tags_round_trip
   FAILED quota_management/spend_tracking/test_spend_tracking_e2e.py::test_tag_spend_matches_sum_of_tagged_logs
   ============= 9 failed, 66 passed, 2 skipped in 506.40s (0:08:26) ==============
   

   On that ref every chat call on a tpm-limited key dies with a 400 before reaching the model, because the proxy of that era leaked an internal field into the outbound Anthropic request body; the tpm and headers tests this PR adds surface it immediately:

   E  Failed: upstream call failed (status 400); body={"error":{"message":"litellm.BadRequestError: AnthropicException -
      {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"_litellm_rate_limit_descriptors:
      Extra inputs are not permitted\"}, ...
   

   test_rpm_limit_blocks_over_limit and test_rpm_limit_resets_after_window pass there (rpm-only keys never trigger the leak), with the license in place model_max_budget passes on that ref too, and the remaining failures are base-branch suites asserting tag attribution, budget-reset advancement, and budget fallback behavior that postdates May 2026

7. Tear each stack down:

   $ docker compose -p e2e_stg_4521 -f docker-compose.yml -f /tmp/override-stg.yml down -v
   $ docker compose -p e2e_pr27001_4522 -f docker-compose.yml -f /tmp/override-pr27001.yml down -v
   

8. Static gates at the same test-code commit, run from the repo root and tests/e2e respectively:

   $ uv run --no-sync basedpyright tests/e2e
   0 errors, 0 warnings, 0 notes
   
   $ cd tests/e2e && uv run python -m coverage_registry.collector --strict
   Quota Management                     31/33       93.9%
   

   The --strict run performs a full pytest collect-only pass over tests/e2e, so every touched module still imports and every covers marker resolves to a registry cell; Quota Management moves from 27/33 on the base branch to 31/33 with the four ratelimit enforcement cells covered here

Type

✅ Test

Changes

The quota_management.ratelimit registry cells rpm.blocks_over_limit and tpm.blocks_over_limit are P0 rows with no covering test: nothing proved that a key's rpm_limit or tpm_limit actually produces a 429 on breach. This adds the ratelimit/ subfolder of the quota_management/ suite family (#32928 moves budgets/ and spend_tracking/ under the same parent; the layout is documented in tests/e2e/CLAUDE.md and matches the registry module these tests claim cells in) with tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py, four live tests following the suite lifecycle contract (create key, assert /key/info echoes the limit, drive real traffic, assert the enforced behavior, teardown)

test_rpm_limit_blocks_over_limit drives a key with rpm_limit 3 to its fourth call and asserts the 429 body carries the api_key scope, the requests limit type, and a positive integer retry-after header. test_tpm_limit_blocks_over_limit enforces tpm_limit 60 with exact token accounting rather than a call-count heuristic: it accumulates each response's actual usage.total_tokens, and once that total reaches the limit the very next call must 429 with the tokens limit type. Because the v3 limiter reserves an estimate pre-call (message chars // 4 + max_tokens, reconciled to actual usage after the call), a block may legitimately land before the actual total crosses the limit; the test accepts such a 429 only when the blocked call's reservation exceeds the remaining budget the 429 body itself reports, so a limiter that blocks while the call still fits fails the test. test_rpm_limit_resets_after_window exhausts an rpm_limit 1 key, asserts the block, then polls until the key serves again and asserts the recovery arrived no sooner than the full 60s window (less a small tolerance for the limiter's integer-second window arithmetic) measured from the send of the first counted call, so a limiter that resets early fails the test. test_headers_report_limit_and_remaining asserts a successful response reports x-ratelimit-api_key-{limit,remaining}-{requests,tokens} so clients can pace

Supporting harness changes: StreamingResponse now captures response headers (lowercased names) so tests can assert retry-after and the pacing headers, KeyInfo gains rpm_limit, and the management key-persistence test now also configures and asserts rpm_limit read-back (it previously covered tpm_limit only). Following review feedback, the suite's model comes from a shared CHEAP_ANTHROPIC_MODEL in e2e_config.py (env E2E_CHEAP_ANTHROPIC_MODEL, default claude-haiku-4-5) instead of a hardcoded name; the other e2e suites hardcode the same name in 25+ places, so migrating them is left as a follow-up to keep this PR isolated

QA runbook

Each test below maps 1:1 to what a user would do by hand against a live proxy (all calls authorized with the key from step 1 except the management routes, which use the master key). One harness-only nuance: a freshly generated key can briefly 401 on its very first chat call until the data plane's auth cache picks it up, so retry that first call for a few seconds; a 401 never reaches the rate limiter, so it consumes no budget

• tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_rpm_limit_blocks_over_limit - a key allowed 3 requests a minute serves exactly 3 and refuses the 4th

  • POST /key/generate with {"models":["claude-haiku-4-5"],"rpm_limit":3} and save the returned key
  • GET /key/info for that key and confirm it echoes "rpm_limit": 3
  • Send 3 short chat completions to /v1/chat/completions with the key inside one minute and confirm all 3 return 200
  • Send a 4th call in the same minute and confirm a 429 whose body contains Rate limit exceeded for api_key and Limit type: requests, with a positive integer retry-after response header
  • POST /key/delete to clean up

• tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_tpm_limit_blocks_over_limit - a key with a 60-token-a-minute allowance serves calls while they fit its token budget and refuses the first one that does not

  • POST /key/generate with {"models":["claude-haiku-4-5"],"tpm_limit":60}
  • GET /key/info and confirm "tpm_limit": 60
  • Send short chat calls with max_tokens: 16 inside one minute, noting each response's usage.total_tokens and keeping a running total
  • When a call is refused, confirm the 429 has Limit type: tokens, the api_key scope, and the retry-after header, and that the Remaining count in its message is smaller than the refused call's reservation (message characters / 4 + 16); with the test's ~33-character prompt that reservation is 24 tokens
  • If instead the running total reaches 60 with no refusal yet, send one more call and confirm that one is refused with the same 429
  • POST /key/delete to clean up

• tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_rpm_limit_resets_after_window - a key that spent its 1-request allowance stays refused until the minute rolls over, then serves again and no sooner

  • POST /key/generate with {"models":["claude-haiku-4-5"],"rpm_limit":1}
  • Send 1 chat completion, note the time you sent it, and confirm 200
  • Send another immediately and confirm the 429 block
  • Keep retrying the same call every few seconds: only 429s while the window drains, then a 200 again around the 60s mark (the default rate-limit window)
  • Confirm the first 200 came no sooner than about 55s after the noted send time (60s window less a 5s tolerance); an earlier recovery means the limiter reset early
  • POST /key/delete to clean up

• tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_headers_report_limit_and_remaining - a key under its limits serves normally and each success tells the client its request and token allowances and how much of each is left

  • POST /key/generate with {"models":["claude-haiku-4-5"],"rpm_limit":5,"tpm_limit":100000}
  • Send 1 chat completion with curl -i so response headers are visible and confirm 200
  • Confirm x-ratelimit-api_key-limit-requests: 5 and x-ratelimit-api_key-remaining-requests: 4
  • Confirm x-ratelimit-api_key-limit-tokens: 100000 and that x-ratelimit-api_key-remaining-tokens is a number below 100000
  • POST /key/delete to clean up

• tests/e2e/management/test_management_e2e.py::TestKeyRoutes::test_generate_persists_to_key_info_and_scopes_chat (extended with rpm_limit) - a new key's saved settings read back exactly as configured and only its allowed model serves traffic

  • POST /key/generate with a unique key_alias, "models":["gemini-2.5-flash"], "tpm_limit":424242, and "rpm_limit":424243
  • GET /key/info and confirm key_alias, models, tpm_limit 424242, and rpm_limit 424243 all echo back
  • Send a chat completion with model gemini-2.5-flash and confirm 200
  • Send a chat completion with model gpt-5.5 and confirm a 403 whose body is the model-access denial
  • POST /key/delete to clean up

[greptile-apps]

Greptile Summary

This PR adds live e2e tests for key-level rpm/tpm rate limiting under tests/e2e/quota_management/ratelimit/, covering four previously untested registry cells: rpm/tpm block-on-breach, window reset timing, and pacing headers. Supporting changes add headers capture to StreamingResponse, a rpm_limit field to KeyInfo, and a CHEAP_ANTHROPIC_MODEL config constant.

• test_rate_limit_e2e.py: four tests asserting the 429 body contract (scope + limit type + retry-after header), exact token-accounting for tpm blocking, and the full 60 s window elapsed before recovery.
• e2e_http.py: StreamingResponse now captures all response headers (lowercased) in _streaming_outcome; the download function still constructs StreamingResponse directly and leaves headers empty — already flagged in a prior review comment.
• management/test_management_e2e.py: extends the key-persistence test to round-trip rpm_limit alongside tpm_limit.

Confidence Score: 5/5

All changes are confined to the tests/e2e directory; no production code is touched, so this is safe to merge.

The PR only adds and extends test infrastructure. The new tests follow the established suite lifecycle contract (generate key → assert /key/info → drive traffic → assert behaviour → cleanup), the timing arithmetic in the reset test is sound, and the tpm token-accounting logic correctly handles the pre-call reservation path. No issues that affect production behaviour were found.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py	New file: four live e2e tests for rpm/tpm rate-limit blocking, window reset, and pacing headers; logic is sound with well-handled edge cases.
tests/e2e/e2e_http.py	Adds headers field to StreamingResponse and populates it in _streaming_outcome; the download function still constructs StreamingResponse directly and leaves headers empty (pre-existing gap, already noted in prior review).
tests/e2e/quota_management/ratelimit/quota_client.py	New thin wrapper around Gateway exposing a chat helper that returns the raw StreamingResponse so tests can inspect status, body, and headers.
tests/e2e/quota_management/ratelimit/conftest.py	Provides session-scoped client fixture for the ratelimit suite; straightforward and consistent with other suite conftest files.
tests/e2e/models.py	Adds `rpm_limit: int
tests/e2e/management/test_management_e2e.py	Extends key persistence test to also configure and assert rpm_limit=424243 read-back alongside the existing tpm_limit check; non-breaking addition.
tests/e2e/e2e_config.py	Adds CHEAP_ANTHROPIC_MODEL constant backed by E2E_CHEAP_ANTHROPIC_MODEL env var; clean centralisation of the model name for this suite.
tests/e2e/CLAUDE.md	Documentation update: adds ratelimit/ to the quota_management suite description.

_{Reviews (7): Last reviewed commit: "test(e2e): source the ratelimit suite's ..." | Re-trigger Greptile}

[greptile-apps]

The remaining-requests assertion is hardcoded to "4". _first_ok retries on 401 (which never reaches the rate limiter), so exactly one request consumes budget — making rpm_limit - 1 the precise expected value. Writing it as a computed expression rather than a literal documents the arithmetic and avoids silent breakage if the rpm_limit value is ever adjusted above.

Suggested change

	assert first.headers.get("x-ratelimit-api_key-remaining-requests") == "4", (
	f"first call against rpm_limit=5 must leave 4 remaining, got "
	f"{first.headers.get('x-ratelimit-api_key-remaining-requests')!r}"
	)
	assert first.headers.get("x-ratelimit-api_key-remaining-requests") == str(5 - 1), (
	f"first call against rpm_limit=5 must leave {5 - 1} remaining, got "
	f"{first.headers.get('x-ratelimit-api_key-remaining-requests')!r}"
	)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_key_rate_limit_coverage (a21669a) with litellm_internal_staging (edd3bce)}

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[ishaan-berri]

this is going to be hard to maintain, can we not keep an ENV var for default cheap anthropic model

[mateo-berri]

fixed

[ishaan-berri]

the proof here looks AI generated, AI can lie. Can we take real screenshots so we confirm a human reviewed it

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

the proof here looks AI generated, AI can lie. Can we take real screenshots so we confirm a human reviewed it

I'm setting up some real proof

[mateo-berri]

@ishaan-berri I've attached real proof to the PR. I tested using the tip of #27001 <- which caused the regression (fails) and the tip of litellm_internal_staging (succeeds). Can you please re-review?

[tin-berri]

nit: can you just like replace TPM_LIMIT with like 60 or whatever so its easy to read/tell

[mateo-berri]

I'd rather keep it a global constant to avoid desyncing

[tin-berri]

can we also prioritize readability here instead of using 3-4 different helper functions

[mateo-berri]

I was thinking on it but I realize functions like _limited_key are repeated 4 separate times and also you can just use GitHub Pull Request extension to easily check out the code and cmd + click and ctrl + - around to view definitions of helpers