test(e2e): assert the long budget window keeps blocking after the short window resets

[ryan-crabbe-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

All proof captured at commit 95c915f against the live tests/e2e/docker-compose.yml stack (published main-latest image, throwaway postgres/redis, real provider calls costing real $). Local-only runtime bend, reverted before commit: this machine only has a Groq key, so the compose litellm_params for gpt-5.5 and claude-haiku-4-5 were pointed at groq/llama-3.3-70b-versatile for the runs (same recipe as PR #33632), and the jaeger host port was remapped to 16687 to dodge a locally running jaeger

The customer-visible walkthrough, exactly what the new key-side test asserts:

KEY=$(curl -s http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"models":["gpt-5.5"],"budget_limits":[{"budget_duration":"30s","max_budget":1e-9},{"budget_duration":"1d","max_budget":5e-7}]}' | jq -r .key)

# call 1 succeeds, call 2 blocks on the 30s window
curl -s http://localhost:4000/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.5","messages":[{"role":"user","content":"proof call"}],"max_tokens":16}'

# watch the 30s window's reset_at in /key/info, then call again once it advanced
curl -s "http://localhost:4000/key/info?key=$KEY" -H "Authorization: Bearer sk-1234" | jq '.info.budget_limits'

Output:

--- call 1:
It seems like you're looking for information on a specific topic, but your request
--- call 2:
{"error":{"message":"ExceededBudget: Key over 30s budget. Spend=$0.0000, Limit=$0.00","type":"budget_exceeded","param":null,"code":"429"}}
--- 30s window reset_at at block time: 2026-07-18T18:22:30+00:00
--- 30s window reset_at advanced to: 2026-07-18T18:23:00+00:00
--- call after 30s window reset:
{"error":{"message":"ExceededBudget: Key over 1d budget. Spend=$0.0000, Limit=$0.00","type":"budget_exceeded","param":null,"code":"429"}}
--- key deleted

The 30s window provably rolled (reset_at moved 18:22:30 to 18:23:00) and the very next refusal is attributed to the 1d window: the long cap enforces on its own schedule

Five consecutive runs of both files against the live stack, 60s cooldowns, zero failures (20/20 test passes):

=== RUN 1 17:51:50Z ===  4 passed in 340.17s
=== RUN 2 17:58:31Z ===  4 passed in 274.53s
=== RUN 3 18:04:07Z ===  4 passed in 300.00s
=== RUN 4 18:10:08Z ===  4 passed in 294.69s
=== RUN 5 18:16:04Z ===  4 passed in 274.54s

each run being

tests/e2e/quota_management/budgets/test_multi_window_budget_e2e.py::test_short_window_blocks_then_resets PASSED
tests/e2e/quota_management/budgets/test_multi_window_budget_e2e.py::test_long_window_blocks_after_short_window_resets PASSED
tests/e2e/quota_management/budgets/test_team_multi_window_budget_e2e.py::test_team_short_window_blocks_then_resets PASSED
tests/e2e/quota_management/budgets/test_team_multi_window_budget_e2e.py::test_team_long_window_blocks_after_short_window_resets PASSED

An intermediate commit hoisted the suite's duplicated drive_to_block and as_datetime helpers into budget_client; a full confirmation pass of every file that commit touched came back green (17 passed in 652.12s: both multi-window files, test_budget_reset_e2e.py's five-case diagonal, all six test_budget_reset_advances_e2e.py rungs, test_team_member_budget_reset_e2e.py, and test_user_budget_across_keys_e2e.py). That hoist was then reverted to keep this PR scoped to the multi-window feature, so the four unrelated budget files are restored to their staging state verbatim and the two multi-window tests carry their own inline drive-to-block loop again; the un-hoist changes no test's behavior and type-checks clean (basedpyright tests/e2e: 0 errors)

The branch was then rebased onto the ProxyClient rename (#33750), re-hashing the commits (proofs above were captured at pre-rebase hashes 95c915f and 1966dde); a post-rebase run of both multi-window files at f4f8af5 came back 4 passed in 315.09s. After merging latest staging and refining the readback (datetime-typed reset_at plus plain window readers, commit 0e793aa), one more live run of both files came back 4 passed in 312.01s; the compose stack for that run was restored from git history since #33837 removed tests/e2e/docker-compose.yml upstream. Note the current staging tip fails basedpyright tests/e2e on its own: the load suite from #33748 still imports the module #33750 renamed, so the e2e type gate shows ~35 errors in tests/e2e/load/ on any PR until that is fixed separately; this branch's own files type-check clean

Type

✅ Test

Changes

The multi-window budget e2e tests proved the tight window blocks and self-heals but never asserted the other direction: a long window whose cap the accumulated spend already crossed must keep refusing calls even inside a fresh short window. This PR adds exactly that direction, one test per existing file (key and team), and nothing else

Each new test mints its entity with budget_limits of a 30s window at 1e-9 and a 1d window at 5e-7 (both far below one real call's cost, so the first burn crosses both caps), drives spend to a 429 budget_exceeded, then records the 30s window's reset_at from /key/info / /team/info and waits for it to advance strictly past that value. The reset job advances a window's reset_at in the same pass that zeroes its spend counter, so once it moved the short window is provably clean. The test then polls until the refusal is attributed to the 1d window (ExceededBudget: ... over 1d budget), failing immediately if any call returns 200 (the long cap failed to hold; the failure message diagnoses the rare case where the 1d window itself rolled at midnight UTC mid-test) or if a non-budget error leaks. Recording reset_at after the block rather than at mint closes a false-pass race where a boundary rolls before the burn's spend lands; the final phase polls rather than single-shots because enforcement reads a cached auth object whose view of the rolled window lags the DB write (run 1 of QA caught this live: the block stayed attributed to the 30s window for a few seconds after reset_at advanced)

Harness support, modelled only to the fields read: BudgetWindowState (a BudgetWindow plus a pydantic-parsed datetime reset_at) in tests/e2e/models.py, budget_limits on KeyInfo, and two plain readers on BudgetClient (key_budget_windows / team_budget_windows) with a pure window_reset_at lookup, so the client exposes state rather than one test's access pattern and the tests compare real datetimes with no hand parsing

Each file's drive-to-block loop returns the blocking response and both long-window tests assert its status is 429 in the test body, which also pins the multi-window enforcement path's HTTP mapping (the enforcement suite's 429 assert covers only the single-budget path); the observed 429s in the proof runs above are the live evidence for that assert

This PR stays scoped to the multi-window feature and does not touch the rest of the suite: the two new tests keep their own inline drive-to-block loop rather than sharing one, so no unrelated budget test file is modified. Both Greptile P2 findings are still addressed: explicit non-None asserts on the long-window reset_at snapshots so the midnight-roll diagnostic cannot misreport when the window is absent from the info response, and the three /team/info readers inside budget_client share a private _team_info. Pulling the duplicated drive_to_block and as_datetime helpers into budget_client is left for its own PR

Both new tests intentionally share their file's existing registry cell (quota_management.budget.key_multi_window.blocks_then_resets and quota_management.budget.team_multi_window.blocks_then_resets): this direction is the unasserted half of those same cells, so no new denominator rows. The collector runs clean with --strict

One adversarial-review finding is accepted rather than fixed: a mutation that wipes every window's Redis counter (long included) while advancing only the expired window's reset_at would not fail these tests, because enforcement re-derives an authoritative floor for the long window from spend logs and blocks anyway. That redundancy is invisible at the customer boundary, and the customer promise (the long cap holds) is exactly what the tests assert; mutations that break the promise itself (long enforcement removed, reset job advancing every window's boundary) are caught

QA runbook

• tests/e2e/quota_management/budgets/test_multi_window_budget_e2e.py::test_long_window_blocks_after_short_window_resets - a key's 1d window keeps blocking after its 30s window provably reset
  • Mints a bare key whose budget_limits carry a 30s window at 1e-9 and a 1d window at 5e-7, both far below one real call's cost, so the first successful call's spend crosses both caps at once
  • Drives /chat/completions until a refusal lands, holding every pre-block response to either a success or a 429 budget_exceeded
  • Reads the 30s window's reset_at out of /key/info budget_limits only after the block (recording it at mint would race a boundary that rolls before the spend lands), also snapshotting the 1d window's reset_at for later diagnosis
  • Polls /key/info until the 30s reset_at is strictly later than the recorded value, which proves the reset job processed the window since it zeroes the window's spend counter in the same pass it advances reset_at
  • Polls one chat call at a time until the 429 body attributes the block to the 1d window ("over 1d budget"); interim blocks still naming the 30s window are tolerated because enforcement reads a cached auth object that lags the DB write
  • Fails immediately if any call in that final phase returns 200 (the 1d cap failed to hold; the message says whether the 1d window itself rolled mid-test) or if any refusal is not a budget block
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/quota_management/budgets/test_team_multi_window_budget_e2e.py::test_team_long_window_blocks_after_short_window_resets - a team's 1d window keeps blocking a team key after the team's 30s window provably reset
  • Creates a team with the same 30s/1e-9 plus 1d/5e-7 window pair and mints a plain team key on it (team_id only, no key-level budget), so the team windows are the only caps in play
  • Drives the key to a 429 budget_exceeded the same way as the key test
  • Reads the 30s window's reset_at from team_info.budget_limits in /team/info after the block and polls until it strictly advances
  • Polls calls until the refusal names the team's 1d window ("ExceededBudget: Team=<team_id> over 1d budget"), with the same immediate failures on any 200 or non-budget error
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Environment prerequisites: the tests/e2e docker-compose stack with provider keys for gpt-5.5 (OpenAI) and claude-haiku-4-5 (Anthropic) in tests/e2e/.env, or the Groq swap described in the proof section. The compose stack already sets proxy_budget_rescheduler_min_time/max_time to 5/10s; without that, the reset wait exceeds the tests' deadlines

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

Adds the "long-window still blocks after short-window resets" direction to the multi-window budget e2e suite, one new test per entity type (key and team). Both previous review findings are addressed: blocked_long_reset_at is now explicitly asserted non-None before comparison, and the three /team/info HTTP calls are unified behind a private _team_info helper.

• New BudgetWindowState model carries a pydantic-parsed datetime reset_at, enabling direct > comparisons with no hand-parsing; TeamInfoRow and a window_reset_at pure helper expose the same state for teams.
• Each new test mints an entity with both a 30s/1e-9 and a 1d/5e-7 window, drives spend to a 429, waits for the short window's reset_at to advance (proving the reset job ran), then polls until the refusal is attributed to the 1d window — failing immediately on any 200 or non-budget error.
• The existing _drive_to_block loop is extracted into a shared local helper in each file and returns the blocking response so the new tests can pin the HTTP status code to 429.

Confidence Score: 5/5

Safe to merge — this PR is additive e2e tests only, with no production code changes.

The change is entirely test-only: new pydantic models, a refactored client helper, and two new live e2e test cases. The refactor of three /team/info calls into _team_info is behavior-preserving, the new models add no required fields, and all previous review findings are resolved. No issues were found.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/models.py	Adds BudgetWindowState (extends BudgetWindow with a pydantic-typed `datetime
tests/e2e/quota_management/budgets/budget_client.py	Adds TeamInfoRow + team_info field on TeamInfoResponse, the pure window_reset_at helper, key_budget_windows/team_budget_windows methods, and consolidates three duplicate /team/info HTTP calls behind a private _team_info method; refactor is behavior-preserving and well-structured.
tests/e2e/quota_management/budgets/test_multi_window_budget_e2e.py	Extracts _drive_to_block into a local helper, adds test_long_window_blocks_after_short_window_resets asserting the 1d cap holds after the 30s window provably rolls; both previous findings (non-None assert on long-window reset_at, 429 status pin) are addressed.
tests/e2e/quota_management/budgets/test_team_multi_window_budget_e2e.py	Team-side analog of the key test: extracts _drive_to_block, adds test_team_long_window_blocks_after_short_window_resets with the same polling-reset pattern and identical non-None diagnostic guards; no issues found.

_{Reviews (6): Last reviewed commit: "docs(e2e): number the long-window team t..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_multi_window_long_cap (e0b1728) with litellm_internal_staging (366ec6f)}

[ryan-crabbe-berri]

@greptileai re review

[ryan-crabbe-berri]

@greptileai re review

[ryan-crabbe-berri]

@greptileai re review

[ryan-crabbe-berri]

@greptileai re review

[ryan-crabbe-berri]

@greptileai re review