feat(sgr): make the gateway middleware the source of truth for successful requests

[yassin-berriai]

TLDR

Problem this solves:

• SGR (successful gateway requests) has two definitions that disagree, and the dashboard reads the weaker one
• The Usage tiles derive it from SpendLogs, so they count what litellm's logging callbacks observed and could attribute, not what the gateway answered
• That count goes to zero when spend logging is off or the database is unavailable
• The request-metrics middleware already counts the real thing at the ASGI edge, but only exported it to OTLP for enterprise metering

How it solves it:

• New LiteLLM_DailyGatewayRequests table written by the middleware
• The Successful Requests and Failed Requests tiles read that table
• New by-endpoint breakdown chart the SpendLogs path cannot produce
• The old SpendLogs path is left running, marked with TODOs

Relevant issues

Linear ticket

Resolves LIT-5147

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on real Postgres against the real OpenAI API, with real spend. The proxy logged Enterprise billing metrics disabled, so this is the unlicensed path, which is the case the middleware's old early return would have switched the counter off in

Both runs used the same config with disable_spend_updates: true, which is the case the ticket is about: a deployment that turned spend tracking off still wants to know how many requests it served. Both sent the same 14 requests, 7 chat completions and 4 embeddings that succeed and 3 for a model that does not exist. The only difference is the branch

$ for i in 1 2 3 4 5 6 7; do curl -s -o /dev/null -w "chat #$i -> %{http_code}\n" $P/v1/chat/completions -H "$K" -H "$J" \
    -d "{\"model\":\"gpt-5.6\",\"messages\":[{\"role\":\"user\",\"content\":\"say ok $i\"}]}"; done
chat #1 -> 200 ... chat #7 -> 200

$ for i in 1 2 3 4; do curl -s -o /dev/null -w "embedding #$i -> %{http_code}\n" $P/v1/embeddings -H "$K" -H "$J" \
    -d "{\"model\":\"text-embedding-3-small\",\"input\":\"gateway counting sample $i\"}"; done
embedding #1 -> 200 ... embedding #4 -> 200

$ for i in 1 2 3; do curl -s -o /dev/null -w "failed chat #$i -> %{http_code}\n" $P/v1/chat/completions -H "$K" -H "$J" \
    -d '{"model":"model-that-does-not-exist","messages":[{"role":"user","content":"hi"}]}'; done
failed chat #1 -> 400
failed chat #2 -> 400
failed chat #3 -> 400

On litellm_internal_staging, 14 answered requests leave nothing behind and the dashboard reports zero:

$ psql -c 'SELECT (SELECT count(*) FROM "LiteLLM_SpendLogs") AS spend_logs,
                  (SELECT count(*) FROM "LiteLLM_DailyUserSpend") AS daily_user_spend;'
 spend_logs | daily_user_spend
------------+------------------
          0 |                0
$ curl -s $P/gateway/daily/activity -H "$K" -o /dev/null -w '%{http_code}\n'
404

On this branch the spend tables are still empty, and the counts are there anyway:

$ psql -c 'SELECT (SELECT count(*) FROM "LiteLLM_SpendLogs") AS spend_logs,
                  (SELECT count(*) FROM "LiteLLM_DailyUserSpend") AS daily_user_spend;'
 spend_logs | daily_user_spend
------------+------------------
          0 |                0

$ psql -c 'SELECT date, category, route, successful_requests, failed_requests
           FROM "LiteLLM_DailyGatewayRequests" ORDER BY route;'
    date    | category |       route       | successful_requests | failed_requests
------------+----------+-------------------+---------------------+-----------------
 2026-08-05 | llm      | /chat/completions |                   7 |               3
 2026-08-05 | llm      | /embeddings       |                   4 |               0

$ curl -s $P/gateway/daily/activity -H "$K"
{
    "total_successful_requests": 11,
    "total_failed_requests": 3,
    "by_date": [{"date": "2026-08-05", "successful_requests": 11, "failed_requests": 3}],
    "by_route": [
        {"category": "llm", "route": "/chat/completions", "successful_requests": 7, "failed_requests": 3},
        {"category": "llm", "route": "/embeddings", "successful_requests": 4, "failed_requests": 0}
    ]
}

The new chart, which the spend path cannot produce at all when spend logging is off:

Full pages for context, base and this branch

A key without an admin role is refused. Two independent gates cover this; the route-level management-route check answers first, which is why the status is 401 rather than the handler's own 403:

$ NEWKEY=$(curl -s $P/key/generate -H "$K" -d '{"models":["gpt-5.6"]}' | jq -r .key)
$ curl -s -w '%{http_code}\n' $P/gateway/daily/activity -H "Authorization: Bearer $NEWKEY"
{"error":{"message":"Authentication Error, Only proxy admin can be used to generate, delete, update info
 for new keys/users/teams. Route=/gateway/daily/activity. Your role=unknown. ...","code":"401"}}
401

The table as the migration file creates it, piped into an empty database rather than pushed from schema.prisma:

$ psql -d litellm_mig -f litellm-proxy-extras/.../20260803000000_add_daily_gateway_requests/migration.sql
CREATE TABLE
CREATE INDEX
$ psql -d litellm_mig -c '\d "LiteLLM_DailyGatewayRequests"'
 date                | text   | not null
 category            | text   | not null
 route               | text   | not null
 successful_requests | bigint | not null | 0
 failed_requests     | bigint | not null | 0
Indexes:
    "LiteLLM_DailyGatewayRequests_pkey" PRIMARY KEY, btree (date, category, route)
    "LiteLLM_DailyGatewayRequests_date_idx" btree (date)

Type

🆕 New Feature

Changes

SGR has two independent definitions. The admin UI derives it from SpendLogs, so it counts what litellm's logging callbacks observed and could attribute and price. BillableRequestMetricsMiddleware counts what the proxy actually answered at the ASGI edge, but only exported to OTLP for enterprise metering. This makes the middleware the source of truth for the dashboard

Table. LiteLLM_DailyGatewayRequests, keyed (date, category, route) with successful_requests and failed_requests. Added to all three schema.prisma copies plus a migration following the add_daily_tool_spend precedent

Every part of that key is chosen by the proxy and drawn from a closed set: the date is the server clock, the category is a three-value enum, and the route is one of a fixed list the classifier maps paths onto rather than the raw path. So the table's size is a function of how long the deployment has been running and how many route classes it serves, never of traffic. There is deliberately no deployment dimension, for two reasons. The read endpoint groups by (date, category, route) and never selects it, so it would be write-only. And the deployment id available at the ASGI edge, the x-litellm-model-id header, is a sha256 over the deployment's litellm_params including credentials, so a caller who puts a credential in the request body mints a fresh one per distinct value; api_base and base_url need allow_client_side_credentials, but api_key is not on _BANNED_REQUEST_BODY_PARAMS and needs no configuration at all. Keying on it would let a caller grow a persisted table

Write path. Requests fold into an in-memory map at record time rather than going through a queue like the spend path. A count is a pure aggregate and the fold is bounded as above, whereas the spend queue caps at 1000 items and blocks when full, which is not acceptable in the response path. A scheduler job drains it on the existing batch interval, with a final flush on shutdown that sits inside the same if prisma_client: guard as disconnect() and immediately before it; a write attempted after disconnect raises ClientNotConnectedError and loses the interval silently, so a test pins the call order. A failed flush merges its counts back into the fold and retries on the next tick, mirroring the neighbouring tool-spend writer, so a database blip undercounts nothing. That buys at-least-once rather than exactly-once and the docstring says so: the batch commits inside its context manager's __aexit__, so a failure raised after the transaction committed restores counts that are already persisted and the next flush increments them again. For a traffic-volume metric a rare overcount on a dropped acknowledgement beats losing a whole interval to every blip

Many workers and many pods. The accumulator is per-process state, so the flush job is deliberately not behind the PodLockManager that gates the key-rotation and spend-reset jobs: every worker and every pod runs its own scheduler and drains its own fold, where a lock would leave every non-holder's counts stranded in memory. Correctness across processes comes from the write rather than from coordination. Each row goes out as one statement, INSERT ... ON CONFLICT ("date","category","route") DO UPDATE SET "successful_requests" = ("successful_requests" + $n), so the arithmetic happens server side under the row lock rather than as a read followed by a write, and any number of processes summing into one row is correct however they interleave. The same statement covers two processes racing to create a row that does not exist yet, since the conflict resolves into the increment instead of a unique violation. Each flush also sorts its rows by (date, category, route) so concurrent writers take row locks in the same order, which a test pins

Verified on a live proxy at --num_workers 4, four distinct worker processes: 24 concurrent requests produced exactly 24 in one row, then 8 more followed immediately by SIGTERM well inside the flush interval produced 32, so each worker's shutdown drain lands. The cost of holding counts in memory is bounded by that interval: a graceful shutdown drains it, while an ungraceful kill (OOM, or a scale-down that outruns terminationGracePeriodSeconds) loses at most one interval for that one process

Middleware. It classifies once and fans out to two sinks, and returns early only when both are absent, which is what lets the SGR sink run on an unlicensed deployment. The billing recorder keeps its 2xx-only gate and its model id, unchanged. The SGR sink takes every status so failed_requests is real, and is not handed the model id at all: the recorder exports to a metering backend that can drop a series, while the sink writes a row that stays

Read and UI. GET /gateway/daily/activity is restricted to proxy admin roles, because the table carries no key, user or team dimension and the totals are deployment-wide. The Successful Requests and Failed Requests tiles read it, with a tooltip saying they will not match the per-key and per-model breakdowns below, and there is a new stacked Gateway Requests by Endpoint chart. Both tiles read the same source deliberately, or the tile row contradicts the chart directly beneath it. The per-key and per-model charts keep reading the daily spend tables. schema.d.ts regenerated via npm run gen:api

Range staleness. Every source the request tiles can read is stamped with the range it answers and selected during render: the gateway counts, the spend aggregate, and the failure flag that gates the paginated pages behind it. Clearing state inside a fetch effect is one render too late, since an effect runs after the render that follows a range change and that render still holds the previous range's numbers. All three need it because the tiles read the first and fall through to the second and then the third, so a stamp on any one of them is escaped by the next. The aggregate's key also names the selected user, since the same dates describe different numbers for different users. The rule, the range key and the chart's route folding live in gatewayActivity.ts with unit tests, following this directory's convention of testing logic directly rather than through a render

Two deliberate choices worth calling out. The DB write is not license-gated even though the OTLP export is, otherwise the dashboard would blank on OSS. And a non-admin keeps seeing the old per-user numbers, since the new table cannot answer "my requests"

One inbound request is counted once however many upstream calls litellm made to serve it, so router retries, fallbacks and internal fan-out do not inflate the total

Docs. BerriAI/litellm-docs#768 documents the new table, the endpoint and why the gateway counts deliberately do not tie out against the spend rollups. Merge this first

Mutation results

Fifteen backend mutants, all killed:

Mutation	Result
sink records only on 2xx	4 failed
middleware returns early when only the billing recorder is absent	3 failed
billing recorder records every status	6 failed
the serving deployment added to the key, so a caller can mint rows	1 failed
success boundary widened past 2xx	1 failed
deterministic commit ordering removed	1 failed
drain stops resetting the fold	3 failed
failed flush drops counts instead of restoring	2 failed
shutdown drain moved after prisma_client.disconnect()	1 failed
admin-only gate also accepts view-only	1 failed
default lookback shortened from 30 days	1 failed
by_route ordered ascending instead of busiest first	2 failed
by_route folds every category together	1 failed
caller-supplied dates ignored	1 failed
the db-not-connected guard removed	1 failed

Seven frontend mutants, all killed. The first two drive the page rather than the helper, because the defect they pin is in the wiring:

Mutation	Result
the spend fallback read without checking its range stamp	1 failed
the failure flag read without checking its range stamp, so the paginated pages leak	1 failed
drop the range-key guard, so a stale range renders	1 failed
drop the admin guard	1 failed
widen the top-N route cap	1 failed
stop prefixing non-llm routes with their category	1 failed
range key ignores the end date	2 failed

157 backend tests across the three feature test files and 48 frontend tests pass unmutated, plus the shutdown-ordering tests in test_lifecycle.py, 189 backend in total. The full dashboard suite is 6169 passing across 585 files. gateway_request_endpoints.py, gateway_request_tracking.py, billable_request_metrics_middleware.py and types/proxy/gateway_requests.py are at 100% line coverage

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR makes middleware-recorded gateway responses the source of truth for deployment-wide successful and failed request metrics.

• Adds a bounded daily gateway request accumulator, persistence table, migration, and admin-only activity endpoint
• Updates the usage dashboard with range-stamped data selection, gateway-backed request tiles, and an endpoint breakdown chart
• Adds backend, lifecycle, middleware, endpoint, and dashboard regression coverage

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains, and the previously reported stale gateway, aggregate, and paginated fallback paths are now withheld during date or user transitions

Important Files Changed

Filename	Overview
litellm/proxy/db/gateway_request_tracking.py	Adds bounded in-memory request aggregation, retry restoration, and batched Prisma upserts without a confirmed correctness issue
litellm/proxy/middleware/billable_request_metrics_middleware.py	Fans classified responses out to gateway and billing sinks while preserving the billing sink's successful-response behavior
litellm/proxy/management_endpoints/gateway_request_endpoints.py	Adds an admin-restricted deployment-wide request-count endpoint with typed query validation and date and route aggregation
litellm/proxy/proxy_server.py	Wires request accumulation into middleware, scheduled persistence, and shutdown ordering
litellm-proxy-extras/litellm_proxy_extras/migrations/20260803000000_add_daily_gateway_requests/migration.sql	Creates the daily gateway request table with the intended bounded composite primary key and date index
ui/litellm-dashboard/src/app/(dashboard)/usage/_components/components/UsagePageView.tsx	Uses render-time range stamps for all fallback sources and displays deployment-wide gateway request totals and endpoint activity for administrators
ui/litellm-dashboard/src/app/(dashboard)/usage/_components/components/gatewayActivity.ts	Centralizes range-key selection and endpoint chart shaping with no remaining stale-range failure identified
ui/litellm-dashboard/src/app/(dashboard)/usage/_components/components/UsagePageView.test.tsx	Adds component-level regression coverage for stale aggregate and paginated fallback behavior during range changes

_{Reviews (11): Last reviewed commit: "feat(sgr): make the gateway middleware t..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_sgr_gateway_request_table (5e5844a) with litellm_internal_staging (2792887)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (d3d3035) during the generation of this report, so 2792887 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yassin-berriai]

@greptileai please review the current head b4c3abe112. Since your 4/5 the branch was rebased onto current staging and the UI finding is fixed: a fetched gateway result now carries the date range it was fetched for, and the render selects it only when that stamp matches the range currently picked, so a previous range's counts are unrepresentable rather than merely unlikely. That rule, the range key and the chart's route folding moved into gatewayActivity.ts with unit tests. The Failed Requests tile also now reads the gateway counter alongside Successful Requests; pointing only one of them at it made the tile row contradict the chart directly below, which the live capture caught.

[yassin-berriai]

@greptileai please review the current head 02b895e56b. The only change since the 5/5 on b4c3abe112 is one added test file: gateway_request_endpoints.py had no mapped test and sat at 57% line coverage, which codecov flagged correctly. It is at 100% now, with six mutants killed (admin gate widened to view-only, shortened default lookback, by_route ordered ascending, by_route folding categories together, caller dates ignored, db-not-connected guard removed). Production code is byte-identical to the commit you scored.

[yassin-berriai]

@greptileai please review the current head 0bf4166f2a. Production code is unchanged from the 02b895e56b you just scored 5/5; the only delta is the test file, which had a flake I introduced. test_omitted_dates_fall_back_to_the_default_window recomputed the expected window in the assertion after the request had already computed its own, so a run crossing UTC midnight between the two evaluations would fail on a one-day-off mismatch. Both date assertions now pin the clock, against a date in the past so they cannot pass on a day the fixture silently failed to patch; neutering the fixture fails them. The file also now pins throwaway DATABASE_URL and LITELLM_MASTER_KEY before importing proxy_server and restores the prior values, matching test_component_allowlists.py, so it does not depend on a tier-zero runner having them set.

[yassin-berriai]

@greptileai please review the current head 42f98c35c2. The only delta from the 02b895e56b you scored 5/5 is one test file plus a docstring; the executable code is identical (parsing both with docstrings stripped gives the same AST). The docstring names a semantic the module previously left implicit: because the batch commits in its context manager's __aexit__, a failure raised after the transaction committed restores counts that are already persisted and the next flush increments them again, so the retry is at-least-once rather than exactly-once. Exactly-once would need a dedup key the upserts could ignore on replay; for a traffic-volume metric a rare overcount beats losing an interval to every database blip, so the trade is deliberate and is now written down instead of only the guarantee it buys.

[yucheng-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 42f98c3. Configure here.}

[cursor]

Shutdown flush after DB disconnect

High Severity

During proxy shutdown, flush_gateway_requests is called after prisma_client has already disconnected. This prevents the final write of accumulated Gateway Request (SGR) counts to the database, resulting in the loss of up to one batch interval of SGR data on every proxy restart or deploy.

 

^{Reviewed by Cursor Bugbot for commit 42f98c3. Configure here.}

[yucheng-berri]

@yassin-berriai

[yassin-berriai]

Confirmed and fixed in ce7e8442be. This was real: proxy_shutdown_event disconnected prisma at the top and the SGR drain ran further down, so the final flush always raised ClientNotConnectedError. I verified that rather than assuming it, by driving a batch upsert on a disconnected client: ClientNotConnectedError: Client is not connected to the query engine. Because flush_gateway_requests catches Exception and merges the counts back onto an accumulator the process is about to discard, it failed silently, so up to one batch interval was lost on every restart exactly as you described.

The drain now runs inside the same if prisma_client: guard, immediately before disconnect(), so the two cannot drift apart again. Added a regression test in tests/test_litellm/proxy/proxy_server/test_lifecycle.py that asserts the call order rather than merely that both ran, plus one covering the no-database case; restoring the original ordering fails exactly that first test and nothing else.

Worth noting this was missed by a Greptile 5/5 and by my live-proxy capture, which never exercised shutdown. Good catch.

[yassin-berriai]

@greptileai please review the current head ce7e8442be. Cursor Bugbot found a real High-severity bug that this pass and the previous 5/5 both missed, so it is worth re-reviewing rather than carrying the score forward: proxy_shutdown_event disconnected prisma at the top of the function while the SGR drain ran further down, so the shutdown flush always raised ClientNotConnectedError, was swallowed by the flush's own except Exception, and lost up to one batch interval on every restart. I confirmed the failure mode by driving a batch upsert on a disconnected client rather than reasoning about it. The drain now sits inside the same if prisma_client: guard immediately before disconnect(), with a regression test asserting the call ORDER (restoring the old ordering fails it) and a second covering the no-database case.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[yassin-berriai]

@greptileai please review the current head 68a77a06be. No code change from the ce7e8442be you scored 5/5; this is a rebase onto current staging to clear a ui-unit-tests failure that was base drift rather than a defect. Staging added a "template presets" suite to add_auto_router_tab.test.tsx after my branch point, and since CI evaluates the merge ref those new tests ran against my tree. The mechanism was that both staging and this branch had edited the generated ui/litellm-dashboard/src/lib/http/schema.d.ts, so git merged the two textually without reporting a conflict and produced a file that was neither side's. Rebasing applies my hunks on top of staging's version instead; regenerating with npm run gen:api afterwards produces a byte-identical file, which confirms the committed one is now correct. That suite is 26/26 locally and the full changed-file run is 3550 passing.

[yassin-berriai]

Note on frontend-lint, which is red on this PR for a reason that is not in this diff.

The committed ceilings on staging are complexity: 121 and local/no-large-inline-object-arg: 469, but staging's own counts are already 122 and 471, so the gate fails for any PR that touches a frontend file. I measured this rather than inferred it: running npx eslint . -f json on a clean origin/litellm_internal_staging worktree gives 122 and 471, and this branch now gives exactly 122 and 471 too, so this PR's delta on both rules is zero. #35956 is failing on the identical two lines, and #35960 (chore(ui): sync dashboard eslint budgets after crossed staging merges) is the fix; its merge tree reads max: 122 and max: 471 with zero headroom.

This branch did add one no-large-inline-object-arg violation of its own, an inline 10-property session object in UsagePageView.test.tsx. That is fixed in 0e0b2ff7aa by hoisting it to a named nonAdminSession fixture, which reads better anyway. I have deliberately not touched eslint-budgets.json; raising a ceiling to absorb someone else's drift does not belong in a feature PR, so frontend-lint here should clear once #35960 merges.

[yassin-berriai]

@greptileai please review the current head 97d24af8da. Since your 5/5 the branch was rebased onto current staging and the one open veria finding is fixed, so the delta is worth a fresh pass rather than carrying the score forward.

LiteLLM_DailyGatewayRequests is now keyed (date, category, route). It previously also carried the serving deployment, taken from the x-litellm-model-id response header, and veria was right that a caller can vary it. I measured that rather than reasoning about it: the deployment id is a hash over the deployment's litellm_params, so with general_settings.allow_client_side_credentials: true three requests to /v1/chat/completions differing only in a body-supplied api_base came back with three distinct ids and wrote three separate rows on the previous head, against one row now. Without that opt-in the proxy rejects the request outright ("api_base is not allowed in request body"), so the finding needed an admin setting to be reachable, which is a supported configuration rather than a misconfiguration.

Dropping the column rather than pinning it to "" was the smaller change: the read endpoint's GROUP BY never included it, so the dimension was written and never read, and removing it also retires the "" sentinel and the "a composite primary key cannot hold a NULL" comment that existed only to support it. The sink Protocol no longer takes a model id at all; the billing recorder still does, since it exports to a metering backend rather than writing a row that stays.

The regression test drives three requests through the real ASGI middleware into a real accumulator with a different x-litellm-model-id on each response and asserts one key. Restoring model_id on GatewayRequestKey, on GatewayRequestAccumulator.record and on the middleware's call into the sink fails exactly that test. The live before/after and the migration's \d output are in the PR body.

[yassin-berriai]

@greptileai please review the current head 7e41b24710. This picks up your 4/5 as well; the previous comment covered the veria cardinality finding only, and the tile fallback is fixed here.

You were right about the mechanism. setAggregatedData(null) sits inside the fetch effect, and an effect runs after the render that follows a date change, so that render still holds the previous range's aggregate and can paint it. Withholding a superseded gateway result while the fallback stayed stale meant the tile showed the previous range's number by the other route, which is the guarantee the range key exists to provide.

The aggregate now carries its own range key and is selected during render through the same helper as the gateway counts, so both go through one rule rather than one rule and one effect. Its key also carries the selected user, since the same dates describe different numbers for different users and the effect's clear was the only thing covering that. The redundant setAggregatedData(null) is gone with it, which removes a render pass per range change. Total Requests and Average Cost per Request read the same aggregate, so they stop showing the previous range too.

The regression test drives the page rather than the helper, because the defect is in the wiring: it resolves one range, changes the date with the next fetch held open, and asserts the first range's total is off screen before releasing it. Reading the aggregate without checking its stamp fails it. The helper's own rule keeps its unit tests, including the new one covering two users over identical dates.

[yassin-berriai]

@greptileai please review the current head c576b7ef05. Your 4/5 on 7e41b24710 was right and this fixes it.

aggregatedFailed was a bare boolean, so on the render following a range change it still held the previous range's failure and the paginated pages behind it were read. It is now a stamped value like the two data sources, selected through the same helper. That keeps the paginated hook disabled while a new range is in flight, and disabled is what empties it, so its previous rows cannot reach a tile.

usePaginatedDailyActivity.ts is unchanged on purpose. It already resets to EMPTY_DATA whenever enabled goes false and it already discards stale in-flight pages by fetch id, so gating enabled on the stamped failure is sufficient; stamping inside the hook as well would be a second mechanism for the same rule. If you think there is a path through it that survives enabled: false, name it and I will drive it rather than argue.

That is three sources now behind one rule, which was the point of your finding: the tiles read the gateway counts, fall through to the aggregate, and fall through again to the paginated pages, so a stamp on any one of them is escaped by the next. The new test drives the page rather than the helper, since the defect lives in the wiring: it fails the first range's aggregated call so the paginated pages render, changes the range with the second aggregated call held open, and asserts the first range's total is off screen before releasing it. Reverting the flag to aggregatedFailure !== null fails exactly that test.

[yassin-berriai]

Correcting myself on the cardinality finding, since I put a wrong claim on this PR earlier today.

I wrote that it "needed an admin setting to be reachable", based on a live probe where a body-level api_base came back 401 with "api_base is not allowed in request body". That generalised from one parameter to the family, and it is wrong. api_base and base_url are on _BANNED_REQUEST_BODY_PARAMS in litellm/proxy/auth/auth_utils.py; api_key is not, so it is never banned and the opt-in never enters the picture for it. Both names then take the identical branch at route_llm_request.py:439 into _handle_clientside_credential, so the id minting is the same either way.

Re-measured on a live proxy with no clientside-credentials setting of any kind, using an ordinary non-admin virtual key, one model configured:

  no api_key in the body            -> 200, x-litellm-model-id: 7bf6346cff2f4b26...
  "api_key": "<a working key>"      -> 200, x-litellm-model-id: 277940395e89bacb...
  "api_key": "<the same key> "      -> 200, x-litellm-model-id: 92f333adb37cc575...

The third is that same working credential with one trailing space. OpenAI accepts it and the sha256 preimage does not, so one valid key is enough to mint unbounded ids; a caller does not need several. On the previous head each of those is its own primary-key row per day. On this head all eight requests of that run, three distinct ids across the successes, sit in a single row.

Two bounds so this is not overstated. The credential has to actually work, because a provider rejection carries no x-litellm-model-id and collapses onto the empty id. And the matching in-memory growth, one upsert_deployment router entry per distinct credential, is pre-existing and out of scope here; what this PR would have added is the durable copy in a table.

The fix does not change, and the PR body now carries the corrected version with the retraction in place rather than a quiet edit.

[yassin-berriai]

@greptileai please review the current head 5e5844abb0. The tree is byte-identical to the c576b7ef05 you scored 5/5 (git diff c576b7ef05 5e5844abb0 is empty); the only delta is the commit message, which still implied the cardinality exposure required an admin opt-in. It does not, and the message now says so, matching the correction already in the PR body. Re-anchoring rather than leaving your score pinned to a commit that is no longer in the PR.

[yassin-berriai]

Flagging a stale approval before anyone acts on the "approved but unmerged" bump.

@yucheng-berri approved this at 68a77a06be. That commit is no longer in the PR, and what has landed since is not cosmetic:

• The table's primary key changed. LiteLLM_DailyGatewayRequests was keyed (date, category, route, model_id) and is now keyed (date, category, route), with the column and its migration line removed. That came out of a veria finding that the model_id dimension was caller-influenceable; a caller putting a credential in the request body mints a fresh deployment id per distinct value, with no configuration required, and the read endpoint never selected the column anyway. Measured live: three requests to one route wrote three rows before and one row now
• Three dashboard fixes for a stale-range defect Greptile raised over two rounds. The request tiles read the gateway counts, fall through to the spend aggregate, and fall through again to the paginated pages; only the first was range-stamped, so a superseded range could still reach a tile through either fallback. All three now go through one rule

Greptile is 5/5 on the current tree and CI is re-running after a commit-message correction, but no human has looked at the schema change. Please re-review rather than merging on the earlier approval.