fix(spend): bound the logs-tab pagination count to stop full-window scans

[yassin-berriai]

Relevant issues

Linear ticket

Resolves LIT-4119

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

A customer reported RDS Aurora ACU spiking after an upgrade, with Performance Insights flagging SELECT COUNT(*) FROM (SELECT request_id FROM "LiteLLM_SpendLogs" WHERE ...) at ~472s average latency against a multi-TB SpendLogs table. That is the spend-logs UI list endpoint computing an exact pagination total over the whole selected time window on every load

Repro on a live proxy backed by Postgres seeded with 30,000 spend-log rows in a 24h window

EXPLAIN (ANALYZE, BUFFERS) on the count and page queries; the row-count actually scanned is the signal (execution time is small only because this is 30k local rows, not the customer's billions)

OLD (pre-fix) standalone exact COUNT over the whole 24h window
  Aggregate  (actual rows=1 loops=1)
    ->  Seq Scan on "LiteLLM_SpendLogs"  (actual rows=30000 loops=1)   <-- scans every matching row

OLD (LIT-4027) page query with COUNT(*) OVER ()
  Limit  (actual rows=50 loops=1)
    ->  WindowAgg  (actual rows=50 loops=1)
          ->  Index Scan Backward ...  (actual rows=30000 loops=1)     <-- still drains every matching row

NEW (fix) bounded count, probes at most cap+1 (10001) rows
  Aggregate  (actual rows=1 loops=1)
    ->  Limit  (actual rows=10001 loops=1)
          ->  Seq Scan on "LiteLLM_SpendLogs"  (actual rows=10001)     <-- stops early at cap+1

NEW (fix) page query without the window count
  Limit  (actual rows=50 loops=1)
    ->  Index Scan Backward ...  (actual rows=50 loops=1)              <-- reads only the page

Same behavior over HTTP through the proxy (GET /spend/logs/v2, 30,000 matching rows, page_size 50)

UNFIXED  {"total": 30000, "total_pages": 600, "total_is_capped": null}    # exact count = full-window scan
FIXED    {"total": 10000, "total_pages": 200, "total_is_capped": true}    # bounded scan, UI renders "10,000+"

When fewer than the cap match, the total stays exact and total_is_capped is false, so normal deployments see no change

UI proof (Logs tab, e2e)

Live proxy on localhost:4000 backed by Postgres seeded with 30,000 spend-log rows in the last-24h window, UI built from this branch. With more than the cap matching, the Logs tab renders Showing 1 - 50 of 10000+ results and Page 1 of 200+ (capped total plus the + indicator). Narrowing the filter to a 1-hour sub-window (1,152 rows, below the cap) shows an exact of 1152 results / Page 1 of 24 with no +, confirming the cap only engages above 10,000

Type

🐛 Bug Fix

Changes

The spend-logs UI list endpoint (/spend/logs/ui, /spend/logs/v2) built its total/total_pages from an exact count of every row in the selected window. On a large LiteLLM_SpendLogs table that scan is what drives the ACU spike. The earlier LIT-4027 change moved the count into COUNT(*) OVER (), but a window count still has to drain every matching row before the LIMIT applies, so the full-window scan stayed

This computes the total with a bounded SELECT COUNT(*) FROM (SELECT 1 FROM "LiteLLM_SpendLogs" WHERE ... LIMIT $cap+1) that probes at most cap+1 rows, and drops the window count from the page query so the page query is a plain indexed ORDER BY "startTime" ... LIMIT/OFFSET. When more than the cap match, the endpoint reports the cap and sets a new total_is_capped flag so the dashboard renders <cap>+ and stops paging past it. The bounded subquery terminates early instead of aggregating across every tablet, so it also stays safe on sharded engines like YugabyteDB (the case LIT-4027 was about)

The cap is 10,000. Deployments whose filtered window has fewer rows than that keep an exact total and unchanged pagination

Link to Devin session: https://app.devin.ai/sessions/66130e7189474b6e9b7346f96199a2a4

[yassin-berriai]

@greptileai

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes a full-table-scan on LiteLLM_SpendLogs that caused Aurora ACU spikes: the pagination count query now probes at most cap+1 (10,001) rows via SELECT COUNT(*) FROM (SELECT 1 … LIMIT $cap+1), and the page query is a plain LIMIT/OFFSET scan with no window function.

• Backend: ui_view_spend_logs runs two separate query_raw calls — a bounded count first, then the page — and returns total_is_capped: true when the result set exceeds 10,000 rows. Parameter indices are correctly maintained across both calls.
• Frontend: LogsTableToolbar renders "N+" and a tooltip when total_is_capped is set; PaginatedResponse gains an optional total_is_capped field to preserve backward compatibility.
• Tests: All mocks are updated to dispatch on COUNT(*) in the SQL string, and new tests cover the capped-total and out-of-range-page paths.

Confidence Score: 5/5

Safe to merge — the bounded count subquery correctly decouples the two query_raw calls and parameter indices are maintained without collision.

The parameter-index reuse between the count and page queries is correct, the capping logic is sound, and tests cover all relevant paths.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/spend_tracking/spend_management_endpoints.py	Replaces COUNT(*) OVER () window-function with a bounded SELECT COUNT(*) FROM (SELECT 1 ... LIMIT cap+1) subquery, decouples count from page query, and threads total_is_capped through to the response. Parameter indexing for count and page queries is correct.
tests/test_litellm/proxy/spend_tracking/test_spend_query_optimization.py	Tests rewritten to use side_effect for two-call query_raw dispatch (count then page), covering exact-total, capped-total, out-of-range-page, and empty-result paths. Old window-function assertions replaced with bounded-count assertions.
tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py	All inline mocks updated to dispatch on "COUNT(*)" in sql_query, return [{"total_count": min(total, cap_plus_one)}] for count calls, and strip the old total_count column from page rows. Coverage preserved for all sort/filter/RBAC cases.
ui/litellm-dashboard/src/components/view_logs/LogsTableToolbar.tsx	Renders "N+" for totals and page counts when total_is_capped is true, adds a hover tooltip explaining the cap, and falls back gracefully when the field is absent (optional chaining throughout).
ui/litellm-dashboard/src/components/view_logs/log_filter_logic.tsx	Adds total_is_capped?: boolean as an optional field on PaginatedResponse, keeping the interface backward-compatible with older API responses.

_{Reviews (3): Last reviewed commit: "test(spend): make zero-total mock reflec..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR replaces the COUNT(*) OVER () window-function approach (which forced a full-window scan on every paginated request) with a two-query pattern: a bounded SELECT COUNT(*) FROM (SELECT 1 ... LIMIT cap+1) subquery that stops early, followed by a plain indexed page query with no window function.

• Backend (spend_management_endpoints.py): introduces SPEND_LOGS_PAGINATION_COUNT_CAP = 10000, computes the total with the bounded count, caps it when exceeded, and surfaces total_is_capped in the response. The $p parameter indexing is correct across both queries; the fallback SpendLogsRepository.count() for out-of-range pages is removed since the bounded count now always runs first.
• Tests: two-query mocking pattern is cleanly implemented in both the shared mock helper and the individual test helpers; new tests for the capped-total scenario are added and existing tests are accurately updated rather than weakened.
• UI: LogsTableToolbar and PaginatedResponse updated to display total_is_capped as a trailing + on both the result count and the page count.

Confidence Score: 4/5

Safe to merge; the two-query approach is correctly parameterized, backward-compatible, and well-covered by tests. The only open items are minor UX polish on the cap-boundary UI affordance and a slightly fragile dispatch heuristic in the test mock.

The core optimization is sound: the bounded subquery count terminates early, the page query no longer drags a window aggregate, and the response contract is purely additive with total_is_capped. Parameter indexing was traced and is correct for both queries. The test mock dispatch works for current SQL shapes but could misbehave if a future filter condition embeds that literal string. The UI silently disables the Next button at the cap boundary without explaining why, which could confuse users on large deployments.

The test mock in test_spend_management_endpoints.py (line 168) uses a string-contains check to distinguish the count query from the page query — worth reviewing if WHERE-clause filtering is ever extended with LIKE patterns containing COUNT(*)

Important Files Changed

Filename	Overview
litellm/proxy/spend_tracking/spend_management_endpoints.py	Replaces the COUNT(*) OVER () window-function with a bounded two-query approach: a capped subquery count followed by a plain paginated SELECT. Adds SPEND_LOGS_PAGINATION_COUNT_CAP constant and total_is_capped response field. Logic and parameter indexing are correct.
tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py	Mock updated to dispatch on COUNT(*) in the SQL string and return bounded count vs. page rows accordingly. WHERE-clause regex updated to accept either ORDER BY or LIMIT as terminator. Changes accurately reflect the new two-query behavior.
tests/test_litellm/proxy/spend_tracking/test_spend_query_optimization.py	New tests added for bounded-count behavior and the capped total scenario; existing tests updated to remove window-count assertions. Coverage improved, no regressions masked.
ui/litellm-dashboard/src/components/view_logs/LogsTableToolbar.tsx	Appends + to both the total-results count and the page-count display when total_is_capped is true. Next-button navigation is already bounded by total_pages, so the capped indicator is purely informational.
ui/litellm-dashboard/src/components/view_logs/log_filter_logic.tsx	Adds optional total_is_capped boolean to PaginatedResponse interface. Additive, backward-compatible change.

Comments Outside Diff (1)

1. ui/litellm-dashboard/src/components/view_logs/LogsTableToolbar.tsx, line 215-218 (link)

   Next-page navigation silently stops at the cap boundary

   When total_is_capped is true the toolbar displays "Page X of 200+" but the Next button becomes disabled the moment currentPage === total_pages (200), giving no indication to the user that the cut-off is intentional rather than a UI glitch. Consider adding a tooltip or a small inline message (e.g. "Showing first 10,000 results — refine your filters to see more") when the cap is active so users understand why forward navigation is unavailable.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (2): Last reviewed commit: "fix(spend): bound the logs-tab paginatio..." | Re-trigger Greptile}

[greptile-apps]

Mock count dispatch is fragile on COUNT(*) substring match

The query-type branch relies on "COUNT(*)" in sql_query. Any future WHERE-clause value that happens to contain the literal string COUNT(*) (e.g. in a LIKE pattern) would cause the mock to handle a page query as a count query, returning [{"total_count": ...}] instead of a row list and silently breaking the tests. A more robust discriminator would be checking whether the outer SQL starts with SELECT COUNT(*) (case-insensitive) or checking for FROM ( in combination with COUNT(*), which is unique to the bounded-count subquery shape.

[yassin-berriai]

@greptileai

[devin-ai-integration]

Devin e2e test — both checks passed

Ran a live proxy on localhost:4000 (this branch, UI built from it) backed by Postgres seeded with 30,000 SpendLogs rows in the last-24h window. Full recording is embedded in the PR description.

• Logs tab with 30,000 rows (24h window): renders Showing 1 - 50 of 10000+ results and Page 1 of 200+ — capped total with the + indicator. passed
• Regression, 1-hour sub-window (1,152 rows, below cap): renders exact Showing 1 - 50 of 1152 results and Page 1 of 24, no +. passed

A broken/unfixed build would show of 30000 results / Page 1 of 600 with no +, so the capped 10000/200 + + only appear when the fix is working.

Capped (30,000 rows, 24h)	Below cap (1,152 rows, 1h)

Also confirmed over HTTP: GET /spend/logs/v2 -> {"total": 10000, "total_pages": 200, "total_is_capped": true} for 30,000 matching rows.

_{Tested by Devin — https://app.devin.ai/sessions/66130e7189474b6e9b7346f96199a2a4}