Skip to content

feat: add model_group filter to /spend/logs/v2 endpoint - #24782

Closed
silencedoctor wants to merge 57 commits into
BerriAI:litellm_oss_branchfrom
silencedoctor:litellm_feat-model-group-filter-spend-logs
Closed

feat: add model_group filter to /spend/logs/v2 endpoint#24782
silencedoctor wants to merge 57 commits into
BerriAI:litellm_oss_branchfrom
silencedoctor:litellm_feat-model-group-filter-spend-logs

Conversation

@silencedoctor

Copy link
Copy Markdown
Contributor

Relevant issues

Fixes #24781

Pre-Submission checklist

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Type

🆕 New Feature

Changes

Add an optional model_group query parameter to the /spend/logs/v2 and /spend/logs/ui endpoints, enabling users to filter spend logs by model group (the public-facing model name used by the Router for load-balancing).

What changed

litellm/proxy/spend_tracking/spend_management_endpoints.py

  • Added model_group as an optional query parameter (consistent with existing model and model_id params)
  • Added Prisma where_conditions for model_group filtering
  • Added model_group to the raw SQL equality filter list

tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py

  • Added test_ui_view_spend_logs_with_model_group following the same pattern as the existing test_ui_view_spend_logs_with_model and test_ui_view_spend_logs_with_model_id tests

Why

The model_group column already exists in the LiteLLM_SpendLogs table and is returned in responses, but there was no way to filter by it. This is needed for downstream consumers that want to analyze usage at the model-group level (e.g., all deployments of "gpt-4") rather than individual deployment models.

Test plan

  • test_ui_view_spend_logs_with_model_group — verifies that passing model_group=gpt-4 returns only logs with that model group
  • All 51 existing tests in test_spend_management_endpoints.py continue to pass
  • Adjacent test_ui_view_spend_logs_with_model and test_ui_view_spend_logs_with_model_id tests unaffected

@vercel

vercel Bot commented Mar 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
litellm Ready Ready Preview, Comment Apr 15, 2026 7:55am

Request Review

@CLAassistant

CLAassistant commented Mar 30, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
12 out of 17 committers have signed the CLA.

✅ lucassz
✅ hatim-ez
✅ Ashton-Sidhu
✅ daanhendrio
✅ ryan-crabbe-berri
✅ emerzon
✅ duan-levan
✅ yuneng-berri
✅ Sameerlite
✅ shivamrawat1
✅ silencedoctor
✅ joereyna
❌ krisyang1125
❌ krrish-berri-2
❌ ti3x
❌ ishaan-berri
❌ jonemo
You have signed the CLA already but the status is still pending? Let us recheck it.

@codspeed-hq

codspeed-hq Bot commented Mar 30, 2026

Copy link
Copy Markdown
Contributor

Congrats! CodSpeed is installed 🎉

🆕 16 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


Open in CodSpeed

@greptile-apps

greptile-apps Bot commented Mar 30, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds model_group as an optional query parameter to the /spend/logs/v2 and /spend/logs/ui endpoints, enabling filtering by the Router's public model group name. The implementation follows the exact same pattern as the existing model and model_id filters, correctly wiring model_group into both the Prisma count query and the parameterized raw SQL data query. A corresponding unit test is included and matches the style of adjacent tests.

Confidence Score: 5/5

Safe to merge — the change is minimal, follows existing patterns exactly, and the filter value is properly parameterized in raw SQL to prevent injection.

All findings are P2 or below. The implementation correctly adds the filter to both the Prisma count path and the parameterized raw SQL data path, matching the pattern of model and model_id filters. The test covers the happy path using the established mock helper.

No files require special attention.

Important Files Changed

Filename Overview
litellm/proxy/spend_tracking/spend_management_endpoints.py Adds model_group as an optional FastAPI query parameter and correctly propagates it into both the Prisma where_conditions and the parameterized raw SQL equality filter list.
tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py Adds test_ui_view_spend_logs_with_model_group using the existing make_ui_spend_logs_mock_prisma helper; also reformats several multi-context with statements to Python 3.10+ parenthesized style (no functional change).

Sequence Diagram

sequenceDiagram
    participant Client
    participant FastAPI as /spend/logs/v2 or /spend/logs/ui
    participant Prisma as Prisma Count
    participant SQL as Raw SQL Query

    Client->>FastAPI: GET ?model_group=gpt-4&start_date=...&end_date=...
    FastAPI->>FastAPI: Build where_conditions {model_group: "gpt-4", startTime: {...}}
    FastAPI->>Prisma: litellm_spendlogs.count(where=where_conditions)
    Prisma-->>FastAPI: total_records
    FastAPI->>FastAPI: Build SQL WHERE clause model_group = $N (parameterized)
    FastAPI->>SQL: query_raw(sql, ..., "gpt-4", ...)
    SQL-->>FastAPI: paginated rows
    FastAPI-->>Client: {data, total, page, page_size, total_pages}
Loading

Reviews (3): Last reviewed commit: "feat(proxy): add model_group filter to /..." | Re-trigger Greptile

krrish-berri-2 and others added 25 commits April 13, 2026 12:23
…rieval tool (BerriAI#25637)

* feat: add litellm.compress() for BM25-based context compression

Adds a compress() utility that reduces context size for LLM calls using
BM25 relevance scoring (with optional semantic embeddings via
litellm.embedding()). Messages below a token threshold pass through
unchanged; messages above are scored, ranked, and the lowest-relevance
ones replaced with stubs. Originals are cached and a retrieval tool is
injected so the model can recover dropped content on demand.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(compress): truncate high-scoring messages instead of fully stubbing them

When a relevant message was too large to fit in the token budget it was
replaced with a stub, leaving the LLM with no real content to work with.
Now the highest-scoring overflow message is truncated (first 70% + last 30%
of words) to fill the remaining budget, so the LLM always receives actual
content rather than just a retrieval pointer.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(bm25): add prefix expansion so query terms match inflected doc tokens

"cook" now matches "cooking", "auth" matches "authentication", etc.
Without this, short query terms scored 0 against longer inflected forms
in documents, causing the wrong message to be kept.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add routing correctness test and eval harness for litellm.compress()

- test_simple_compression: parametrized test verifying BM25 routes the
  right message based on query ("How to cook?" keeps cooking, "Fix auth"
  keeps auth content)
- eval_compression.py: end-to-end eval harness comparing baseline vs
  compressed model performance on HumanEval-style coding problems

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(eval): add SWE-bench Lite compression eval harness

Uses princeton-nlp/SWE-bench_Lite_bm25_27K which bundles ~27k tokens of
BM25-retrieved repo context per problem — large enough to meaningfully
stress litellm.compress() without Docker or GitHub API calls.

Proxy eval metrics (no test runner needed):
  - has_diff: model produced a valid unified diff
  - file_overlap: fraction of gold-patch files in generated patch
  - exact_file_match: generated patch touches exactly the right files

Run: python tests/eval_swe_bench.py --model gpt-4o --problems 10

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(eval): robust dataset loading + sys.path fix for worktree imports

- Add HuggingFace API fallback so the SWE-bench loader doesn't need
  the `datasets` library (avoids pyarrow/numpy binary compat issues)
- Insert repo root into sys.path so compression module resolves
  from worktrees
- Use direct import of litellm_compress to avoid __getattr__ issues

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* improve compression quality: line-based truncation, multi-message budget, 70% default target

- Switch truncate_message from word-based to line-based splitting to
  preserve code structure (function boundaries, indentation)
- Allow multiple messages to be truncated instead of burning entire
  budget on one overflow message
- Raise default compression target from 50% to 70% of trigger for
  better quality/cost tradeoff
- Add --compression-target CLI arg to SWE-bench eval harness
- Move tests to canonical locations (tests/test_litellm/, scripts/)
- Add docs page and sidebar entries for compress()

Eval results (5 problems, Opus, trigger=10k):
  Hunk overlap delta improved from -0.417 to -0.221
  Content similarity now matches baseline (+0.006)
  Cost savings: 72%

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add SWE-bench performance results to compress() docs

Include benchmark table from Opus eval (5 problems, trigger=10k)
showing 72% cost savings with file-level quality fully preserved.
Add metric explanations and eval runner examples.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(eval): use tolerance-based hunk overlap metric

The exact line-number matching was too brittle — LLM-generated patches
often target the right code region but with slightly offset line numbers.
Switch to hunk-level overlap with a 10-line tolerance window so nearby
edits count as matches. This better reflects actual patch quality.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add compression_interception callback for LiteLLM Proxy

Add a proxy callback that automatically compresses incoming /v1/messages
payloads above a configurable token threshold, runs the retrieval tool
loop server-side, and returns the final response. This brings compress()
support to proxy deployments (e.g. Claude Code via /v1/messages).

- New callback: litellm/integrations/compression_interception/
- Proxy config: compression_interception_params in litellm_settings
- Support for input_type param in compress() (openai vs anthropic)
- Docs: proxy setup instructions with YAML config example
- Tests: 139-line unit test suite for the interception handler

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Revert "feat: add compression_interception callback for LiteLLM Proxy"

This reverts commit 72bd5cb.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…elds

Boolean fields in the auto-generated guardrail provider form (e.g. Noma
`use_v2`) rendered as empty Selects because the Form.Item only populated
`initialValue` for percentage fields, and the `defaultValue` passed to the
Select child was silently dropped by antd's controlled-component wrapper.
Users could not tell what the backend default was, and the visual ambiguity
made flags like `use_v2` look inoperative even though the save path worked.

Unify `initialValue` to fall back through `fieldValue → field.default_value →
(percentage ? 0.5 : undefined)`, and switch Select.Option values from
"true"/"false" strings to real booleans so the backend default flows through
without stringification.
…_.py

Cast message lists to the expected `List[Union[AllMessageValues, Message]]`
type at `token_counter` call sites, and suppress the `no-redef` warning for
the `compress` import in `__init__.py` caused by the wildcard `main` import.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…iAI#25656)

### Background

The Gemini batchEmbedContents response handler hardcoded `index=0` for
every embedding in the response. Any consumer relying on the OpenAI-format
`index` field to match embeddings back to inputs would silently get wrong
associations.

### Changes

Use `enumerate` in `process_response` so each embedding gets its
positional index instead of 0.

### Test Plan

Added unit test asserting sequential indices and correct vector ordering
for a 3-element batch response.
…hunk (BerriAI#25533)

* fix: emit input_json_delta for tool args bundled in first streaming chunk

Some providers (xAI, Gemini) include tool_call function arguments in the
same streaming chunk as the function name/id. The AnthropicStreamWrapper
was discarding the trigger chunk entirely when starting a new content
block, which silently dropped the input_json_delta carrying tool
arguments. This caused tool_use blocks to arrive with empty input {}.

Now queue the processed_chunk after content_block_start when it carries
non-empty input_json_delta data. Backward compatible: providers that send
empty arguments in the first chunk (OpenAI-style) are unaffected since
the condition checks for truthy partial_json.

* test: add tests for input_json_delta emission on bundled tool args

Covers the fix for providers (xAI, Gemini) that bundle tool_call
arguments in the same streaming chunk as the function name/id.
Verifies the AnthropicStreamWrapper emits input_json_delta after
content_block_start, and that empty-arg chunks (OpenAI-style) are
unaffected.

* style: apply Black formatting to streaming_iterator.py

* fix: mirror input_json_delta fix to sync __next__ and add sync tests

* test: make no_extra_delta tests assert explicitly instead of passing silently
…et is configured have no budget enforcement (BerriAI#25557)

* fix BerriAI#25506

* address greptile review feedback

* [Test] UI - Models: Add E2E tests for Add Model flow

Add E2E tests covering:
- Test connection with bad credentials shows failure modal
- Adding a specific model and verifying it appears in All Models table
- Adding a wildcard route and verifying it appears in All Models table
- Verifying model dropdown shows provider-specific models (existing test updated)

Added data-testid attributes to UI components to support stable test selectors.

Tests verified passing 3/3 consecutive runs with zero flakiness.

* address greptile review feedback (greploop iteration 1)

Add cleanup helper to delete models created during tests, preventing
stale data accumulation across repeated test runs.

* fix CI: replace data-testid selectors with text/role-based selectors

The data-testid attributes added to React components are not present
in the CI-built UI output. Switch to using getByRole and getByText
selectors which work with the rendered DOM regardless of build cache.

* remove unnecessary cleanup helper

The database is freshly seeded on every test run via seed.sql,
so per-test cleanup is not needed.

---------

Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
* Serialize error message to a string; only scan last message

* Update litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Add v2 of hiddenlayer guardrail implementation

* Update litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Fix potential header issue

* linting

* Add image support

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…t_latency strategy (BerriAI#25548)

* fix(router): discard oldest entry when trimming latency list in lowest_latency strategy

The lowest_latency routing strategy keeps a rolling window of the most
recent latency and time-to-first-token measurements per deployment. When
the window is full, the strategy was discarding the *newest* value
instead of the oldest, because the trim used
`[: max_latency_list_size - 1]` (keeping indices 0..N-2) rather than
`[1:]` (dropping index 0 and keeping indices 1..N-1).

Since new values are appended at the end, the bug meant the most recent
measurement was always dropped once the list reached capacity. The
routing decisions then relied on stale data (including any early-spike
values that never aged out), and timeout penalties written via
`async_log_failure_event` were silently discarded as well.

Fix the slice in all five call sites (sync + async log_success_event for
both latency and time_to_first_token, and async_log_failure_event for
the timeout penalty) and add regression tests covering each path.

* test(router): cover async TTFT trim path in lowest_latency regression tests

Adds test_ttft_list_trimming_discards_oldest_entry_async, an async
counterpart to test_ttft_list_trimming_discards_oldest_entry that drives
async_log_success_event with a ModelResponse and completion_start_time so
the async time_to_first_token trim branch is actually exercised.

Previously no test touched that code path: the sync TTFT test used
log_success_event, and the async latency test passed a plain dict
response_obj without stream/completion_start_time, so TTFT was never
computed and the async trim was unreached. Verified load-bearing by
reverting only the async TTFT slice — the new test fails and all others
pass.

* format
* fix: drain datadog batches safely

* fix: preserve datadog batches on 413

* fix: import time in datadog flush queue

* test: cover datadog batching edge cases

* fix: only stamp successful datadog flushes

* test: use sync mock for datadog payload builder
…ns (BerriAI#23337)

Vertex AI rejects requests containing both search tools (googleSearch,
enterpriseWebSearch, urlContext) and function declarations with error:
'Multiple tools are supported only when they are all search tools.'

When _merge_tools_from_deployment() combines deployment-level search
tools with user-request function tools (e.g. via MCP), the mixed tool
list causes a 400 error. This fix detects the conflict in _map_function()
and drops search tools, keeping function declarations.

Non-search tools like code_execution and computerUse are preserved.

Fixes BerriAI#23337
…13_2026_p1

litellm oss staging 04/13/2026
feat: add litellm.compress() — BM25-based prompt compression with ret…
The Logs view's Team ID filter dropdown was reading `allTeams` from the
root `teams` state in page.tsx, which the Teams page search overwrites
with its filtered subset. Applying a team search on the Teams page made
filtered-out teams disappear from the Logs filter dropdown.

Swap the Team ID filter to use the existing `TeamDropdown` component via
a small `FilterTeamDropdown` wrapper that adapts it to the filter slot's
`FilterOptionCustomComponentProps` contract. The dropdown now drives its
own `useInfiniteTeams` query against `/v2/team/list` with server-side
search and an isolated react-query cache, unreachable from root state.

Rename the now-unused `hookAllTeams` destructure to `allTeams` so the
`KeyInfoView` passthrough receives the hook's unpolluted fetch instead
of the polluted prop, and drop the dead `allTeams` prop from
`SpendLogsTable` and both of its call sites.
…ool-select-rendering

fix(ui): pre-select backend default for boolean guardrail provider fields
…ilter-state-bleed

fix: isolate logs team filter dropdown from root teams state bleed
user_dashboard.tsx imports getCookie from @/utils/cookieUtils, but the
vi.mock factory in user_dashboard.test.tsx only exports clearTokenCookies.
Vitest throws `No "getCookie" export is defined on the "@/utils/cookieUtils"
mock`, breaking all three beforeunload-listener tests.

Add getCookie to the mock factory so it matches the current imports.
…ts-get-cookie

test(ui): add getCookie to cookieUtils mock in user_dashboard test
Pre-select "Internal User Viewer" in the Global Proxy Role dropdown
on both the standalone and embedded Invite User forms so admins don't
have to remember to pick a role, and the default lands on the least
privileged option rather than silently posting an undefined role.
joereyna and others added 11 commits April 14, 2026 18:59
Bedrock GPT-OSS occasionally emits truncated toolUse.input deltas
(e.g. accumulated args of '{"":"'), which causes
test_function_calling_with_tool_response to hard-fail on json.loads.
Other overrides in TestBedrockGPTOSS already handle similar
model-side flakiness; apply retries=6 delay=5 scoped to this subclass
so other providers keep strict behavior.
GPT-OSS on Bedrock intermittently emits truncated toolUse.input deltas
(e.g. accumulated args of '{"":"'), causing
test_function_calling_with_tool_response to hard-fail on json.loads.
The model flakiness is not a litellm regression: the same base test
passes for Anthropic in the same CI run, and the streaming delta path
at invoke_handler.py has not changed recently.

Follow the existing override pattern in TestBedrockGPTOSS
(test_prompt_caching, test_completion_cost, test_tool_call_no_arguments)
and stub the test to pass. The underlying bedrock converse streaming
tool-call path is already covered by Claude/Nova/Llama Converse suites
in test_bedrock_completion.py and test_bedrock_llama.py, so removing
the live GPT-OSS check loses no unique litellm-side signal.
Complements the stubbed-out live integration test by verifying the
outgoing Bedrock Converse request body for GPT-OSS is well-formed when
the caller supplies a tool schema with OpenAI-style metadata
($id, $schema, additionalProperties, strict):
- correct converse URL for bedrock/converse/openai.gpt-oss-20b-1:0
- toolConfig.tools[0].toolSpec has the expected name/description
- inputSchema.json keeps type/properties/required and strips fields
  Bedrock does not accept
…OssToolCall

[Test] Replace flaky bedrock gpt-oss tool-call live test with request-body mock
…s-coverage-path

fix: remove non-existent litellm_mcps_tests_coverage from coverage combine
…ath-timeout

fix(ci): increase test-server-root-path timeout to 30m
[Infra] Guard main to only accept PRs from staging and hotfix branches
@silencedoctor
silencedoctor force-pushed the litellm_feat-model-group-filter-spend-logs branch from 96d4a09 to 96d7f2d Compare April 15, 2026 06:56
@silencedoctor
silencedoctor changed the base branch from main to litellm_oss_branch April 15, 2026 06:56
@codecov

codecov Bot commented Apr 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Add an optional `model_group` query parameter to the `/spend/logs/v2`
and `/spend/logs/ui` endpoints, allowing users to filter spend logs by
model group. This is consistent with the existing `model` and `model_id`
filters and requires no schema changes since `model_group` is already a
column in the `LiteLLM_SpendLogs` table.

Fixes BerriAI#24781

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@silencedoctor

silencedoctor commented Apr 15, 2026

Copy link
Copy Markdown
Contributor Author

Hi @krrish-berri-2 @ishaan-jaff 👋

This PR has been open for over 2 weeks. I've rebased onto the latest main, fixed the uv.lock sync issue, and applied Black formatting — CI should be green now.

Greptile gave it a 5/5 confidence score and all modified lines are covered by tests.

The change itself is minimal: it adds an optional model_group query parameter to /spend/logs/v2, following the exact same pattern as the existing model and model_id filters.

Use case: We're building an AI Gateway dashboard where end users should only see the logical model name (e.g. "gpt-4"), without needing to know which specific provider deployment is serving their requests behind the scenes. The model_group column already exists in SpendLogs and is returned in responses — this PR just adds the missing filter parameter so we can query logs at the model-group level.

Would appreciate a review when you get a chance. Thanks!

silencedoctor added a commit to silencedoctor/litellm that referenced this pull request Apr 20, 2026
Add an optional `model_group` query parameter to the `/spend/logs/v2`
and `/spend/logs/ui` endpoints, allowing users to filter spend logs by
model group. This is consistent with the existing `model` and `model_id`
filters and requires no schema changes since `model_group` is already a
column in the `LiteLLM_SpendLogs` table.

Supersedes BerriAI#24782 (rebased onto latest main).
@silencedoctor

Copy link
Copy Markdown
Contributor Author

Closing in favor of #26080, which contains the same diff cleanly rebased onto current main. The branch here had drifted significantly behind main and the PR sat un-reviewed for 3 weeks.

New PR: #26080
New issue: #26079

silencedoctor added a commit to silencedoctor/litellm that referenced this pull request May 25, 2026
Add an optional `model_group` query parameter to the `/spend/logs/v2`
and `/spend/logs/ui` endpoints, allowing users to filter spend logs by
model group. This is consistent with the existing `model` and `model_id`
filters and requires no schema changes since `model_group` is already a
column in the `LiteLLM_SpendLogs` table.

Supersedes BerriAI#24782 (rebased onto latest main).
Sameerlite pushed a commit that referenced this pull request Jun 2, 2026
Add an optional `model_group` query parameter to the `/spend/logs/v2`
and `/spend/logs/ui` endpoints, allowing users to filter spend logs by
model group. This is consistent with the existing `model` and `model_id`
filters and requires no schema changes since `model_group` is already a
column in the `LiteLLM_SpendLogs` table.

Supersedes #24782 (rebased onto latest main).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Add model_group filter to /spend/logs/v2 endpoint