feat: add model_group filter to /spend/logs/v2 endpoint - #24782
feat: add model_group filter to /spend/logs/v2 endpoint#24782silencedoctor wants to merge 57 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
Greptile SummaryThis PR adds Confidence Score: 5/5Safe to merge — the change is minimal, follows existing patterns exactly, and the filter value is properly parameterized in raw SQL to prevent injection. All findings are P2 or below. The implementation correctly adds the filter to both the Prisma count path and the parameterized raw SQL data path, matching the pattern of No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/proxy/spend_tracking/spend_management_endpoints.py | Adds model_group as an optional FastAPI query parameter and correctly propagates it into both the Prisma where_conditions and the parameterized raw SQL equality filter list. |
| tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py | Adds test_ui_view_spend_logs_with_model_group using the existing make_ui_spend_logs_mock_prisma helper; also reformats several multi-context with statements to Python 3.10+ parenthesized style (no functional change). |
Sequence Diagram
sequenceDiagram
participant Client
participant FastAPI as /spend/logs/v2 or /spend/logs/ui
participant Prisma as Prisma Count
participant SQL as Raw SQL Query
Client->>FastAPI: GET ?model_group=gpt-4&start_date=...&end_date=...
FastAPI->>FastAPI: Build where_conditions {model_group: "gpt-4", startTime: {...}}
FastAPI->>Prisma: litellm_spendlogs.count(where=where_conditions)
Prisma-->>FastAPI: total_records
FastAPI->>FastAPI: Build SQL WHERE clause model_group = $N (parameterized)
FastAPI->>SQL: query_raw(sql, ..., "gpt-4", ...)
SQL-->>FastAPI: paginated rows
FastAPI-->>Client: {data, total, page, page_size, total_pages}
Reviews (3): Last reviewed commit: "feat(proxy): add model_group filter to /..." | Re-trigger Greptile
…rieval tool (BerriAI#25637) * feat: add litellm.compress() for BM25-based context compression Adds a compress() utility that reduces context size for LLM calls using BM25 relevance scoring (with optional semantic embeddings via litellm.embedding()). Messages below a token threshold pass through unchanged; messages above are scored, ranked, and the lowest-relevance ones replaced with stubs. Originals are cached and a retrieval tool is injected so the model can recover dropped content on demand. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(compress): truncate high-scoring messages instead of fully stubbing them When a relevant message was too large to fit in the token budget it was replaced with a stub, leaving the LLM with no real content to work with. Now the highest-scoring overflow message is truncated (first 70% + last 30% of words) to fill the remaining budget, so the LLM always receives actual content rather than just a retrieval pointer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(bm25): add prefix expansion so query terms match inflected doc tokens "cook" now matches "cooking", "auth" matches "authentication", etc. Without this, short query terms scored 0 against longer inflected forms in documents, causing the wrong message to be kept. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add routing correctness test and eval harness for litellm.compress() - test_simple_compression: parametrized test verifying BM25 routes the right message based on query ("How to cook?" keeps cooking, "Fix auth" keeps auth content) - eval_compression.py: end-to-end eval harness comparing baseline vs compressed model performance on HumanEval-style coding problems Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(eval): add SWE-bench Lite compression eval harness Uses princeton-nlp/SWE-bench_Lite_bm25_27K which bundles ~27k tokens of BM25-retrieved repo context per problem — large enough to meaningfully stress litellm.compress() without Docker or GitHub API calls. Proxy eval metrics (no test runner needed): - has_diff: model produced a valid unified diff - file_overlap: fraction of gold-patch files in generated patch - exact_file_match: generated patch touches exactly the right files Run: python tests/eval_swe_bench.py --model gpt-4o --problems 10 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(eval): robust dataset loading + sys.path fix for worktree imports - Add HuggingFace API fallback so the SWE-bench loader doesn't need the `datasets` library (avoids pyarrow/numpy binary compat issues) - Insert repo root into sys.path so compression module resolves from worktrees - Use direct import of litellm_compress to avoid __getattr__ issues Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * improve compression quality: line-based truncation, multi-message budget, 70% default target - Switch truncate_message from word-based to line-based splitting to preserve code structure (function boundaries, indentation) - Allow multiple messages to be truncated instead of burning entire budget on one overflow message - Raise default compression target from 50% to 70% of trigger for better quality/cost tradeoff - Add --compression-target CLI arg to SWE-bench eval harness - Move tests to canonical locations (tests/test_litellm/, scripts/) - Add docs page and sidebar entries for compress() Eval results (5 problems, Opus, trigger=10k): Hunk overlap delta improved from -0.417 to -0.221 Content similarity now matches baseline (+0.006) Cost savings: 72% Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add SWE-bench performance results to compress() docs Include benchmark table from Opus eval (5 problems, trigger=10k) showing 72% cost savings with file-level quality fully preserved. Add metric explanations and eval runner examples. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(eval): use tolerance-based hunk overlap metric The exact line-number matching was too brittle — LLM-generated patches often target the right code region but with slightly offset line numbers. Switch to hunk-level overlap with a 10-line tolerance window so nearby edits count as matches. This better reflects actual patch quality. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add compression_interception callback for LiteLLM Proxy Add a proxy callback that automatically compresses incoming /v1/messages payloads above a configurable token threshold, runs the retrieval tool loop server-side, and returns the final response. This brings compress() support to proxy deployments (e.g. Claude Code via /v1/messages). - New callback: litellm/integrations/compression_interception/ - Proxy config: compression_interception_params in litellm_settings - Support for input_type param in compress() (openai vs anthropic) - Docs: proxy setup instructions with YAML config example - Tests: 139-line unit test suite for the interception handler Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Revert "feat: add compression_interception callback for LiteLLM Proxy" This reverts commit 72bd5cb. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…elds Boolean fields in the auto-generated guardrail provider form (e.g. Noma `use_v2`) rendered as empty Selects because the Form.Item only populated `initialValue` for percentage fields, and the `defaultValue` passed to the Select child was silently dropped by antd's controlled-component wrapper. Users could not tell what the backend default was, and the visual ambiguity made flags like `use_v2` look inoperative even though the save path worked. Unify `initialValue` to fall back through `fieldValue → field.default_value → (percentage ? 0.5 : undefined)`, and switch Select.Option values from "true"/"false" strings to real booleans so the backend default flows through without stringification.
…_.py Cast message lists to the expected `List[Union[AllMessageValues, Message]]` type at `token_counter` call sites, and suppress the `no-redef` warning for the `compress` import in `__init__.py` caused by the wildcard `main` import. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…iAI#25656) ### Background The Gemini batchEmbedContents response handler hardcoded `index=0` for every embedding in the response. Any consumer relying on the OpenAI-format `index` field to match embeddings back to inputs would silently get wrong associations. ### Changes Use `enumerate` in `process_response` so each embedding gets its positional index instead of 0. ### Test Plan Added unit test asserting sequential indices and correct vector ordering for a 3-element batch response.
…hunk (BerriAI#25533) * fix: emit input_json_delta for tool args bundled in first streaming chunk Some providers (xAI, Gemini) include tool_call function arguments in the same streaming chunk as the function name/id. The AnthropicStreamWrapper was discarding the trigger chunk entirely when starting a new content block, which silently dropped the input_json_delta carrying tool arguments. This caused tool_use blocks to arrive with empty input {}. Now queue the processed_chunk after content_block_start when it carries non-empty input_json_delta data. Backward compatible: providers that send empty arguments in the first chunk (OpenAI-style) are unaffected since the condition checks for truthy partial_json. * test: add tests for input_json_delta emission on bundled tool args Covers the fix for providers (xAI, Gemini) that bundle tool_call arguments in the same streaming chunk as the function name/id. Verifies the AnthropicStreamWrapper emits input_json_delta after content_block_start, and that empty-arg chunks (OpenAI-style) are unaffected. * style: apply Black formatting to streaming_iterator.py * fix: mirror input_json_delta fix to sync __next__ and add sync tests * test: make no_extra_delta tests assert explicitly instead of passing silently
…et is configured have no budget enforcement (BerriAI#25557) * fix BerriAI#25506 * address greptile review feedback * [Test] UI - Models: Add E2E tests for Add Model flow Add E2E tests covering: - Test connection with bad credentials shows failure modal - Adding a specific model and verifying it appears in All Models table - Adding a wildcard route and verifying it appears in All Models table - Verifying model dropdown shows provider-specific models (existing test updated) Added data-testid attributes to UI components to support stable test selectors. Tests verified passing 3/3 consecutive runs with zero flakiness. * address greptile review feedback (greploop iteration 1) Add cleanup helper to delete models created during tests, preventing stale data accumulation across repeated test runs. * fix CI: replace data-testid selectors with text/role-based selectors The data-testid attributes added to React components are not present in the CI-built UI output. Switch to using getByRole and getByText selectors which work with the rendered DOM regardless of build cache. * remove unnecessary cleanup helper The database is freshly seeded on every test run via seed.sql, so per-test cleanup is not needed. --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
* Serialize error message to a string; only scan last message * Update litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Add v2 of hiddenlayer guardrail implementation * Update litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Fix potential header issue * linting * Add image support --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…t_latency strategy (BerriAI#25548) * fix(router): discard oldest entry when trimming latency list in lowest_latency strategy The lowest_latency routing strategy keeps a rolling window of the most recent latency and time-to-first-token measurements per deployment. When the window is full, the strategy was discarding the *newest* value instead of the oldest, because the trim used `[: max_latency_list_size - 1]` (keeping indices 0..N-2) rather than `[1:]` (dropping index 0 and keeping indices 1..N-1). Since new values are appended at the end, the bug meant the most recent measurement was always dropped once the list reached capacity. The routing decisions then relied on stale data (including any early-spike values that never aged out), and timeout penalties written via `async_log_failure_event` were silently discarded as well. Fix the slice in all five call sites (sync + async log_success_event for both latency and time_to_first_token, and async_log_failure_event for the timeout penalty) and add regression tests covering each path. * test(router): cover async TTFT trim path in lowest_latency regression tests Adds test_ttft_list_trimming_discards_oldest_entry_async, an async counterpart to test_ttft_list_trimming_discards_oldest_entry that drives async_log_success_event with a ModelResponse and completion_start_time so the async time_to_first_token trim branch is actually exercised. Previously no test touched that code path: the sync TTFT test used log_success_event, and the async latency test passed a plain dict response_obj without stream/completion_start_time, so TTFT was never computed and the async trim was unreached. Verified load-bearing by reverting only the async TTFT slice — the new test fails and all others pass. * format
* fix: drain datadog batches safely * fix: preserve datadog batches on 413 * fix: import time in datadog flush queue * test: cover datadog batching edge cases * fix: only stamp successful datadog flushes * test: use sync mock for datadog payload builder
…ns (BerriAI#23337) Vertex AI rejects requests containing both search tools (googleSearch, enterpriseWebSearch, urlContext) and function declarations with error: 'Multiple tools are supported only when they are all search tools.' When _merge_tools_from_deployment() combines deployment-level search tools with user-request function tools (e.g. via MCP), the mixed tool list causes a 400 error. This fix detects the conflict in _map_function() and drops search tools, keeping function declarations. Non-search tools like code_execution and computerUse are preserved. Fixes BerriAI#23337
…h_tool_conflict method
…13_2026_p1 litellm oss staging 04/13/2026
feat: add litellm.compress() — BM25-based prompt compression with ret…
The Logs view's Team ID filter dropdown was reading `allTeams` from the root `teams` state in page.tsx, which the Teams page search overwrites with its filtered subset. Applying a team search on the Teams page made filtered-out teams disappear from the Logs filter dropdown. Swap the Team ID filter to use the existing `TeamDropdown` component via a small `FilterTeamDropdown` wrapper that adapts it to the filter slot's `FilterOptionCustomComponentProps` contract. The dropdown now drives its own `useInfiniteTeams` query against `/v2/team/list` with server-side search and an isolated react-query cache, unreachable from root state. Rename the now-unused `hookAllTeams` destructure to `allTeams` so the `KeyInfoView` passthrough receives the hook's unpolluted fetch instead of the polluted prop, and drop the dead `allTeams` prop from `SpendLogsTable` and both of its call sites.
…ool-select-rendering fix(ui): pre-select backend default for boolean guardrail provider fields
…ilter-state-bleed fix: isolate logs team filter dropdown from root teams state bleed
user_dashboard.tsx imports getCookie from @/utils/cookieUtils, but the vi.mock factory in user_dashboard.test.tsx only exports clearTokenCookies. Vitest throws `No "getCookie" export is defined on the "@/utils/cookieUtils" mock`, breaking all three beforeunload-listener tests. Add getCookie to the mock factory so it matches the current imports.
…ts-get-cookie test(ui): add getCookie to cookieUtils mock in user_dashboard test
Pre-select "Internal User Viewer" in the Global Proxy Role dropdown on both the standalone and embedded Invite User forms so admins don't have to remember to pick a role, and the default lands on the least privileged option rather than silently posting an undefined role.
Bedrock GPT-OSS occasionally emits truncated toolUse.input deltas
(e.g. accumulated args of '{"":"'), which causes
test_function_calling_with_tool_response to hard-fail on json.loads.
Other overrides in TestBedrockGPTOSS already handle similar
model-side flakiness; apply retries=6 delay=5 scoped to this subclass
so other providers keep strict behavior.
GPT-OSS on Bedrock intermittently emits truncated toolUse.input deltas
(e.g. accumulated args of '{"":"'), causing
test_function_calling_with_tool_response to hard-fail on json.loads.
The model flakiness is not a litellm regression: the same base test
passes for Anthropic in the same CI run, and the streaming delta path
at invoke_handler.py has not changed recently.
Follow the existing override pattern in TestBedrockGPTOSS
(test_prompt_caching, test_completion_cost, test_tool_call_no_arguments)
and stub the test to pass. The underlying bedrock converse streaming
tool-call path is already covered by Claude/Nova/Llama Converse suites
in test_bedrock_completion.py and test_bedrock_llama.py, so removing
the live GPT-OSS check loses no unique litellm-side signal.
Complements the stubbed-out live integration test by verifying the outgoing Bedrock Converse request body for GPT-OSS is well-formed when the caller supplies a tool schema with OpenAI-style metadata ($id, $schema, additionalProperties, strict): - correct converse URL for bedrock/converse/openai.gpt-oss-20b-1:0 - toolConfig.tools[0].toolSpec has the expected name/description - inputSchema.json keeps type/properties/required and strips fields Bedrock does not accept
…OssToolCall [Test] Replace flaky bedrock gpt-oss tool-call live test with request-body mock
…uardrail_fallbacks docs update
…s-coverage-path fix: remove non-existent litellm_mcps_tests_coverage from coverage combine
…ath-timeout fix(ci): increase test-server-root-path timeout to 30m
bump: version 1.83.7 → 1.83.8
[Infra] Guard main to only accept PRs from staging and hotfix branches
96d4a09 to
96d7f2d
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Add an optional `model_group` query parameter to the `/spend/logs/v2` and `/spend/logs/ui` endpoints, allowing users to filter spend logs by model group. This is consistent with the existing `model` and `model_id` filters and requires no schema changes since `model_group` is already a column in the `LiteLLM_SpendLogs` table. Fixes BerriAI#24781 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
96d7f2d to
0fad9e2
Compare
|
Hi @krrish-berri-2 @ishaan-jaff 👋 This PR has been open for over 2 weeks. I've rebased onto the latest Greptile gave it a 5/5 confidence score and all modified lines are covered by tests. The change itself is minimal: it adds an optional Use case: We're building an AI Gateway dashboard where end users should only see the logical model name (e.g. "gpt-4"), without needing to know which specific provider deployment is serving their requests behind the scenes. The Would appreciate a review when you get a chance. Thanks! |
Add an optional `model_group` query parameter to the `/spend/logs/v2` and `/spend/logs/ui` endpoints, allowing users to filter spend logs by model group. This is consistent with the existing `model` and `model_id` filters and requires no schema changes since `model_group` is already a column in the `LiteLLM_SpendLogs` table. Supersedes BerriAI#24782 (rebased onto latest main).
Add an optional `model_group` query parameter to the `/spend/logs/v2` and `/spend/logs/ui` endpoints, allowing users to filter spend logs by model group. This is consistent with the existing `model` and `model_id` filters and requires no schema changes since `model_group` is already a column in the `LiteLLM_SpendLogs` table. Supersedes BerriAI#24782 (rebased onto latest main).
Add an optional `model_group` query parameter to the `/spend/logs/v2` and `/spend/logs/ui` endpoints, allowing users to filter spend logs by model group. This is consistent with the existing `model` and `model_id` filters and requires no schema changes since `model_group` is already a column in the `LiteLLM_SpendLogs` table. Supersedes #24782 (rebased onto latest main).
Relevant issues
Fixes #24781
Pre-Submission checklist
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewType
🆕 New Feature
Changes
Add an optional
model_groupquery parameter to the/spend/logs/v2and/spend/logs/uiendpoints, enabling users to filter spend logs by model group (the public-facing model name used by the Router for load-balancing).What changed
litellm/proxy/spend_tracking/spend_management_endpoints.pymodel_groupas an optional query parameter (consistent with existingmodelandmodel_idparams)where_conditionsformodel_groupfilteringmodel_groupto the raw SQL equality filter listtests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.pytest_ui_view_spend_logs_with_model_groupfollowing the same pattern as the existingtest_ui_view_spend_logs_with_modelandtest_ui_view_spend_logs_with_model_idtestsWhy
The
model_groupcolumn already exists in theLiteLLM_SpendLogstable and is returned in responses, but there was no way to filter by it. This is needed for downstream consumers that want to analyze usage at the model-group level (e.g., all deployments of "gpt-4") rather than individual deployment models.Test plan
test_ui_view_spend_logs_with_model_group— verifies that passingmodel_group=gpt-4returns only logs with that model grouptest_spend_management_endpoints.pycontinue to passtest_ui_view_spend_logs_with_modelandtest_ui_view_spend_logs_with_model_idtests unaffected