Repository navigation
chore: sync upstream 2026-10-03 - #293
Conversation
…ration (BerriAI#43996) * test(proxy): migrate DB and Redis backed proxy tests into tests/integration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop a type suppression comment from the key metadata integration test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): scope integration test cleanup to owned rows and wait for backend stats flush Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): seed NULL cache_hit and bound recovery reads from below Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(lens): bill worker analysis through virtual keys * fix(lens): pin the verified worker image and add setup proof * fix(lens): preserve network checks and redact billed analysis logs * test(lens): preserve legacy worker result submission during upgrade * fix(lens): enforce trusted worker IPs and restore coverage uploads * docs(lens): explain trusted proxy requirements for worker allowlists * fix(lens): yield to worker disconnects after the synthetic body
…sts/unit/proxy (BerriAI#43998) * test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): stub HIBP through respx by disabling the aiohttp transport Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): share the httpx transport fixture across proxy unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): restore proxy globals without a missing-value sentinel Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): package moved dirs and stub the login breach check at the HTTP boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): isolate the mcp server manager per test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ci): add used_client_oauth_token to the GCS pub/sub spend-log golden BerriAI#43063 stamps used_client_oauth_token into spend-log metadata, so test_async_gcs_pub_sub_v1 failed on main with an extra metadata key * test(ui): give the auto-router threshold save wait room for the availability debounce BerriAI#42625 keeps Save disabled while a 300ms-debounced availability check runs. This test waits for Save right after the change, so the whole debounce lands inside waitFor's 1s default and it times out under CI load. It is the recurring UI Unit Tests failure on main since BerriAI#42625 landed * test(e2e): expect no pricing tier on bills for streamed calls OpenAI served at default BerriAI#42870 added both the rule that a served default or standard tier bills at base pricing and records no service_tier, and streamed tests expecting the row to record 'default'. They have failed on every scheduled litellm-e2e run since. The tests now map the served tier to the pricing basis the bill must record and check input is billed at that basis's rate; the messages case registers custom rates so the rate check has something to compare against * test(e2e-ui): wait for the call-id search before hovering the logs row The row the spec hovers is already on the unfiltered first page, so it was found before the search request returned. The search response then re-rendered the table under the mouse, and the Base UI tooltip never opened. Reproduced with Playwright against a local proxy: hovering right after the fill never shows the tooltip, hovering after the search response shows the call id every time * test(e2e): run the Together structured-output case on the hybrid Qwen with reasoning off The case picked the cheapest Together row flagged supports_response_schema. DeepSeek-V4-Flash-0731 hit its cost-map deprecation date on 2026-09-29, so the pick moved to GLM-5.3-Flash, a reasoning-only model that spends the 1024-token budget thinking and returns content=None. Qwen3.5-9B is the pinned hybrid model the reasoning_effort=none case already exercises, and Together lists it with structured output support * test(integration): read the agent 365 guardrail status by its own name in spend logs The MCP shard runs under xdist against one database, and a sibling file creates a default_on pre_mcp_call content filter there. The owned proxy reloads DB guardrails, so that filter's 'success' entry could land first in guardrail_information and the test read it instead of the agent 365 verdict * test(unit): ignore asyncio's leaked-task records in the budget limiter push-failure log check gc.collect() inside the caplog window can collect a pending task an earlier test left on a closed loop, and asyncio logs 'Task was destroyed but it is pending' into this test's records. The check still counts every LiteLLM logger, and unretrieved task exceptions on this loop still go through the asserted exception handler * test(e2e-ui): fill the create-tag fields inside the dialog BerriAI#42949 added 'Filter by tag name' and 'Filter by description' inputs to the Tag Management page, so page-wide getByLabel('Tag Name') and getByLabel('Description') match two elements and Playwright's strict mode fails the create step * test(integration): run integration proxies with the CI license Multi-worker proxies start each uvicorn worker in a fresh process, so every worker reads the license from its environment. Forward LITELLM_LICENSE into the proxy and test runner environments * ci: save GitHub Actions caches only from main and bump codecov-action to 5.5.5 Every pull request saved its own uv, maturin, Rust and Prisma caches, about 4.5 GB per PR, so the repository's 10 GB cache budget evicted main's entries within minutes. Pull request jobs then missed every cache, downloaded all dependencies from PyPI and hit the install step timeouts. Pull requests now restore only, and main keeps the caches warm for them. test-linting and check-ui-api-types run only on pull requests and keep saving codecov-action 5.5.4 imports its signing key from the deleted codecovsecurity keybase account, so every upload failed signature verification. 5.5.5 reads it from codecovsecops; the key ID matches the one signing the current CLI * test(unit): join the session-minting thread before collecting the handler asyncio.to_thread resumes the test as soon as the worker sets its result, while the pool thread can still hold the work item and through it the handler. gc.collect() then cannot finalize the handler and the session stays open. A pool that shuts down before the test continues drops that reference * test(integration): relaunch owned proxies that lose their port, expire idle gateway connections early owned_proxy_process released its reserved port and the proxy bound it only after full startup, so another xdist worker or an outgoing connection could take it first and the proxy exited with 'address already in use'. The launch now retries on a fresh port when that happens and stops every failed attempt. uvicorn closes idle keep-alive connections after 5 seconds and httpx expired them at the same 5 seconds, so a request sent right at that mark could reuse a socket the server was closing and get 'Connection reset by peer'. Gateway clients now drop idle connections after 2 seconds * ci(circleci): give the base SDK wheel build the same 30 minute no-output window as the Windows build The release profile builds with fat LTO and one codegen unit, so the final link of litellm-cache-s3 runs silently for minutes. Successful builds take 711 to 749 seconds, right at the default 10 minute no-output limit, and about 30% of recent runs were killed there * test(integration): model the budget-reset database outage as 10 seconds instead of 5 refused connections The proxy retries the database about every 30 seconds and each retry opens roughly one connection, so a 5-connection outage took 3 to 4 retries to clear and recovery landed between 60 and 90 seconds, straddling the test's 80 second reset window. A fixed 10 second outage still refuses the immediate reconnect and recovers on the next retry * ci: move the unit-test uv cache split into a composite action check_workflow_startup_safety sums every setup step's timeout, so the save and restore variants each counted 5 minutes although only one runs. One composite step keeps the setup ceiling at 35 minutes * test(unit): point tiktoken at the bundled cache for every unit test The rust_bridge tokenizer tests loaded o200k_base before any test in their xdist worker had imported default_encoding, so tiktoken fell back to the temp cache and tried to download under pytest-socket. Move the session fixture from litellm_core_utils/conftest.py to the root unit conftest. * test(integration): answer model discovery probes in the hosted_vllm wire tests The router's periodic upstream model info refresh sends GET /v1/models to hosted_vllm deployments, so a wire server that is live during a refresh sees an extra request. Answer the probe with an empty model list and leave it out of the provider-call assertions, matching the responses bridge tests.
…ails tests into tests/unit/proxy (BerriAI#44003) * test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): stub HIBP through respx by disabling the aiohttp transport Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): share the httpx transport fixture across proxy unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): restore proxy globals without a missing-value sentinel Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): package moved dirs and stub the login breach check at the HTTP boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): isolate the mcp server manager per test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): move management_endpoints, management_helpers and guardrails tests into tests/unit/proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): reuse the shared httpx transport fixture in moved proxy tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): stub outbound HTTP and package moved test dirs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): restore the config server hostname in the mcp resolution test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): pin the completion tokenizer model in the straiker screening test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ew (BerriAI#43972) * feat(ui): redesign agent trace run view with chat-style detail pane Tree with connector lines, typed icon tiles and provider logos, hover cards with timing, and Input/Output sections rendered as message cards. * fix(tracing): show text for block-list message content and split normalizers per convention OpenAI responses-style content (reasoning + text blocks) rendered as raw JSON in the trace view. Keep the text blocks and drop opaque reasoning. Move each convention into litellm/tracing/normalizers with an ordered registry so new frameworks plug in without touching OTLP decoding. * fix(tracing): keep long message histories as valid JSON and parse function_call blocks * feat(ui): open agent traces in a resizable side drawer with a devtool-style tree Clicking a run opens it in a drawer over the list instead of a full page. j/k and the header arrows switch runs, Esc closes. The tree gets dashed connectors, per-span waterfall bars, mono tool names and real provider logos. AI messages with reasoning/function_call blocks render as text. * fix(trace-ui): address review: valid JSON trimming, drawer keys, reduced motion, narrow screens * feat(tracing): serve span content in a standard LiteLLM UI format GET /v1/traces/{trace_id}/spans/{span_id} now also returns input_ui and output_ui, a tagged union of messages, fields or text built server side by litellm/tracing/ui_format.py. The trace UI renders from those fields and only falls back to client-side parsing when talking to an older proxy. The raw input and output strings are unchanged, and so is storage Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(tracing): fall back to an elision marker when shortened messages still exceed the size limit * fix(tracing): keep both messages when tool_calls are oversized and keep failed-tool styling --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
BerriAI#43748) * feat(s3_v2): add s3_partition_granularity option for hourly S3 folders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover s3 v2 partition granularity across surfaces, settings and chaos Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover previous_response_id history rebuilt from an hourly cold storage object Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3_v2): reuse the cold storage key only when s3_v2 owns cold storage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): cover hour rollover, postgres outage, in-flight switches, key/team vars and real S3 layout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(liccheck): authorize libfaketime, the GPLv2 dev-only clock the s3 rollover integration test preloads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): wait for the rejected-request cell's payloads by id, not by line count Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): declare the postgres outage cell's models in config and trip the relay on burst ids Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): drop the libfaketime hour rollover cell and its dev dependency Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): deselect the s3_v2 live e2e on the stage-mirror stack The stage-mirror config enables no s3_v2 callback, so every test in test_s3_log_e2e.py fails its readiness check there. The file keeps running in the Buildkite e2e lane, which configures s3_v2 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(s3_v2): declare the sink outage burst models in config Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(s3_v2): read cold storage metadata without an empty dict default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for the proxy to reconnect before the postgres outage recovery request Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng <yucheng@berri.ai>
* chore(lens): remove deployment screenshots * refactor(lens)!: rename internal engine package and API * fix(lens): pin worker image for renamed API * test(lens): cover fresh and populated rename migrations * fix(lens): protect db-push upgrades and restore routing and CI * fix(lens): resolve migration tables across schemas and include database driver
…s/unit/proxy (BerriAI#44006) * test(proxy): move utils, agent_endpoints and endpoint tests into tests/unit/proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): package moved unit test directories Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): exclude proxy-db-owned files from the misc target Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop the redundant fixture docstrings in the proxy conftest Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…/unit/proxy (BerriAI#44012) * test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): keep tuple identity in proxy state restore and fix misc target paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ils and root proxy tests into tests/unit/proxy (BerriAI#44015) Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…roxy by glob (BerriAI#44018) * test(proxy): delete the legacy proxy test tree and serve the redirect test from loopback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): exercise the shard check directly for unit_selection-owned children Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): serve the redirect test from respx instead of a socket Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): credit shard ownership only to unit flags wired in gha Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): implement the wired-flag shard crediting the tests assert Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): split the root proxy test files into their own unit shard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): point the rate-limit skip reason at the usage-based-routing-v2 RPM tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Bumps [pypdf](https://github.com/py-pdf/pypdf) from 6.16.2 to 6.19.0. - [Release notes](https://github.com/py-pdf/pypdf/releases) - [Changelog](https://github.com/py-pdf/pypdf/blob/main/CHANGELOG.md) - [Commits](py-pdf/pypdf@6.16.2...6.19.0) --- updated-dependencies: - dependency-name: pypdf dependency-version: 6.19.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
) The pytest-postgresql based proxy tests moved to tests/integration on the real Postgres harness in BerriAI#43996, so nothing loads the plugin anymore. uv.lock is edited by hand to drop the package and its now orphaned mirakuru and port-for deps; uv lock --check passes and a full relock resolves the same package set
…rriAI#44055) * docs(proxy): point mcp_server test references at tests/unit/proxy The legacy tests/test_litellm/proxy tree was removed in BerriAI#44018. Repoint the mcp_server AGENTS.md mirror path, swap its auth example for a module that still exists, and drop the utils.py comment block that named the old test path * docs(proxy): fix remaining mcp_server legacy test path and note import-time env reads Repoint the second tests/test_litellm reference in the mcp_server AGENTS.md Tests section and move the import-time env guidance there from the removed utils.py comment
…4044) * feat(ui): bucket model leaderboard usage by day or week * test(ui): cover daily buckets in model leaderboard series * feat(ui): add daily/weekly toggle and per-bucket total to model leaderboard chart * test(ui): cover the daily/weekly toggle on the model leaderboard * feat(model-insights): add gateway-wide daily totals to the response type * feat(model-insights): return per-day totals across every model, not just the top ranked ones * test(model-insights): daily totals include models outside the top ranking * chore(model-insights): regenerate lazy openapi snapshot for daily totals * chore(ui): regenerate api types for model insights daily totals * fix(ui): compute leaderboard bucket totals from gateway-wide daily totals * fix(ui): show the gateway total, not the top-ten subtotal, in the leaderboard tooltip * test(ui): cover gateway-wide bucket totals on the model leaderboard * fix(model-insights): type daily totals as an immutable tuple * fix(model-insights): build daily totals without new mutable collections
* chore(lint): remove the LIT002 mutable-construction rule Drop LIT002 from scripts/check_type_discipline.py along with its helpers, its budget entry, its unit tests, and the AGENTS.md and gate docstring mentions. `# mutable-ok` now only suppresses LIT001, so the markers that only existed to silence LIT002 became LIT013 stale suppressions and are removed. The files whose layout depended on those trailing comments are reformatted with ruff format. Every other LIT rule count is unchanged and the ASTs of all touched litellm/ files match main apart from one docstring. * chore(lint): keep the leftover mutable-ok markers for a follow-up Restore the ~1.4k `# mutable-ok` markers stripped in the previous commit so this PR only touches the checker, its tests, the budget, and docs. Those markers no longer suppress anything, so `# mutable-ok` is exempt from LIT013 until a follow-up strips them. * Revert "chore(lint): keep the leftover mutable-ok markers for a follow-up" This reverts commit c35bc0b. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
…rriAI#43967) Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…rriAI#44058) Sail now rejects completion_window "flex" on synchronous requests with a 400 saying flex is only for background responses or Batch work. The chat flex case and the responses flex case have failed on every scheduled litellm-e2e run in builds 337, 340 and 341. The chat cases keep balanced and auto, and the responses case sends a caller metadata.completion_window of balanced, so both still prove the window reaches Sail and the bill uses that window's distinct rates
… tracebacks (BerriAI#44062) * ci(circleci): test Redis behavior against local Redis and print short tracebacks The redis_caching_unit_tests job ran three legacy files against the shared remote Redis. The DualCache and batch-read logic that never needed a server now lives in tests/unit/caching/test_dual_cache.py with a mocked RedisCache, and the behavior that does need one (the increment-with-floor Lua script, read-through, deletes, batch reads) moved to tests/integration, which starts a local Redis. test_returned_settings only read REDIS_PORT and is replaced by a unit test of Router.get_settings CircleCI pytest runs now use --tb=short so failure output stays readable in the test results tab * ci(integration): print short tracebacks from run.py and allow Redis in the sdk shard
…rriAI#44064) Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…nd fix registry metadata (BerriAI#43907) * fix(cost-map): add nebius qwen3.8-27b and correct nebius context limits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): add and correct provider deprecation dates for deepseek, gemini and azure models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): add perplexity, openrouter and voyage models and correct gemini, nebius and perplexity metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… tool (BerriAI#43892) * feat(tool-policies): show the user who owns the key that discovered a tool GET /v1/tool/list and GET /v1/tool/{tool_name} resolve the discovering key's owner from the verification token and user tables at response time and return it as a nullable user field. The Tool Policies page adds a User column that shows alias, then email, then ID, with the same cell the Virtual Keys page uses. Keys without an owner, deleted owners, and rows without a key hash show no user, and a database failure in the owner lookup keeps the tools listed with user null Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tool-policies): bound the owner lookup with chunked membership queries The key-by-token and user-by-id lookups behind the tool rows' user field put every distinct key hash into one IN list. BaseRepository gains find_many_in, which runs the repository's chunked membership query and converts the rows like find_many does, and the owner lookup uses it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(tool-policies): cover the owner column across the tool routes and the dashboard Integration cells for the direct, detail and filtered tool routes, owners without alias or email, deleted owners and keys, keyless and unknown-key historical rows, more keys than one membership chunk, repeated reads, two-worker reads during discovery and a failed owner lookup. A Playwright cell drives the bundled Tool Policies page against the live proxy and follows the owner link Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…iAI#44052) * feat(proxy): gzip buffered responses for clients that accept it Large JSON reads like /user/daily/activity/aggregated shipped tens of MB uncompressed. Compress single-message bodies of 500B or more when the client's Accept-Encoding allows gzip (q-values and the wildcard honored). Streamed and etagged responses pass through untouched, every negotiable response carries Vary: Accept-Encoding, and bodies of 1MB or more are compressed in a worker thread * fix(proxy): skip partial and no-transform responses in gzip and always release the held start The gzip gate now also skips 206 Partial Content and Cache-Control: no-transform, since compressing either breaks byte ranges or ignores an explicit ban on transforms. A response start without a headers key no longer raises, and a start the app never follows with a body message is forwarded when the app returns instead of being dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…gs (BerriAI#44057) The usage card hid actual and baseline spend unless every older session could be rebuilt from SpendLogs within two seconds, which on a real gateway it never was. Each complexity router's actual spend is now its rollup spend and its baseline is spend plus recorded savings, for old and new requests alike, so the benchmarks and session endpoints never scan SpendLogs. Adaptive and quality routers record no savings baseline and stay out of the compared totals; savings_estimated_classifier_cost is kept and covers the same compared requests
…44035) * refactor(proxy): inject tracing receiver and access context * refactor(proxy): own tracing resources through FastAPI lifespan * test(proxy): pass tracing dependency in Lens lifecycle * refactor(proxy): stop tracing logger cooperatively * refactor(proxy): derive tracing permissions in one place * refactor(proxy): compose application lifespan state * refactor(proxy): give Lens tracing storage directly * refactor(tracing): name shared ClickHouse storage explicitly * refactor(tracing): extract shared ClickHouse storage crate * test(proxy): isolate db push timeout from Lens safety check * fix(tracing): drain spend retries during shutdown
* refactor: separate OTLP HTTP decoding from trace codec * feat: complete trace ingestion and read paths * fix: encode OTLP protobuf errors in Rust * fix: raise OTLP body limit to 16 MiB * test: cover OTLP auth body parsing boundary * refactor: parse OTLP media type into enum * fix: enforce OTLP body size at HTTP boundary * perf: preserve shared OTLP metadata across ingestion * bench: compare owned and shared trace resource fanout * refactor: extract shared storage and Python conversion caches * refactor: keep shared storage owned by traces * test: keep trace loopback coverage in Rust * test(proxy): adapt trace coverage to injected access context * fix(tracing): satisfy stacked branch lint checks * refactor(tracing): use immutable ingestion payloads * fix(tracing): declare native error encoder export * test(proxy): resolve trace access through dependency * fix(tracing): align merged normalizer types and bridge tests * fix(tracing): address ingestion and diagnostic review findings * fix(proxy): preserve body parsing for partial request scopes * test(proxy): use valid HTTP scopes in request fixtures * test(proxy): complete auth request flow scopes
…he Azure retired models page (BerriAI#44455) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
…ic Messages API (BerriAI#44419) * fix(health): probe Bedrock Mantle Claude deployments over the Anthropic Messages API Bedrock Mantle serves Claude ids only on /anthropic/v1/messages, but health checks probed every chat-mode deployment over /v1/chat/completions, so a bedrock_mantle Claude deployment showed unhealthy while real /v1/messages traffic to it succeeded Add an anthropic_messages health check mode and make it the default for bedrock_mantle Claude models. An explicit model_info.mode still wins, and /health/test_connection and the Add Model form accept the new mode * fix(health): resolve the test connection mode from the deployment when the request omits it The Admin UI model page sent the mode /model/info had filled in from the cost map back as the probe mode, so Test Connection on a Bedrock Mantle Claude deployment still went over chat completions. The page now forwards only the row's id, and /health/test_connection resolves a missing mode the way /health does: the stored model_info.mode, then the mode the provider requires, then the cost map. * fix(health): resolve an omitted ahealth_check mode the way the proxy does * fix(health): test connection honors a stored mode only for the stored model and rejects a non-string mode A request that selects a stored deployment and sends a different litellm_params.model now resolves the probe mode from that model instead of the stored model_info.mode. A litellm_params.mode that is not a string answers 400 instead of 500. The Bedrock Mantle rule that Claude models are probed over the Messages API moves into the provider package. * fix(health): shape test connection probe params for the model the request probes A request that selects a stored deployment by id and overrides the model resolved its probe mode from the overridden model but still injected max_tokens from the stored mode, so an embedding override of an anthropic_messages deployment failed with a Mistral 422 extra_forbidden * fix(health): report an early ahealth_check failure as itself, not as a missing mode With the mode resolved automatically when the caller omits it, a failure before that resolution (no model, a non-string model, a provider that does not resolve) was wrapped as "Missing mode", a hint that pointed at the wrong fix and dropped raw_request_typed_dict from the result. Every failure now returns the same shape. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
BerriAI#44462) * fix(tests): match the OS bind error in the owned-proxy port-race retry * test(integration): keep the port-race predicate pure so its unit tests stay in-process --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…sources (BerriAI#44448) * feat(anthropic): workload identity federation and pluggable identity sources Backend half of BerriAI#38818 (internal copy of the fork PR BerriAI#38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from BerriAI#39613 rides along on the backend side with the same server-owned handling. Fixes BerriAI#28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * fix(proxy): let a deployment write name a federated credential reject_federated_credential_reference runs from is_request_body_safe, which pre_db_read_auth_checks calls on every route, so it also fired on POST /model/new, /model/update, /model/{id}/update and /health/test_connection. A proxy admin could no longer attach a federated credential to a deployment over the API or the Admin UI, leaving a static config.yaml entry as the only way to configure the feature the rejection told the caller to go configure, and _reject_non_admin_wif_write never got to make the call it exists to make. is_request_body_safe now takes the route and skips only the credential-reference check on the routes that reach can_user_make_model_call. Federation fields typed inline into a body stay refused everywhere, and a call naming a federated credential still cannot pick the identity it mints as. * refactor(proxy): derive health display policy from the federation key sets The health check module hand-copied the five workload identity fields whose value is a credential, so a shared proxy surface named provider-specific parameters and a newly added secret-bearing field would have gone on being displayed until someone remembered both places WIF_SECRET_BEARING_KEYS now sits beside the key sets it splits out of, types/utils derives secret_bearing_wif_litellm_params from it, and the health layer splats that tuple the same way it already splats the admin-only one * fix(anthropic_wif): treat blank identity-source fields as unset * test(proxy): classify the federation params in the credential slot registry main's registry test (BerriAI#43298) now fails the build for any credential-named deployment param without a classification. The five federation fields that carry a token, a token file path, or a signing or client secret reference are Unplanted, matching WIF_SECRET_BEARING_KEYS; the four remaining Keycloak settings name a URL, a client id, an auth method, or a scope and are NotSecret * fix(anthropic_wif): declare federation params as owned connection leaves and chart their metrics Register the 18 Anthropic and 3 OpenAI federation params as frozen ConnectionSettings leaves so the owned-kwarg registry, the kwargs funnel and the request-body ban list read one declaration. Pass the deployment api_base through to the count-tokens handler instead of a pre-suffixed URL, which doubled the /count_tokens path on main's prompt-cache predictor. Add the five litellm_anthropic_wif_* families to the all-metrics Grafana dashboard. * fix(credentials): gate PATCH on WIF fields resolved from model_id The credential PATCH handler checked server-owned workload identity federation fields only on the values the caller sent, while a body that named a deployment through model_id had its credential values resolved after that check. A non-admin could therefore copy a federated deployment's WIF fields onto an ordinary credential. Resolve the incoming values first and run the non-admin gate on them, matching the POST path * fix(anthropic): count tokens with ANTHROPIC_AUTH_TOKEN through the shared auth header Count-tokens walked its own credential ladder: a static key, else skip minting when ANTHROPIC_AUTH_TOKEN is set, else mint a federated token. With only the auth token set it forwarded nothing and the proxy silently fell back to its local tokenizer while chat on the same deployment authenticated with that token. The handler now takes the auth header that AnthropicModelInfo.aget_auth_header resolves, the same ladder chat, files, batches and skills use, and merges the oauth beta a minted or consumer token carries with the token-counting beta --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> Co-authored-by: mateo-berri <happymvw@gmail.com>
… provider and shared YAML deployment (BerriAI#44451) * test(integration): exact four-part translation cases on a shared fake provider and shared YAML deployment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): compare every non-transport provider header and check for late provider requests at session end Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): name TranslationTestCase fields after litellm and provider sides and drop regressions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prefix checked TranslationTestCase fields with expected_ and name the fake reply mock_provider_response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(integration): name TranslationTestCase fields in the translation README Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): add a claude-opus-5-5 base case and deployment next to claude-sonnet-4-6 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): name translation cases <MODEL>_TEST_CASE and document the naming rule Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(integration): move translation test rules into tests/integration/translation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…AI#44465) * feat(roi): show matched people by default in contributor lists * fix(roi): keep matched filter tabs readable on narrow screens * fix(roi): retain spend-only users and support older browsers
…erriAI#44444) * feat(enterprise): bundle LiteAdmin Slack worker with native gateway login * fix(enterprise): preserve gateway prefixes during native Slack linking * fix(enterprise): retain native Slack linking on the admin backend * fix(enterprise): reuse shared native Slack connection services
…nsus (BerriAI#43556) * fix(caching): count tool_call cache_control marks in the injection census Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): remove cache census casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): only skip injection on message or content marks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): skip injection on messages whose tool calls carry marks Reverts 9f08d8a. A default 5m mark injected on assistant text lands before the client's 1h tool_use mark, which Anthropic rejects with a 400 because a 1h breakpoint must not follow a 5m one. Keeping the full census in the skip check leaves the client's tool_call breakpoint as the only one on that message. * fix(caching): count every client tool_call cache mark in the breakpoint census The census gated tool_call marks on type function and dict shape, so a client mark on a call without a type or with a string cache_control slipped past the count and injection overflowed the 4 breakpoint cap. Count any non-None tool_call mark, keep the server tool exclusion, and add integration cells for the capped surfaces, the yaml stand-down, Bedrock and Gemini, and router affinity * test(integration): hold the upstream so the worker kill lands mid-burst * refactor(caching): reuse the transform's server tool lookup in the breakpoint census The census now calls the same helper the Anthropic transform uses to decide whether a marked tool call becomes a server tool block, so the two cannot drift apart. The owned-proxy burst test waits up to 90 seconds for the burst to reach the wire before it kills a worker * refactor(anthropic): move the server tool rebuild check under llms/anthropic * test(integration): audit the tool call mark census across chat, messages, responses, and chaos Adds the /audit cells for the breakpoint census on assistant tool_calls marks: the Responses stream bridge, the OpenAI and Anthropic SDK clients, in-process Pydantic messages, response cache twins, request-level points, a provider 401 on a capped request, malformed provider_specific_fields and tool_call ids, null or empty points, a points update mid-burst, a proxy restart mid-burst, and a worker SIGKILL that picks the worker holding the burst's upstream connections * test(integration): close the SDK clients the cache census cells open --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…44466) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
…riAI#44467) * fix(lens): pin worker dependencies and support approved image digests * fix(lens): include locked dependencies and release identity in build context * fix(lens): select the dev worker package for SHA-tagged charts
* feat: seed Lens dev with configurable load profiles * feat: run Lens UI live through the dev launcher * fix: verify Lens UI startup before seeding * fix(ui): render run timestamps on one compact line The agent runs table printed the long locale form with timezone, which wrapped to two lines per row. Use a fixed-width 24h form with milliseconds that matches the timeline axis, keep the long form in the hover title, and show the timezone once in the column header Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(ui): use the root query client for the Lens demo Drop the demo's nested QueryClient. Cache keys are already partitioned by scope, and the root client now skips retries on 4xx ApiErrors, which covers the demo's not-in-demo and read-only rejections and live 4xx alike. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: drop unused synthetic_spend reference from query help Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(traces): drop the deeplite fixtures and map every SDK to a logo The deeplite captures predate the example repository and carried synthetic spend rows, which leaked a fixture-only column into the query help SQL and pinned tests to its shape. Replace them with the SDK captures in the ClickHouse round trip and query API tests, and derive the seed tenant lookup from the capture metadata The runs table only knew the two Anthropic framework slugs. Register the slugs the normalizer emits for LangChain, LangGraph, Deep Agents, CrewAI, Google ADK, LlamaIndex, OpenAI Agents, Pydantic AI, Strands, Vercel AI SDK, Codex, Cursor and Copilot, and fall back to the generic agent glyph when a run has no known framework Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(ui): inject the Lens sample through data sources instead of demo checks Components no longer ask whether they are in the demo. The traces source carries live and handoff, navigation state comes from a URL or memory route, and the preview action comes from context instead of onDemo props Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): infinite scroll for the agent runs list Replace the Load more button with a sentinel that fetches the next cursor page as the list nears its end. Placeholder rows hold the tail while more runs exist, and a failed page stops auto-loading until Retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): keep the whole Lens view in the URL Lens navigation now lives entirely in query params through nuqs: the sample session (demo=true, with a Demo data switch in the header), the open run (trace, trace_ref), the selected step, view and detail section (span, view, span_tab) and the list filters and range (q, agent, status, hours). Any Lens view is a shareable link and the back button walks runs RunView takes its selection injected: the drawer feeds it URL state and the investigations evidence sheet keeps a local one, so a finding's original run never writes step ids into the URL. Leaving the sample session clears every Lens key except the tab so sample ids never point at live data Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * style(traces): cargo fmt captures tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…erriAI#44470) * test(ui): wait for step search value to settle in TraceDrawer test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): dispatch search typing and nav keys deterministically in TraceDrawer test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…erriAI#44473) * feat(ui): extract trace drawer into a shared SidePanel that closes on outside press Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): center Lens mode switch in a notch joined to the content card Larger Traces/Investigations switch, a subtle dot when an investigation is running or queued, and a bigger Lens title with a docs link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): stronger Lens frame border and header spacing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): align Lens notch fillet with the notch border Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): centered Lens loading/error states and proxy JSON calls in dev The dev server answered GET /lens with the Lens page HTML because the UI route shadowed the proxy fallback rewrite. JSON API requests now go to the proxy before page routes, and a non-JSON success body raises a readable ApiError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(ui): theme-scale type and shape in trace views, aligned pane bars Add local/no-arbitrary-design-value, scoped to TraceView and Lens, banning arbitrary font size, tracking, leading, radius, border and CSS property values. Map the Figma-export values onto the theme scale and replace hex colors with info/destructive tokens. Add PaneBar, a fixed-height bordered row, and build the step tree and span detail headers from it so their borders line up across the split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): restore AgentTracesSection emptied in e7c3092 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): move Set up tracing into the empty runs state Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(ui): let the runs table gate the setup CTA on an empty range Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): mark Lens demo mode with a blue toggle and frame instead of a banner Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): draw Lens notch corners with CSS borders and thicken the demo frame The SVG corner strokes did not snap to the same device pixels as the tab and card borders, leaving a visible offset at the join. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(ui): drop the redundant Lens timeline header The status, run counts, truncated agent legend and range span all repeated the Live toggle, runs table and range picker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): keep drawer shortcuts out of open menus and use usehooks-ts for timers and observers SidePanel J/K/Esc now yields to menus and listboxes, not just dialogs. The step tree shortcut footer wraps instead of clipping in narrow columns. Replace hand-rolled timeout, keydown, media query and ResizeObserver effects with usehooks-ts, and drop routine doc comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): offer tracing setup when filters hide every run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): replace Lens runs filters with a single ProseMirror query box Agent and status dropdowns are gone. One query box (react-prosemirror) takes free text plus key:value clauses (-key:value, key:*glob*) over name, agent, status, model, input and trace_id, with field and value autocomplete. The editor emits after a 150ms pause so typing no longer re-renders the runs view per keystroke, and the URL keeps only q. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): keep the open trace on screen while the next one loads Switching to an unvisited trace remounted the panel and flashed a loading skeleton. The drawer now keeps the previous trace visible, dimmed and inert, until the new one arrives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): virtualize the Lens runs table and drop its status footer The footer's run count only tracked how many pages had loaded and "Updated just now" never changed, so it carried no signal. The zoom clear button moves onto the timeline. Rows now render through @tanstack/react-virtual so scrolling deep into a range stays cheap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(ui): load traces with Suspense and show the previous run via useDeferredValue Replaces keepPreviousData with the React pattern for showing stale content while fresh content loads. Load failures go through an error boundary that retries the query on reset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): open investigation details from list rows instead of the edit dialog Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…4476) * fix(lens): document working preview setup before coordinated releases * docs(lens): keep current setup guidance factual and preserve Helm options * fix(lens): support explicit worker images on Compose 2
* feat(sdk): add run_tool_loop and arun_tool_loop helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(tests): allow-list bounded tool-loop recursion in recursive detector Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(sdk): harden run_tool_loop per review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(deps): keep uv.lock at revision 3 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(tests): authorize typing-extensions PSF-2.0 license Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(lens): guide setup through the first investigation * fix(lens): restore the onboarding reference visuals * fix(lens): compact onboarding and animate gateway flow * feat(lens): refine onboarding motion and linked examples * feat(lens): turn the LED swarm into organized dot groups * fix(lens): make the LED dot flow visibly animate * feat(lens): refine the swarm scale palette and motion * fix(lens): preserve investigations during activity refresh errors
* test(ci): fix four CircleCI regressions on main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ci): stop reloading auth_checks in unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(constants): cover CLI JWT expiry env parsing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(gateway): restore proxy lifespan after importing gateway.main in launch tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ERROR shows them (BerriAI#44202) * fix(proxy-extras): log v1 migration failures at ERROR so LITELLM_LOG=ERROR shows them Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy-extras): reuse litellm secret redaction and mask configured DB passwords exactly Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy-extras): name the password alternation in _redact_credentials Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy-extras): wrap v1 migration ERROR lines at 120 columns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy-extras): integration cells for v1 migration ERROR logging and password redaction Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy-extras): integration cells for component DB env vars, JSON logs, migration Job and v2 resolver Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy-extras): give every v1 migration integration cell the 900s timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy-extras): bind the recovery forwarder before migrating so the retry cannot race it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy-extras): drop the slow P3005 integration cell and bound migration subprocesses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy-extras): keep command repr and tolerate non-sequence cmd in migration error logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy-extras): double the subprocess boundary in the cmd=None retry test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng <yucheng@berri.ai>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline findings, I also checked two other candidates this run and ruled them out: the migration-lock rolling-deploy boot race in litellm-proxy-extras/litellm_proxy_extras/migration_lock.py (held_migration_lock uses a non-blocking advisory-lock try that simply yields False on contention rather than blocking a new replica's boot), and a possible managed-file id mix-up in enterprise/litellm_enterprise/proxy/hooks/managed_files.py's _iter_provider_file_id_pairs (iterates rows independently, so a shared provider file id across two accessible rows isn't actually a cross-tenant leak path here).
Extended reasoning...
This run's diff spans CI/build config, Dockerfiles, Rust trace-ingestion/ClickHouse crates, proxy/lens/harness Python, and Prisma migrations; two new CONFIRMED findings (a Dockerfile ARG-override regression and an unbounded ancestor/descendant graph walk in trace resolution) are being posted inline. Given those unresolved findings and the long open history of prior unaddressed inline threads on this PR, approval is not appropriate; this note only adds the two additional candidates investigated and ruled out in this pass so the author knows they were checked.
This pull request has been reviewed before and this review found new issues. Where they share a root cause, one fix may close them together.
Still open from earlier reviews (3):
- 🔴
litellm/harness/sandbox/docker.py:151—Operators using litellm.harness with DockerSandbox get the per-session model-endpoint bearer token, and any other secre… - 🔴
litellm/harness/sandbox/local.py:50—Fix 205f243d widened LocalSandbox's env filter with a bare "CERT" substring, which now also strips non-secret TLS-trust… - 🔴
litellm/harness/sandbox/local.py:212—Callers of a long multi-turn litellm.harness Session (LocalSandbox or DockerSandbox) leak one SubprocessHandle per turn…
If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.
|
|
||
| FROM $LITELLM_RUNTIME_IMAGE AS runtime | ||
| ARG PROXY_EXTRAS_SOURCE | ||
| FROM cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d AS runtime |
There was a problem hiding this comment.
🔴 Operators who build this Dockerfile with --build-arg LITELLM_RUNTIME_IMAGE=<custom-image> silently get the hardcoded chainguard wolfi-base digest instead, losing the override that worked on the base branch. The runtime stage's FROM $LITELLM_RUNTIME_IMAGE AS runtime was replaced with a literal FROM cgr.dev/chainguard/wolfi-base@ sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d AS runtime, while the ARG LITELLM_RUNTIME_IMAGE=... declaration at line 5 is kept, still implying it is configurable. Fix: keep the runtime stage's FROM referencing $LITELLM_RUNTIME_IMAGE (as deploy/lens/Dockerfile:16 still does), or drop the now-unused ARG so the Dockerfile doesn't advertise a build-time override it ignores.
Why this was flagged
Trigger: an operator builds docker/Dockerfile.non_root with --build-arg LITELLM_RUNTIME_IMAGE=<internal-mirror-or-pinned-digest> to swap the runtime base image, a documented-by-ARG capability that worked at the pre-diff commit (confirmed: base commit 4a3f30a line 139 was FROM $LITELLM_RUNTIME_IMAGE AS runtime). After this diff, line 125 hardcodes FROM cgr.dev/chainguard/wolfi-base@ sha256:1d95...ef216d AS runtime, and grep confirms LITELLM_RUNTIME_IMAGE is no longer referenced anywhere else in the file. The build arg is silently ignored: no warning, no error, the image is built from the hardcoded digest regardless of what the caller passed. Base behavior: the same build arg actually changed the runtime base image.
Verification: normal (regression). docker/Dockerfile.non_root line 125 is now FROM cgr.dev/chainguard/wolfi-base@ sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d AS runtime, replacing the base branch's FROM $LITELLM_RUNTIME_IMAGE AS runtime. The ARG LITELLM_RUNTIME_IMAGE=... at line 5 is retained but referenced nowhere else, so --build-arg LITELLM_RUNTIME_IMAGE=<custom> is now silently ignored.
| pub(super) fn call_requests(&self, call: usize) -> Option<Requests<'a>> { | ||
| let wrappers = self.graph.ancestors(call).into_iter().filter(|ancestor| { | ||
| self.kind(*ancestor) == ObservationType::Llm | ||
| && self | ||
| .graph | ||
| .descendants(*ancestor) | ||
| .into_iter() | ||
| .all(|descendant| { | ||
| self.graph.id(descendant) == self.graph.id(call) | ||
| || self.kind(descendant) != ObservationType::Llm | ||
| }) | ||
| }); |
There was a problem hiding this comment.
🔴 Reading a trace with a long parent-child span chain can make a single get_trace/get_trace_page call burn CPU for a very long time, a cost the base branch (no such endpoint) never had. call_requests (resolution.rs:88-99) calls graph.ancestors(call), then for every ancestor returned calls graph.descendants(*ancestor); neither Graph::ancestors nor Graph::descendants (graph.rs:56-85) memoize, so each call re-walks the chain. Invoked once per LLM leaf span, this makes resolution cost grow roughly quadratically to cubically with chain depth, up to MAX_GRAPH_SPANS=100,000 (span_batches.rs:12). Fix: precompute ancestor/descendant relationships once per Resolution so cost stays near-linear regardless of chain shape, covering both call_requests and the similar per-row ancestors() call in resolved_type (resolution.rs:210-213).
Why this was flagged
Trigger: a trace with many spans linked in one long parent-child chain (e.g. a looping agent emitting one span per iteration), up to MAX_GRAPH_SPANS=100,000 (span_batches.rs:12), the only cap enforced before resolve_trace runs. get_trace/get_trace_page (traces-clickhouse/src/reads.rs:293,355) call resolve_trace -> Resolution::new -> call_requests (resolution.rs:88-99): it computes graph.ancestors(call) then, for every ancestor, graph.descendants(*ancestor) (resolution.rs:90-93). Neither Graph::ancestors nor Graph::descendants (graph.rs:56-85) cache results. For chain depth D≈N this makes one call_requests cost O(N) ancestors x O(N) descendants-per-ancestor, invoked once per LLM leaf span, so total resolution cost approaches O(N^3). This synchronous CPU work runs inside the async handler with no yield point, stalling the worker thread servicing it.
Verification: graph.rs:56-65 (ancestors) and graph.rs:67-85 (descendants) each allocate a fresh HashSet+Vec keyed on string span_ids and re-walk from scratch, with no memoization. resolution.rs:88-99 call_requests computes ancestors(call), then calls descendants(*ancestor) for every returned ancestor, making ancestor-list cost O(N) and per-ancestor subtree walks sum to O(N²).
There was a problem hiding this comment.
Beyond the inline findings, I checked three other candidates raised this run and ruled them out: the liteadmin-builder Dockerfile stage's apk add without an explicit USER root (the wolfi-base image at that digest runs as root by default there, consistent with other stages' USER root being defensive rather than required, and the build succeeds in CI); the two new spend-logs index migrations (idempotent SQL, no conflict with already-applied migrations); and the auto-router daily-spend cleanup's date-string cutoff (handled by the existing Cutoff/_cutoff_cast split already used for other tables, not a type mismatch).
Extended reasoning...
This is a very large, long-running PR (15+ pushed versions) spanning Rust tracing/ClickHouse crates, proxy DB migrations, Docker/CI, and a new litellm.harness sandbox feature, with security-sensitive surface (ClickHouse RBAC provisioning, OTLP ingestion limits, secret handling in sandbox env) already flagged in prior rounds. This run surfaced two new CONFIRMED bugs (silent ClickHouse-tracing disablement on upgrade, and a Python 3.9 crash in the new migration-index default) that will post as inline comments. Numerous prior findings across many rounds remain only author-resolved (not independent sign-off), so an outstanding-objections concern persists; decision is defer, not approve.
This pull request has been reviewed before and this review found new issues. Where they share a root cause, one fix may close them together.
Still open from earlier reviews (5):
- 🔴
docker/Dockerfile.non_root:125—Operators who build this Dockerfile with --build-arg LITELLM_RUNTIME_IMAGE=<custom-image> silently get the hardcoded ch… - 🔴
litellm-rust/crates/traces/src/resolve/resolution.rs:99—Reading a trace with a long parent-child span chain can make a single get_trace/get_trace_page call burn CPU for a very… - 🔴
litellm/harness/sandbox/docker.py:151—Operators using litellm.harness with DockerSandbox get the per-session model-endpoint bearer token, and any other secre… - 🔴
litellm/harness/sandbox/local.py:48—Fix 205f243d widened LocalSandbox's env filter with a bare "CERT" substring, which now also strips non-secret TLS-trust… - 🔴
litellm/harness/sandbox/local.py:210—Callers of a long multi-turn litellm.harness Session (LocalSandbox or DockerSandbox) leak one SubprocessHandle per turn…
If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.
| def is_clickhouse_tracing_enabled(settings: object) -> bool: | ||
| if not isinstance(settings, Mapping): | ||
| return False | ||
| typed_settings: Final = STORE_SETTINGS.validate_python(settings) | ||
| store: Final = typed_settings.get("store") | ||
| if not isinstance(store, Mapping): | ||
| return False | ||
| return STORE_SETTINGS.validate_python(store).get("type") == "clickhouse" |
There was a problem hiding this comment.
🔴 Operators who already enabled ClickHouse agent tracing with the previously-shipped config general_settings.tracing.store: clickhouse (a plain string, exactly what base docker/tracing-config.yaml had) get tracing silently disabled after upgrading, with no error or log line. is_clickhouse_tracing_enabled (config.py:13-20) now requires store to be a Mapping with type: clickhouse; a plain string fails isinstance(store, Mapping) and returns False. proxy_server.py:1589 feeds that into manage_tracing(enabled=False, ...) which just yields None (tracing_runtime.py:52-54) with no warning logged either place. POST /v1/traces then 503s with "Agent tracing is not enabled" (tracing_endpoints.py:38), which is misleading since tracing: is in fact set. …
Why this was flagged
…Fix: when store is a non-mapping truthy value (old string format), log a clear deprecation/error instead of silently treating it as disabled, or accept the legacy string form for backward compatibility.
Trigger: an existing proxy config has general_settings.tracing.store: clickhouse (string), the exact format base commit 4a3f30a shipped in docker/tracing-config.yaml, reached via ProxyStartupEvent at proxy_server.py:1583-1589. Bug: is_clickhouse_tracing_enabled (litellm/tracing/config.py:13-20) returns False for a non-Mapping store, so tracing_enabled is False. manage_tracing (litellm/proxy/tracing_runtime.py:46-54) takes the enabled=False branch, yields None, and logs nothing (the info log at line 64 is only on the enabled path). On base, general_settings.get("tracing") being merely truthy was enough to start TraceReceiver.from_env() (base proxy_server.py:11412-11429), so tracing worked. Now POST/GET /v1/traces returns 503 "Agent tracing is not enabled" (tracing_endpoints.py:38) with zero diagnostic indicating the config format changed.
Verification: normal (regression against base). The base commit 4a3f30a shipped general_settings.tracing.store: clickhouse as a plain string in docker/tracing-config.yaml, and base init_tracing enabled tracing precisely on that string: if not isinstance(settings, dict) or settings.get("store") != "clickhouse": return (base proxy_server.py:11413-11415), then logged "Agent tracing enabled (store=clickhouse)".
| def run_migration_job( | ||
| use_migrate: bool = False, | ||
| use_v2_resolver: bool = False, | ||
| setup: Callable[[bool, bool], bool] = setup_database, | ||
| build: Callable[[], bool] = build_request_log_indexes, |
There was a problem hiding this comment.
🔴 Users running litellm-proxy-extras on Python 3.9 (the floor declared by requires-python = ">=3.9" in litellm-proxy-extras/pyproject.toml) get a crash in the new migration-index feature instead of it working. run_migration_job's defaults setup=setup_database and build=build_request_log_indexes (utils.py:1399-1400), and start_request_log_index_build's build=build_request_log_indexes (utils.py:1408), bind the bare class-body name to the raw staticmethod descriptor, not the function, since staticmethod only became directly callable in Python 3.10. Fix: make every sibling-staticmethod default a plain callable at all 3 sites, so run_migration_job() (migrations/run.py:61) and start_request_log_index_build() (prisma_client.py:1021) don't raise TypeError: 'staticmethod' object is not callable.
Why this was flagged
Trigger: litellm-proxy-extras installed and imported under CPython 3.9 (explicitly supported per pyproject.toml), then migrations/run.py:61 calls ProxyExtrasDBManager.run_migration_job(use_migrate=..., use_v2_resolver=...) with no setup/build override, or litellm/proxy/db/prisma_client.py:1021 calls ProxyExtrasDBManager.start_request_log_index_build() with no override. Both rely on utils.py:1399/1400/1408 defaults setup_database/build_request_log_indexes, which at class-body-execution time are staticmethod wrapper objects, not plain functions, since bare-name lookup in a class body bypasses get. Calling setup(...) or build() inside the function body then raises TypeError: 'staticmethod' object is not callable — a hard failure of the migration job's main entrypoint and the proxy's startup index-build thread; no test in this diff exercises the zero-argument call path.
Verification: setup_database (utils.py:1347-1348) and build_request_log_indexes (1380-1381) are @ staticmethod, so the names resolve to staticmethod descriptors in the class namespace.
| #[serde(untagged)] | ||
| pub enum TracingStoreSettings { | ||
| ClickHouse(ClickHouseStoreSettings), | ||
| } | ||
|
|
||
| #[derive(Clone, Default, Debug, Deserialize)] | ||
| #[serde(default)] | ||
| pub struct TracingSettings { | ||
| pub store: Option<TracingStoreSettings>, |
There was a problem hiding this comment.
🔴 Operators upgrading the litellm-rust gateway with their existing general_settings.tracing.store: clickhouse config (a plain string, the exact format docker/tracing-config.yaml shipped pre-diff) now get the whole gateway process crash at startup instead of just tracing being affected. TracingStoreSettings (settings.rs:36-38) is an untagged enum with one variant, ClickHouseStoreSettings, which requires a mapping; a bare string fails to deserialize, confirmed by the new test tracing_settings_reject_string_store. Config::load (lib.rs:90-92) returns that error, and gateway/src/main.rs:47 does Config::load(settings.litellm_config)?, so parsing the whole config.yaml fails and the binary exits non-zero. Fix: do not let an unrecognized/legacy tracing.store shape fail the entire config load — accept the legacy string form or fall back to tracing: None with a logged warning.
Why this was flagged
Trigger: an operator runs the litellm-rust gateway against a config.yaml containing general_settings:\n tracing:\n store: clickhouse — the exact format this PR's own docker/tracing-config.yaml shipped before being rewritten to the object form. Before this diff, GeneralSettings had no tracing field at all, so that key landed in the catch-all additional_fields flatten bag and parsing always succeeded. After this diff, tracing: Option<TracingSettings> is strongly typed (settings.rs:58), and TracingStoreSettings (settings.rs:36-38) is #[serde(untagged)] with only a ClickHouse(ClickHouseStoreSettings) mapping variant, so a bare string fails deserialization — proven by the new test tracing_settings_reject_string_store. Config::load (lib.rs:90) propagates that Err, and gateway/src/main.rs:47 does Config::load(settings.litellm_config)?, so main() returns Err and the process exits immediately: the whole gateway never starts, not merely tracing being disabled.
Verification: normal — this change turns a booting gateway into a startup crash for the config format the repo itself shipped in the base branch. Base GeneralSettings had no tracing field, so store: clickhouse parsed via the flatten catch-all. After the diff, TracingStoreSettings (settings.rs:36-39), an untagged enum requiring a mapping, rejects the bare string, and gateway/src/main.rs:47 Config::load(...)? propagates that error, crashing the whole gateway at startup.
There was a problem hiding this comment.
This pull request has been reviewed before and this review found new issues. Where they share a root cause, one fix may close them together.
Still open from earlier reviews (7):
- 🔴
docker/Dockerfile.non_root:125—Operators who build this Dockerfile with --build-arg LITELLM_RUNTIME_IMAGE=<custom-image> silently get the hardcoded ch… - 🔴
litellm-proxy-extras/litellm_proxy_extras/utils.py:1400—Users running litellm-proxy-extras on Python 3.9 (the floor declared by requires-python = ">=3.9" in litellm-proxy-extr… - 🔴
litellm-rust/crates/traces/src/resolve/resolution.rs:99—Reading a trace with a long parent-child span chain can make a single get_trace/get_trace_page call burn CPU for a very… - 🔴
litellm/harness/sandbox/docker.py:151—Operators using litellm.harness with DockerSandbox get the per-session model-endpoint bearer token, and any other secre… - 🔴
litellm/harness/sandbox/local.py:48—Fix 205f243d widened LocalSandbox's env filter with a bare "CERT" substring, which now also strips non-secret TLS-trust… - 🔴
litellm/harness/sandbox/local.py:210—Callers of a long multi-turn litellm.harness Session (LocalSandbox or DockerSandbox) leak one SubprocessHandle per turn… - 🔴
litellm/tracing/config.py:20—Operators who already enabled ClickHouse agent tracing with the previously-shipped config general_settings.tracing.stor…
If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🟣
litellm/proxy/_lazy_features.py:379— pre-existing (security-relevant, in code this diff substantially rewrites): on the new split gateway/backend deployment, any single request for a path owned by a lazy feature that the split excludes (e.g./guardrails/applyon the gateway, a backend-only admin surface per gateway/routes/allowlist.py) makes LazyFeatureMiddleware.call (_lazy_features.py:358-379) import and app.include_router() that feature's full router at runtime via _register_feature (line 484), undoing the one-time route trim gateway/main.py:69 / backend/main.py:58 already applied, and then serving the request. …Why this was flagged
…Fix: have the middleware (or register_fn) check the active component's allowlist (gateway/routes/allowlist.py or backend/routes/allowlist.py) before registering, so lazy-loading can never reintroduce a route the split excluded.
Trigger: an operator runs the split deployment (gateway.main:app / backend.main:app, new Dockerfiles/helm in this diff) and a caller sends any request whose path matches a LAZY_FEATURES prefix the other component owns, e.g. GET /guardrails/apply on the gateway. LazyFeatureMiddleware.call (_lazy_features.py:358-379) matches on LAZY_FEATURES only, not on GATEWAY_PATH_PREFIXES/BACKEND_PATH_PREFIXES, so it calls _force_load -> _register_feature (line 445-500) which app.include_router()s the admin router into the live gateway app, reversing gateway/main.py:69's startup-time route trim. The request is then dispatched to and served by that now-registered admin router on the gateway. _eager_route_wins (line ~413) does not stop this: the reserved anchor route was itself removed by the trim, so it treats every remaining route as 'ahead' and still proceeds to load.
Verification: Severity: pre-existing (real and reachable, but the base branch fails identically by the same route — does not block).
The behavior is real. proxy_server.py:20141 calls attach_lazy_features(app) on the single shared app object that both gateway/main.py and backend/main.py import (
from litellm.proxy.proxy_server import app). With lazy routes enabled (default), that attaches LazyFeatureMiddleware to the ASGI stack.
There was a problem hiding this comment.
Beyond the inline finding on deploy/lens/compose.yaml, I checked two other candidates raised this run. The new spend_logs index migrations (20260823000000 and 20260831120001) are deliberate no-op placeholders (SELECT 1) — actual index creation is deferred to request_log_indexes.py, so neither fails prisma migrate deploy for already-upgraded operators. And the baseline-accounting "missing spend log" path (litellm/proxy/db/baseline_accounting.py:580-587) returns "unavailable" and retries on the next flush rather than permanently dropping savings, so that understatement is transient, not permanent.
Extended reasoning...
This run's two ruled-out candidates were verified directly against the code: the migration SQL files are confirmed no-op placeholders with index building moved to request_log_indexes.py, and baseline_accounting.py's project() method explicitly returns "unavailable" on a missing spend log and relies on flush_baseline_accounting's retry-on-exception comment to re-attempt later, rather than silently and permanently under-crediting savings. These are informational only; the large outstanding thread of prior unresolved/author-only-resolved objections on this PR already means a human look remains warranted regardless.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Still open from earlier reviews (8):
- 🔴
docker/Dockerfile.non_root:125—Operators who build this Dockerfile with --build-arg LITELLM_RUNTIME_IMAGE=<custom-image> silently get the hardcoded ch… - 🔴
litellm-proxy-extras/litellm_proxy_extras/utils.py:1400—Users running litellm-proxy-extras on Python 3.9 (the floor declared by requires-python = ">=3.9" in litellm-proxy-extr… - 🔴
litellm-rust/crates/config/src/settings.rs:44—Operators upgrading the litellm-rust gateway with their existing general_settings.tracing.store: clickhouse config (a p… - 🔴
litellm-rust/crates/traces/src/resolve/resolution.rs:99—Reading a trace with a long parent-child span chain can make a single get_trace/get_trace_page call burn CPU for a very… - 🔴
litellm/harness/sandbox/docker.py:151—Operators using litellm.harness with DockerSandbox get the per-session model-endpoint bearer token, and any other secre… - 🔴
litellm/harness/sandbox/local.py:48—Fix 205f243d widened LocalSandbox's env filter with a bare "CERT" substring, which now also strips non-secret TLS-trust… - 🔴
litellm/harness/sandbox/local.py:210—Callers of a long multi-turn litellm.harness Session (LocalSandbox or DockerSandbox) leak one SubprocessHandle per turn… - 🔴
litellm/tracing/config.py:20—Operators who already enabled ClickHouse agent tracing with the previously-shipped config general_settings.tracing.stor…
If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.
| services: | ||
| lens-worker: | ||
| image: ${LENS_WORKER_IMAGE:-ghcr.io/berriai/litellm-lens-worker@sha256:40fdb82113dd4474cb6e833cf28552487d87c8baf61693a1c3fc2863b7968c6a} | ||
| image: ${LENS_WORKER_IMAGE:-${LITELLM_VERSION:+ghcr.io/berriai/litellm-lens-worker:v}${LITELLM_VERSION:-}} |
There was a problem hiding this comment.
🟡 nit (optional): operators who run docker compose -f deploy/lens/compose.yaml up without LENS_WORKER_IMAGE or LITELLM_VERSION set get a bare Docker "invalid reference format" error instead of the working pinned-digest image the base branch shipped. image: ${LENS_WORKER_IMAGE:-${LITELLM_VERSION:+ghcr.io/berriai/litellm-lens-worker:v}${LITELLM_VERSION:-}} resolves to an empty string when both vars are unset, unlike LITELLM_URL/LENS_WORKER_TOKEN on lines 5-6 which use the :? form with a clear message. Fix: give the image field the same :?Set LENS_WORKER_IMAGE... required-variable treatment (or restore a real default) so the failure matches the file's own convention instead of silently producing an empty image reference.
Why this was flagged
Trigger: a user downloads deploy/lens/compose.yaml directly and runs docker compose up without exporting LENS_WORKER_IMAGE or LITELLM_VERSION (e.g. missed the README step at deploy/lens/README.md requiring 'an explicit LENS_WORKER_IMAGE'). compose.yaml:3's default ${LITELLM_VERSION:+...}${LITELLM_VERSION:-} evaluates to '' when LITELLM_VERSION is unset, so image: "". Base branch had a working fallback: a pinned digest ghcr.io/berriai/litellm-lens-worker@ sha256:40fdb8.... Now the service fails with a cryptic 'invalid reference format' from Docker instead of either working or a clear ':?'-style required-var message, which lines 5-6 in the same file use for LITELLM_URL/LENS_WORKER_TOKEN.
Verification: On deploy/lens/compose.yaml:3, when both LENS_WORKER_IMAGE and LITELLM_VERSION are unset, the default expands to "", so image is empty and docker compose up fails with a bare "invalid reference format" error. This is inconsistent with lines 5-6, which use the :? form with explicit operator messages. The diff confirms the base shipped a concrete digest fallback that was removed.
20 participants
Full -X theirs sync of https://github.com/BerriAI/litellm main (now at HEAD, 252 commits since litellm_internal_staging).
2026-10-04 refresh of this sync PR: 71 more upstream commits had landed since the 2026-10-03 merge (which was green but not yet mergeable at the start of today's run). Re-merged
upstream/maininto this branch and reconciled the new conflicts and merge corruptions:Conflict resolutions (resolved toward upstream, per the fork's own policy on modify/delete conflicts):
Fork patches re-applied after today's merge wiped or broke them:
RouterErrors/RouterRateLimitError/RouterRateLimitErrorBasic/_get_deployment_orderimports the merge silently dropped from this file's import block.decideclosure in favor of the module-level_decidehelper trio; today's further upstream commits flipped which copy is live, leaving both the now-dead trio and an orphaned= appliedunpack referencing a variable that no longer exists. Upstream's current file has already done the same dead-code cleanup, so restored verbatim rather than hand-patching.attempts=min(64, max(8, job.settings.concurrency * 4))scaled retry budget on thereserve_budgetCAS loop, dropped when upstream wrapped it in@asynccontextmanagerwith settle-on-failure handling this sync.runtimestage'sFROM $LITELLM_RUNTIME_IMAGEback to a literal digest matching its ARG default (reverted to the vanilla$VARform by this sync).Dropped/retired (superseded by upstream, not fork patches anymore):
OTLP_MAX_SPANS+ the span.rs "keep first N spans" truncation patch: upstream's own newDecodeLimits/Budgetsystem (litellm-rust/crates/traces/src/otlp/limits.rs) ships its own test suite asserting every limit dimension fails fast and uniformly on overflow (environment_decode_limits_are_used_and_invalid_values_fail,configurable_decode_limits_apply_to_both_wire_formats). Restoring yesterday's "truncate gracefully" patch broke those 2 upstream tests, so reverted to upstream'sErr(Error::TooLarge)and deleted the fork's own now-superseded regression test. The underlying OTel-Collector-default-batch-size concern is still real but is now an operator-config fix (OTLP_MAX_SPANS>=8192), not a code patch.python:3.12-slimtag onto the same wolfi-base ARGs the main Dockerfile uses, already digest-pinned at the ARG default.Verification before pushing:
python3 scripts/verify_fork_patches.py: OK, all 234 current-assertion patches present, all literal digest pins current.cargo check --workspaceandcargo test -p litellm-traces --test otlp(166 passed) in litellm-rust/.ruff check --select F821,F401across litellm/ and litellm-rust/: clean (no new undefined-name/unused-import regressions).pyteston the touched areas: lens/analysis, lens/inference, fallback_event_handlers (142 passed), router tests (1861 passed), tracing tests (92+ passed), proxy routes/realtime/http-parsing (212 passed).Follow-up push after the first CI run on this head: fixed a ruff-format violation in litellm/harness/sandbox/local.py (a nearby upstream line shifted this sync), and quarantined a new fork-infra gap: upstream's brand-new
roi-databasetest shard (BerriAI#44426) hit the same un-registered-Codecov-repo gap as the existing Lens coverage upload, so addedcontinue-on-error: trueto its own coverage-upload step, same fix already applied there.Second follow-up: the merged head also breached all 3 lint-budget gates documented in REPO_RECURRING_FIXES (ruff-strict, type-discipline, basedpyright), each confirmed as upstream's own growth (not fork hunks) via a base-vs-head diff and raised per that section's procedure, with two calling out pre-existing drift already present before this sync touched anything: ruff-strict C901 306->308, TID251 1035->1037; type-discipline LIT001 22174->22233, LIT010 16398->17885 (base already 1171 over before this sync); basedpyright reportUnusedFunction 136->141, reportUnusedVariable 137->201 (base already 58 over before this sync). All three gates re-run clean locally after the raise.
🤖 Generated with Claude Code
https://claude.ai/code/session_011vvp2gsj31HM4iGNhKKvk6
Generated by Claude Code