chore: litellm oss staging - #31185
Merged
Merged
Conversation
…abels aren't clipped
The Total Tokens Over Time and Total Requests Over Time AreaCharts on the
Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow
once totals pass the hundred-million mark — leading digits of labels like
"100.00M" / "4500.00M" got clipped against the chart edge. The requests
chart was worse: it formatted with toLocaleString(), so billion-scale
request counts produced "1,000,000,000" (13 chars) and overflowed
immediately.
Fix in two places so neither alone has to carry the whole margin:
- activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and
switch the requests chart to the shared valueFormatter so it uses the
same compact k/M/B suffixes as the tokens chart.
- value_formatters.tsx: add a >= 1e9 branch to valueFormatter /
valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping
every formatted label at most 7 chars.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
…atters.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
[Infra] Promote internal staging to main
[Infra] Promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
Adds a quickstart for the two published Terraform modules on the public registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste main.tf for each cloud, the one-time GCP Artifact Registry remote-repo command, and pointers to the registry pages for the full input surface. Sits inside the Get Started section, between the gateway/SDK table and Run in Developer Mode -- where someone scanning the README for "how do I deploy this" will land. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo and walks through `terraform apply` via the existing DeployStack tutorial (already shipped at terraform/litellm/gcp/examples/default/ TUTORIAL.md). User just picks a project. AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an in-browser, already-authenticated shell. User runs four commands (clone + cd + cp tfvars + terraform apply) once inside. There's no native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN "Launch Stack" + CodeBuild would be needed for that, and that's a separate piece of work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…on font GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px. Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.
The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.
Backports only the model map changes from #30064 so deployments on released litellm versions pick up Fable 5 pricing, context window, and the adaptive thinking flag through the hosted cost map fetch without upgrading. Includes the supports_sampling_params flag on the 28 Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read by the gating that ships with the next release) and the matching one-line schema declaration so the map validation test passes. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map)
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
Three bugs in model_prices_and_context_window.json: 1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K max output, but the values were set as max_input=128000, max_tokens=272000. This caused token limit errors when sending prompts over 128K tokens to GPT-5 Pro. 2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was 272000, but GPT-5.4 Mini shares the same 1,050,000 token context window as GPT-5.4. This was inconsistent with the azure/ variants which already correctly had 1,050,000. 3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini, max_input_tokens was 272000 instead of 1,050,000. Source: OpenAI model documentation and contextwindows.dev which aggregates official context window sizes. Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini should be 400K; their 272K values are correct per OpenAI docs)
Per reviewer feedback, max_output_tokens was left at 272000 while max_tokens was corrected to 128000, causing an internal inconsistency. Both should be 128000 per OpenAI docs.
Contributor
|
bugbot run Generated by Claude Code |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 4 issues found in the latest run.
- ✅ Fixed: Unparseable marker triggers catch-up
- Catch-up now treats only empty metrics markers as fresh connectors and skips catch-up for unparseable non-empty markers.
- ✅ Fixed: Upload succeeds marker not updated
- Metrics marker PATCH failures now raise so delivery does not complete with stale connector state.
- ✅ Fixed: Dangling tool_choice after tool drop
- DeepSeek tool sanitization now removes a specific tool_choice when it no longer matches a surviving function tool.
- ✅ Fixed: Thinking blocks only at signature
- Anthropic streaming thinking deltas now emit thinking_blocks alongside reasoning_content before the signature arrives.
You can send follow-ups to the cloud agent here.
Contributor
Optional router strategy that picks a deployment tier from request_kwargs.metadata.lar1 (confidence, evidence, time). Deployments are tagged with model_info.type (cloud-smart, cloud-fast, local, deep). Thresholds are configurable via routing_strategy_args. Includes 30 unit tests and an Ollama example config. Co-authored-by: Cursor <cursoragent@cursor.com>
feat: add LAR-1 semantic routing strategy
Contributor
Author
Contributor
|
It looks like your message got cut off. Could you clarify what you'd like to ask or discuss? |
When deliver() receives empty content (no spend data for a date), it now registers with Mavvrik and PATCHes the metricsMarker before returning instead of short-circuiting. Dates with zero spend no longer stall marker advancement, preventing unnecessary catch-up API calls on subsequent runs.
Contributor
Author
Contributor
Author
Contributor
|
bugbot run |
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Empty export skips marker
- Mavvrik empty export windows now call the destination with an empty payload so deliver advances the metrics marker.
- ✅ Fixed: LAR-1 routing sticks after switch
- Routing strategy initialization now removes instance-bound custom selectors before switching back to built-in routing strategies.
You can send follow-ups to the cloud agent here.
…itellm_oss_staging # Conflicts: # litellm/proxy/spend_tracking/spend_management_endpoints.py
Contributor
Author
Contributor
Author
Contributor
Author
|
@greptileai review |
…n't leave partial state apply_lar1_routing_strategy set router.routing_strategy to "lar1" before constructing LAR1RoutingStrategy, whose __init__ validates thresholds via _normalize_thresholds and raises on a misconfigured (out-of-order or out-of-range) set. On a live update_settings call with bad thresholds the router was left advertising routing_strategy="lar1" with no custom selector bound, while the previous strategy's selectors stayed registered. Build (and validate) the strategy before mutating any router state, so a threshold error leaves the router exactly as it was. Add a regression test that asserts a failed switch keeps the prior strategy intact.
Contributor
Contributor
|
bugbot run Generated by Claude Code |
Contributor
|
@greptileai the summary still lists two concerns; walking through the current head (26ac40c), both look stale. Mavvrik catch-up double-export: for a fresh connector the marker is empty/0/None, so LAR-1 sync Neither is a functional bug, consistent with the 5/5 Generated by Claude Code |
Contributor
|
Both explanations are correct — I was wrong on both points.
Neither is a bug. The 5/5 confidence stands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
make test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes
Note
Medium Risk
Touches billing export state (Mavvrik markers), auth-adjacent guardrails on rerank, and several high-traffic chat/routing paths (Bedrock, DeepSeek, Copilot, Anthropic streaming), plus a very large pricing JSON diff where catalog mistakes could misreport costs.
Overview
This staging merge bundles provider and proxy fixes with a large model pricing/catalog refresh.
Mavvrik FOCUS export now PATCHes
metricsMarkerafter each successful (or intentionally empty) daily upload, treats 0/None markers as fresh connectors for catch-up, widens export windows tonowso late-flushed spend rows are included, and eager-inits the Mavvrik logger at scheduler startup when only the"mavvrik"callback string is configured.GitHub Copilot / OpenAI SDK path adds
transform_parsed_response_dicton the base chat config and wires it throughopenai.py, synthesizing OpenAI-stylechoicesfrom Anthropic-native Copilot responses when missing.Bedrock routes application inference profile ARNs to converse and adds
amazon.titan-embed-g1-text-02embedding support. DeepSeek strips non-functiontools (and invalidtool_choice) before/chat/completions. Anthropic streaming refines thinking /reasoning_contenthandling in deltas.Rerank gains an
instructionparameter across configs; Cohere rerank guardrails scaninstructionas well asquery. Moonshot fixestool_choice: requiredso the synthetic user message is not appended twice.Adds
examples/lar1_ollama_config.yamlfor LAR1 multi-Ollama routing.cold_storage_handleraccepts an injectable cold-storage logger for tests.model_prices_and_context_window_backup.jsonupdates SambaNova, Snowflake, Scaleway, Pinstripes, search providers, deprecations, and related entries.Reviewed by Cursor Bugbot for commit 26ac40c. Bugbot is set up for automated code reviews on this repo. Configure here.