Repository navigation
docs(proxy): document PTU shares per team and the per-team PTU ceiling - #1700
Merged
Merged
Conversation
…PTU-hours on team usage
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Author
|
bugbot run |
…rvations get the sizing warning
Contributor
Author
|
bugbot run |
11 tasks done
… quarter of the share
Contributor
Author
|
bugbot run |
Contributor
Author
|
bugbot run |
Contributor
Author
|
bugbot run |
…r_hour is refused
Contributor
Author
|
bugbot run |
…rovider model is sized like the group
Contributor
Author
|
bugbot run |
Contributor
Author
|
bugbot run |
… wildcards included
Contributor
Author
|
bugbot run |
Contributor
Author
|
@litereview code-review |
Contributor
Author
|
bugbot run |
…es, and unfittable requests
Contributor
Author
|
bugbot run |
…icit max_tokens cap, large output budgets, and dated PTU-hours
Contributor
Author
|
bugbot run |
…am rows keep ptu_hours at 0
Contributor
Author
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 8c4abab. Configure here.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Documents the per-team PTU shares shipping in BerriAI/litellm#43043:
model_info.ptu_shareson an Azure provisioned deployment, the per-minute ceiling each share becomes, the flat cost split by share, andptu_hoursonGET /team/daily/activityand the Team Usage pagedocs/proxy/ptu_flat_cost.mdgains a "Share a deployment across teams" section (the API and config.yaml shapes, the exclusivity withteam_idand how a deployment one team already owns is split (delete it and register it again with the samemodel_info.id), the sum rule, who can call the deployment, the ceiling formula and its 429, the names and routes that count against it with wildcards included,base_model), the cost split in "How the cost is calculated",ptu_hoursin "Read the cost back",ptu_sharesandbase_modelin the field table, and three limitations.docs/proxy/dynamic_rate_limit.mdgains a short section pointing at it, since the ceiling replaces a hand-convertedmodel_tpm_limit. "Read the cost back" also says a usage row keyed by an alias, a routing group, a deployment id, or a provider model is sized like the group, and that one team's page sizes such a row by the deployment that team is served from, never one reserved for another team, while a page spanning teams uses the first reserved deployment behind the name, matching the read-path fixes in the code PR. The section also says a team holding a share cannot be removed throughDELETE /organization/deleteor a SCIM group delete either, that a small share writesmax_tokensfor an uncapped request (987 on 3 PTUs of gpt-5.4-mini, 100 on 1 PTU of gpt-5.4), what share a largemax_tokensneeds, and that PTU-hours count only inside a reservation's effective windowMerge after BerriAI/litellm#43043 lands so the docs never lead the code
Type
Note
Low Risk
Documentation-only changes with no runtime or security impact.
Overview
Documents Azure PTU multi-team sharing (
model_info.ptu_shares) and the proxy behaviors that ship with it, intended to merge after the implementation PR.ptu_flat_cost.mdis expanded with a Share a deployment across teams guide:ptu_sharesvsteam_id, validation and access rules (including team/org/SCIM delete guards), how flat cost splits per share,ptu_hourson/team/daily/activityand Team Usage, secret-manager override forLITELLM_ENABLE_PTU_COST_ATTRIBUTION, andbase_modelfor ceiling sizing. A nested section explains the per-minute normalized-token ceiling (Azure sizing table, 429/400 behavior,max_tokensrewriting on small shares, routing aliases/wildcards, and what routes bypass the limit).dynamic_rate_limit.mdadds a short cross-link showing PTU shares as an alternative to hand-tunedmodel_tpm_limit. Limitations call out PTU-hours scope, mixed model groups, alias sizing on usage pages, and no Admin UI editor for shares yet.Reviewed by Cursor Bugbot for commit 8c4abab. Bugbot is set up for automated code reviews on this repo. Configure here.