Repository navigation
feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting - #43043
devin-ai-integration[bot] wants to merge 34 commits into
Conversation
A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on, routing hides the deployment from teams holding no share, the v3 limiter enforces each share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team Usage page report PTU-hours next to tokens.
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…r_team # Conflicts: # tests/test_litellm/proxy/management_endpoints/test_team_endpoints.py
Merging this PR will not alter performance
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…deployments by public name The PTU flag is read from the process environment on every check instead of through the secret-manager path, so a proxy with a hosted secret manager does not pay a round trip per request. A team's ceiling now carries the model's cached-input ratio and settlement charges cache reads at that ratio, matching how Azure sizes a PTU. A deployment registered for one team is found by its team_public_model_name for PTU-hours and the ceiling, since its model_name is rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and parsed_ptu_shares refuses booleans, strings, and non-string team ids, a fractional ptu_count reserves nothing, and models_processed counts distinct deployments rather than holdings.
…ting flag-independent, and warn only shared or Azure reservations
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Comments Outside DiffThese findings could not be posted inline.
|
…aw output tokens and escape the exported team id
|
bugbot run |
1 similar comment
…r picks, and a deployment id before any wildcard A team's wildcard reservation behind an open proxy-wide * route was charged and sized for traffic the * route served, and naming a shared deployment by id next to a catch-all wildcard escaped its per-team ceiling. Router.wildcard_route_deployments now owns the most-specific-proxy-wide-else-team precedence, and both the limiter and the consumption page resolve through it after listed rows and deployment ids.
|
bugbot run |
# Conflicts: # litellm/types/router.py
…count already does A share of 2.0 now reads as 2 on POST /model/new and in config.yaml, where it was refused with a 422 and dropped at load. Fractional, infinite, boolean and string shares stay refused. Adds unit tests for the malformed-rate, provider-over-prefix, zero-token Responses usage, fractional-count and sizing-warning paths.
| return self._build_reservation_aware_tpm_ops( | ||
| targets=reserved_ptu_scopes, | ||
| reserved_scopes=reserved_scopes, | ||
| actual_tokens=self._ptu_settlement_tokens( | ||
| reserved_ceiling, self._resolve_reconciled_usage(usage_source), total_tokens | ||
| ), | ||
| reserved_tokens=reserved_tokens, |
There was a problem hiding this comment.
A PTU request that finishes after its minute expires can refund tokens into the next minute. _build_team_ptu_tpm_ops subtracts the old reservation from the current counter without checking which minute owns it. If another request has started a new window, the late refund reduces that request's counter and lets the team exceed its share.
Save the window returned at admission and check it before applying settlement or refunds, as the project token limits already do.
| team_id=team_id, | ||
| ptu_count=share, | ||
| cost_per_ptu_per_hour=terms.cost_per_ptu_per_hour, | ||
| effective_from=terms.effective_from, | ||
| effective_to=terms.effective_to, | ||
| ) | ||
| for team_id, share in sorted(terms.shares.items()) |
There was a problem hiding this comment.
Reassigning a share to a new team can add a second charge to already billed days. _parse_ptu_models gives the new holder the deployment's original effective_from. The nightly backfill adds that holder's missing historical charges while retaining the former holder's rows.
For example, changing a 30/20 split from A/B to A/C makes older days carry 30 PTUs for A, 20 for B, and another 20 for C. Keep already attributed deployment-days unchanged when backfilling a changed split, or store when each split takes effect.
| model: Final = self._rate_limited_model(requested_model) | ||
| if model is None or not user_api_key_dict.team_id: | ||
| return | ||
| ceiling: Final = self._ptu_team_ceiling_resolver(user_api_key_dict.team_id, model.group) | ||
| if ceiling is None: | ||
| return |
There was a problem hiding this comment.
A fallback from an open model to a shared PTU group skips the group's ceiling at admission. _add_team_ptu_rate_limit_descriptor checks only the original requested model. Upstream failures switch groups inside run_async_fallback, which calls the router again without rerunning the proxy's limiter hook. A holder can therefore reach its shared deployment even when its PTU counter is full.
Check and reserve the selected shared group's ceiling before each fallback attempt, rather than only before the original request.
There was a problem hiding this comment.
Team per-model TPM limits on main also check only the requested model. Re-checking fallback hops is a separate limiter change, listed in Caveats
| page = _ptu_activity_page() | ||
|
|
||
| with ( | ||
| patch("litellm.proxy.management_endpoints.team_endpoints.get_daily_activity", AsyncMock(return_value=page)), |
There was a problem hiding this comment.
The new team activity tests replace LiteLLM's own get_daily_activity with an AsyncMock. The unit-test guide says “No doubles on our own code.” These tests cannot catch a change that stops the real query from supplying the model-group rows PTU reporting needs.
Use the real activity builder with a fake database boundary before merging to satisfy the repository requirement. The same pattern appears at lines 17156 and 17188.
File Used: tests/unit/AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| return ModelResponse( | ||
| id="ptu-share", | ||
| object="chat.completion", | ||
| created=int(datetime.now().timestamp()), |
There was a problem hiding this comment.
_ptu_response reads the real clock even though the new tests need only a fixed response timestamp. The unit-test guide requires “No network, clock or subprocess.” Use a fixed created value before merging to satisfy that requirement and keep these tests independent of the machine's time.
File Used: tests/unit/AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
… billed PTU days when shares move
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| requested_model=requested_model if isinstance(requested_model, str) else None, | ||
| descriptors=descriptors, | ||
| ) | ||
| self._add_team_ptu_rate_limit_descriptor( |
There was a problem hiding this comment.
Low: Native Azure requests bypass PTU ceilings
Native Azure inference (POST /azure/openai/deployments/<group>/chat/completions) goes through _relay_router_model → llm_router.allm_passthrough_route without running this proxy pre-call limiter. A share-holding team can therefore keep consuming the deployment after exhausting its PTU ceiling; apply the same PTU reservation before forwarding native requests and reconcile it when they complete.
| servable: Final = team_servable_deployments( | ||
| routed_deployments(listed_rows, deployments, requested_model, wildcard_route), team_id | ||
| ) | ||
| return next( |
There was a problem hiding this comment.
Medium: Routing groups charge the wrong PTU allowance
With shared groups A and B, A first in model_list, and a callable routing group that prioritizes B, this next(...) resolves the request to A's ceiling even though the router selects B. An authorized caller can exhaust B through its direct model name and continue consuming B through the routing-group name because admission and settlement charge A instead; reserve against the selected deployment's PTU scope, or enforce the relevant candidate scopes.
| at the model's cached ratio, output weighted by the output-to-input ratio.""" | ||
| cached: Final = min(max(cache_read_tokens, 0), max(prompt_tokens, 0)) | ||
| uncached: Final = max(prompt_tokens, 0) - cached | ||
| return uncached + weights.cached_input_ratio * cached + weights.output_to_input_ratio * max(completion_tokens, 0) |
There was a problem hiding this comment.
Low: Long-context GPT-6 requests undercount PTU usage
Azure's long-context GPT-6 PTU weights are 2 for uncached input and 7.5 for output, but this normalizer always uses input weight 1 and the table's output weight 5. A caller with a 200-PTU GPT-6 Sol share (600,000 normalized tokens/minute) can submit 400,000 uncached input tokens with a 10,000-token output cap: the current estimator reserves 550,000 and settlement charges 450,000, while that usage consumes 875,000 under the long-context weights. Apply context-tier weights consistently to admission, settlement, and PTU-hour reporting.
PR overviewThis PR adds support for sharing Azure PTU deployments across teams, with per-team PTU ceilings and PTU-hour usage reporting. Authorized teams can exceed their PTU allocations through native Azure requests or routing groups that charge the wrong deployment's allowance, consuming shared capacity beyond their intended limits. Long-context GPT-6 usage is also undercounted, weakening ceiling enforcement and understating PTU-hour usage. All three issues remain open, with none addressed yet. Open issues (3)
Fixed/addressed: 0 · PR risk: 5/10 |
|
@litereview code-review |
1 similar comment
|
@litereview code-review |
Review already requestedcode-review is already reviewing |
✅ code-review5/5 · Ready for human reviewScope: full · Full PR review. No correctness findings. The earlier backfill finding is fixed. Each deployment-day's team rows now go through one Prisma batch transaction, and a regression test covers a partial failure. The 4 failures in Checks and execution details
View review · Reviewed |
![litereview[bot]](https://avatars.githubusercontent.com/in/5255007?s=60&v=4){kind=small}
| return tuple(start + timedelta(days=offset) for offset in range((end - start).days + 1)) | ||
|
|
||
|
|
||
| def _unpriced_charges( |
There was a problem hiding this comment.
Medium · Backfill skips a split deployment's day when one team's charge failed to write
- An admin splits a deployment with
ptu_shares: {"team-a": 30, "team-b": 20}. The daily rollup then writes one sentinel row per team. - One night, team A's upsert lands. Team B's upsert uses up its retries because of a transient DB error. The rollup alerts and moves on.
- The scheduled catch-up pass (
run_ptu_flat_cost_backfill) treats a day as priced once a sentinel row exists for that deployment id and date for any team. This PR droppedteam_idfrom_existing_sentinel_keys, and_unpriced_chargesfilters on(model_id, date). - Team A's row marks the day as priced, so the backfill never writes team B's 480 charge. Team B's usage keeps showing no flat cost for that day unless someone notices the alert and reruns that date by hand. Before this PR, a failed write left no row behind, and the next night's backfill filled the day in automatically.
You can keep the reassignment guarantee and still catch partial days: treat (model_id, date) as priced only when the stored rows' ptu_flat_cost adds up to the deployment's full daily cost. Another option is to record the share map that was billed.
Evidence
Reproducer: a temporary test in tests/unit/proxy/spend_tracking/ that reuses the existing _FakeSentinelTable and _prisma_for helpers. It seeds only ("team-a", 2026-07-30, "shared") for a 30/20 split effective from that day, then calls run_ptu_flat_cost_backfill(prisma, today=TODAY).
Observed output: upserts: [] rows_written: 0. The assertion that team-b's row gets written fails.
_existing_sentinel_keys now returns frozenset[(model, date)], and _unpriced_charges checks (charge.model_id, day.isoformat()) not in priced.
_run_and_alert shows the backfill as the automatic path for unpriced days: if target_date is None: await _backfill_and_alert(...).
The PR adds test_backfill_does_not_bill_a_priced_day_again_after_its_shares_moved_to_other_teams, which covers reassignment. No test covers a partial write within one split day.
Latest summary by code-review · View this review · Reviewed 17abc6e26c58
…action The backfill counts a deployment day as priced once any team's row exists for it, so a rollup that wrote team A's share and lost team C's left team C unbilled for good. Each deployment day's team rows now go through one Prisma batch with the retry around the whole batch, so a day is fully written or not written at all and the next backfill prices it for every team.
|
@litereview code-review |
|
bugbot run |
|
@litereview code-review |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 8a97ba4. Configure here.
This branch has not been deployed
1 participant
TLDR
Problem this solves:
How it solves it:
model_info.ptu_sharessplitsptu_countacross teams in whole PTUsDocs for the new fields, the ceiling, and the cost split are in BerriAI/litellm-docs#1700 (
docs/proxy/ptu_flat_cost.mdanddocs/proxy/dynamic_rate_limit.md), to merge after this PRUser Flow
Before: a proxy admin cannot split one PTU deployment across two departments, and neither department ever sees PTU consumption
model_info: {"ptu_count": 50, "cost_per_ptu_per_hour": 1.0, "ptu_effective_from": "2026-09-23T00:00:00Z", "ptu_shares": {"<team-a>": 30, "<team-b>": 20}}and gets 400team_id is required when PTU fields are set (one model maps to one team)"team_id": "<team-a>"instead, and team B's key then gets 400Invalid model name passed in model=gpt-4.1-ptuon POST https://litellm-domain/v1/chat/completionsmetadata.model_tpm_limitthrough POST https://litellm-domain/team/update to a TPM number they worked out by hand, and nothing keeps that number in step with the reservationflat_costunder team A, nothing under team B, and no PTU figure anywhere; the Team Usage view at https://litellm-domain/ui/usage shows tokens onlyAfter: the admin declares the split in PTUs once, the proxy enforces it per team, and cost and PTU-hours land on each team
ptu_sharesand noteam_idand gets 200; shares that do not add up toptu_count, or ateam_idnext toptu_shares, get a 400 saying sogpt-4.1-ptu, while a key from a team holding no share gets 400Deployment gpt-4.1-ptu is reserved for the teams holding a PTU share of itRate limit exceeded for model_per_team_ptu: <team-a>:gpt-4.1-ptuwith aretry-after, and team B keeps its own 20 PTUflat_costsplit 720 under team A and 480 under team B (30 and 20 of the $1,200 day) plusptu_hoursper team and per model group next to the tokens, and the Team Usage page shows a PTU Hours tileDesign decisions
ptu_sharesandteam_idare mutually exclusive and the shares must add up toptu_countexactly. The alternative, letting a partial split leave the rest unowned, would silently drop cost, so the deployment either belongs to one team the old way or is fully split. For the same reasonptu_shareswithoutptu_countandcost_per_ptu_per_houris refused too: a split with no count has nothing to add up to, and the deployment would route and cap by share while its flat cost never rolled upptu_sharesdeployments. A single-teamteam_iddeployment gets no new 429s, since the team already owns all of it, but its usage does report PTU-hourslitellm/llms/azure/ptu_capacity.py) with the source URL and the date it was read. A deployment declaring shares, or an Azure deployment reserved for one team, whose model has no row logs a warning at registration and fails open (no ceiling, zero PTU-hours); a single-team reservation on another provider only ever fed the flat-cost rollup, which needs no sizing, so it is not warned about.model_info.base_modelis the preferred lookup key because an Azure deployment name is arbitrarymax_tokensgets the proxy's usual output floor, but measured against the share expressed in raw output tokens (the ceiling divided by the model's output ratio), so the floor and the implicitmax_tokenscap the proxy writes stay in raw tokens and an uncapped request never reserves more than a quarter of the share: one PTU of gpt-4.1 admits four uncapped requests a minute instead of refusing the firstGET /team/daily/activityfrom the tokens already stored, so no schema change and no new rollup. Per-key, per-user and per-tag rows keepptu_hoursat zero because those rollups carry no model group todayptu_sharesis declared, flag or no flag, because a declared split is an access rule and a proxy that ignored it would hand one department's reservation to another. The ceiling, the cost split, PTU-hours, and the registration validation stay gated onLITELLM_ENABLE_PTU_COST_ATTRIBUTION, like the existing flat-cost attribution, so a proxy with neither the flag nor shares behaves exactly as before, with one carve-out: a config.yaml entry that declaresptu_shareshas its PTU fields validated with the flag off too, since the split it declares is enforced either way and a malformed one is better dropped at load, with the startup log naming the model and the reason, than honoured or ignored without a word (the model endpoints already refuse every PTU field with the flag off). The drop follows how the proxy has handled a malformed PTU entry with the flag on since before this PR: the router raises, the proxy'signore_invalid_deploymentslogs the error and starts without that deployment, and a request to its group gets a 400 with no healthy deployments, so this release adds no startup refusalmodel_per_team_ptu:<team_id>:<model group>) on the same window as the team'smodel_tpm_limit, so a 429 from it reads and resets the same way as the limits admins already know. A rolling hour was considered and rejected because Azure sizes a PTU per minuteLITELLM_ENABLE_PTU_COST_ATTRIBUTIONis read from the process environment, never through the secret-manager pathget_secrettakes, because the flag is checked on every request in the router and the limiter. Caching a secret-manager read was the alternative and would add a cache to invalidate for a value that only ever lives in the pod's environmentptu_countand every share are whole PTUs, which is how Azure sells them:100.0is accepted as 100, and100.5is refused at registration fromPOST /model/newand config.yaml alikemodel_nameand keeps the public one inmodel_info.team_public_model_name, so the sizing lookup for PTU-hours and the ceiling matches on either name; usage rows carry the public nameptu_sharesis checked against the team table onPOST /model/newandPATCH /model/{model_id}/update, the same checkteam_idalready gets there, so a mistyped id is a 400 naming it instead of a share no key can ever use. config.yaml registration stays unchecked, since the router loads before the database, exactly as it does forteam_idteam_idis not switched toptu_sharesby a PATCH: the row is team-scoped (internal routing name, entry in the team's model list) and a shared row is proxy-wide, so the switch is a delete and aPOST /model/newwith the samemodel_info.id, which keeps the id the flat-cost rows are keyed on, and the docs say so. The alternative, letting an explicitteam_id: nullclear it in place, would need the PATCH path to repeat the delete path's team cleanup on a field the update endpoint deliberately keeps team admins from clearingmodel_info.idor provider model (litellm_params.model), or through a routing group ormodel_group_alias, counts against the ceiling of the group where the calling team holds its share, and settlement credits that same counter. A group name wins over a colliding deployment id, and a deployment id wins over any wildcard, as in the router. A name served through a wildcard counts against the group of the deployment behind the one pattern the router picks for it, the most specific proxy-wide pattern or, when none matches, the team's own, so a team's wildcard reservation behind an open*route is never charged or sized for traffic*serves. A provider model or routing group that also reaches an open deployment counts against the shared group even when the router serves it from the open one, which over-counts rather than leaks; naming the group keeps the count exact. A team with shares in two groups over one provider model is counted against the first in config order. A usage row keyed by any of those names (the router stamps the row with the name the request used) is sized by the same resolution, aliases included, so the PTU-hours a team reads back match what the ceiling charged however it named the deployment. One team's usage page sizes the row by the deployment that team is served from behind the name, its shared one first and never one reserved for another team (the router's own team-ownership rule, now one shared predicate), so a team with no share on a name, or served by an open deployment next to another team's reservation, reads no PTU-hours on it; a page spanning several or all teams merges every team into one row per name and sizes it by the first reserved deployment behind the namePre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or moreDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Last updated: 8a97ba4. QA ran on this tip (after) and on its merge base aa879f7 (before) on 2026-10-14 UTC
Both legs ran the same script against fresh databases. Each leg is two proxies, P1 and P2, each booted with
--num_workers 2, sharing one Postgres database and one Redis, withLITELLM_ENABLE_PTU_COST_ATTRIBUTION=TrueandLITELLM_LICENSEexported, and calling the real Azure deployment behindazure/gpt-5.4-mini. Teams are created on P1 and keys on P2, and model calls alternate between the two. The deployment under test is 5 PTU of gpt-5.4-mini at $10 per PTU-hour, so $1,200 a day. Azure sizes gpt-5.4-mini at 7,900 tokens per PTU per minute with output counted 6x, so team A's 3 PTU is a ceiling of 23,700 and team B's 2 PTU is 15,800. Each P1 serves the dashboard built from its own worktree's ui/ treeshow.pyprints the status,retry-after,ptu R/Lfrom thex-ratelimit-model_per_team_ptu-remaining-tokensand-limit-tokensresponse headers, thex-litellm-model-groupheader, the usage and the error message.flat_cost_by_team.pyprintstotal_flat_cost,total_ptu_hoursand each day'sflat_costper team, andptu_by_group.pyprintstotal_tokens,total_ptu_hoursandptu_hoursper model group. Team ids print as team A, B and C. The daily flat-cost rollup and its backfill are the proxy's scheduled job, so the rig calls them once from the leg's own worktree:run_rollup.py rollup <day>...awaitsrun_ptu_flat_cost_rollupfor each day andrun_rollup.py backfill <today>awaitsrun_ptu_flat_cost_backfill, and every verdict is then read back throughGET /team/daily/activityor the Admin UIBefore (aa879f7)
Setup
Case 1: register one PTU deployment split between team A (3 PTU) and team B (2 PTU)
ptu_sharesand noteam_id, then, since that is refused, the documented single-team registration, then GET /model/info on P2 after its database pollCase 2: who may call the deployment, on all three unified endpoints
Team B, which should hold 2 PTU, cannot call the deployment at all
Case 3: team A's share is a per-minute ceiling, and team B keeps its own
No ceiling applies to team A, and team B has no access
Case 4: a call that settles after its minute rolled over leaves the next minute alone
$LONGasks for a 3,000 word story) in the background, at T0+61s a small call in the next minute, and two more small calls once the long one returnsNo per-minute PTU accounting exists to check
Case 5: PTU-hours per team on today's usage
Case 6: the daily rollup splits the flat cost by share, and moving a share does not bill a paid day again
Case 7: a split deployment day whose write fails for one team is billed to every team once the database recovers
BEFORE INSERT OR UPDATEtrigger onLiteLLM_DailyTeamSpendraising for team C's flat-cost rows, run the rollup for 2026-10-09, drop the trigger, read the day, run the next backfill and read the day againThe split cannot be registered, so team C is never billed
Case 8: Team Usage page in the Admin UI
All teams read $3,600.00 and team A reads $3,600.00, the whole flat cost on team A, with five tiles and no PTU figure
After (8a97ba4)
Setup
Case 1: register one PTU deployment split between team A (3 PTU) and team B (2 PTU)
ptu_sharesand noteam_id, then shares adding up to 4 of 5 PTU, then GET /model/info on P2 after its database pollCase 2: who may call the deployment, on all three unified endpoints
Teams A and B are served, each against its own share, on every endpoint, and team C gets a 400 naming the reason
Case 3: team A's share is a per-minute ceiling, and team B keeps its own
The 24,000-token reservation is refused before it reaches Azure, team B is unaffected, and a call that fits is still served
Case 4: a call that settles after its minute rolled over leaves the next minute alone
The long call took its reservation from the first minute (244 left) and settled in the second. The next two calls read 23,541 and 23,503, which leaves room only for c4-w2-open and their own reservations, so the late settlement did not move the second minute's counter
Case 5: PTU-hours per team on today's usage
Case 6: the daily rollup splits the flat cost by share, and moving a share does not bill a paid day again
Each day is split 720 and 480 by share, and team C is not billed again for days team B already paid
Case 7: a split deployment day whose write fails for one team is billed to every team once the database recovers
The failed day wrote nothing for either team, so the backfill billed both once the database recovered
Case 8: Team Usage page in the Admin UI
All teams read $4,800.00 (the three split days plus the 2026-10-09 day) with a PTU Hours tile of 0.05, and team A reads $2,880.00, four days at $720, with the same tile
Surprises
Earlier rounds as controls
At 2d1d021 a settlement landing after its minute rolled over moved the next minute's counter, which Case 4 now covers. At 17abc6e a split day whose write failed for one team kept the other team's row, so the backfill counted the day as priced and never billed the missing team, which Case 7 now covers
Recent commits
17abc6e guards every PTU settle and refund by the window the request reserved in, the same guard project ITPM/OTPM already use, and keys the backfill by deployment-day, so moving a share no longer bills the new holder for days already paid
1a4b393 writes each deployment-day's team charges in one Prisma batch with the retry around the whole batch, so a day is fully written or not at all (design decision 18)
8a97ba4 merges main at aa879f7 with no conflicts. Main touched no PR file in that range except the generated
schema.d.ts, and the PR diff is unchangedType
🆕 New Feature
Caveats (if any)
Medium
max_tokens= min(1024, a quarter of the share in output tokens, the ceiling divided by the model's output ratio)max_tokensyourself to pick the cap; capped requests are untouchedptu_hours: 0breakdown.model_groupsknow their model groupLow
ptu_shareseditor yetPOST /model/newGET /v1/modelsfor share-less teams, andGET /health?model=<group>from such a team still probes it onceGET /team/daily/activitycarries the full valueptu_shareshas no cap on how many teams it names, and the team existence check looks them up in chunks under the database'sINlist boundteam_idis split by deleting it and registering it again withptu_sharesand the samemodel_info.id, not by a PATCH, which refuses the mix with a 400 (design decision 16, documented)GET /team/daily/activityquery reportsptu_hours: 0underbreakdown.entities.<team_id>fail_closed_rate_limit_enforcement: trueand Redis down, share-holding teams get a 503 on the shared deploymentPATCH /model/{id}/updatewith aptu_sharesmap summing toptu_countreassigns itLITELLM_ENABLE_PTU_COST_ATTRIBUTIONis read from the process environment only (design decision 12)ptu_sharesin config.yaml is not checked against the team table, onlyPOST /model/newand the PATCH are*route also matches it/azurepass-through gets a 500 instead of the 400, and nothing reaches Azureptu_sharesmap written to the model table before this release, by a client posting the then-unknown key, is enforced at load: the deployment serves only the teams it names, sized from their shares, with no flat cost attributed until its terms existPATCH /model/{id}/updatewithptu_count,cost_per_ptu_per_hourandptu_effective_fromcompletes it, or delete and re-register the deployment without the mapPOST /model/newwrote, so it bills nothing on either path until the PATCHPOST /model/newwhile the flag was on serves for free once the flag is off: its token prices were zeroed at registration and no flat cost accrues without the flag, while a config.yaml entry keeps its token pricesteam_idreservations, now also the case for shared ones; turn the flag back on or re-register the deploymentptu_shareswithoutptu_countandcost_per_ptu_per_houris dropped at load, flag on or off: the startup log names the model and the reason, the proxy starts without it, and a request to its group gets a 400 with no healthy deploymentsPOST /v1/messageswithoutmax_tokensis a 500 from the Anthropic handler on main and on this branch for a team under no ceiling; under a ceiling this branch writes the implicit cap before the handler runs, so the same call is 200osv-scanfails on the tip for an upstream advisory on a dashboard dev dependency with no fixed release, in a lockfile this PR does not touch; it is not a required checkGET /team/daily/activitycarry the figureFinal Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
b41eed4 passes /live-pr-risk
e4b33bc passes /live-pr-risk
6f3915b passes /live-pr-risk
Note
High Risk
Changes request routing, per-team rate limiting in normalized tokens, and daily cost rollup for reserved capacity—mistakes could block traffic, mis-attribute spend, or leak capacity across teams.
Overview
Adds
model_info.ptu_sharesso one Azure PTU deployment can split whole PTUs across teams (exclusive withteam_id), with validation that shares sum toptu_countand that named teams exist on model create/update.Routing & limits: Share-holding teams only get routed to split deployments; the parallel request limiter enforces a per-team
model_per_team_ptuceiling in Azure normalized tokens (reservation, success/failure settlement, and window-aware refunds). Newlitellm/llms/azure/ptu_capacity.pyandrouter_utils/ptu_shares.pydrive ceilings and access; the router adds share filtering, wildcard resolution order, and sizing warnings.Cost & reporting: Daily flat-cost rollup writes one sentinel row per team share per deployment-day (batched upserts). Team daily activity APIs attach
ptu_hoursfrom stored tokens when PTU attribution is on (ptu_consumption.py).PTUTermsnow carries asharesmap instead of a singleteam_id.Ops:
LITELLM_ENABLE_PTU_COST_ATTRIBUTIONis read from the process environment only (not the secret manager). OpenAPI and spend metrics types gainptu_hours/total_ptu_hours.Reviewed by Cursor Bugbot for commit 8a97ba4. Bugbot is set up for automated code reviews on this repo. Configure here.