From f1f3af84e28193ef65bb0cbe66aefb5947f8a4d4 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Thu, 24 Sep 2026 14:48:22 -0700 Subject: [PATCH 01/17] docs(proxy): document PTU shares per team, the per-team ceiling, and PTU-hours on team usage --- docs/proxy/dynamic_rate_limit.md | 35 ++++++++++++ docs/proxy/ptu_flat_cost.md | 97 ++++++++++++++++++++++++++++++-- 2 files changed, 126 insertions(+), 6 deletions(-) diff --git a/docs/proxy/dynamic_rate_limit.md b/docs/proxy/dynamic_rate_limit.md index 16f404e38..ee502c6d1 100644 --- a/docs/proxy/dynamic_rate_limit.md +++ b/docs/proxy/dynamic_rate_limit.md @@ -343,3 +343,38 @@ This video walks through setting up dynamic rate limiting with priority reservat +## PTU share ceilings on Azure provisioned deployments + +An Azure provisioned throughput (PTU) deployment split across teams with `model_info.ptu_shares` gets a per-team, per-minute ceiling without any `model_tpm_limit` worked out by hand. The proxy converts each team's share through the Azure sizing table it ships (input TPM per PTU and the output-to-input ratio for the model) and charges each request in Azure's normalized tokens, so the ceiling stays in step with the reservation when the shares change: + +```yaml +model_list: + - model_name: gpt-4.1-ptu + litellm_params: + model: azure/ + api_key: os.environ/AZURE_API_KEY + api_base: os.environ/AZURE_API_BASE + model_info: + id: gpt-4.1-ptu-shared + base_model: gpt-4.1 + ptu_count: 50 + cost_per_ptu_per_hour: 1.0 + ptu_effective_from: "2026-01-01T00:00:00Z" + ptu_shares: + : 30 + : 20 +``` + +Team A's 30 PTUs of gpt-4.1 are 30 x 3,000 = 90,000 normalized tokens a minute (output counted at 4x), and the first request past that gets a 429 while team B keeps its own 20 PTUs: + +```json +{ + "error": { + "message": "Rate limit exceeded for model_per_team_ptu: :gpt-4.1-ptu. Limit type: tokens. Current limit: 90000, Remaining: 0. Limit resets at: ...", + "type": "throttling_error", + "code": "429" + } +} +``` + +The ceiling shares the per-minute window of a team's `model_tpm_limit` and needs only `LITELLM_ENABLE_PTU_COST_ATTRIBUTION=True`. Setup, the cost split by share, and PTU-hours on the usage routes are covered in [Azure PTU Flat Cost Attribution](./ptu_flat_cost.md#share-a-deployment-across-teams) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 88d0a8fe4..96c4b8c03 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -6,6 +6,8 @@ Azure provisioned throughput is billed by the hour for reserved capacity, not pe A PTU deployment is billed by its reserved capacity alone. LiteLLM stores zero per-token pricing on it, so the traffic the capacity serves is never charged on top of the flat cost. +One deployment can also be split across several teams in PTUs. Each team's share becomes a per-minute ceiling the proxy enforces, the flat cost is attributed by share, and usage reports the PTU-hours each team consumed next to its tokens. See [Share a deployment across teams](#share-a-deployment-across-teams) + ## Enable it The feature is off by default and inert until you opt in: @@ -70,16 +72,91 @@ Upgrading an existing reservation that has already accrued cost, set `id` to the | Field | Required | Meaning | | --- | --- | --- | | `id` | in `config.yaml` | The deployment's stable identity. Not needed through the API or the UI, where one is stored for you | -| `team_id` | yes | The team the capacity belongs to. One deployment maps to one team | -| `ptu_count` | yes | Provisioned throughput units reserved | +| `team_id` | one of the two | The team the capacity belongs to, when one team owns the whole deployment | +| `ptu_shares` | one of the two | Team id to whole PTUs, adding up to `ptu_count`, when several teams share it | +| `ptu_count` | yes | Provisioned throughput units reserved, a whole number | | `cost_per_ptu_per_hour` | yes | Your contracted hourly rate per unit | | `ptu_effective_from` | yes | When the reservation starts accruing | | `ptu_effective_to` | no | When it stops. Leave unset for an open reservation | +| `base_model` | for the ceiling | The Azure model name when the deployment name is not one, e.g. `gpt-4.1` | `ptu_count` and `cost_per_ptu_per_hour` must be set together, and `ptu_effective_from` is required because flat cost accrues from that instant. Without it a deployment configured today would bill for days it did not exist. Take `cost_per_ptu_per_hour` from your Azure agreement rather than a list price; PTU rates are negotiated and vary by region and commitment term. +## Share a deployment across teams + +A department that reserves 50 PTUs and lets two teams use them declares the split once, in PTUs, with `model_info.ptu_shares` in place of `team_id`: + +```bash +curl -X POST http://localhost:4000/model/new \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ + -H 'Content-Type: application/json' \ + -d '{ + "model_name": "gpt-4.1-ptu", + "litellm_params": { + "model": "azure/", + "api_key": "os.environ/AZURE_API_KEY", + "api_base": "os.environ/AZURE_API_BASE" + }, + "model_info": { + "base_model": "gpt-4.1", + "ptu_count": 50, + "cost_per_ptu_per_hour": 1.0, + "ptu_effective_from": "2026-01-01T00:00:00Z", + "ptu_shares": {"": 30, "": 20} + } + }' +``` + +Or in `config.yaml`: + +```yaml +model_list: + - model_name: gpt-4.1-ptu + litellm_params: + model: azure/ + api_key: os.environ/AZURE_API_KEY + api_base: os.environ/AZURE_API_BASE + model_info: + id: gpt-4.1-ptu-shared + base_model: gpt-4.1 + ptu_count: 50 + cost_per_ptu_per_hour: 1.0 + ptu_effective_from: "2026-01-01T00:00:00Z" + ptu_shares: + : 30 + : 20 +``` + +`ptu_shares` and `team_id` cannot both be set, and the shares have to add up to `ptu_count` exactly, in whole PTUs. A split that leaves capacity unowned or hands out more than was reserved is refused with a 400 saying how many of the PTUs were allocated, from `POST /model/new` and `config.yaml` alike + +A shared deployment keeps its public model name and stays visible in model lists, but the proxy serves it only to the teams named in `ptu_shares`. A key from any other team, a key with no team, and the master key all get a 400 on it: + +``` +Deployment gpt-4.1-ptu is reserved for the teams holding a PTU share of it +``` + +### Each share is a per-minute ceiling + +Azure sizes a provisioned deployment in normalized tokens per minute: uncached input tokens in full, cached input at the model's cached ratio (a tenth for the GPT-6 family, nothing for the rest), and output tokens weighted by the model's output-to-input ratio, divided by the model's input TPM per PTU. LiteLLM ships that table, read from [Azure's provisioned throughput sizing page](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/provisioned-throughput-sizing#deployment-parameters-and-throughput-values-by-model), and turns each share into a per-minute ceiling in the same units: + +``` +team ceiling per minute = share x input TPM per PTU for the model +tokens charged per request = uncached input + cached input x cached ratio + output x output ratio +``` + +Team A's 30 PTUs of gpt-4.1 (3,000 input TPM per PTU, output counted at 4x) are 90,000 normalized tokens a minute. The first request past that gets a 429 with a `retry-after` header, and team B's 20 PTUs are untouched: + +``` +Rate limit exceeded for model_per_team_ptu: :gpt-4.1-ptu. Limit type: tokens. Current limit: 90000, Remaining: 0. Limit resets at: ... +``` + +The ceiling lives on the same per-minute window as a team's `model_tpm_limit`, so it reads and resets the way the limits you already set do, and it needs nothing beyond `LITELLM_ENABLE_PTU_COST_ATTRIBUTION`. The sizing row is looked up by `model_info.base_model` first, then by the model in `litellm_params`, because an Azure deployment name is arbitrary. A reserved deployment whose model has no row logs a warning at startup, sets no ceiling, and reports no PTU-hours; set `base_model` to the Azure model name to fix it + +A deployment owned by one team through `team_id` gets no ceiling, since that team already owns all of it, but its usage does report PTU-hours + + ## How the cost is calculated A job runs at 00:15 UTC and writes one row per team and model for the previous day: @@ -88,7 +165,7 @@ A job runs at 00:15 UTC and writes one row per team and model for the previous d flat cost = ptu_count x cost_per_ptu_per_hour x hours active that day ``` -Active hours are the overlap between the day and the reservation window, so a reservation starting at noon accrues 12 hours on its first day and 24 thereafter. The rows are written into `LiteLLM_DailyTeamSpend` under the reserved key `__ptu_flat_cost__`, which keeps flat cost separate from the per-request spend recorded against real API keys. +A deployment split with `ptu_shares` writes one row per team instead, each carrying that team's share in place of `ptu_count`: 30 and 20 of the 50 PTUs at $1 per PTU-hour are $720 and $480 for a full day, and the deployment's $1,200 lands on nobody else. Active hours are the overlap between the day and the reservation window, so a reservation starting at noon accrues 12 hours on its first day and 24 thereafter. The rows are written into `LiteLLM_DailyTeamSpend` under the reserved key `__ptu_flat_cost__`, which keeps flat cost separate from the per-request spend recorded against real API keys. A reservation that starts before the job first sees it is filled in as well: the catch-up pass prices each elapsed day back to `ptu_effective_from`, up to 91 days. A deployment configured today with a backdated start therefore accrues its whole window on the first run @@ -108,17 +185,19 @@ This needs `LITELLM_ENABLE_PTU_COST_ATTRIBUTION` set and a pricing map entry for ## Read the cost back -`/team/daily/activity` reports `flat_cost` per day and `total_flat_cost` for the range, alongside the usual per-token `spend`: +`/team/daily/activity` reports `flat_cost` per day and `total_flat_cost` for the range, alongside the usual per-token `spend`, and `ptu_hours` next to the tokens: on each day, on each PTU model group and the API keys under it, and as `total_ptu_hours` for the range. PTU-hours are the team's prompt, cached, and completion tokens converted through the model's sizing row, so a team can compare what it consumed with what it reserved: ```bash curl -s "http://localhost:4000/team/daily/activity?team_ids=&start_date=2026-01-01&end_date=2026-01-31" \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '{ total_spend: .metadata.total_spend, - total_flat_cost: .metadata.total_flat_cost + total_flat_cost: .metadata.total_flat_cost, + total_ptu_hours: .metadata.total_ptu_hours, + gpt41_ptu_hours: [.results[].breakdown.model_groups["gpt-4.1-ptu"].metrics.ptu_hours] }' ``` -The Usage page in the Admin UI shows the same figures under Team Usage, charting flat cost separately from request cost, and CSV export carries them: +The Usage page in the Admin UI shows the same figures under Team Usage, charting flat cost separately from request cost, with a PTU Hours tile for a team whose window consumed any, and CSV export carries them: @@ -140,3 +219,9 @@ Web search rates are handled the same way. Note that xAI models bill their list The legacy `POST /model/update` does not run the rules above, so a PTU deployment configured through it keeps billing per token and never accrues flat cost. Use `POST /model/new`, `PATCH /model/{model_id}/update`, the Admin UI, or `config.yaml` The Python `Router` used on its own zeroes per-token pricing at registration, but nothing schedules the daily job outside the proxy, so flat cost is not accrued there + +PTU-hours are reported on team activity only. The per-key, per-user, and per-tag activity routes carry no model group in their rows today, so they report none + +An unshared deployment in the same model group as a shared one counts toward the teams' ceilings and PTU-hours on that group, and a group mixing models is sized by its first reserved deployment's row. Keep a shared deployment in a model group of its own + +`ptu_shares` is edited through `POST /model/new`, `PATCH /model/{model_id}/update`, or `config.yaml`; the model form in the Admin UI has no shares editor yet From 4bfd5afe9aad66e08bce119f4246be6d4d7db045 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Thu, 24 Sep 2026 15:31:13 -0700 Subject: [PATCH 02/17] docs(ptu): say the share filter holds without the flag and which reservations get the sizing warning --- docs/proxy/dynamic_rate_limit.md | 2 +- docs/proxy/ptu_flat_cost.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/proxy/dynamic_rate_limit.md b/docs/proxy/dynamic_rate_limit.md index ee502c6d1..83dbbc502 100644 --- a/docs/proxy/dynamic_rate_limit.md +++ b/docs/proxy/dynamic_rate_limit.md @@ -377,4 +377,4 @@ Team A's 30 PTUs of gpt-4.1 are 30 x 3,000 = 90,000 normalized tokens a minute ( } ``` -The ceiling shares the per-minute window of a team's `model_tpm_limit` and needs only `LITELLM_ENABLE_PTU_COST_ATTRIBUTION=True`. Setup, the cost split by share, and PTU-hours on the usage routes are covered in [Azure PTU Flat Cost Attribution](./ptu_flat_cost.md#share-a-deployment-across-teams) +The ceiling shares the per-minute window of a team's `model_tpm_limit` and needs only `LITELLM_ENABLE_PTU_COST_ATTRIBUTION=True`; the share filter itself (which teams the deployment is served to) applies without the flag. Setup, the cost split by share, and PTU-hours on the usage routes are covered in [Azure PTU Flat Cost Attribution](./ptu_flat_cost.md#share-a-deployment-across-teams) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 96c4b8c03..63ae77294 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -131,7 +131,7 @@ model_list: `ptu_shares` and `team_id` cannot both be set, and the shares have to add up to `ptu_count` exactly, in whole PTUs. A split that leaves capacity unowned or hands out more than was reserved is refused with a 400 saying how many of the PTUs were allocated, from `POST /model/new` and `config.yaml` alike -A shared deployment keeps its public model name and stays visible in model lists, but the proxy serves it only to the teams named in `ptu_shares`. A key from any other team, a key with no team, and the master key all get a 400 on it: +A shared deployment keeps its public model name and stays visible in model lists, but the proxy serves it only to the teams named in `ptu_shares`. A key from any other team, a key with no team, and the master key all get a 400 on it, whether or not `LITELLM_ENABLE_PTU_COST_ATTRIBUTION` is set, because a declared split is an access rule; the ceiling, the cost split, and PTU-hours below need the flag: ``` Deployment gpt-4.1-ptu is reserved for the teams holding a PTU share of it @@ -146,13 +146,13 @@ team ceiling per minute = share x input TPM per PTU for the model tokens charged per request = uncached input + cached input x cached ratio + output x output ratio ``` -Team A's 30 PTUs of gpt-4.1 (3,000 input TPM per PTU, output counted at 4x) are 90,000 normalized tokens a minute. The first request past that gets a 429 with a `retry-after` header, and team B's 20 PTUs are untouched: +Team A's 30 PTUs of gpt-4.1 (3,000 input TPM per PTU, output counted at 4x) are 90,000 normalized tokens a minute. Each request reserves its input plus its output budget (`max_tokens`, or the proxy's estimate without one) in those units before the call and settles at the usage the response reports, so a burst of concurrent requests cannot together pass the share. The first request past it gets a 429 with a `retry-after` header, and team B's 20 PTUs are untouched: ``` Rate limit exceeded for model_per_team_ptu: :gpt-4.1-ptu. Limit type: tokens. Current limit: 90000, Remaining: 0. Limit resets at: ... ``` -The ceiling lives on the same per-minute window as a team's `model_tpm_limit`, so it reads and resets the way the limits you already set do, and it needs nothing beyond `LITELLM_ENABLE_PTU_COST_ATTRIBUTION`. The sizing row is looked up by `model_info.base_model` first, then by the model in `litellm_params`, because an Azure deployment name is arbitrary. A reserved deployment whose model has no row logs a warning at startup, sets no ceiling, and reports no PTU-hours; set `base_model` to the Azure model name to fix it +The ceiling lives on the same per-minute window as a team's `model_tpm_limit`, so it reads and resets the way the limits you already set do, and it needs nothing beyond `LITELLM_ENABLE_PTU_COST_ATTRIBUTION`. The sizing row is looked up by `model_info.base_model` first, then by the model in `litellm_params`, because an Azure deployment name is arbitrary. A shared deployment, or an Azure deployment reserved for one team, whose model has no row logs a warning at startup, sets no ceiling, and reports no PTU-hours; set `base_model` to the Azure model name to fix it. A single-team reservation on another provider only feeds the flat-cost rollup, which needs no sizing, so it is not warned about A deployment owned by one team through `team_id` gets no ceiling, since that team already owns all of it, but its usage does report PTU-hours From 169852e8755c60d087823cf3f12c831fb767ec77 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Thu, 24 Sep 2026 16:02:27 -0700 Subject: [PATCH 03/17] docs(ptu): say the output floor of an uncapped request stays within a quarter of the share --- docs/proxy/ptu_flat_cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 63ae77294..963664766 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -146,7 +146,7 @@ team ceiling per minute = share x input TPM per PTU for the model tokens charged per request = uncached input + cached input x cached ratio + output x output ratio ``` -Team A's 30 PTUs of gpt-4.1 (3,000 input TPM per PTU, output counted at 4x) are 90,000 normalized tokens a minute. Each request reserves its input plus its output budget (`max_tokens`, or the proxy's estimate without one) in those units before the call and settles at the usage the response reports, so a burst of concurrent requests cannot together pass the share. The first request past it gets a 429 with a `retry-after` header, and team B's 20 PTUs are untouched: +Team A's 30 PTUs of gpt-4.1 (3,000 input TPM per PTU, output counted at 4x) are 90,000 normalized tokens a minute. Each request reserves its input plus its output budget (`max_tokens`, or without one the proxy's output floor, sized so it never takes more than a quarter of the share) in those units before the call and settles at the usage the response reports, so a burst of concurrent requests cannot together pass the share. The first request past it gets a 429 with a `retry-after` header, and team B's 20 PTUs are untouched: ``` Rate limit exceeded for model_per_team_ptu: :gpt-4.1-ptu. Limit type: tokens. Current limit: 90000, Remaining: 0. Limit resets at: ... From 58a78df6fbe41e10e0f094cbe145d278233dcba2 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Thu, 24 Sep 2026 17:47:30 -0700 Subject: [PATCH 04/17] docs(ptu): say the API checks that every ptu_shares team exists --- docs/proxy/ptu_flat_cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 963664766..a2d1c7c97 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -129,7 +129,7 @@ model_list: : 20 ``` -`ptu_shares` and `team_id` cannot both be set, and the shares have to add up to `ptu_count` exactly, in whole PTUs. A split that leaves capacity unowned or hands out more than was reserved is refused with a 400 saying how many of the PTUs were allocated, from `POST /model/new` and `config.yaml` alike +`ptu_shares` and `team_id` cannot both be set, and the shares have to add up to `ptu_count` exactly, in whole PTUs. A split that leaves capacity unowned or hands out more than was reserved is refused with a 400 saying how many of the PTUs were allocated, from `POST /model/new` and `config.yaml` alike. Through the API every team in the split has to exist already: `POST /model/new` and `PATCH /model/{model_id}/update` refuse a split naming an unknown team with a 400 that names it, the same way they refuse an unknown `team_id` A shared deployment keeps its public model name and stays visible in model lists, but the proxy serves it only to the teams named in `ptu_shares`. A key from any other team, a key with no team, and the master key all get a 400 on it, whether or not `LITELLM_ENABLE_PTU_COST_ATTRIBUTION` is set, because a declared split is an access rule; the ceiling, the cost split, and PTU-hours below need the flag: From 5943902114c03d2f29e0e477e22484c007c32643 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Thu, 24 Sep 2026 19:19:50 -0700 Subject: [PATCH 05/17] docs(ptu): say how a single-team PTU deployment is split across teams --- docs/proxy/ptu_flat_cost.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index a2d1c7c97..57843df76 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -131,6 +131,8 @@ model_list: `ptu_shares` and `team_id` cannot both be set, and the shares have to add up to `ptu_count` exactly, in whole PTUs. A split that leaves capacity unowned or hands out more than was reserved is refused with a 400 saying how many of the PTUs were allocated, from `POST /model/new` and `config.yaml` alike. Through the API every team in the split has to exist already: `POST /model/new` and `PATCH /model/{model_id}/update` refuse a split naming an unknown team with a 400 that names it, the same way they refuse an unknown `team_id` +A deployment one team already owns through `team_id` is not switched to `ptu_shares` by a `PATCH`: a `team_id` row is team-scoped, with an internal routing name and an entry in that team's model list, and a shared row is proxy-wide, so the patch is refused with the same 400. To split it, delete the deployment and register it again with `ptu_shares` and the same `model_info.id`, which keeps the flat-cost rows already written under that id + A shared deployment keeps its public model name and stays visible in model lists, but the proxy serves it only to the teams named in `ptu_shares`. A key from any other team, a key with no team, and the master key all get a 400 on it, whether or not `LITELLM_ENABLE_PTU_COST_ATTRIBUTION` is set, because a declared split is an access rule; the ceiling, the cost split, and PTU-hours below need the flag: ``` From d73936ce29f0fb955a89ca951c4befeaa909d14a Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Thu, 1 Oct 2026 16:53:30 -0700 Subject: [PATCH 06/17] docs(ptu): say a split declared without ptu_count and cost_per_ptu_per_hour is refused --- docs/proxy/ptu_flat_cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 57843df76..6c7e8b961 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -129,7 +129,7 @@ model_list: : 20 ``` -`ptu_shares` and `team_id` cannot both be set, and the shares have to add up to `ptu_count` exactly, in whole PTUs. A split that leaves capacity unowned or hands out more than was reserved is refused with a 400 saying how many of the PTUs were allocated, from `POST /model/new` and `config.yaml` alike. Through the API every team in the split has to exist already: `POST /model/new` and `PATCH /model/{model_id}/update` refuse a split naming an unknown team with a 400 that names it, the same way they refuse an unknown `team_id` +`ptu_shares` and `team_id` cannot both be set, and the shares have to add up to `ptu_count` exactly, in whole PTUs. A split that leaves capacity unowned or hands out more than was reserved is refused with a 400 saying how many of the PTUs were allocated, from `POST /model/new` and `config.yaml` alike, and so is a split declared without `ptu_count` and `cost_per_ptu_per_hour`, since it would have nothing to add up to and no flat cost to roll up. Through the API every team in the split has to exist already: `POST /model/new` and `PATCH /model/{model_id}/update` refuse a split naming an unknown team with a 400 that names it, the same way they refuse an unknown `team_id` A deployment one team already owns through `team_id` is not switched to `ptu_shares` by a `PATCH`: a `team_id` row is team-scoped, with an internal routing name and an entry in that team's model list, and a shared row is proxy-wide, so the patch is refused with the same 400. To split it, delete the deployment and register it again with `ptu_shares` and the same `model_info.id`, which keeps the flat-cost rows already written under that id From ebcffd9cb3ea10e86ee04a346bf17f678b33c0ef Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Fri, 2 Oct 2026 15:22:46 -0700 Subject: [PATCH 07/17] docs(ptu): say a usage row keyed by an alias, a deployment id, or a provider model is sized like the group --- docs/proxy/ptu_flat_cost.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 6c7e8b961..596a94ee9 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -224,6 +224,8 @@ The Python `Router` used on its own zeroes per-token pricing at registration, bu PTU-hours are reported on team activity only. The per-key, per-user, and per-tag activity routes carry no model group in their rows today, so they report none +A request that names a shared deployment by an alias, a routing group, its deployment id, or its provider model lands in a usage row keyed by that name. The row is sized by the deployment behind the name, so its PTU-hours count the same as a request naming the model group + An unshared deployment in the same model group as a shared one counts toward the teams' ceilings and PTU-hours on that group, and a group mixing models is sized by its first reserved deployment's row. Keep a shared deployment in a model group of its own `ptu_shares` is edited through `POST /model/new`, `PATCH /model/{model_id}/update`, or `config.yaml`; the model form in the Admin UI has no shares editor yet From ca7b80ac60932a02a1cb56ff4af96b0ba5197e09 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Fri, 2 Oct 2026 15:56:09 -0700 Subject: [PATCH 08/17] docs(ptu): size a team's usage row by the deployment it is served from --- docs/proxy/ptu_flat_cost.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 596a94ee9..a29397efe 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -224,8 +224,8 @@ The Python `Router` used on its own zeroes per-token pricing at registration, bu PTU-hours are reported on team activity only. The per-key, per-user, and per-tag activity routes carry no model group in their rows today, so they report none -A request that names a shared deployment by an alias, a routing group, its deployment id, or its provider model lands in a usage row keyed by that name. The row is sized by the deployment behind the name, so its PTU-hours count the same as a request naming the model group +A request that names a shared deployment by an alias, a routing group, its deployment id, or its provider model lands in a usage row keyed by that name. On one team's page the row is sized by the deployment that team is served from behind the name, its shared one first, so its PTU-hours count the same as a request naming the model group and a team with no share on a name reads no PTU-hours on it. A page spanning several or all teams sizes the row by the first reserved deployment behind the name -An unshared deployment in the same model group as a shared one counts toward the teams' ceilings and PTU-hours on that group, and a group mixing models is sized by its first reserved deployment's row. Keep a shared deployment in a model group of its own +An unshared deployment in the same model group as a shared one counts toward the teams' ceilings and PTU-hours on that group, and a group mixing models is sized on one team's page by the deployment that team holds its share on and on a page spanning teams by its first reserved deployment's row. Keep a shared deployment in a model group of its own `ptu_shares` is edited through `POST /model/new`, `PATCH /model/{model_id}/update`, or `config.yaml`; the model form in the Admin UI has no shares editor yet From d9592e116e4801492f4a2129368bd759bc9d219b Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Fri, 2 Oct 2026 16:38:33 -0700 Subject: [PATCH 09/17] docs(ptu): a team's usage row never sizes by another team's reservation --- docs/proxy/ptu_flat_cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index a29397efe..9db66900a 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -224,7 +224,7 @@ The Python `Router` used on its own zeroes per-token pricing at registration, bu PTU-hours are reported on team activity only. The per-key, per-user, and per-tag activity routes carry no model group in their rows today, so they report none -A request that names a shared deployment by an alias, a routing group, its deployment id, or its provider model lands in a usage row keyed by that name. On one team's page the row is sized by the deployment that team is served from behind the name, its shared one first, so its PTU-hours count the same as a request naming the model group and a team with no share on a name reads no PTU-hours on it. A page spanning several or all teams sizes the row by the first reserved deployment behind the name +A request that names a shared deployment by an alias, a routing group, its deployment id, or its provider model lands in a usage row keyed by that name. On one team's page the row is sized by the deployment that team is served from behind the name, its shared one first and never one reserved for another team, so its PTU-hours count the same as a request naming the model group and a team with no share on a name, or served by an open deployment next to another team's reservation, reads no PTU-hours on it. A page spanning several or all teams sizes the row by the first reserved deployment behind the name An unshared deployment in the same model group as a shared one counts toward the teams' ceilings and PTU-hours on that group, and a group mixing models is sized on one team's page by the deployment that team holds its share on and on a page spanning teams by its first reserved deployment's row. Keep a shared deployment in a model group of its own From 78dd0373fdf894bada05abac3e903bd9c6d3d728 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Fri, 2 Oct 2026 20:23:43 -0700 Subject: [PATCH 10/17] docs(ptu): say a malformed config.yaml split is dropped at load, not refused with a 400 --- docs/proxy/ptu_flat_cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 9db66900a..2fa38848e 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -129,7 +129,7 @@ model_list: : 20 ``` -`ptu_shares` and `team_id` cannot both be set, and the shares have to add up to `ptu_count` exactly, in whole PTUs. A split that leaves capacity unowned or hands out more than was reserved is refused with a 400 saying how many of the PTUs were allocated, from `POST /model/new` and `config.yaml` alike, and so is a split declared without `ptu_count` and `cost_per_ptu_per_hour`, since it would have nothing to add up to and no flat cost to roll up. Through the API every team in the split has to exist already: `POST /model/new` and `PATCH /model/{model_id}/update` refuse a split naming an unknown team with a 400 that names it, the same way they refuse an unknown `team_id` +`ptu_shares` and `team_id` cannot both be set, and the shares have to add up to `ptu_count` exactly, in whole PTUs. A split that leaves capacity unowned or hands out more than was reserved is refused from `POST /model/new` with a 400 saying how many of the PTUs were allocated, and so is a split declared without `ptu_count` and `cost_per_ptu_per_hour`, since it would have nothing to add up to and no flat cost to roll up. In `config.yaml` the same entry is dropped when the proxy loads its models, whether or not `LITELLM_ENABLE_PTU_COST_ATTRIBUTION` is set: the startup log names the deployment and the reason, the proxy starts without it, and a request to that model name gets a 400 with no healthy deployments. Through the API every team in the split has to exist already: `POST /model/new` and `PATCH /model/{model_id}/update` refuse a split naming an unknown team with a 400 that names it, the same way they refuse an unknown `team_id` A deployment one team already owns through `team_id` is not switched to `ptu_shares` by a `PATCH`: a `team_id` row is team-scoped, with an internal routing name and an entry in that team's model list, and a shared row is proxy-wide, so the patch is refused with the same 400. To split it, delete the deployment and register it again with `ptu_shares` and the same `model_info.id`, which keeps the flat-cost rows already written under that id From fa06b544800e1850802b7d4affb172569df9883f Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Fri, 9 Oct 2026 19:24:07 -0700 Subject: [PATCH 11/17] docs(ptu): name every route that counts against a team's PTU ceiling, wildcards included --- docs/proxy/ptu_flat_cost.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 2fa38848e..215db1050 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -156,6 +156,8 @@ Rate limit exceeded for model_per_team_ptu: :gpt-4.1-ptu. Limit type: The ceiling lives on the same per-minute window as a team's `model_tpm_limit`, so it reads and resets the way the limits you already set do, and it needs nothing beyond `LITELLM_ENABLE_PTU_COST_ATTRIBUTION`. The sizing row is looked up by `model_info.base_model` first, then by the model in `litellm_params`, because an Azure deployment name is arbitrary. A shared deployment, or an Azure deployment reserved for one team, whose model has no row logs a warning at startup, sets no ceiling, and reports no PTU-hours; set `base_model` to the Azure model name to fix it. A single-team reservation on another provider only feeds the flat-cost rollup, which needs no sizing, so it is not warned about +A request counts against its team's ceiling however it names the shared deployment: the model group, a `model_group_alias`, a routing group, the deployment's `model_info.id`, its provider model, the `/azure` pass-through route, or a wildcard pattern such as `ptu-*` that the router serves it through. A wildcard name counts only against the pattern the router serves it from, the most specific one that matches, so a request an open `*` route serves never charges a share, and a team holding no share gets the 400 on a name the shared pattern matches even when `*` matches it too + A deployment owned by one team through `team_id` gets no ceiling, since that team already owns all of it, but its usage does report PTU-hours From 43251cb588ef97ca9974debbba3b5426019a1b66 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 14 Oct 2026 04:54:58 +0000 Subject: [PATCH 12/17] docs(ptu): say that moving a share never bills an already priced day again --- docs/proxy/ptu_flat_cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 215db1050..fd5622899 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -171,7 +171,7 @@ flat cost = ptu_count x cost_per_ptu_per_hour x hours active that day A deployment split with `ptu_shares` writes one row per team instead, each carrying that team's share in place of `ptu_count`: 30 and 20 of the 50 PTUs at $1 per PTU-hour are $720 and $480 for a full day, and the deployment's $1,200 lands on nobody else. Active hours are the overlap between the day and the reservation window, so a reservation starting at noon accrues 12 hours on its first day and 24 thereafter. The rows are written into `LiteLLM_DailyTeamSpend` under the reserved key `__ptu_flat_cost__`, which keeps flat cost separate from the per-request spend recorded against real API keys. -A reservation that starts before the job first sees it is filled in as well: the catch-up pass prices each elapsed day back to `ptu_effective_from`, up to 91 days. A deployment configured today with a backdated start therefore accrues its whole window on the first run +A reservation that starts before the job first sees it is filled in as well: the catch-up pass prices each elapsed day back to `ptu_effective_from`, up to 91 days. A deployment configured today with a backdated start therefore accrues its whole window on the first run. A day the job has already priced keeps its rows when shares later move between teams, so the team taking over a share is billed only for days priced after the move, never for days the previous holder already paid for. Flat cost does not count against team or key budgets. Reserved capacity is already paid for, so a team cannot exhaust a budget by using the capacity it reserved. From 00fdc5c04d0d18b9a23a74e9455e2489e0458e67 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 14 Oct 2026 07:22:50 +0000 Subject: [PATCH 13/17] docs(ptu): say that a failed day is priced again for every team --- docs/proxy/ptu_flat_cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index fd5622899..ebf4ccfe0 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -171,7 +171,7 @@ flat cost = ptu_count x cost_per_ptu_per_hour x hours active that day A deployment split with `ptu_shares` writes one row per team instead, each carrying that team's share in place of `ptu_count`: 30 and 20 of the 50 PTUs at $1 per PTU-hour are $720 and $480 for a full day, and the deployment's $1,200 lands on nobody else. Active hours are the overlap between the day and the reservation window, so a reservation starting at noon accrues 12 hours on its first day and 24 thereafter. The rows are written into `LiteLLM_DailyTeamSpend` under the reserved key `__ptu_flat_cost__`, which keeps flat cost separate from the per-request spend recorded against real API keys. -A reservation that starts before the job first sees it is filled in as well: the catch-up pass prices each elapsed day back to `ptu_effective_from`, up to 91 days. A deployment configured today with a backdated start therefore accrues its whole window on the first run. A day the job has already priced keeps its rows when shares later move between teams, so the team taking over a share is billed only for days priced after the move, never for days the previous holder already paid for. +A reservation that starts before the job first sees it is filled in as well: the catch-up pass prices each elapsed day back to `ptu_effective_from`, up to 91 days. A deployment configured today with a backdated start therefore accrues its whole window on the first run. A day the job has already priced keeps its rows when shares later move between teams, so the team taking over a share is billed only for days priced after the move, never for days the previous holder already paid for. A day whose write fails is written for none of its teams, and the next catch-up pass prices it for all of them. Flat cost does not count against team or key budgets. Reserved capacity is already paid for, so a team cannot exhaust a budget by using the capacity it reserved. From 87ddf8b78d496a023d38eae14af3c0da37d0b616 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 14 Oct 2026 08:28:47 +0000 Subject: [PATCH 14/17] docs(proxy): cover PTU share holders, secret-manager flag, team deletes, and unfittable requests --- docs/proxy/ptu_flat_cost.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index ebf4ccfe0..a641ee317 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -16,6 +16,8 @@ The feature is off by default and inert until you opt in: export LITELLM_ENABLE_PTU_COST_ATTRIBUTION=True ``` +A configured secret manager can hold the variable instead. When it is not set in the environment, the proxy reads it from the secret manager at most once a minute, and a value set in the environment always wins. + With the variable unset the daily job is not scheduled, the model endpoints reject PTU configuration, the usage read path reports zero flat cost, and the PTU inputs stay hidden in the model form. ## Configure a deployment @@ -139,6 +141,10 @@ A shared deployment keeps its public model name and stays visible in model lists Deployment gpt-4.1-ptu is reserved for the teams holding a PTU share of it ``` +A team holding a share is served only from the deployments it holds a share on. An open deployment under the same model name serves the teams holding none, so a team's ceiling and its PTU-hours count only the traffic its reservation serves. To send a team's overflow past its share to pay-as-you-go capacity, use Azure's spillover target, described under [Spillover requests](#spillover-requests), rather than an open deployment beside the shared one. + +A team named in `ptu_shares` cannot be deleted while it holds the share. `POST /team/delete` returns a 400 naming the deployment until the share is moved to another team, since a deleted team would keep being billed for PTUs nobody can use. + ### Each share is a per-minute ceiling Azure sizes a provisioned deployment in normalized tokens per minute: uncached input tokens in full, cached input at the model's cached ratio (a tenth for the GPT-6 family, nothing for the rest), and output tokens weighted by the model's output-to-input ratio, divided by the model's input TPM per PTU. LiteLLM ships that table, read from [Azure's provisioned throughput sizing page](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/provisioned-throughput-sizing#deployment-parameters-and-throughput-values-by-model), and turns each share into a per-minute ceiling in the same units: @@ -154,9 +160,13 @@ Team A's 30 PTUs of gpt-4.1 (3,000 input TPM per PTU, output counted at 4x) are Rate limit exceeded for model_per_team_ptu: :gpt-4.1-ptu. Limit type: tokens. Current limit: 90000, Remaining: 0. Limit resets at: ... ``` +A request that could never fit, such as one whose `max_tokens` alone is more than the share allows in a minute, gets a 400 instead, since no empty minute would admit it and a `retry-after` would only make the client retry it. + The ceiling lives on the same per-minute window as a team's `model_tpm_limit`, so it reads and resets the way the limits you already set do, and it needs nothing beyond `LITELLM_ENABLE_PTU_COST_ATTRIBUTION`. The sizing row is looked up by `model_info.base_model` first, then by the model in `litellm_params`, because an Azure deployment name is arbitrary. A shared deployment, or an Azure deployment reserved for one team, whose model has no row logs a warning at startup, sets no ceiling, and reports no PTU-hours; set `base_model` to the Azure model name to fix it. A single-team reservation on another provider only feeds the flat-cost rollup, which needs no sizing, so it is not warned about -A request counts against its team's ceiling however it names the shared deployment: the model group, a `model_group_alias`, a routing group, the deployment's `model_info.id`, its provider model, the `/azure` pass-through route, or a wildcard pattern such as `ptu-*` that the router serves it through. A wildcard name counts only against the pattern the router serves it from, the most specific one that matches, so a request an open `*` route serves never charges a share, and a team holding no share gets the 400 on a name the shared pattern matches even when `*` matches it too +A request counts against its team's ceiling however it names the shared deployment: the model group, a `model_group_alias`, a routing group, the deployment's `model_info.id`, its provider model, or a wildcard pattern such as `ptu-*` that the router serves it through. A wildcard name counts only against the pattern the router serves it from, the most specific one that matches, so a request an open `*` route serves never charges a share, and a team holding no share gets the 400 on a name the shared pattern matches even when `*` matches it too. + +The ceiling is checked for the model a request names, before the proxy routes it. A fallback hop into the shared group is not checked against it, and neither is a call through the `/azure` pass-through route, which runs no rate limit checks. A routing group that spans shared deployments in more than one model group counts all of its traffic against one of those groups, the first in the model list, so keep each routing group to one shared model group. A deployment owned by one team through `team_id` gets no ceiling, since that team already owns all of it, but its usage does report PTU-hours @@ -171,7 +181,7 @@ flat cost = ptu_count x cost_per_ptu_per_hour x hours active that day A deployment split with `ptu_shares` writes one row per team instead, each carrying that team's share in place of `ptu_count`: 30 and 20 of the 50 PTUs at $1 per PTU-hour are $720 and $480 for a full day, and the deployment's $1,200 lands on nobody else. Active hours are the overlap between the day and the reservation window, so a reservation starting at noon accrues 12 hours on its first day and 24 thereafter. The rows are written into `LiteLLM_DailyTeamSpend` under the reserved key `__ptu_flat_cost__`, which keeps flat cost separate from the per-request spend recorded against real API keys. -A reservation that starts before the job first sees it is filled in as well: the catch-up pass prices each elapsed day back to `ptu_effective_from`, up to 91 days. A deployment configured today with a backdated start therefore accrues its whole window on the first run. A day the job has already priced keeps its rows when shares later move between teams, so the team taking over a share is billed only for days priced after the move, never for days the previous holder already paid for. A day whose write fails is written for none of its teams, and the next catch-up pass prices it for all of them. +A reservation that starts before the job first sees it is filled in as well: the catch-up pass prices each elapsed day back to `ptu_effective_from`, up to 91 days. A deployment configured today with a backdated start therefore accrues its whole window on the first run. A day the job has already priced keeps its rows when shares later move between teams, so the team taking over a share is billed only for days priced after the move, never for days the previous holder already paid for. A day whose write fails is written for none of its teams, and the next catch-up pass prices it for all of them. The job splits a day by the shares as they stand when it runs, so shares moved during a day bill that whole day by the new split. Flat cost does not count against team or key budgets. Reserved capacity is already paid for, so a team cannot exhaust a budget by using the capacity it reserved. From 22eea15b1a1838daa50e7305e20d060f85d7aa87 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 14 Oct 2026 08:59:31 +0000 Subject: [PATCH 15/17] docs(proxy): note that /azure pass-through tokens do not count against a PTU share --- docs/proxy/ptu_flat_cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index a641ee317..40611bf1a 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -166,7 +166,7 @@ The ceiling lives on the same per-minute window as a team's `model_tpm_limit`, s A request counts against its team's ceiling however it names the shared deployment: the model group, a `model_group_alias`, a routing group, the deployment's `model_info.id`, its provider model, or a wildcard pattern such as `ptu-*` that the router serves it through. A wildcard name counts only against the pattern the router serves it from, the most specific one that matches, so a request an open `*` route serves never charges a share, and a team holding no share gets the 400 on a name the shared pattern matches even when `*` matches it too. -The ceiling is checked for the model a request names, before the proxy routes it. A fallback hop into the shared group is not checked against it, and neither is a call through the `/azure` pass-through route, which runs no rate limit checks. A routing group that spans shared deployments in more than one model group counts all of its traffic against one of those groups, the first in the model list, so keep each routing group to one shared model group. +The ceiling is checked for the model a request names, before the proxy routes it. A fallback hop into the shared group is not checked against it, and neither is a call through the `/azure` pass-through route, which runs no rate limit checks and whose tokens do not count against the ceiling. A routing group that spans shared deployments in more than one model group counts all of its traffic against one of those groups, the first in the model list, so keep each routing group to one shared model group. A deployment owned by one team through `team_id` gets no ceiling, since that team already owns all of it, but its usage does report PTU-hours From a2d767a121353565861dccf2695b870fc9d87185 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 14 Oct 2026 10:25:13 +0000 Subject: [PATCH 16/17] docs(proxy): cover PTU share guards on org and SCIM deletes, the implicit max_tokens cap, large output budgets, and dated PTU-hours --- docs/proxy/ptu_flat_cost.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 40611bf1a..474a69787 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -143,7 +143,7 @@ Deployment gpt-4.1-ptu is reserved for the teams holding a PTU share of it A team holding a share is served only from the deployments it holds a share on. An open deployment under the same model name serves the teams holding none, so a team's ceiling and its PTU-hours count only the traffic its reservation serves. To send a team's overflow past its share to pay-as-you-go capacity, use Azure's spillover target, described under [Spillover requests](#spillover-requests), rather than an open deployment beside the shared one. -A team named in `ptu_shares` cannot be deleted while it holds the share. `POST /team/delete` returns a 400 naming the deployment until the share is moved to another team, since a deleted team would keep being billed for PTUs nobody can use. +A team named in `ptu_shares` cannot be deleted while it holds the share. `POST /team/delete`, `DELETE /organization/delete` for an organization that owns such a team, and the SCIM `DELETE /scim/v2/Groups/{id}` each return a 400 naming the deployment until the share is moved to another team, since a deleted team would keep being billed for PTUs nobody can use. The organization delete is checked before anything is removed, so a refused call leaves every team in place. ### Each share is a per-minute ceiling @@ -160,7 +160,9 @@ Team A's 30 PTUs of gpt-4.1 (3,000 input TPM per PTU, output counted at 4x) are Rate limit exceeded for model_per_team_ptu: :gpt-4.1-ptu. Limit type: tokens. Current limit: 90000, Remaining: 0. Limit resets at: ... ``` -A request that could never fit, such as one whose `max_tokens` alone is more than the share allows in a minute, gets a 400 instead, since no empty minute would admit it and a `retry-after` would only make the client retry it. +A request that could never fit, such as one whose `max_tokens` alone is more than the share allows in a minute, gets a 400 instead, since no empty minute would admit it and a `retry-after` would only make the client retry it. Large output budgets need large shares: gpt-5.4 and gpt-5.4-mini count output at 6x, so `max_tokens: 32000` alone is 192,000 normalized tokens and needs a share of more than 80 PTUs of gpt-5.4 or 24 of gpt-5.4-mini. + +A request without `max_tokens` gets one written for it when the share is small. When the share counted in output tokens (the ceiling divided by the output ratio) is under 4,096, the proxy sends `max_tokens` as 1,024 or a quarter of that, whichever is smaller, the same cap it writes under any `tpm_limit` that small, and a longer reply stops with `finish_reason: length`. Three PTUs of gpt-5.4-mini cap an uncapped reply at 987 output tokens and one PTU of gpt-5.4 at 100. A client that sends its own `max_tokens` is never rewritten, so a team on a small share that needs long replies sets it. The ceiling lives on the same per-minute window as a team's `model_tpm_limit`, so it reads and resets the way the limits you already set do, and it needs nothing beyond `LITELLM_ENABLE_PTU_COST_ATTRIBUTION`. The sizing row is looked up by `model_info.base_model` first, then by the model in `litellm_params`, because an Azure deployment name is arbitrary. A shared deployment, or an Azure deployment reserved for one team, whose model has no row logs a warning at startup, sets no ceiling, and reports no PTU-hours; set `base_model` to the Azure model name to fix it. A single-team reservation on another provider only feeds the flat-cost rollup, which needs no sizing, so it is not warned about @@ -199,7 +201,7 @@ This needs `LITELLM_ENABLE_PTU_COST_ATTRIBUTION` set and a pricing map entry for ## Read the cost back -`/team/daily/activity` reports `flat_cost` per day and `total_flat_cost` for the range, alongside the usual per-token `spend`, and `ptu_hours` next to the tokens: on each day, on each PTU model group and the API keys under it, and as `total_ptu_hours` for the range. PTU-hours are the team's prompt, cached, and completion tokens converted through the model's sizing row, so a team can compare what it consumed with what it reserved: +`/team/daily/activity` reports `flat_cost` per day and `total_flat_cost` for the range, alongside the usual per-token `spend`, and `ptu_hours` next to the tokens: on each day, on each PTU model group and the API keys under it, and as `total_ptu_hours` for the range. PTU-hours are the team's prompt, cached, and completion tokens converted through the model's sizing row, so a team can compare what it consumed with what it reserved. A day's tokens are sized only by a reservation whose `ptu_effective_from` to `ptu_effective_to` window covers that day, so usage before a reservation starts or after it ends reads no PTU-hours: ```bash curl -s "http://localhost:4000/team/daily/activity?team_ids=&start_date=2026-01-01&end_date=2026-01-31" \ From 8c4abab45977cbbb2d6474410354b729c6f5daa8 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 14 Oct 2026 12:47:08 +0000 Subject: [PATCH 17/17] docs(proxy): secret manager outranks the env for the PTU flag, and team rows keep ptu_hours at 0 --- docs/proxy/ptu_flat_cost.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/proxy/ptu_flat_cost.md b/docs/proxy/ptu_flat_cost.md index 474a69787..4644c2d70 100644 --- a/docs/proxy/ptu_flat_cost.md +++ b/docs/proxy/ptu_flat_cost.md @@ -16,7 +16,7 @@ The feature is off by default and inert until you opt in: export LITELLM_ENABLE_PTU_COST_ATTRIBUTION=True ``` -A configured secret manager can hold the variable instead. When it is not set in the environment, the proxy reads it from the secret manager at most once a minute, and a value set in the environment always wins. +A configured secret manager can hold the variable instead. When the proxy would ask a secret manager for it, as it does for other settings, the secret manager's value wins over the environment and is read at most once a minute. The environment decides only when no secret manager is consulted. With the variable unset the daily job is not scheduled, the model endpoints reject PTU configuration, the usage read path reports zero flat cost, and the PTU inputs stay hidden in the model form. @@ -201,7 +201,7 @@ This needs `LITELLM_ENABLE_PTU_COST_ATTRIBUTION` set and a pricing map entry for ## Read the cost back -`/team/daily/activity` reports `flat_cost` per day and `total_flat_cost` for the range, alongside the usual per-token `spend`, and `ptu_hours` next to the tokens: on each day, on each PTU model group and the API keys under it, and as `total_ptu_hours` for the range. PTU-hours are the team's prompt, cached, and completion tokens converted through the model's sizing row, so a team can compare what it consumed with what it reserved. A day's tokens are sized only by a reservation whose `ptu_effective_from` to `ptu_effective_to` window covers that day, so usage before a reservation starts or after it ends reads no PTU-hours: +`/team/daily/activity` reports `flat_cost` per day and `total_flat_cost` for the range, alongside the usual per-token `spend`, and `ptu_hours` next to the tokens: on each day, on each PTU model group and the API keys under it, and as `total_ptu_hours` for the range. PTU-hours are the team's prompt, cached, and completion tokens converted through the model's sizing row, so a team can compare what it consumed with what it reserved. A day's tokens are sized only by a reservation whose `ptu_effective_from` to `ptu_effective_to` window covers that day, so usage before a reservation starts or after it ends reads no PTU-hours. Each team's row under `breakdown.entities` keeps `ptu_hours` at 0, since that row carries no model group to size by, so read one team's PTU-hours from a request naming only that team: ```bash curl -s "http://localhost:4000/team/daily/activity?team_ids=&start_date=2026-01-01&end_date=2026-01-31" \