Skip to content

feat(ptu): runtime UI flag for enable_ptu_cost_attribution + always-registered rollup - #33439

Closed
yucheng-berri wants to merge 1 commit into
litellm_lit1697_stage4_uifrom
litellm_lit1697_stage5_rollout
Closed

feat(ptu): runtime UI flag for enable_ptu_cost_attribution + always-registered rollup#33439
yucheng-berri wants to merge 1 commit into
litellm_lit1697_stage4_uifrom
litellm_lit1697_stage5_rollout

Conversation

@yucheng-berri

@yucheng-berri yucheng-berri commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Relevant issues

Stage 5 of the LIT-1697 stack, sitting on top of #33302

Linear ticket

Resolves LIT-1697

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

QA runbook below has the concrete steps against a live proxy hitting real Azure PTU + gpt-4. Paste output from steps 1 through 5 back once run

Type

New Feature

Changes

Wires enable_ptu_cost_attribution into the runtime-flippable UI settings pipeline so an admin can turn the whole PTU cost attribution feature on and off from the Admin Settings page without a proxy restart. The flag is now declared on UISettings, added to ALLOWED_UI_SETTINGS_FIELDS, and added to _RUNTIME_GENERAL_SETTINGS_FLAGS, which is the list /update/ui_settings and /get/ui_settings sync into general_settings; the same sync also runs on startup via _sync_ui_settings_to_general_settings, so a value persisted through the UI survives a restart.

Moves the daily rollup job's scheduler.add_job call outside the flag gate. run_ptu_reservation_rollup already returns early when the flag is off (added in #33137 as skipped_flag_off=True), so scheduling the job unconditionally is a no-op until the flag flips. This is what makes the flag actually live-flippable; before this PR, flipping the flag would only affect the endpoints and UI, and the rollup would still need a restart to attach.

Adds a Switch on the Admin Settings > UI Settings page ("Enable PTU cost attribution") next to the existing enable_chat_ui toggle, with the same page-refresh notice pattern used for enable_projects_ui and enable_chat_ui. The page-refresh handles reflowing the leftnav to show the PTU Reservations item and the Usage page column.

Behavior changes

  • PATCH /update/ui_settings now accepts enable_ptu_cost_attribution: bool; unknown-field validation keeps the same shape it did for the other UI flags
  • GET /get/ui_settings returns enable_ptu_cost_attribution in values and its description in field_schema
  • The ptu_reservation_rollup_job cron is registered on every startup regardless of flag state; it fires daily at 00:15 UTC and no-ops (returning RollupResult(skipped_flag_off=True)) while the flag is off. Zero-write, zero-read while disabled
  • The nav item and the Usage page's Flat Cost column continue to gate on the runtime flag through useIsPtuCostAttributionEnabled; no config-file change required to unlock them once the UI toggle is on

Nothing else changes for tenants that never touch the flag. The per-request spend hot path is not modified in this PR (or any earlier stage in the stack).

QA runbook

Prereq: real Azure PTU deployment. Run against a live proxy started with python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log and a fresh DB.

  1. Confirm the flag is off by default. curl -H "Authorization: Bearer sk-1234" http://localhost:4000/get/ui_settings | jq '.values.enable_ptu_cost_attribution' returns false (or absent). curl -H "Authorization: Bearer sk-1234" -X POST http://localhost:4000/ptu_reservation/new -H "Content-Type: application/json" -d '{"team_id":"any","model":"gpt-4","ptu_count":1,"cost_per_ptu":100.0,"effective_from":"2026-08-01T00:00:00Z"}' returns 403 with PTU cost attribution is not enabled.

  2. Flip it on via the UI. Open http://localhost:4000/ui/ > Admin Settings > UI Settings. Toggle "Enable PTU cost attribution" on. The page reloads. The leftnav now shows "PTU Reservations".

  3. Confirm the sync happened without a restart. curl -H "Authorization: Bearer sk-1234" http://localhost:4000/get/ui_settings | jq '.values.enable_ptu_cost_attribution' returns true.

  4. Create a reservation and see the flat cost land. Follow steps 2 through 5 of 05-stage-rollout.md's original runbook (create reservation for a real Azure gpt-4 team, run python scripts/ptu_reservation_backfill.py --date 2026-07-15, then GET /team/daily/activity for that team). Confirm metrics.flat_cost is nonzero for that day.

  5. Flip it off. Toggle back off in the UI, refresh. Nav item disappears. POST /ptu_reservation/new returns 403 again. GET /team/daily/activity still returns the historical row from step 4 unchanged (flag-off does not delete data).

  6. Restart the proxy after step 2 (with the flag still on in the DB). Confirm on restart that the log shows Synced UI settings to general_settings on startup: ['enable_ptu_cost_attribution'] and step 3's curl still returns true. Persistence survives restart.

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Open in Devin Review

@greptile-apps

greptile-apps Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR wires enable_ptu_cost_attribution into the runtime UI settings pipeline and unconditionally registers the daily rollup cron job so the feature can be toggled from the Admin Settings page without a proxy restart.

  • enable_ptu_cost_attribution is added to UISettings, ALLOWED_UI_SETTINGS_FIELDS, and _RUNTIME_GENERAL_SETTINGS_FLAGS, making it persist to the DB and sync into general_settings on both write and startup — which is the correct data path for a flag checked server-side by the rollup job and PTU endpoints.
  • The scheduler.add_job call for run_ptu_reservation_rollup is moved outside the flag gate; the job itself continues to return early with skipped_flag_off=True while the flag is off, keeping the no-op promise.
  • A Switch toggle with page-refresh behavior is added to the Admin Settings > UI Settings page, following the same pattern as enable_chat_ui and enable_projects_ui.

Confidence Score: 4/5

Safe to merge — the core logic change (unconditional job registration + flag-gated execution) is correct, the sync path is consistent with other runtime flags, and the UI follows established patterns.

The implementation is clean and well-tested. The one minor concern is that test_ptu_rollup_job_registered_regardless_of_flag asserts against the module-level ps.scheduler object after the with block exits, which relies on that global not being reset between tests. If test ordering changes, a later test could see a scheduler in an unexpected state. It is not a production defect but is worth noting.

tests/test_litellm/proxy/test_proxy_server.py — the new test reads module-level scheduler state after context-manager patches exit; worth confirming test isolation holds across the full suite run.

Important Files Changed

Filename Overview
litellm/proxy/proxy_server.py Moves PTU rollup job registration outside the feature-flag gate; import and scheduler.add_job now run unconditionally. The job's own flag check in run_ptu_reservation_rollup handles the no-op path cleanly.
litellm/proxy/ui_crud_endpoints/proxy_setting_endpoints.py Adds enable_ptu_cost_attribution to UISettings model, ALLOWED_UI_SETTINGS_FIELDS, and _RUNTIME_GENERAL_SETTINGS_FLAGS. The flag correctly lands in general_settings on both write and startup sync, which is required for the rollup's runtime flag check.
tests/test_litellm/proxy/test_proxy_server.py Adds test asserting PTU rollup job is registered even when enable_ptu_cost_attribution is False at startup. Test relies on module-level scheduler state; has a minor concern around test isolation but is otherwise well-structured.
tests/test_litellm/proxy/ui_crud_endpoints/test_proxy_setting_endpoints.py Adds two tests: one verifying PATCH persists and syncs enable_ptu_cost_attribution to general_settings, and one asserting membership in both ALLOWED_UI_SETTINGS_FIELDS and _RUNTIME_GENERAL_SETTINGS_FLAGS. Good coverage of the sync path.
ui/litellm-dashboard/src/components/Settings/AdminSettings/UISettings/UISettings.tsx Adds the enable_ptu_cost_attribution Switch toggle with page-refresh behavior, matching the existing enable_chat_ui and enable_projects_ui patterns. aria-label defaults to the schema description.
ui/litellm-dashboard/src/components/Settings/AdminSettings/UISettings/UISettings.test.tsx Adds test verifying the PTU toggle calls updateSettings with the correct payload and triggers the page-refresh success notification. Mock schema uses 'Enable PTU cost attribution' as the description (matching the aria-label query), so the test is internally consistent.

Reviews (1): Last reviewed commit: "feat(ptu): runtime UI flag for enable_pt..." | Re-trigger Greptile

@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@yucheng-berri
yucheng-berri force-pushed the litellm_lit1697_stage4_ui branch from 4100e34 to 06d8b45 Compare July 15, 2026 20:24
@yucheng-berri
yucheng-berri force-pushed the litellm_lit1697_stage5_rollout branch 2 times, most recently from a06bcf7 to 0ffa82f Compare July 15, 2026 21:11
…egistered rollup

Add enable_ptu_cost_attribution to UISettings so admins can flip it from
the UI's Admin Settings page without a proxy restart. Register the daily
rollup job unconditionally at startup; it already no-ops on flag-off,
so the runtime toggle takes effect on the next scheduled fire without
any restart.

Existing per-request guards in the /ptu_reservation endpoints and in
run_ptu_reservation_rollup remain the source of truth. Persisted
settings sync into general_settings on both PATCH and GET paths as
well as on server startup via _sync_ui_settings_to_general_settings.
@yucheng-berri
yucheng-berri force-pushed the litellm_lit1697_stage5_rollout branch from 0ffa82f to 1b85426 Compare July 21, 2026 18:53

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

Closing as superseded. This is part of the v1 PTU design, which stored PTU config in a separate reservation table. The shipped design puts that config on the model deployment instead, merged as #35341, #35343, #35391, #35393 and #36829.

The runtime gate landed as litellm/proxy/spend_tracking/ptu_feature_flag.py.

The branch is kept, so nothing here is lost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant