feat(proxy): add native ROI calculator for gateway spend vs merged PRs by devin-ai-integration[bot] · Pull Request #43669 · BerriAI/litellm

devin-ai-integration · 2026-09-29T06:14:49Z

TLDR

Problem this solves:

Compare gateway spend with estimated merged-PR effort inside LiteLLM
Preserve the standalone calculator's simple setup and scheduled updates

How it solves it:

Three steps: GitHub, repositories, estimator and schedule
Daily reruns, persistent estimate reuse, progress and cancellation
Overview, People, email matching, CSV export and sample preview
Admin permissions, encrypted credentials and normal gateway inference accounting

Intentional product change: the calculator runs inside the gateway, using a GitHub token instead of the standalone GitHub App. Full gateway admins configure it; read-only admins view results. Team admins and regular users cannot access the calculator

Scope: ROI calculator only. Dependency manifests, lockfiles, Google API tests, and shared encryption behavior match main. Includes main through 50f5cc9bbb

User Flow

Before: the initial native implementation exposes all settings together and requires manual refreshes

Open /roi-calculator/ as a gateway admin
Enter GitHub connection, repositories, model and prompt together
Save settings and start analysis
Return later and manually refresh the report

After: guided setup leads to the first report and scheduled updates

Open /roi-calculator/ as a gateway admin
Connect GitHub, select repositories, then choose an estimator
Keep daily updates or select manual updates, then start backfill
Review Overview, inspect PR reasoning, and match emails in People
Return to automatically refreshed results; change preferences in Settings

Parity with the standalone calculator

Capability	Gateway implementation
Calculation	Same matched-spend / estimated-hours calculation and cohort exclusions
Estimator	Same default prompt, temperature 0, metadata-only evidence and three concurrent PRs
Initial window and schedule	Seven-day backfill, daily refresh, manual option, minimum five-minute schedule
Reuse	Successful unchanged PRs reused before details or inference; failed estimates retried
Progress	Elapsed time, estimated remaining time, reused count and cancellation
Reporting	Overview, People, calculation details, PR reasoning and CSV export
Email matching	GitHub profile/commit emails plus manual matches to observed gateway identities
Setup tools	Connection checks, sample preview, advanced prompt/key options and restart setup

Intentional differences:

The gateway connection is already available, so no separate URL/key setup or standalone service is needed
GitHub token authentication replaces the GitHub App creation/install flow; Enterprise API URLs remain supported
Gateway PostgreSQL replaces local SQLite/files, and secrets are encrypted using gateway secret storage
Shared database leases coordinate multiple gateway workers, including cancellation and recovery after interruption
Gateway admin roles protect reports and settings; the standalone requires an external access-control layer
Superseded cache versions are removed after successful re-estimation; estimates outside the selected date/repository scope are retained
One unreadable PR is marked for review while other results publish; a complete metadata outage preserves the last report, and failed metadata is retried
Unlike standalone, a repository outage permits an explicitly incomplete report from healthy repositories, with ratios withheld; complete outages preserve the previous report
Profile emails refresh for reused PRs without another estimate; transient lookup failures retain known addresses, while confirmed email removal clears them
Total estimator failure and partial repository outages with no usable PRs preserve the previous report

Screenshot walkthrough

Screenshots use an isolated local gateway, two real public PRs from BerriAI/litellm-admin-agent, and paid estimator calls. engineer@example.com is a synthetic QA identity used only to demonstrate matching

Setup: GitHub, repositories, estimator, first backfill

Overview, pull requests and estimate reasoning

People: before matching, matching dialog, matched result

Settings, advanced options, restart confirmation and sample preview

Matching calculator icons in the sidebar and page heading

Screenshots / Proof of Fix

Shared setup: local gateway on port 4036, dashboard on port 3036, isolated PostgreSQL database, real public GitHub PRs, roi-estimator routed to paid gpt-6-luna. Authentication below uses the local admin key through an environment variable, never a committed secret

Before (`1afcc91`, initial PR implementation)

Open /roi-calculator/ with no saved configuration: all setup fields appear together, shown above
Configure and analyze the public repository: a manual refresh is required for another run; the initial implementation has no scheduled refresh setting

After (`5d642e6`)

Complete the guided setup shown above; the live report contains saved estimates of 36 and 32 hours from paid gateway calls
Send curl -X POST http://localhost:4036/roi-calculator/connections/test -H "Authorization: Bearer $ROI_ADMIN_KEY": HTTP 200
Send curl -X POST http://localhost:4036/roi-calculator/sync -H "Authorization: Bearer $ROI_ADMIN_KEY", then GET the same URL: {"phase":"complete","total":2,"estimated":2,"reused":2,"needs_attention":0}
Set a five-minute schedule and age the isolated test database's last-run timestamp: the real gateway timer starts a run without a button press, reusing both estimates; restore the daily schedule afterward
Compare the saved real report with both standalone and native calculation functions: every metric matches exactly with automatic matching and with a manual QA mapping
Test a restricted inference key: connection validation returns HTTP 409 and unauthorized estimates fail instead of using admin privileges; restore the valid key and complete both estimates
Add BerriAI/roi-qa-missing-repository alongside the real public repository in the isolated QA configuration and start analysis: two existing PR estimates remain, the unavailable repository is named, and cost-per-hour is absent
Remove that unavailable repository and rerun: both estimates are reused, the warning clears, and the ratio returns
Select the empty BerriAI/litellm-roi-calculator repository alongside the unavailable QA repository and analyze: an error explains that no report was published, and both previous estimates remain; restore the original repositories
In isolated QA, use an invalid estimator key with a changed prompt and run analysis: the authentication failure preserves the previous report; restore the original prompt and key, then rerun successfully using both cached estimates
Send PUT /roi-calculator/identity-map with {"github_login":"invalid.name","email":"engineer@example.com"}: HTTP 422, with settings unchanged

Additional regression checks: 77 ROI backend/database tests and 15 ROI UI tests passed using the unchanged dependencies from main. Real PostgreSQL coverage verifies scope-preserving cache cleanup, exclusive sync ownership, writer routing with a separate reader, the scheduled interval gate, expired-lease fencing, and UTC timestamp recovery. A repeated-startup regression also verifies the scheduler retains exactly one ROI job; it fails with the original duplicate-job error before the fix. Full make check passes

Validation on 5d642e6211: all required CI checks pass, Codecov patch coverage passes at 84.13% against 82.51%, Greptile is 5/5 with no outstanding findings, and Veria reports no security issues. All reported review findings have tested fixes

Outage regressions cover unavailable repositories, empty partial results, estimator failures, profile lookup failures versus confirmed email removal, and recovery. Refreshed identities persist across restarts; unchanged identities skip database writes. Live PostgreSQL row versions confirm no cache writes during an unchanged rerun

Bugbot found no issues on deb9d2d5e5; its final rerun is unavailable because the team spending limit was reached. The only subsequent change skips unchanged cache writes and is covered by regression tests and live database checks

Validation limitation: Python CodeQL fails because Security/CWE-117/LogInjection.ql exceeds its 2 GiB result-set limit on both this PR and main. This is an incomplete scan, not a reported security finding; scan configuration and protections are unchanged. Required checks pass on 5d642e6211

OSV baseline: a fresh scan of main at 264b09ac8d reproduces the same five findings as this PR, affecting the existing PyJWT, urllib3 and Next.js versions. All scanned lockfiles and scan configuration are identical to main; dependency upgrades remain outside this ROI-only change

Unrelated CI baseline: the unchanged Google API compliance tests fail against upstream schema changes. Those tests and dependencies remain identical to main

Pre-Submission checklist

Added meaningful behavioral and real-database regression tests
Focused backend and UI tests pass locally
Latest required CI checks pass
Changes are scoped to the ROI calculator, gateway integration, tests and screenshots
Latest Greptile confidence is at least 4/5

Caveats

Medium

GitHub App onboarding is replaced by the accepted token flow
Scheduled updates require a running gateway and available database
Partial repository outages hide ratios until access recovers
Authorized admins share the connection and repository report visibility

Low

Estimated effort is not measured savings or financial return
Source patches and personal emails are excluded from estimator evidence
PR descriptions and commit messages may themselves contain code
Exact model outputs can vary despite temperature 0
CSV content is tested; native browser download completion remains unverified

Type

New feature, bug fixes and regression tests

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

devin-ai-integration · 2026-09-29T06:14:56Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

CLAassistant · 2026-09-29T06:14:58Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ moe-berri
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

greptile-apps · 2026-09-29T06:21:24Z

Confidence Score: 5/5

[Medium risk] Adds ROI calculator feature with GitHub integration and spend analysis.

The PR appears safe to merge based on this review; the latest cache-write change leaves no outstanding finding.

Summary

The PR adds a gateway-native ROI calculator with guided setup, scheduled analysis, persistent estimate reuse, reporting, and admin-only access.

The latest change avoids rewriting saved PR records when a reused PR’s identity has not changed.
Changed identities continue to be persisted, including confirmed email removal.

_{Reviews (18) · Last reviewed commit: "perf(roi): skip writes for unchanged cac..."}

codspeed · 2026-09-29T06:25:12Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_roi_calculator_v2 (5d642e6) with main (50f5cc9)¹}

No successful run was found on main (657bb77) during the generation of this report, so 50f5cc9 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

codecov · 2026-09-29T06:26:14Z

Codecov Report

❌ Patch coverage is 86.01494% with 206 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...y/management_endpoints/roi_calculator_endpoints.py	60.41%	114 Missing ⚠️
litellm/proxy/roi_calculator/github.py	84.30%	43 Missing ⚠️
litellm/proxy/roi_calculator/sync.py	93.33%	19 Missing ⚠️
litellm/proxy/roi_calculator/sync_store.py	60.46%	17 Missing ⚠️
litellm/proxy/roi_calculator/estimator.py	88.57%	8 Missing ⚠️
litellm/types/roi_calculator.py	99.27%	3 Missing ⚠️
litellm/proxy/roi_calculator/pull_cache.py	88.88%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

moe-berri · 2026-09-30T16:05:47Z

bugbot run

moe-berri · 2026-09-30T16:05:48Z

@greptileai

moe-berri · 2026-09-30T16:05:49Z

@veria-ai

cursor

Stale Bugbot comment from a previous run.

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

moe-berri · 2026-09-30T19:55:33Z

bugbot run

moe-berri · 2026-09-30T19:55:34Z

@greptileai

moe-berri · 2026-09-30T19:55:35Z

@veria-ai

cursor

Stale Bugbot comment from a previous run.

moe-berri · 2026-09-30T20:05:28Z

bugbot run

moe-berri · 2026-09-30T20:05:30Z

@greptileai

moe-berri · 2026-09-30T20:05:31Z

@veria-ai

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit deb9d2d. Configure here.}

moe-berri · 2026-09-30T20:13:32Z

bugbot run

moe-berri · 2026-09-30T20:13:35Z

@greptileai

cursor · 2026-09-30T20:13:35Z

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

moe-berri · 2026-09-30T20:13:37Z

@veria-ai

moe-berri · 2026-09-30T20:45:28Z

bugbot run

cursor · 2026-09-30T20:45:30Z

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

…route sweep GET /roi-calculator/repositories (#43669) lists repositories from the configured GitHub API, api.github.com by default, so the S2 sweep's GET of every route made the owned proxy reach an external host and failed the egress check in 31 integration-security tests. It joins /get/latest_release_info in the deny list

…fixtures (#43958) * test(ci): repair stale request fakes, spend-log golden, auto-router labels, and Interactions spec lookups Request fakes now carry the scope a real Starlette request has, the GCS pub/sub spend-log golden gains the agent identity keys from #43722, the auto-router session tests follow the baseline_models contract from #43348, and the Interactions spec checks resolve the create body and resource paths from the live spec instead of hardcoded names * test(ci): move retired OpenAI text-completion fixtures to live vehicles OpenAI still serves native /v1/completions on the gpt-5.4 family, so the single-prompt cases move to text-completion-openai/gpt-5.4-nano. Multi-prompt batches and echo with logprobs now 500 on every OpenAI model, so those cases keep the same text-completion-openai transport pointed at Fireworks, which documents both. The optional-params test asserts the request body actually sent instead of a success callback whose assertions were swallowed * test(ci): use a serverless Fireworks model for the text-completion batch and echo cases gpt-oss-20b is on-demand only on Fireworks, so the CI key got 404 model not deployed; glm-5p3-flash is listed as serverless * test(ci): skip the ROI calculator repository listing in the security route sweep GET /roi-calculator/repositories (#43669) lists repositories from the configured GitHub API, api.github.com by default, so the S2 sweep's GET of every route made the owned proxy reach an external host and failed the egress check in 31 integration-security tests. It joins /get/latest_release_info in the deny list

feat(proxy): add native ROI calculator for gateway spend vs merged PRs

49871ce

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

devin-ai-integration Bot requested a review from a team September 29, 2026 06:14

devin-ai-integration Bot assigned ishaan-berri Sep 29, 2026

greptile-apps Bot reviewed Sep 29, 2026

View reviewed changes

github-advanced-security Bot found potential problems Sep 29, 2026

View reviewed changes

Comment thread litellm/proxy/roi_calculator/analytics.py Fixed

Comment thread litellm/proxy/roi_calculator/sync.py Fixed

devin-ai-integration Bot and others added 8 commits September 29, 2026 06:29

fix(proxy): serialize ROI Prisma inputs with builtin containers

1e6854a

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

style(proxy): format ROI calculator backend files

7286f84

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

fix: parse fenced ROI estimates and retain completed reports

99bc544

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

fix(roi-calculator): correct estimator and dashboard behavior

738a136

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

chore(ui): drop next dev generated AGENTS.md block

0ba9e67

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

fix(proxy): chunk ROI spend user lookup

a8679be

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

fix(ui): show reused ROI estimates after sync

8f0d8bb

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

fix(security): address ROI CodeQL alerts

8426563

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 07:28 Error

test(proxy): make ROI calculator unit tests discoverable

4a825ce

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 07:53 Error

test(ci): run ROI calculator tests in proxy infra shard

3ecedc8

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 08:15 Error

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread ui/litellm-dashboard/src/app/(dashboard)/roi-calculator/_components/ROICalculatorView.tsx Outdated

cursor Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread litellm/proxy/roi_calculator/github.py Outdated

Comment thread ui/litellm-dashboard/src/app/(dashboard)/roi-calculator/_components/ROICalculatorView.tsx Outdated

fix(roi): page repository search and recover polling errors

1afcc91

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

devin-ai-integration Bot had a problem deploying to e2e-changed September 30, 2026 16:26 Error

moe-berri had a problem deploying to e2e-changed September 30, 2026 19:55 — with GitHub Actions Error

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread litellm/proxy/roi_calculator/sync.py

cursor Bot reviewed Sep 30, 2026

View reviewed changes

fix(roi): persist refreshed identities for reused estimates

deb9d2d

moe-berri had a problem deploying to e2e-changed September 30, 2026 20:05 — with GitHub Actions Error

cursor Bot reviewed Sep 30, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread litellm/proxy/roi_calculator/sync.py Outdated

perf(roi): skip writes for unchanged cached identities

5d642e6

moe-berri deployed to e2e-changed September 30, 2026 20:13 — with GitHub Actions Active

tin-berri approved these changes Sep 30, 2026

View reviewed changes

moe-berri merged commit 6b9766f into main Sep 30, 2026
102 of 104 checks passed

moe-berri deleted the litellm_roi_calculator_v2 branch September 30, 2026 21:16

yuneng-berri mentioned this pull request Oct 1, 2026

test(ci): repair stale tests and move retired OpenAI text-completion fixtures #43958

Merged

devin-ai-integration Bot mentioned this pull request Oct 1, 2026

refactor: clean up fresh tech debt from 2026-09-30 #43993

Merged

8 of 9 tasks

doramirdor mentioned this pull request Oct 5, 2026

feat(complexity-router): add a bundled Nadir classifier plugin #41797

Open

5 tasks done

moe-berri mentioned this pull request Oct 10, 2026

chore: remove committed PR screenshots #45816

Merged

3 of 4 tasks

Conversation

devin-ai-integration Bot commented Sep 29, 2026 • edited by moe-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Parity with the standalone calculator

Screenshot walkthrough

Screenshots / Proof of Fix

Before (1afcc91, initial PR implementation)

After (5d642e6)

Pre-Submission checklist

Caveats

Medium

Low

Type

Uh oh!

devin-ai-integration Bot commented Sep 29, 2026

Uh oh!

CLAassistant commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codspeed Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

codecov Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

moe-berri commented Sep 30, 2026

Uh oh!

devin-ai-integration Bot commented Sep 29, 2026 •

edited by moe-berri

Loading

Before (`1afcc91`, initial PR implementation)

After (`5d642e6`)

CLAassistant commented Sep 29, 2026 •

edited

Loading

greptile-apps Bot commented Sep 29, 2026 •

edited

Loading

codspeed Bot commented Sep 29, 2026 •

edited

Loading

codecov Bot commented Sep 29, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading