feat(prometheus): configure deployment caller identity by mphilippnv · Pull Request #38221 · BerriAI/litellm

mphilippnv · 2026-08-25T16:42:40Z

TLDR

Problem this solves:

Deployment metrics cannot identify callers by email
Early deployment failures can lose caller email context

How it solves it:

Adds alias, email, and combined identity modes
Preserves the existing alias schema by default
Resolves failure emails through deterministic fallback sources

User Flow

Before: a proxy operator cannot build an email-scoped deployment dashboard

They set prometheus_deployment_and_latency_caller_identity: user_email and restart the proxy
Their application sends POST https://litellm.example.com/v1/chat/completions with a virtual key linked to an email
They scrape GET https://litellm.example.com/metrics and the nine deployment/latency families still expose api_key_alias, not user_email

After: the same configuration exposes the resolved email on those families

They set prometheus_deployment_and_latency_caller_identity: user_email and restart the proxy
Their application sends POST https://litellm.example.com/v1/chat/completions with a virtual key linked to an email
They scrape GET https://litellm.example.com/metrics and the nine deployment/latency families expose user_email, including early failure counters

Relevant issues

Closes #38159

Companion documentation: BerriAI/litellm-docs#1012

Linear ticket

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: each case ran in a clean Python process with the public setting set to user_email, a real PrometheusLogger, one emitted request outcome, and a scrape of prometheus_client.REGISTRY. No mocks or network providers were used.

Before (`bb27bfd`)

Successful request

Run the logger emission and registry scrape against origin/litellm_internal_staging
Observe litellm_deployment_total_requests_total{...,api_key_alias="dashboard-user",...} 1.0; user_email is absent

Deployment failure using UserAPIKeyAuth fallback

Emit a deployment failure whose email is available only as UserAPIKeyAuth.user_email, then scrape the registry
Observe litellm_deployment_failure_responses_total{...,api_key_alias="dashboard-user",...} 1.0; user_email is absent

After (`9b331ae`)

Successful request

Run the identical logger emission and registry scrape against this PR
Observe litellm_deployment_total_requests_total{...,user_email="alice@example.com",...} 1.0; api_key_alias is absent

Deployment failure using UserAPIKeyAuth fallback

Run the identical failure emission and registry scrape against this PR
Observe litellm_deployment_failure_responses_total{...,user_email="fallback@example.com",...} 1.0; api_key_alias is absent

Type

🆕 New Feature
✅ Test

Caveats (if any)

Mode changes require restarting every proxy/logger process
Email labels expose sensitive caller information
Only the nine approved metric families change

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

Medium Risk
Changing Prometheus label names on core deployment/latency metrics can break dashboards and recording rules; default api_key_alias preserves behavior, but user_email/both expose PII and require process restart.

Overview
Adds prometheus_deployment_and_latency_caller_identity (api_key_alias default, user_email, or both) so nine deployment/latency Prometheus families can label callers by API key alias, user email, or both instead of always using api_key_alias.

Label schemas are resolved in PrometheusMetricLabels.get_labels without mutating class-level lists; invalid modes fail at PrometheusLogger init and when the proxy loads litellm_settings (before Prometheus callbacks). user_email mode rejects prometheus_metrics_config that still include_labels api_key_alias on those metrics.

PrometheusLogger now resolves user_email from standard logging metadata, litellm params, or UserAPIKeyAuth, including deployment failure paths. Comprehensive tests cover schema modes, boot order, and scraped samples.

^{Reviewed by Cursor Bugbot for commit cdaaf40. Bugbot is set up for automated code reviews on this repo. Configure here.}

CLAassistant · 2026-08-25T16:42:52Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ yucheng-berri
❌ mphilipp622
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

codecov · 2026-08-25T16:45:46Z

Codecov Report

❌ Patch coverage is 87.93103% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/types/integrations/prometheus.py	89.74%	4 Missing ⚠️
litellm/integrations/prometheus.py	84.61%	2 Missing ⚠️
litellm/proxy/proxy_server.py	75.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-08-25T16:46:01Z

Greptile Summary

The PR adds configurable alias, email, or combined caller identity labels for nine Prometheus deployment and latency metric families while retaining alias-only behavior by default.

Validates the identity mode before Prometheus collectors are registered.
Loads the setting before callback initialization and checks incompatible metric-label configuration.
Adds email fallback resolution for deployment-failure metrics.
Adds focused schema, emission, fallback, and configuration-order tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/types/integrations/prometheus.py	Defines caller-identity modes, validation, and mode-dependent label-schema resolution for the targeted metrics.
litellm/integrations/prometheus.py	Validates configuration during logger initialization and propagates fallback email identity into deployment-failure metric labels.
litellm/proxy/proxy_server.py	Applies and validates the caller-identity setting before callback construction.
litellm/init.py	Adds the public setting with the backward-compatible api_key_alias default.
tests/test_litellm/integrations/test_prometheus_caller_identity.py	Covers all identity modes, real registry samples, failure fallback precedence, exclusions, validation, and callback initialization order.

_{Reviews (6): Last reviewed commit: "fix(prometheus): validate caller identit..." | Re-trigger Greptile}

codspeed-hq · 2026-08-25T17:25:50Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing mphilippnv:mphilipp/38159-prometheus-caller-identity (cdaaf40) with litellm_internal_staging (3e2927d)¹}

No successful run was found on litellm_internal_staging (e300822) during the generation of this report, so 3e2927d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

yucheng-berri · 2026-08-25T20:05:13Z

@greptileai review latest head

yucheng-berri · 2026-08-25T20:05:17Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a3ce23e. Configure here.}

yucheng-berri · 2026-08-26T00:22:41Z

@mphilippnv
Scope question: in user_email mode, co-emitted
litellm_llm_api_latency_metric and
litellm_request_queue_time_seconds keep api_key_alias, so
identity joins within one request break. #38159 doesn't mention those
two, are they deliberate, or worth adding to the frozenset?

mphilippnv · 2026-08-26T00:59:26Z

Good catch. The original seven were deliberate only in that they matched the families observed and listed in #38159; there was no semantic reason to exclude these two. Both are emitted from _set_latency_metrics with the same UserAPIKeyLabelValues, so retaining api_key_alias in user_email mode would make the request latency breakdown internally inconsistent.

I added litellm_llm_api_latency_metric and litellm_request_queue_time_seconds to the frozenset in 9b331ae, updated the docs PR, and extended the real-sample test to assert both histogram outputs in all three modes. The focused suite is now 55/55, and the strict Ruff, format, type-discipline, test-quality, and basedpyright delta gates pass. Default api_key_alias behavior remains unchanged.

@greptileai review latest head

mphilippnv · 2026-08-26T01:10:11Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9b331ae. Configure here.}

yucheng-berri · 2026-08-26T05:19:25Z

@greptileai review latest head

yucheng-berri · 2026-08-26T05:19:30Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cc89ded. Configure here.}

yucheng-berri · 2026-08-26T05:44:35Z

@greptileai review latest head

yucheng-berri · 2026-08-26T05:44:40Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1e3ab6a. Configure here.}

…ster Fail config load on an invalid prometheus_deployment_and_latency_caller_identity value (including null) and on include_labels entries the selected mode removes from a target metric, instead of booting green with an empty /metrics. Validate the mode at the top of PrometheusLogger.__init__ so an invalid value raises before any collector lands in the process-global registry, keeping retries free of duplicated-timeseries errors. Label-validation errors now name the mode setting alongside the rejected label.

yucheng-berri · 2026-08-26T06:00:54Z

@greptileai review latest head

yucheng-berri · 2026-08-26T06:01:02Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cdaaf40. Configure here.}

yucheng-berri · 2026-08-26T06:05:56Z

Hi @mphilippnv, I hit one failure mode while live-testing this PR and pushed a small fix to your branch in cdaaf40. Hope that’s okay. Happy to move it to a follow-up if you prefer.

A bad value for the new setting (user-email, YAML on/null) could let the proxy boot successfully while silently disabling all Prometheus metrics. The same happened when user_email mode conflicted with an existing include_labels: [api_key_alias].

The commit makes these fail fast instead:

validate the mode during config load
validate again before any Prometheus collectors are registered
reject user_email + include_labels: [api_key_alias] with a clear error

Valid configs are unchanged. I verified all three modes on a live proxy with a real key/email/provider, plus the failure cases above.

feat(prometheus): configure deployment caller identity

8d0c23e

mphilippnv mentioned this pull request Aug 25, 2026

docs: explain Prometheus caller identity modes BerriAI/litellm-docs#1012

Merged

test(prometheus): satisfy strict caller identity lint

a3ce23e

cursor Bot reviewed Aug 25, 2026

View reviewed changes

fix(prometheus): align caller identity on latency metrics

9b331ae

cursor Bot reviewed Aug 26, 2026

View reviewed changes

yucheng-berri force-pushed the mphilipp/38159-prometheus-caller-identity branch 2 times, most recently from c9b9d81 to 1e3ab6a Compare August 26, 2026 05:43

cursor Bot reviewed Aug 26, 2026

View reviewed changes

yucheng-berri force-pushed the mphilipp/38159-prometheus-caller-identity branch from 1e3ab6a to cdaaf40 Compare August 26, 2026 05:56

cursor Bot reviewed Aug 26, 2026

View reviewed changes

yucheng-berri approved these changes Aug 26, 2026

View reviewed changes

yucheng-berri merged commit e52f055 into BerriAI:litellm_internal_staging Aug 26, 2026
79 of 80 checks passed

yucheng-berri mentioned this pull request Aug 26, 2026

test(prometheus): cover caller-identity config failure cases #38380

Merged

6 tasks

Uh oh!

Uh oh!

Conversation

mphilippnv commented Aug 25, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (bb27bfd)

Successful request

Deployment failure using UserAPIKeyAuth fallback

After (9b331ae)

Successful request

Deployment failure using UserAPIKeyAuth fallback

Type

Caveats (if any)

Final Attestation

Uh oh!

CLAassistant commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

codspeed-hq Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

yucheng-berri commented Aug 25, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Aug 26, 2026

Uh oh!

mphilippnv commented Aug 26, 2026

Uh oh!

mphilippnv commented Aug 26, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Aug 26, 2026

Uh oh!

yucheng-berri commented Aug 26, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Aug 26, 2026

Uh oh!

yucheng-berri commented Aug 26, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Aug 26, 2026

Uh oh!

yucheng-berri commented Aug 26, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Aug 26, 2026

Uh oh!

Uh oh!

mphilippnv commented Aug 25, 2026 •

edited by cursor Bot

Loading

Before (`bb27bfd`)

After (`9b331ae`)

CLAassistant commented Aug 25, 2026 •

edited

Loading

codecov Bot commented Aug 25, 2026 •

edited

Loading

greptile-apps Bot commented Aug 25, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 25, 2026 •

edited

Loading