test(prometheus): cover caller-identity config failure cases

[yucheng-berri]

TLDR

Problem this solves:

• The caller-identity config validation added in feat(prometheus): configure deployment caller identity #38221 had no unit tests
• A regression in those guards would ship without any test failing

How it solves it:

• Extends the mapped test file with the config failure cases
• Covers typo, null, and include_labels-conflict boot failures end to end
• Pins the error messages, registry retry safety, and accepted values

User Flow

This PR only adds tests, so the user-visible flow is identical before and after. The flow below is what the new tests pin

Before and after: a proxy admin who misconfigures the caller-identity mode gets a failed boot naming the bad setting instead of a silently empty /metrics

1. The admin sets prometheus_deployment_and_latency_caller_identity: user-email (a typo) in config.yaml and starts the proxy
2. The boot exits non-zero with ValueError: Invalid prometheus_deployment_and_latency_caller_identity='user-email'. Accepted values: api_key_alias, user_email, both.
3. Same for a null value, and for include_labels: ["api_key_alias"] on a target metric while the mode is user_email, whose error names the conflicting metrics and the fix
4. With a valid mode the proxy boots and serves traffic

Relevant issues

Follow-up to #38221 (refs #38159): adds the unit tests requested in review for the config failure cases

Linear ticket

Resolves LIT-6134

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

This PR changes no runtime code (287 net added lines, all in one test file), so there is no Before leg to capture: the guarded behavior shipped in #38221 and is identical at the merge base and the tip. The run below shows the live proxy behavior the new unit tests pin, captured at the PR tip

Shared setup: config files as described per case, each with one model_list entry and callbacks: ["prometheus"], proxy started with python litellm/proxy/proxy_cli.py --config <case>.yaml --port 46134

Behavior pinned by the new tests (5bf21fa)

typo mode

1. config.yaml sets prometheus_deployment_and_latency_caller_identity: user-email
2. Boot exits with code 3: ValueError: Invalid prometheus_deployment_and_latency_caller_identity='user-email'. Accepted values: api_key_alias, user_email, both.

null mode

1. config.yaml sets prometheus_deployment_and_latency_caller_identity: null
2. Boot exits with code 3: ValueError: Invalid prometheus_deployment_and_latency_caller_identity=None. Accepted values: api_key_alias, user_email, both.

include_labels conflict

1. config.yaml sets the mode to user_email and prometheus_metrics_config with include_labels: ["api_key_alias"] for litellm_deployment_total_requests
2. Boot exits with code 3: ValueError: prometheus_metrics_config include_labels contains 'api_key_alias' for litellm_deployment_total_requests, but prometheus_deployment_and_latency_caller_identity='user_email' replaces that label on these metrics. Use 'user_email' in include_labels or change the mode.

valid mode (control)

1. config.yaml sets prometheus_deployment_and_latency_caller_identity: user_email
2. Proxy boots, and curl http://127.0.0.1:46134/health/liveliness returns "I'm alive!"

Type

✅ Test

Behavior changes

None, this PR only adds tests

Caveats (if any)

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[yucheng-berri]

bugbot run

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[greptile-apps]

Greptile Summary

This test-only PR expands coverage for Prometheus caller-identity configuration validation without changing runtime behavior.

• Covers accepted, default, invalid, and null caller-identity modes.
• Verifies conflicts between caller identity and metric label configuration.
• Exercises proxy boot failure ordering and registry-safe retries.
• Pins caller-identity-specific validation messages.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/test_litellm/integrations/test_prometheus_caller_identity.py	Adds focused unit and proxy-loading tests for caller-identity validation, conflict reporting, initialization ordering, and retry behavior without introducing an eligible blocking finding.

_{Reviews (2): Last reviewed commit: "test(prometheus): narrow pytest.raises w..." | Re-trigger Greptile}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8d239a8. Configure here.}

[yucheng-berri]

@greptileai please review the current head 5bf21fa

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5bf21fa. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!