fix(prometheus): fold auth/pre-call time into litellm_request_total_latency_metric

[yassin-berriai]

TLDR

Problem this solves:

• litellm_request_total_latency_metric silently excludes authentication and pre-call setup time
• litellm_request_queue_time_seconds has the same problem despite its own doc comment claiming otherwise

How it solves it:

• Stamp a true request-arrival timestamp unconditionally at the top of auth, and fold it into both metrics

User Flow

Before: an operator's dashboard says request latency is fine while their users experience multi-second delays from a slow auth backend

1. A customer's Redis/DB-backed auth path becomes slow under load (project-scoped key checks, budget lookups, etc.)
2. Their litellm_request_total_latency_metric Grafana panel stays flat and low, because the metric's clock only starts after auth already finished
3. Their own external tracing (e.g. Datadog's trace.fastapi.request) reports several seconds higher for the same window, with no explanation from LiteLLM's own metrics
4. litellm_request_queue_time_seconds, despite its name and docs, also reads near-zero during the same slow-auth window

After: the total-latency metric reflects what the customer's users actually experienced

1. The same slow auth path occurs under load
2. litellm_request_total_latency_metric rises to match real end-to-end latency, including the auth delay
3. litellm_request_queue_time_seconds rises by (approximately) the auth delay itself, correctly isolating that phase
4. The operator can now alert on and diagnose an auth regression from LiteLLM's own metrics, without needing external tracing

Relevant issues

Linear ticket

Resolves LIT-6012

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy, real OpenAI call (gpt-4o-mini), no mocks. general_settings.custom_auth points at a small local function that await asyncio.sleep(3.0)s before returning a valid key -- a controllable stand-in for a slow Redis/DB-backed auth backend, run inside the real user_api_key_auth flow (the sleep happens after the point this PR's timestamp is stamped, so it lands in the "auth" window either way).

Setup (shared by both runs): litellm_settings.callbacks: ["prometheus"], one real openai/gpt-4o-mini deployment, custom_auth sleeping 3.0s, proxy on port 24012.

Before (490c9f9)

1. curl -H "Authorization: Bearer sk-test-key" http://localhost:24012/v1/chat/completions -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Say the word OK and nothing else."}]}' -> HTTP 200, real completion ("content":"OK"), wall-clock 7.454s total
2. curl -sL http://localhost:24012/metrics | grep '_sum':
   • litellm_request_total_latency_metric_sum{...} 4.430722 -- misses ~3s of the 7.454s real request
   • litellm_request_queue_time_seconds_sum{...} 0.0001049... -- reads as essentially zero despite the real 3s auth delay

After (bcfd107)

1. Same command, same config, same 3.0s auth sleep -> HTTP 200, real completion, wall-clock 4.572s total
2. curl -sL http://localhost:24012/metrics | grep '_sum':
   • litellm_request_total_latency_metric_sum{...} 4.552430082550049 -- now matches the real wall-clock time
   • litellm_request_queue_time_seconds_sum{...} 3.009190082550049 -- now matches the 3.0s simulated auth delay almost exactly

Type

🐛 Bug Fix

Caveats (if any)

• This closes the auth/pre-call gap. A second, smaller gap remains: end_time for these metrics is captured when the LLM response is received, not when the response finishes being sent to the client (post-call hooks, guardrails, serialization). Not addressed here; filed as a follow-up consideration in LIT-6012 if it turns out to matter in practice
• Companion docs fix (litellm-docs) updates the Prometheus metrics reference to describe the corrected behavior and to be more readable in general

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR stamps request arrival before authentication and carries that timestamp through queue-time and total-latency Prometheus metrics

• Uses a shared boundary between queue time and processing latency to prevent overlap
• Preserves the first request timestamp independently of OpenTelemetry configuration
• Adds focused regression coverage for timestamp propagation and metric observations

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/integrations/prometheus.py	Adds non-overlapping queue time to total request latency and updates metric descriptions
litellm/proxy/auth/user_api_key_auth.py	Stamps an idempotent request-arrival timestamp before authentication and fully fixes the prior local-variable rebinding
litellm/proxy/common_request_processing.py	Aligns the queue interval endpoint with the logging start timestamp, resolving the previously reported overlap
litellm/proxy/litellm_pre_call_utils.py	Prefers the authentication-entry timestamp while preserving a fallback for callers outside the normal auth flow
tests/test_litellm/integrations/test_prometheus_queue_guardrail_metrics.py	Adds coverage for queue-time inclusion, fallback behavior, and invalid negative values
tests/test_litellm/proxy/auth/test_user_api_key_auth.py	Verifies unconditional and idempotent request-arrival stamping
tests/test_litellm/proxy/test_common_request_processing.py	Verifies queue and processing intervals share an exact boundary
tests/test_litellm/proxy/test_litellm_pre_call_utils.py	Verifies stamped arrival-time preference and fallback timestamp behavior

_{Reviews (2): Last reviewed commit: "fix(prometheus): fold auth/pre-call time..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 88.88889% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/auth/user_api_key_auth.py	81.81%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai both findings were right: fixed the overlapping-interval double-count (queue_time_seconds now ends exactly at start_time) and the Final-annotation rebinding in the timestamp helper. Please review the current head bcfd107.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit6012_request_total_latency_metric (bcfd107) with litellm_internal_staging (490c9f9)}