feat(prometheus): add litellm_total_overhead_latency_metric (SDK overhead + guardrails)

[kunal2002]

Relevant issues

Surfaces LiteLLM's own per-request internal overhead, including guardrails, as a single Prometheus metric, so dashboards can read it directly instead of subtracting two unrelated percentiles of separate metrics.

Pre-Submission checklist

• I have added meaningful tests
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• My PR passes all unit tests on make test-unit
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5

Type

New Feature

Changes

LiteLLM already exposes litellm_overhead_latency_metric, but it only measures the SDK wrapper window: it reads litellm_overhead_time_ms = (end_time - start_time) - llm_api_duration, and end_time is captured the instant the provider response returns into the wrapper (litellm/utils.py, right after result = await original_function(...)). Proxy guardrails run outside that window (pre-call before the wrapper starts, post-call after end_time, during-call concurrently with the LLM call), so no existing metric reflects guardrail latency as part of LiteLLM's overhead. There was no single number for "how much latency does LiteLLM itself add around the call, including guardrails".

This PR adds a new histogram, litellm_total_overhead_latency_metric, recorded once per successful request alongside the existing overhead metric in litellm/integrations/prometheus.py:

litellm_total_overhead_latency_metric
    = litellm_overhead_time_ms / 1000                 # existing SDK overhead
    + sum(pre_call + post_call guardrail durations)   # from StandardLoggingPayload.guardrail_information

A new _get_guardrail_overhead_seconds helper sums the guardrail durations already recorded on the StandardLoggingPayload and excludes during_call (moderation) guardrails, because those run concurrently with the LLM API call (asyncio.gather); their wall-clock time overlaps the provider call and is not additive overhead, so counting it would over-report.

The metric reuses the exact label set and latency buckets of litellm_overhead_latency_metric, and is registered in litellm/types/integrations/prometheus.py (DEFINED_PROMETHEUS_METRICS and PrometheusMetricLabels) so the label machinery and metric-name validation accept it. No existing metric's value or math is changed.

Key additions:

# litellm/integrations/prometheus.py - metric definition
self.litellm_total_overhead_latency_metric = self._histogram_factory(
    "litellm_total_overhead_latency_metric",
    "Total internal latency (seconds) added by LiteLLM, including "
    "pre/post-call guardrails (excludes the LLM API call)",
    labelnames=self.get_labels_for_metric("litellm_total_overhead_latency_metric"),
    buckets=self.latency_buckets,
)

# litellm/integrations/prometheus.py - additive guardrail time (during_call excluded)
@staticmethod
def _get_guardrail_overhead_seconds(standard_logging_payload: StandardLoggingPayload) -> float:
    total = 0.0
    for info in standard_logging_payload.get("guardrail_information") or []:
        mode = info.get("guardrail_mode")
        mode_value = getattr(mode, "value", mode)
        if mode_value == GuardrailEventHooks.during_call.value:
            continue
        total += float(info.get("duration") or 0.0)
    return total

# litellm/integrations/prometheus.py - recorded next to the existing overhead metric
guardrail_overhead_seconds = self._get_guardrail_overhead_seconds(standard_logging_payload)
self.litellm_total_overhead_latency_metric.labels(**total_overhead_labels).observe(
    (litellm_overhead_time_ms / 1000) + guardrail_overhead_seconds
)  # set as seconds

Because it is a histogram (same buckets as the other latency metrics), p95/p99 are available server side and aggregate correctly across replicas:

histogram_quantile(0.95, sum(rate(litellm_total_overhead_latency_metric_bucket[5m])) by (le))

Tests and tooling

Added tests/test_litellm/integrations/test_prometheus_total_overhead.py (4 tests): the helper sums pre and post and excludes during_call; tolerates plain-string guardrail_mode (post serialization) and missing durations; returns 0.0 with no guardrails; and the histogram registers in the Prometheus registry. The existing test_prometheus_metric_name_consistency.py and test_prometheus_missing_metrics.py suites (which enumerate every defined metric) pass with the new metric, confirming no name or label collision.

Proof

Computed value and live /metrics scrape (250 ms SDK overhead plus pre 100 ms plus post 50 ms guardrails; during-call 500 ms correctly excluded, total 0.4s):

guardrail_overhead_seconds (during_call excluded) = 0.15
total_overhead_seconds = overhead(0.25) + guardrails(0.15) = 0.4

# HELP litellm_total_overhead_latency_metric Total internal latency (seconds) added by LiteLLM, including pre/post-call guardrails (excludes the LLM API call)
# TYPE litellm_total_overhead_latency_metric histogram
litellm_total_overhead_latency_metric_bucket{...,le="0.5",...} 1.0
litellm_total_overhead_latency_metric_count{...} 1.0
litellm_total_overhead_latency_metric_sum{...} 0.4

Unit tests:

tests/test_litellm/integrations/test_prometheus_total_overhead.py ....  [100%]
4 passed

[greptile-apps]

Greptile Summary

This PR adds litellm_total_overhead_latency_metric, a new Prometheus histogram that surfaces LiteLLM's full per-request internal overhead — SDK wrapper time plus sequential (pre/post-call) guardrail latency — as a single scrapeable metric. It is recorded via a new _set_total_overhead_metric helper that runs outside the existing SDK-overhead walrus gate, so guardrail-only overhead is captured even when litellm_overhead_time_ms is zero or absent.

• _get_guardrail_overhead_seconds sums only pre_call and post_call guardrail durations; during_call, logging_only, MCP-specific, and realtime modes are excluded because they either overlap with the provider call or don't block the user-facing response.
• GuardrailMode TypedDict (unhashable dict at runtime) and list modes are handled safely: non-string values are silently dropped from the mode set, preventing the TypeError that would have aborted downstream metric updates.
• 12 unit tests cover the helper logic, registration, and the integration between _set_total_overhead_metric and the histogram, with no real network calls.

Confidence Score: 5/5

Safe to merge — the change is additive (new metric only), no existing metric values or logging paths are altered, and all previously identified edge cases are handled correctly.

The three previously flagged issues (walrus-operator zero-overhead gate, GuardrailMode dict TypeError, and list-mode during_call exclusion) are all resolved in this revision. The _set_total_overhead_metric helper is now called outside the SDK-overhead gate, the mode resolution strips non-string values before set membership tests, and only the two unambiguously additive modes (pre_call, post_call) are counted. No production behavior is changed for existing metrics.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/integrations/prometheus.py	Adds litellm_total_overhead_latency_metric histogram and supporting helpers; addresses all previously flagged edge cases (zero SDK overhead gate, GuardrailMode dict TypeError, list modes with during_call, logging_only/MCP exclusions).
litellm/types/integrations/prometheus.py	Registers the new metric name in DEFINED_PROMETHEUS_METRICS and PrometheusMetricLabels with the same label set as litellm_overhead_latency_metric; straightforward and correct.
tests/test_litellm/integrations/test_prometheus_total_overhead.py	12 unit tests covering helper correctness, edge cases (dict mode, list mode, zero overhead, no guardrails), and histogram registration; all use mocks with no real network calls.

_{Reviews (3): Last reviewed commit: "fix(prometheus): avoid TypeError on Guar..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 88.46154% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/prometheus.py	88.23%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[kunal2002]

Thanks for the review. Addressed all three points in ad834d7f8b:

1. during_call exclusion for list-typed guardrail_mode (P1): _get_guardrail_overhead_seconds now uses an allowlist instead of a denylist. A duration is counted only when every mode it carries is pre_call or post_call, so a list such as ["pre_call", "during_call"] is excluded (the previous getattr(list, "value", list) comparison could never match the string). Covered by test_get_guardrail_overhead_seconds_excludes_list_mode_with_during_call and test_get_guardrail_overhead_seconds_counts_pure_pre_post_list_mode.

2. logging_only and MCP modes (P2): the same allowlist now excludes logging_only, pre_mcp_call and during_mcp_call, since they do not block the user-facing response and should not inflate user-visible overhead. Covered by test_get_guardrail_overhead_seconds_excludes_logging_only_and_mcp.

3. Metric skipped when SDK overhead is exactly 0 (P2): extracted _set_total_overhead_metric and call it outside the if litellm_overhead_time_ms := gate, so guardrail-only overhead is recorded when litellm_overhead_time_ms is 0 or absent. Covered by test_total_overhead_recorded_when_sdk_overhead_is_zero, test_total_overhead_recorded_when_only_guardrails_no_sdk_overhead, and test_total_overhead_skipped_when_no_overhead_and_no_guardrails.

The existing litellm_overhead_latency_metric value and gate are unchanged. All unit tests pass.

@greptileai

[kunal2002]

Addressed the GuardrailMode TypedDict issue in ac3d7d8f4b.

guardrail_mode can be a GuardrailMode TypedDict (a plain dict at runtime) when an enterprise tag-based Mode guardrail records without an explicit event_type (custom_guardrail.py:788). The previous set comprehension would have put that unhashable dict into a set and raised TypeError: unhashable type: 'dict', aborting the rest of async_log_success_event and skipping later metric updates.

_get_guardrail_overhead_seconds now resolves each mode to its string value and keeps only strings, so dict and None modes can never reach the set. A dict-typed mode is excluded from the overhead sum because it is a per-tag phase config, not a single resolved phase, so its duration cannot be unambiguously attributed to a blocking pre/post phase (counting it could over-report overhead).

Added regression tests (both would raise under the previous code):

• test_get_guardrail_overhead_seconds_ignores_dict_mode_without_error
• test_get_guardrail_overhead_seconds_ignores_dict_inside_list_mode

All 12 tests in test_prometheus_total_overhead.py pass. Thanks for the catch.

@greptileai