feat: add in_flight_requests metric to /health/backlog + prometheus

[ishaan-jaff]

Relevant issues

Pre-Submission checklist

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

• Branch creation CI run Link:
• CI run for the last commit Link:
• Merge / cherry-pick CI run Links:

Type

🆕 New Feature

Changes

Problem

Users were reporting 20s total latency while LiteLLM only logged 10s. The missing 10s was the request sitting in uvicorn's event loop waiting to be dispatched — LiteLLM's clock doesn't start until its handler runs, so that queue wait is completely invisible.

T=0   Request arrives at ALB / uvicorn
      [10s gap — LiteLLM never sees this]
T=10  LiteLLM handler starts → starts its own timer
T=20  Response sent

LiteLLM logs: 10s   User experiences: 20s

What this adds

GET /health/backlog — new per-pod endpoint returning the number of HTTP requests currently in-flight on this uvicorn worker:

{"in_flight_requests": 47}

litellm_in_flight_requests Prometheus gauge — same value, scraped at /metrics. Uses multiprocess_mode="livesum" so it aggregates correctly when running with multiple workers.

Implementation

• litellm/proxy/middleware/in_flight_requests_middleware.py — pure ASGI middleware (not BaseHTTPMiddleware). Increments a class-level counter on request start, decrements in finally so errors never leak. Prometheus gauge is lazily initialised on first request (so PROMETHEUS_MULTIPROC_DIR is already set), with a sentinel flag so the import is attempted exactly once.
• litellm/proxy/proxy_server.py — InFlightRequestsMiddleware registered as the outermost middleware (wraps everything, including CORS and Prometheus auth).
• litellm/proxy/health_endpoints/_health_endpoints.py — /health/backlog endpoint, protected by user_api_key_auth.
• litellm/types/integrations/prometheus.py — litellm_in_flight_requests added to DEFINED_PROMETHEUS_METRICS.

POC results

start — no traffic                          0
low  — +5  requests fired                   5  █████
med  — +5  more (10 total)                 10  ██████████
high — +10 more (20 total)                 20  ████████████████████
peak — +10 more (30 total)                 30  ██████████████████████████████
peak — holding                             30  ██████████████████████████████
drained — all requests done                 0

How SREs use this

in_flight_requests	ALB TargetResponseTime	What it means
High	High	Pod overloaded → scale out
Low	High	Delay is pre-ASGI (network, event loop blocking)
High	Normal	Pod is busy but healthy

Works out of the box with AWS ALB — no nginx header injection needed.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 28, 2026 1:59am

[greptile-apps]

Greptile Summary

Adds an InFlightRequestsMiddleware ASGI middleware to track the number of concurrent HTTP requests per uvicorn worker, exposed via a new GET /health/backlog endpoint and as a litellm_in_flight_requests Prometheus gauge. This addresses a real observability gap where request queuing time inside the event loop was invisible to LiteLLM's internal latency tracking.

• New pure ASGI middleware (InFlightRequestsMiddleware) with class-level counter and lazy Prometheus gauge initialization with sentinel pattern to avoid repeated import attempts
• New authenticated /health/backlog health endpoint returning {"in_flight_requests": N}
• litellm_in_flight_requests added to DEFINED_PROMETHEUS_METRICS type
• Middleware registered as the outermost layer in proxy_server.py (correct for capturing full request lifecycle)
• Includes model pricing data updates for gpt-audio-1.5 and gpt-realtime-1.5 (unrelated to the main feature)
• Test for exception handling (test_counter_decrements_after_error) doesn't actually raise an exception — it returns a 500 response, missing the key try/finally code path

Confidence Score: 4/5

• This PR is safe to merge — the middleware is lightweight, well-isolated, and the feature addresses a real observability need.
• The implementation is clean and follows existing patterns (pure ASGI middleware, lazy initialization). The one issue is a test that doesn't actually exercise the exception path it claims to cover, which is worth fixing but doesn't affect runtime safety.
• tests/test_litellm/proxy/middleware/test_in_flight_requests_middleware.py — the error-handling test needs correction to actually raise an exception.

Important Files Changed

Filename	Overview
litellm/proxy/middleware/in_flight_requests_middleware.py	New ASGI middleware tracking in-flight HTTP requests via a class-level counter and optional Prometheus gauge. Clean implementation with lazy gauge initialization and sentinel pattern to avoid repeated import attempts.
litellm/proxy/health_endpoints/_health_endpoints.py	Adds /health/backlog GET endpoint that returns the in-flight request count. Straightforward, auth-protected, consistent with existing health endpoints.
litellm/proxy/proxy_server.py	Registers InFlightRequestsMiddleware as outermost ASGI middleware (last add_middleware call = first to run). Correct placement for capturing full request lifecycle.
litellm/types/integrations/prometheus.py	Adds litellm_in_flight_requests to DEFINED_PROMETHEUS_METRICS Literal type for metric registration.
tests/test_litellm/proxy/middleware/test_in_flight_requests_middleware.py	Good test coverage for basic counter behavior, but test_counter_decrements_after_error has a misleading name — it tests a 500 response, not an actual exception, missing the key try/finally logic path.
litellm/model_prices_and_context_window_backup.json	Adds model pricing entries for gpt-audio-1.5 and gpt-realtime-1.5 — unrelated to the in-flight requests feature.

Sequence Diagram

sequenceDiagram
    participant Client
    participant InFlightMW as InFlightRequestsMiddleware
    participant PrometheusGauge as Prometheus Gauge
    participant ASGI as ASGI App (FastAPI)
    participant Handler as /health/backlog

    Client->>InFlightMW: HTTP Request
    InFlightMW->>InFlightMW: _in_flight += 1
    InFlightMW->>PrometheusGauge: gauge.inc()
    InFlightMW->>ASGI: await app(scope, receive, send)
    ASGI->>Handler: Route to endpoint
    Handler->>InFlightMW: get_in_flight_requests()
    Handler-->>Client: {"in_flight_requests": N}
    InFlightMW->>InFlightMW: _in_flight -= 1
    InFlightMW->>PrometheusGauge: gauge.dec()

Loading

_{Last reviewed commit: c9e84f5}

[greptile-apps]

_{6 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Repeated import attempt on every request when prometheus_client is missing

When prometheus_client is not installed, _get_gauge() catches the ImportError and leaves _gauge as None. Since the check on line 22 is if _gauge is not None, every subsequent HTTP request will re-enter the try block and attempt the import again, failing each time. Consider using a sentinel value (e.g., _gauge_initialized = False) to avoid retrying after the first failure:

Suggested change

	_in_flight: int = 0
	# Lazily created on first request so PROMETHEUS_MULTIPROC_DIR is already set
	# by the time we register the metric.
	_gauge: Optional[object] = None
	def _get_gauge() -> Optional[object]:
	global _gauge
	if _gauge is not None:
	return _gauge
	try:
	from prometheus_client import Gauge
	kwargs = {}
	if "PROMETHEUS_MULTIPROC_DIR" in os.environ:
	# livesum aggregates across all worker processes in the scrape response
	kwargs["multiprocess_mode"] = "livesum"
	_gauge = Gauge(
	"litellm_in_flight_requests",
	"Number of HTTP requests currently in-flight on this uvicorn worker",
	**kwargs,
	)
	except Exception:
	pass
	return _gauge
	_in_flight: int = 0
	# Lazily created on first request so PROMETHEUS_MULTIPROC_DIR is already set
	# by the time we register the metric.
	_gauge: Optional[object] = None
	_gauge_initialized: bool = False
	def _get_gauge() -> Optional[object]:
	global _gauge, _gauge_initialized
	if _gauge_initialized:
	return _gauge
	_gauge_initialized = True
	try:
	from prometheus_client import Gauge
	kwargs = {}
	if "PROMETHEUS_MULTIPROC_DIR" in os.environ:
	# livesum aggregates across all worker processes in the scrape response
	kwargs["multiprocess_mode"] = "livesum"
	_gauge = Gauge(
	"litellm_in_flight_requests",
	"Number of HTTP requests currently in-flight on this uvicorn worker",
	**kwargs,
	)
	except Exception:
	pass
	return _gauge

[greptile-apps]

Gauge bypasses PrometheusLogger metric-enable/disable framework

The existing PrometheusLogger (in litellm/integrations/prometheus.py) uses _create_metric_factory which checks _is_metric_enabled() and respects the enabled_metrics configuration — allowing users to disable specific metrics. This new gauge is created directly via prometheus_client.Gauge(...), so it will always be registered regardless of the user's Prometheus metric configuration. Although this metric is added to DEFINED_PROMETHEUS_METRICS, the enable/disable check is never applied here.

This is worth aligning for consistency with the existing metric management pattern, though it may be intentionally standalone since it's middleware-level rather than logger-level.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[ishaan-jaff]

@greptileai review

[greptile-apps]

Test doesn't cover the exception path it claims to test

The docstring says "Counter must reach 0 even when the handler raises" but the handler returns a normal Response("boom", status_code=500) — it never raises an exception. This means the finally block in InFlightRequestsMiddleware.__call__ (which is the whole reason for using try/finally) is never exercised by this test.

To actually test exception resilience, the handler should raise:

Suggested change

	def test_counter_decrements_after_error():
	"""Counter must reach 0 even when the handler raises."""
	async def handler(request: Request) -> Response:
	return Response("boom", status_code=500)
	TestClient(_make_app(handler)).get("/")
	assert get_in_flight_requests() == 0
	def test_counter_decrements_after_error():
	"""Counter must reach 0 even when the handler raises."""
	async def handler(request: Request) -> Response:
	raise RuntimeError("boom")
	client = TestClient(_make_app(handler), raise_server_exceptions=False)
	client.get("/")
	assert get_in_flight_requests() == 0