fix(prometheus): bound per-request budget metric emission with a timeout

[fernando-izar]

Relevant issues

N/A

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Deterministic, self-contained reproduction — no Redis/DB, no live load. It calls the patched method directly and captures verbose_logger into an in-memory buffer, exercising both the timeout branch and the happy path. Run from the repo root.

Why timeout=0.0 is a reliable trigger: on CPython, asyncio.wait_for(coro, timeout) short-circuits when timeout <= 0 and raises TimeoutError immediately, before the coroutine runs. So PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT=0.0 deterministically forces the timeout branch on every call — simulating an "infinitely slow Redis" with zero real load. Production default is 5.0s.

for T in 0.0 5.0; do echo "===== timeout=$T ====="; uv run --no-sync python -c "
import os, asyncio, logging, io
os.environ['PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT'] = '$T'
from litellm._logging import verbose_logger
buf = io.StringIO()
h = logging.StreamHandler(buf); h.setLevel(logging.DEBUG)
verbose_logger.addHandler(h); verbose_logger.setLevel(logging.DEBUG)
from litellm.integrations.prometheus import PrometheusLogger
pl = PrometheusLogger()
raised = None
async def main():
    global raised
    try:
        await pl._increment_remaining_budget_metrics(
            user_api_team=None, user_api_team_alias=None,
            user_api_key=None, user_api_key_alias=None,
            litellm_params={}, response_cost=0.0,
            user_id=None, user_api_key_org_id=None)
    except BaseException as e:
        raised = repr(e)
asyncio.run(main())
out = buf.getvalue()
print('Exception raised?:', raised, '(expected: None)')
print('Skip line present?:', 'budget metric emission exceeded' in out)
"; done

Output:

===== timeout=0.0 =====
LiteLLM:DEBUG: prometheus.py:1670 - [Non-Blocking] Prometheus: per-request budget metric emission exceeded 0.0s under load; skipping (values are refreshed by the periodic budget-metrics cron job).
Exception raised?: None (expected: None)   # returns clean, does NOT propagate TimeoutError/CancelledError
Skip line present?: True                   # patched branch executed

===== timeout=5.0 =====
Exception raised?: None (expected: None)
Skip line present?: False                  # happy path unchanged, no regression

Criterion	Expected	Observed
Under timeout (0.0): no exception propagated	none	raised = None ✅
Under timeout (0.0): skips and logs	line present	True ✅
No timeout (5.0): completes normally	no skip line	False ✅
No timeout (5.0): no exception	none	raised = None ✅

The two cases together prove the exact patch behavior: when budget emission exceeds the timeout it is dropped in isolation and silently, without tearing down the whole success-logging event (the original failure mode); when it doesn't exceed, nothing changes. The 5.0 case is the control proving the wrapper adds no regression on the happy path.

Type

🐛 Bug Fix

Changes

Problem

PrometheusLogger._increment_remaining_budget_metrics awaits an asyncio.gather over the key/team/user/org budget branches with no timeout. Each branch reads the Redis cache and the Prisma DB. Under load these reads stall, the gather is awaited unbounded, and the success-logging coroutine exceeds the LoggingWorker watchdog (LOGGING_WORKER_MAX_TIME_PER_COROUTINE, default 20s). The watchdog then cancels the whole event, so every metric emitted after this await (latency, cache, total requests) is dropped too, not only the budget gauges

Requests still return 200, but the Prometheus gauges go incomplete and the logs fill with LoggingWorker error / CancelledError

Fix

Bound only the budget-metric gather with its own asyncio.wait_for. The branches are already non-blocking (return_exceptions=True); on timeout we log and return, letting async_log_success_event keep emitting the remaining metrics instead of being cancelled wholesale. The budget gauges are independently refreshed by the periodic cron, so a skipped per-request emission only loses sub-cron real-time detail, not correctness

The bound is configurable via a new env var PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT (default 5.0s, well below the 20s watchdog). Only asyncio.TimeoutError is caught so an injected CancelledError (cooperative shutdown / outer watchdog) still propagates. The env value is validated to be a finite positive number; anything else (a typo, 0, a negative, nan, or inf) falls back to the default instead of being used, so a bad value cannot silently disable emission (<= 0 skips every call) or reintroduce the unbounded wait (inf)

Tests

tests/test_litellm/integrations/test_prometheus_budget_metrics_timeout.py covers the behaviors: a branch slower than the timeout is skipped without propagating and the skip is logged; with a generous timeout every branch is awaited and nothing is skipped; the env parser returns a finite positive value as-is and falls back to the default for unusable inputs (not-a-number, 0, -1, nan, inf, -inf) and when unset; an outer cancellation while awaiting still propagates instead of being swallowed

[CLAassistant]

All committers have signed the CLA.

[greptile-apps]

Greptile Summary

This PR bounds the asyncio.gather call in PrometheusLogger._increment_remaining_budget_metrics with asyncio.wait_for, preventing slow Redis/DB lookups from consuming the entire LoggingWorker watchdog budget and causing downstream Prometheus metrics to be dropped.

• A new _get_budget_metrics_per_request_timeout() helper reads the timeout from PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT (default 5 s) and validates that the parsed value is finite and positive; invalid values (non-numeric, zero, negative, nan, inf) fall back to the default rather than silently disabling or unbounding the wait.
• Only asyncio.TimeoutError is caught on timeout so that an outer CancelledError (watchdog / cooperative shutdown) still propagates; budget gauges are independently refreshed by the periodic cron job, so a skipped per-request emission loses only sub-cron granularity, not correctness.

Confidence Score: 5/5

Safe to merge; the change is narrowly scoped to wrapping one gather call in a timeout and adds no new code paths that affect the main request flow.

The implementation is logically correct: the timeout is bounded and validated, only TimeoutError is swallowed (CancelledError propagates), the gather branches already used return_exceptions=True so individual failures were already isolated, and the env-var parser guards against all degenerate inputs. The tests cover every specified behavior deterministically without real network calls.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/integrations/prometheus.py	Wraps asyncio.gather for budget metrics in asyncio.wait_for with a configurable timeout; env-var parser correctly validates finiteness and positivity before use.
tests/test_litellm/integrations/test_prometheus_budget_metrics_timeout.py	New test file covering timeout branch, happy path, invalid/valid env parsing, and CancelledError propagation; all network calls are mocked, satisfying the no-real-network-calls rule.

_{Reviews (4): Last reviewed commit: "fix(prometheus): reject non-finite and n..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[fernando-izar]

@greptileai

[fernando-izar]

@greptileai please re-review the latest commits. The earlier P1 (env parser accepting 0, negatives, nan, inf) is fixed in 250bf7f: the value is now validated as finite and positive, otherwise it falls back to the default. 74ac661 only reformats to line-length 88.

[Sameerlite]

@greptileai