fix(prometheus): track spend + suppress failure counter for cancellations

[songkuan-zheng]

Tier classification

• D — Universal mechanism + company opinion in litellm/ core

Tried upstream first?

• No — closing a metric gap left by Phase 3 (PR feat(cancel): billing accuracy + orthogonal delivery/billing status taxonomy #78). Same dogfood window as the Tier-D entry in UPSTREAM_PR_QUEUE.md; will be bundled into the same upstream issue after one internal release.

Summary

After PR #78 / #81, the SpendLogs DB row for a cancelled request has correct status="success" + cancellation_indicator + positive spend (via compute_prompt_only_cost). But Prometheus was still divergent: cancelled requests counted as failures AND litellm_spend_metric undercounted.

Two layers fixed in one PR:

Layer 1 — _failure_handler_helper_fn (litellm_logging.py)

response_cost was hardcoded to 0 before building the StandardLoggingPayload. Every SLP-reading callback (Prometheus, Langfuse, Custom Callback API, OTel, Helicone, …) saw 0 for cancellations even though the DB row carried a positive spend.

Fix: detect asyncio.CancelledError and call compute_prompt_only_cost inline. Best-effort — falls back to 0 on cost-map miss or tokenizer failure, never raises.

Layer 2 — async_log_failure_event (prometheus.py)

The failure-event path didn't increment litellm_spend_metric at all, AND it bumped litellm_llm_api_failed_requests_metric (which pollutes failure-rate alerting with what are really client cancellations).

Fix: branch on standard_logging_payload.metadata.cancellation_indicator. When present:

• Route the SLP's response_cost (now correctly populated by Layer 1) into litellm_spend_metric so Prometheus spend stays in lockstep with the SpendLogs spend column
• SKIP litellm_llm_api_failed_requests_metric and set_llm_deployment_failure_metrics — cancels are not system failures
• Track the cancel cost on the org-budget metric too

Non-cancel failures: original behaviour preserved exactly (gated on the indicator).

Verification

Unit tests (Layer 1 — SLP cost population)

• ✅ 3 new tests in test_litellm_logging.py::TestFailureHandlerCancelCost:
  • test_cancelled_error_populates_prompt_only_cost — Anthropic Haiku + 20×repeated long prompt → response_cost > 0
  • test_cancelled_error_falls_back_to_zero_on_unknown_model — unknown model, no raise, finite result
  • test_non_cancel_exception_keeps_response_cost_zero — ValueError keeps 0
• ✅ 217/217 cancel + spend_tracking regression suite stays green
• ✅ One pre-existing failure in test_logfire_logger_accepts_env_vars_for_base_url is unrelated — missing opentelemetry optional dep on the local Python env (confirmed reproducible BEFORE this PR via git stash).

E2E (Layer 2 — Prometheus routing topology)

• ✅ New e2e/cases/data/35_cancel_prometheus_metrics.sh verifies the routing against the real Prometheus client state inside the running proxy (no Counter mocking — would have been theater). Two-probe discrimination:
  • Probe A — streaming cancel → litellm_llm_api_failed_requests_metric_total delta = 0 (cancel suppressed) + SpendLogs row with cancellation marker
  • Probe B — X-Mock-Fail: 503 control → delta ≥ 1 (real failure DOES count) + SpendLogs row status=failure
• ✅ Wired into run-all-cases after case 32. Full cancel suite (26/27/28/29/30/32/35) all PASS end-to-end after the fix.

Case 35 deliberately doesn't assert spend-metric VALUES because the mock provider's mock-claude isn't in the litellm cost map (cost=0 makes Counter.inc(0) a no-op that's indistinguishable from "Layer 2 didn't fire"). Spend-value coverage lives in:

• The unit tests above (Anthropic Haiku in the cost map → positive response_cost)
• e2e/cases/data/33_real_anthropic_cancel.sh (real provider, asserts the billed SpendLogs spend column)

Impact

Metric	Before	After
Cancelled request → litellm_spend_metric	❌ 0 (cancel via failure_hook)	✅ compute_prompt_only_cost
Cancelled request → litellm_llm_api_failed_requests_metric	❌ +1 (counted as failure)	✅ skipped
Cancelled request → org budget metric	❌ 0	✅ compute_prompt_only_cost
Cancelled request → SLP response_cost for OTHER callbacks (Langfuse, Custom Callback API, OTel)	❌ 0	✅ real prompt-only cost
Non-cancel failure → everything	unchanged	unchanged ✓

Conflict resolutions

N/A — single commit, branch linear from ship/v1.87.0 HEAD 083f325241 (post-#81 merge).

Pre-submission

• Tests in tests/test_litellm/
• make test-unit equivalent passes (217/217 cancel + spend tests; one pre-existing unrelated logfire failure documented)
• PR scope: 1 commit, 2 cohesive layers of the same metric-gap fix