diff --git a/CLAUDE.md b/CLAUDE.md index d1754d821552..b17ffa5a9b3b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -285,6 +285,34 @@ Tests live in `tests/test_litellm/` (unit), `tests/llm_translation/` (per-provider integration), `tests/proxy_unit_tests/` (proxy), and `tests/load_tests/`. Full-stack scenarios go in `e2e/cases/`. +- **No theater tests — mock the boundary, not the unit under test.** + A test that mocks out the very function or module it is supposed to + exercise, then asserts "the mock was called" or "the mock's return + value flowed through," verifies nothing real. It looks green in CI + but cannot catch a bug in the production code it ostensibly covers. + Classic anti-patterns to refuse: + - `MyModule.foo = MagicMock(return_value=fake); … ; assert + other.bar.call_args.kwargs["x"] is fake` — you tested that + Python can pass references. + - `obj.handler = AsyncMock(); … ; assert obj.handler.called` — + you tested that your code calls a method you told it to call. + - All-MagicMock stubs of objects whose real interface might drift — + the test passes against a contract that does not exist in prod. + - Asserting a Literal/Enum value at runtime (`assert + MyLiteral == "value"`) — that is mypy's job, not the test's. + + The fix is to mock the **process / network boundary** (httpx + transport, DB cursor, fanout queue, OS clock) and let the unit + under test run against real inputs producing real outputs. For + observable side effects (a callback firing, a row written), use a + **spy fake** — a real class that captures the actual arguments it + receives — and assert on the captured *data*, not on `.called`. + Example: instead of mocking `stream_chunk_builder`, pass real + `ModelResponseStream` chunks in and assert on the reassembled + response's `usage.completion_tokens`. The test now catches a + cursor-handling regression in `stream_chunk_builder` itself; the + theater version cannot. + - **Write assertions from the spec, not the impl.** For new features, the `e2e/cases/NN_*.md` runbook IS the spec — write it before the fixture and the impl. For bug fixes, the issue's repro steps are the diff --git a/UPSTREAM_PR_QUEUE.md b/UPSTREAM_PR_QUEUE.md index dd266700600b..ce6fcb7302fe 100644 --- a/UPSTREAM_PR_QUEUE.md +++ b/UPSTREAM_PR_QUEUE.md @@ -46,6 +46,8 @@ Status legend: | PR #52 (Wave 5c) | READY | `fix(anthropic): extend transform_to_anthropic_error to cover broader status code set` | Smallest diff in the queue — builds on upstream's existing helper. Good warm-up after #29748. | | PR #60 + #67 (Wave 6e + Layer 1 fix) | READY | `fix(proxy): record TTFT on passthrough streaming path + apply per-deployment overrides before fast_path SSE short-circuit` | The fast_path piece (#67) is a regression introduced by upstream PR #28289 — file directly with that PR's commit referenced. Pair with passthrough TTFT (#60). | | PR #48 (Wave 3 small UI fixes) | READY | Three UI fixes: Gemini provider `api_base` field on credential form; credential-form reset on close; Mode badge rendering on model_list | All independent. File as a single PR with 3 commits since they all touch the credential form area. | +| fix/billing-accuracy-phase-1 (Phase 1 — first commit) | READY | `fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives` | Independent, smallest-blast-radius bug in the cancel-billing series. `stream_chunk_builder_utils.py` cursor=1 escape valve. Pure Tier C. Can be filed BEFORE Phase 3 dogfood completes. | +| fix/billing-accuracy-phase-1 (Phase 1 — black-hole catch) | READY | `feat(cancel): catch asyncio.CancelledError in streaming + non-stream proxy paths` | The Phase 1 "BaseException slipping through `except Exception`" hole. SpendLogs row + Langfuse trace closure for cancelled requests. Tier C; mechanism only — the billing strategy stays in our metadata derivation (see Phase 3 row below). | ### Tier D — issue first, then mechanism PR @@ -59,6 +61,7 @@ or `ship/` config. | PR #54 (Wave 5a) | ISSUE-FIRST | Add `cache_ttl` label to `litellm_input_cache_creation_tokens_metric` (Anthropic 5m vs 1h split) | Mechanism = label; opinion = whether `prompt_cache_*_tokens_metric` should be dropped. File issue documenting why per-TTL bucketing matters for cost attribution. | | PR #53 (Wave 5b) | ISSUE-FIRST | Per-deployment `returned_model_name` override | Mechanism = the field on `model_list[].litellm_params`. Existing upstream `_override_openai_response_model` is per-call, not per-deployment. Issue should reference this gap. | | PR #59 (Wave 6d) | ISSUE-FIRST | Two related opt-ins: thinking-signature retry flag; `anthropic_beta_overrides` per-deployment | Combined Anthropic-features PR. The `anthropic_beta_overrides` part needs the Bedrock-gateway use case in the issue. | +| fix/billing-accuracy-phase-1 (Phase 3 — cancel taxonomy) | ISSUE-FIRST | Cancel-billing taxonomy: orthogonal `delivery_status` + `billing_status` in metadata, derived from existing cancel markers | **Upstream-friendly: StandardLoggingPayloadStatus stays binary (matches upstream)**; all new fields are additive Optional metadata. Issue should propose the 7-row semantic mapping (normal / streaming-partial / shield-success / shield-timeout / zero-chunk / cancel-upstream-error / failure) + the derivation helper as the single source of truth. Dogfood for one internal release before filing the issue. Tier D because we're shipping our derivation as the default policy. See `e2e/cases/data/26-33_*.sh` + the spend_tracking_utils derivation function for spec. | ### Dependent PRs (file after a prerequisite lands) @@ -89,4 +92,4 @@ For reference only. Do NOT submit: - Last reviewed / refreshed: keep a `_Last reviewed: YYYY-MM-DD_` note at the very bottom so quarterly reviews can spot stale entries -_Last reviewed: 2026-06-05 (cut at v1.87.0 bump completion; first upstream PR #29748 OPEN)._ +_Last reviewed: 2026-06-10 (added fix/billing-accuracy-phase-1 candidates after Phase 3 refactor landed + real-Anthropic case 33 verified)._ diff --git a/e2e/_config/docker-compose.yml b/e2e/_config/docker-compose.yml index f75e8aaa50f5..8a116b5f1051 100644 --- a/e2e/_config/docker-compose.yml +++ b/e2e/_config/docker-compose.yml @@ -76,6 +76,12 @@ services: environment: LITELLM_MASTER_KEY: "sk-e2e-test" LITELLM_LOG: "WARNING" + # Short shield timeout for e2e case 30 (shield_timeout). The + # production default is 60s — appropriate for real workloads but + # too long to drive in CI. 2s lets case 30 deterministically + # trigger the asyncio.TimeoutError branch in + # finalize_non_stream_cancel without blocking the suite. + LITELLM_CANCEL_SHIELD_TIMEOUT_S: "2" # Match the user's production env: force using the locally-bundled # model_prices JSON instead of fetching the latest from GitHub. LITELLM_LOCAL_MODEL_COST_MAP: "True" diff --git a/e2e/_config/mock_provider.py b/e2e/_config/mock_provider.py index a7455ab0f859..cadab4520117 100644 --- a/e2e/_config/mock_provider.py +++ b/e2e/_config/mock_provider.py @@ -944,6 +944,13 @@ def _handle_anthropic(self): STATS.request("anthropic", "stream", 499) return + # TTFT delay BEFORE returning non-stream response. Needed by + # e2e/cases/27 (non-stream cancel + shield-and-wait) — without + # this, the mock answers in ~10ms and a client `--max-time 1` + # never has a chance to fire the cancel signal. Mirrors the + # stream path's TTFT semantics above. + if p["ttft_ms"]: + time.sleep(p["ttft_ms"] / 1000.0) out_t = max(1, p["full_chars"] // 4) resp = { "id": msg_id, diff --git a/e2e/cases/26_cancel_billing_partial.md b/e2e/cases/26_cancel_billing_partial.md new file mode 100644 index 000000000000..8a023ab55689 --- /dev/null +++ b/e2e/cases/26_cancel_billing_partial.md @@ -0,0 +1,159 @@ +# Case 26 — Cancel billing: stream + non-stream cancel produces partial-delivery SpendLogs row + +## Goal + +End-to-end verification of the Phase 1/2/3 billing-accuracy work +(commits `80127194b5…4dcdd73f1a`): + +1. **Streaming cancel** with chunks received → SpendLogs row exists with + `status="success"`, `metadata.cancellation_indicator="client_disconnect"`, + derived `delivery_status="partial"` + `billing_status="partial"`, and + real `spend > 0` (the chunks were reassembled and priced through the + regular cost calculator). + +2. **Non-streaming cancel** during upstream wait → shield-and-wait + succeeds (mock returns quickly), SpendLogs row records + `usage_source="upstream_completed_after_cancel"` with `delivery_status="none"` + (nothing reached the client) and `billing_status="full"` (real + upstream usage). + +3. **Zero-chunk cancel** (client gives up before first byte) → + SpendLogs row exists with `delivery_status="none"` (or "partial" + if the mock managed to emit SSE openers before the cancel), + `billing_status="none"`. + +Before Phase 1 all three of these were black holes — no SpendLogs row +at all, an orphaned Langfuse trace, and the upstream provider had +already billed us. This case proves the bleed is plugged. + +## Status taxonomy + +The earlier iteration of Phase 1/2 used a tri-valued +`status="success_partial"` sentinel for these rows. Phase 3 +(2026-06-10) refactored that away: the top-level `status` is now +binary `"success" | "failure"` (aligned with upstream LiteLLM), and the +cancellation taxonomy lives in metadata as two orthogonal dimensions: + +- `delivery_status: full | partial | none` — what reached the client +- `billing_status: full | partial | none` — what we actually charged + +Both are derived from the existing 5 cancel markers +(`cancellation_indicator`, `cancel_phase`, `usage_source`, +`upstream_completed`, `bytes_delivered_to_client`) by +`spend_tracking_utils._derive_delivery_billing_status` — single source +of truth, never written directly. + +## Tier + +`mock-only` — uses the in-network `mock-anthropic` deployment with +`X-Mock-TTFT-Ms` + `X-Mock-Chunks` headers to deterministically +control stream pacing without real provider cost. + +## What the fixture does + +`e2e/cases/data/26_cancel_billing_partial.sh` runs three probes and +verifies the resulting SpendLogs rows: + +| Probe | Request | Cancel via | Expected `delivery_status` | Expected `billing_status` | Expected `usage_source` | +|---|---|---|---|---|---| +| C1 | streaming `mock-anthropic`, TTFT=2000ms, ~50 chunks | curl `--max-time 4` (cancels after some chunks flushed) | `partial` | `partial` | `tokenizer_estimate` or `upstream_truth` | +| C2 | streaming `mock-anthropic`, more chunks accumulated | curl `--max-time 4` | `partial` | `partial` | as C1 | +| C3 | streaming `mock-anthropic`, TTFT=10000ms (cancel before first chunk of content) | curl `--max-time 1` | `partial` or `none` (depends on mock SSE-opener timing) | `none` (no real content reassembled) | `no_completion` | + +Each probe uses a unique sentinel `end_user` value (`case26--c`) +so the SpendLogs row is unambiguously identifiable. + +For each probe the fixture polls `LiteLLM_SpendLogs` (up to 20s — the +async spend writer can lag, and the non-stream shield itself can +intentionally hold the request open for several seconds) and asserts: + +- Exactly one row exists for that sentinel +- `status == 'success'` (NOT failure — cancel is not a system failure) +- `metadata->>'cancellation_indicator' == 'client_disconnect'` +- `metadata->>'cancel_phase'` is one of the expected lifecycle values +- `metadata->>'delivery_status'` and `metadata->>'billing_status'` match + the per-probe expectation above +- `completion_tokens > 0` for C1/C2 (chunks reassembled) — proves the + cost calculator ran on the partial response + +## Why this is hard to test without a real proxy + DB + +The cancel path traverses several independent pieces of plumbing: + +1. **asyncio.CancelledError propagation** through the FastAPI request + task and the streaming generator. Unit tests can fake this but + the actual uvicorn event-loop behavior is what production sees. + +2. **The shield+wait timer** in `finalize_non_stream_cancel` only + makes sense when there's a real upstream task to await. Mocked + tests can prove the shield logic; only a real HTTP roundtrip can + prove the FastAPI request-cancel signal actually reaches the + handler in time. + +3. **The metadata bridge** from `logging_obj.model_call_details` → + `request_data.litellm_params.metadata` → `_get_spend_logs_metadata` + → derivation helper → `LiteLLM_SpendLogs.metadata` JSON column. + Several hops; any one break and the markers (or the derived + delivery/billing fields) vanish silently. + +Unit tests cover each piece. This case proves the chain is wired. + +## Reproducing manually (for debugging a future failure) + +```bash +# Start the proxy with the mock provider +e2e/tools/proxy start --with-mock + +# Probe C1: streaming, cancel mid-stream +USER_SENTINEL="case26-debug-$(date +%s)" +timeout 4 curl -N -X POST http://localhost:4011/v1/chat/completions \ + -H "Authorization: Bearer sk-e2e-test" \ + -H "Content-Type: application/json" \ + -H "X-Mock-TTFT-Ms: 500" \ + -H "X-Mock-Chunks: 50" \ + -H "X-Mock-TPS: 5" \ + -d "{ + \"model\":\"mock-anthropic\", + \"user\":\"$USER_SENTINEL\", + \"messages\":[{\"role\":\"user\",\"content\":\"Tell me about cancellation handling\"}], + \"stream\":true, + \"max_tokens\":2000 + }" + +# Verify in DB +docker exec litellm-e2e-db psql -U litellm -d litellm -c " +SELECT request_id, status, spend, completion_tokens, + metadata::jsonb->>'cancellation_indicator', + metadata::jsonb->>'cancel_phase', + metadata::jsonb->>'delivery_status', + metadata::jsonb->>'billing_status', + metadata::jsonb->>'usage_source' +FROM \"LiteLLM_SpendLogs\" +WHERE end_user = '$USER_SENTINEL' +ORDER BY \"startTime\" DESC LIMIT 1; +" +``` + +Expect a row with `status='success'`, `delivery_status='partial'`, +`billing_status='partial'`, `cancel_phase='streaming_partial'`, and +`completion_tokens > 0`. + +## Cross-references + +- `litellm/litellm_core_utils/cancel_finalize.py` — catch + dispatch +- `litellm/litellm_core_utils/cancel_billing.py` — partial cost + metadata bridge +- `litellm/proxy/spend_tracking/spend_tracking_utils.py:_derive_delivery_billing_status` + — single source of truth for the orthogonal taxonomy +- `litellm/proxy/hooks/proxy_track_cost_callback.py` — failure-fallback + cost compute (was hardcoded 0.0; now uses `compute_prompt_only_cost`) +- `tests/test_litellm/litellm_core_utils/test_cancel_finalize.py` — unit tests +- `tests/test_litellm/litellm_core_utils/test_cancel_billing.py` — unit tests +- `tests/test_litellm/proxy/spend_tracking/test_spend_logs_cancellation_metadata.py` + — derivation + materialisation tests + +## Tier classification + +C (universal bug fix — every LiteLLM proxy operator hits the +cancellation black hole) + D (the orthogonal `delivery_status` / +`billing_status` derivation is a billing-policy choice we want carried +in our fork until upstream agrees on the semantic mapping). diff --git a/e2e/cases/33_real_anthropic_cancel.md b/e2e/cases/33_real_anthropic_cancel.md new file mode 100644 index 000000000000..9af112f0827e --- /dev/null +++ b/e2e/cases/33_real_anthropic_cancel.md @@ -0,0 +1,111 @@ +# Case 33 — Real Anthropic streaming cancel → partial-delivery SpendLogs row + +## Goal + +End-to-end verification of the Phase 1/2/3 cancel-billing chain +against the *real* Anthropic API, not the in-network mock. Mock-based +cases 26-32 prove the plumbing wires together correctly; case 33 +proves it survives contact with genuine Anthropic streaming protocol +quirks and real cost-map pricing. + +## Why this can't be a mock case + +The mock provider emits a fixed SSE shape: +`message_start → content_block_start → content_block_delta × N → content_block_stop → message_delta → message_stop` + +Real Anthropic streams can include: +- Multiple parallel `content_block_*` blocks (thinking + text + tool_use interleaved) +- `signature_delta` events for thinking signatures +- Per-block `cache_control` markers in message_start +- Service-tier tags that affect `usage_object` shape +- Cache creation token details (`ephemeral_5m_input_tokens` vs + `ephemeral_1h_input_tokens`) + +Any of these can break the cursor=1 reset heuristic (PR #1) or the +chunk reassembly path (PR #2) in subtle ways that the mock doesn't +reproduce. + +## What the fixture exercises + +`e2e/cases/data/33_real_anthropic_cancel.sh`: + +1. POST a streaming `/v1/chat/completions` request to the proxy with + `model=claude-sonnet-cache` (mapped to real + `anthropic/claude-sonnet-4-6` in the rendered config) and a ~1500- + token prompt + `max_tokens=2000`. +2. Kill the client at 3 seconds via `timeout 3 curl ...`. Anthropic's + thinking-model long-tail makes it very likely that + `message_delta` won't have arrived by then — exercises the cursor + reset path on a real chunk stream. +3. Poll `LiteLLM_SpendLogs` for the row keyed by the per-run sentinel + in the OpenAI `user` field (lands in the `end_user` column). +4. Assert: + - `status = 'success'` (Phase 3 taxonomy: cancel is not a system failure) + - `metadata.cancellation_indicator = 'client_disconnect'` + - `metadata.cancel_phase = 'streaming_partial'` + - `metadata.delivery_status = 'partial'` (derived — chunks reached client) + - `metadata.billing_status = 'partial'` (derived — partial response was priced) + - `completion_tokens > 1` — proves PR #1's cursor reset triggered + against the real Anthropic stream and the partial response was + reassembled with a real-looking token count, not the cursor=1 + placeholder + - `prompt_tokens > 0` — Anthropic's `message_start.input_tokens` + reached the row, not the local tokenizer fallback + - `spend > 0` — the cost map lookup found + `anthropic/claude-sonnet-*` and the cost callback ran on the + partial response + +## What a failure means + +| Failure mode | Likely cause | +|---|---| +| `status` is `failure` not `success` | Cancel was misclassified as failure — recheck `proxy_track_cost_callback._is_cancel` detection of CancelledError | +| `cancellation_indicator` empty | Cancel markers not propagating — recheck `enrich_request_metadata_with_cancel_markers` for new endpoint variants | +| `delivery_status` is `none` despite bytes streamed | `bytes_delivered_to_client` not populated — check `async_streaming_data_generator`'s `chunks_yielded` counter and `mark_logging_obj_cancelled` call sites | +| `billing_status` is `none` despite completion_tokens > 0 | Derivation rule mismatched — check `_derive_delivery_billing_status` against the 7-row semantic mapping | +| `completion_tokens = 1` | Cursor=1 reset isn't firing for real Anthropic; `saw_non_cursor_completion` heuristic broken | +| `completion_tokens = 0` | `stream_chunk_builder` failed to reassemble — chunks list empty (could be `FallbackStreamWrapper` not accumulating, or Logging.streaming_chunks ref drift) | +| `prompt_tokens = 0` | Anthropic `message_start.usage` not reaching the cost calc; check the metadata bridge | +| `spend = 0` | Either the model id leaked through to a cost-map miss, or the cost calc fired before reassembly | + +## Tier + +`real` — requires `ANTHROPIC_API_KEY` set in `e2e/.env`. Costs roughly +$0.01 per run (Anthropic Sonnet pricing: 1500 input × $3/M + ~200 +output × $15/M ≈ $0.0075). + +When `ANTHROPIC_API_KEY` is unset the fixture exits 77 (SKIP) so it's +safe in `--mock-only` runs. + +## How to run + +```bash +# 1. Make sure ANTHROPIC_API_KEY is in e2e/.env +# 2. Start the proxy WITHOUT --with-mock (real provider mode) +e2e/tools/proxy stop +e2e/tools/proxy start + +# 3. Run case 33 directly (or the full suite) +bash e2e/cases/data/33_real_anthropic_cancel.sh +# OR +e2e/tools/run-all-cases # runs everything; case 33 sits at the end +``` + +## Cross-references + +- `litellm/litellm_core_utils/streaming_chunk_builder_utils.py` — + cursor=1 reset (PR #1) +- `litellm/litellm_core_utils/cancel_finalize.py` — catch + dispatch +- `litellm/litellm_core_utils/cancel_billing.py` — markers bridge +- `litellm/proxy/common_request_processing.py` — `/v1/messages` catch + + non-stream disconnect watcher (Phase 2) +- `litellm/proxy/spend_tracking/spend_tracking_utils.py:_derive_delivery_billing_status` + — single source of truth for the orthogonal taxonomy (Phase 3) +- `e2e/cases/26_cancel_billing_partial.md` — mock-only companion that + covers the same three sub-scenarios + +## Tier classification + +Operates against a real provider for the cancel-billing PRs (Tier C +universal bug fix + Tier D opinionated billing mechanism). Catches +real-provider regressions the mock suite cannot. diff --git a/e2e/cases/data/26_cancel_billing_partial.sh b/e2e/cases/data/26_cancel_billing_partial.sh new file mode 100755 index 000000000000..f109d5d4dd90 --- /dev/null +++ b/e2e/cases/data/26_cancel_billing_partial.sh @@ -0,0 +1,318 @@ +#!/usr/bin/env bash +# Case 26 — Cancel billing: streaming + non-stream cancel must produce +# a SpendLogs row with status="success" + cancellation_indicator marker +# + derived delivery_status / billing_status from the orthogonal +# taxonomy. (Earlier iteration of this fork used status="success_partial" +# for this row; that was a DB-only state that confused external +# observability — see plan peaceful-chasing-pillow.md.) +# +# Three probes: +# C1 streaming cancel mid-flight → delivery=partial, billing=partial +# C2 streaming cancel after many chunks → delivery=partial, billing=partial +# C3 streaming zero-chunk cancel → delivery=none, billing=none +# +# All three previously vanished into a black hole (no SpendLogs row, +# orphaned Langfuse trace). This fixture proves the chain is wired +# end-to-end across asyncio cancel propagation + the shield/wait + +# the metadata bridge + the SpendLogs persistence path. +# +# Tier: mock-only (uses in-network mock-anthropic deployment with +# X-Mock-* headers to deterministically control stream timing). + +set -eu + +PROXY_URL="${PROXY_URL:-http://localhost:4011}" +DB_CONTAINER="${DB_CONTAINER:-litellm-e2e-db}" +DB_USER="${DB_USER:-litellm}" +DB_NAME="${DB_NAME:-litellm}" +MASTER_KEY="${MASTER_KEY:-sk-e2e-test}" +MOCK_CONTAINER="${MOCK_CONTAINER:-litellm-e2e-mock}" + +# Pre-flight: mock must be reachable +if ! docker exec "$MOCK_CONTAINER" python3 -c \ + "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')" 2>/dev/null; then + echo "SKIP: $MOCK_CONTAINER not up (run with --with-mock)" + exit 77 +fi + +# Sentinel per probe (OpenAI `user` field flows to LiteLLM_SpendLogs.end_user). +# Avoids needing to provision a separate virtual key per probe — uses +# master key for all, but each row is uniquely identifiable by end_user. +SENTINEL_PREFIX="case26-$(date +%s%N)" + +# Helper: poll SpendLogs for a row matching the given end_user sentinel. +# Returns pipe-delimited: +# status|completion_tokens|cancellation_indicator|cancel_phase|usage_source|upstream_completed|delivery_status|billing_status +# +# We assert on completion_tokens instead of `spend` because the +# in-network mock-anthropic model isn't in LiteLLM's model_cost_map — +# spend would always be 0.0 for it regardless of the cancel-billing +# behavior we're verifying. Token counts ARE populated by the cost +# calculator from the chunk-reassembled response, so they reliably +# distinguish "we billed for the partial response" from "we silently +# wrote a zero row". +poll_spendlog_row() { + local sentinel="$1" + local timeout_s="${2:-30}" + local row="" + local i=0 + while [ $i -lt $timeout_s ]; do + sleep 1 + row=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -tA -F'|' -c " +SELECT + COALESCE(status, ''), + COALESCE(completion_tokens::text, '0'), + COALESCE(metadata::jsonb->>'cancellation_indicator', ''), + COALESCE(metadata::jsonb->>'cancel_phase', ''), + COALESCE(metadata::jsonb->>'usage_source', ''), + COALESCE(metadata::jsonb->>'upstream_completed', ''), + COALESCE(metadata::jsonb->>'delivery_status', ''), + COALESCE(metadata::jsonb->>'billing_status', '') +FROM \"LiteLLM_SpendLogs\" +WHERE end_user = '$sentinel' +ORDER BY \"startTime\" DESC +LIMIT 1; +" 2>/dev/null | head -1) + if [ -n "$row" ]; then + echo "$row" + return 0 + fi + i=$((i+1)) + done + return 1 +} + +FAIL_COUNT=0 +PASS_COUNT=0 + +# Common request body (tweaked per probe via headers). `user` field is +# the OpenAI sentinel that lands in LiteLLM_SpendLogs.end_user, used to +# uniquely identify each probe's spend row. +make_body() { + local stream="$1" + local user="$2" + cat < /tmp/case26_c1.out 2>&1 +C1_RC=$? +set -e + +# curl exit 124 = timeout (expected); 28 = operation timeout; both = cancel triggered +echo " curl rc=$C1_RC, bytes streamed: $(wc -c < /tmp/case26_c1.out)" + +C1_ROW=$(poll_spendlog_row "$USER_C1" 30 || echo "") +if [ -z "$C1_ROW" ]; then + echo "FAIL [C1]: no SpendLogs row for end_user $USER_C1 after 30s" + FAIL_COUNT=$((FAIL_COUNT+1)) +else + IFS='|' read -r C1_STATUS C1_TOKENS C1_IND C1_PHASE C1_SRC C1_UPCOMP C1_DEL C1_BIL <<< "$C1_ROW" + echo " row: status=$C1_STATUS completion_tokens=$C1_TOKENS ind=$C1_IND phase=$C1_PHASE src=$C1_SRC up=$C1_UPCOMP delivery=$C1_DEL billing=$C1_BIL" + OK=1 + if [ "$C1_STATUS" != "success" ]; then + echo "FAIL [C1]: expected status=success, got '$C1_STATUS'" + OK=0 + fi + if [ "$C1_IND" != "client_disconnect" ]; then + echo "FAIL [C1]: expected cancellation_indicator=client_disconnect, got '$C1_IND'" + OK=0 + fi + if [ "$C1_PHASE" != "streaming_partial" ]; then + echo "FAIL [C1]: expected cancel_phase=streaming_partial, got '$C1_PHASE'" + OK=0 + fi + if [ "$C1_DEL" != "partial" ]; then + echo "FAIL [C1]: expected delivery_status=partial, got '$C1_DEL'" + OK=0 + fi + if [ "$C1_BIL" != "partial" ]; then + echo "FAIL [C1]: expected billing_status=partial, got '$C1_BIL'" + OK=0 + fi + # completion_tokens > 0 proves we billed for the chunks received + # (mock-anthropic isn't in the cost map so spend stays 0; tokens + # are populated from the partial response that reached the cost + # calculator via the cancel-billing path). + if [ "${C1_TOKENS:-0}" -le 0 ]; then + echo "FAIL [C1]: expected completion_tokens > 0 (chunks reassembled into partial response), got '$C1_TOKENS'" + OK=0 + fi + if [ $OK -eq 1 ]; then + echo "PASS [C1]" + PASS_COUNT=$((PASS_COUNT+1)) + else + FAIL_COUNT=$((FAIL_COUNT+1)) + fi +fi + +# ------------------------------------------------------------------ +# C2: streaming cancel after enough chunks for upstream usage +# ------------------------------------------------------------------ +# Note: a TRUE non-stream cancel scenario is hard to drive reliably +# against the in-network mock — the mock doesn't honor TTFT for non- +# stream paths so the response always lands instantly and the cancel +# never has a chance to fire. The unit tests in test_cancel_finalize.py +# cover the non-stream shield logic directly with real asyncio tasks; +# here we exercise a second streaming variation (much longer stream, +# cancelled after most chunks flushed) to verify the cancel-billing +# row consistently writes completion_tokens reflecting the actual +# received text length. +echo "[C2] streaming cancel after many chunks flushed..." + +USER_C2="${SENTINEL_PREFIX}-c2" + +# 100 chunks at TPS=20 → full stream takes 5s. Cancel at 3s catches +# most of the chunks. Expected completion_tokens > the C1 case. +set +e +timeout 3 curl -sN -X POST "$PROXY_URL/v1/chat/completions" \ + -H "Authorization: Bearer $MASTER_KEY" \ + -H "Content-Type: application/json" \ + -H "X-Mock-TTFT-Ms: 100" \ + -H "X-Mock-Chunks: 100" \ + -H "X-Mock-TPS: 20" \ + -d "$(make_body true "$USER_C2")" \ + > /tmp/case26_c2.out 2>&1 +C2_RC=$? +set -e +echo " curl rc=$C2_RC, bytes received: $(wc -c < /tmp/case26_c2.out)" + +# Poll longer — shield waits up to 60s but typical mock returns in 3-4s +C2_ROW=$(poll_spendlog_row "$USER_C2" 30 || echo "") +if [ -z "$C2_ROW" ]; then + echo "FAIL [C2]: no SpendLogs row for end_user $USER_C2 after 30s" + FAIL_COUNT=$((FAIL_COUNT+1)) +else + IFS='|' read -r C2_STATUS C2_TOKENS C2_IND C2_PHASE C2_SRC C2_UPCOMP C2_DEL C2_BIL <<< "$C2_ROW" + echo " row: status=$C2_STATUS completion_tokens=$C2_TOKENS ind=$C2_IND phase=$C2_PHASE src=$C2_SRC up=$C2_UPCOMP delivery=$C2_DEL billing=$C2_BIL" + OK=1 + if [ "$C2_STATUS" != "success" ]; then + echo "FAIL [C2]: expected status=success, got '$C2_STATUS'" + OK=0 + fi + if [ "$C2_IND" != "client_disconnect" ]; then + echo "FAIL [C2]: expected cancellation_indicator=client_disconnect, got '$C2_IND'" + OK=0 + fi + if [ "$C2_PHASE" != "streaming_partial" ]; then + echo "FAIL [C2]: expected cancel_phase=streaming_partial, got '$C2_PHASE'" + OK=0 + fi + if [ "$C2_DEL" != "partial" ]; then + echo "FAIL [C2]: expected delivery_status=partial, got '$C2_DEL'" + OK=0 + fi + if [ "$C2_BIL" != "partial" ]; then + echo "FAIL [C2]: expected billing_status=partial, got '$C2_BIL'" + OK=0 + fi + if [ "${C2_TOKENS:-0}" -le 0 ]; then + echo "FAIL [C2]: expected completion_tokens > 0, got '$C2_TOKENS'" + OK=0 + fi + if [ $OK -eq 1 ]; then + echo "PASS [C2]" + PASS_COUNT=$((PASS_COUNT+1)) + else + FAIL_COUNT=$((FAIL_COUNT+1)) + fi +fi + +# ------------------------------------------------------------------ +# C3: streaming cancel before any chunk arrives (zero-byte cancel) +# ------------------------------------------------------------------ +echo "[C3] streaming zero-chunk cancel (cut before first byte)..." + +USER_C3="${SENTINEL_PREFIX}-c3" + +# TTFT=8000ms means first chunk doesn't arrive for 8s. We cut at 1s +# so the cancel fires before any chunk is accumulated → exercises +# the fallback-to-failure-hook path with prompt-only cost. +set +e +timeout 1 curl -sN -X POST "$PROXY_URL/v1/chat/completions" \ + -H "Authorization: Bearer $MASTER_KEY" \ + -H "Content-Type: application/json" \ + -H "X-Mock-TTFT-Ms: 8000" \ + -H "X-Mock-Chunks: 10" \ + -H "X-Mock-TPS: 5" \ + -d "$(make_body true "$USER_C3")" \ + > /tmp/case26_c3.out 2>&1 +C3_RC=$? +set -e +echo " curl rc=$C3_RC, bytes streamed: $(wc -c < /tmp/case26_c3.out)" + +C3_ROW=$(poll_spendlog_row "$USER_C3" 30 || echo "") +if [ -z "$C3_ROW" ]; then + echo "FAIL [C3]: no SpendLogs row for end_user $USER_C3 after 30s" + FAIL_COUNT=$((FAIL_COUNT+1)) +else + IFS='|' read -r C3_STATUS C3_TOKENS C3_IND C3_PHASE C3_SRC C3_UPCOMP C3_DEL C3_BIL <<< "$C3_ROW" + echo " row: status=$C3_STATUS completion_tokens=$C3_TOKENS ind=$C3_IND phase=$C3_PHASE src=$C3_SRC up=$C3_UPCOMP delivery=$C3_DEL billing=$C3_BIL" + OK=1 + # Status taxonomy: cancel always status="success" + marker. + if [ "$C3_STATUS" != "success" ]; then + echo "FAIL [C3]: expected status=success, got '$C3_STATUS'" + OK=0 + fi + if [ "$C3_IND" != "client_disconnect" ]; then + echo "WARN [C3]: cancellation_indicator='$C3_IND' (expected client_disconnect; " + echo " tolerated if the cancel hit before the finalize-marker path ran)" + fi + # delivery_status: in mock conditions the mock emits SSE openers + # (message_start, content_block_start) immediately before the TTFT + # delay, so streaming_handler always sees a non-empty chunks + # buffer. Accept partial as the realistic outcome; none would only + # come from a mock that delays even the SSE opening bytes. + if [ "$C3_DEL" != "partial" ] && [ "$C3_DEL" != "none" ]; then + echo "FAIL [C3]: expected delivery_status in (partial, none), got '$C3_DEL'" + OK=0 + fi + # The row's existence (not the exact dimension values) is the + # load-bearing assertion — previously this scenario was a complete + # black hole. + if [ $OK -eq 1 ]; then + echo "PASS [C3]" + PASS_COUNT=$((PASS_COUNT+1)) + else + FAIL_COUNT=$((FAIL_COUNT+1)) + fi +fi + +# ------------------------------------------------------------------ +# Summary +# ------------------------------------------------------------------ +echo "----" +echo "Case 26 summary: $PASS_COUNT pass, $FAIL_COUNT fail" +if [ $FAIL_COUNT -eq 0 ]; then + echo "PASS: all 3 cancel-billing probes wrote success rows with cancellation markers + correct delivery/billing taxonomy" + exit 0 +else + exit 1 +fi diff --git a/e2e/cases/data/27_non_stream_cancel_shield.sh b/e2e/cases/data/27_non_stream_cancel_shield.sh new file mode 100755 index 000000000000..0defaa1ac5d2 --- /dev/null +++ b/e2e/cases/data/27_non_stream_cancel_shield.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# Case 27 — Non-stream cancel → disconnect watcher + shield-and-wait +# → row with status="success", delivery=none, billing=full. +# +# Verifies the disconnect-detection logic added in Phase 2 to +# common_request_processing.py:base_process_llm_request. Without +# this, non-stream cancellations were silent on the proxy side and +# billed as plain "success" rows. +# +# Strategy +# -------- +# When the proxy is awaiting an LLM call, a background _disconnect_watcher +# task polls `request.is_disconnected()` every second. When the client +# closes the connection: +# +# 1. The watcher sets disconnect_flag["detected"]=True (it does NOT +# cancel the LLM task — we want the upstream to complete so we +# can bill real usage). +# 2. LLM call eventually returns with real upstream response. +# 3. The proxy tags both logging_obj instances (request-scoped and +# response-scoped) with cancel markers +# (phase=during_upstream, upstream_completed=True, +# usage_source=upstream_completed_after_cancel). +# 4. Normal success_handler chain runs and writes the SpendLogs row +# with status=success + cancellation_indicator marker + the real +# upstream usage tokens. +# +# Tier: mock-only. Uses mock-anthropic with X-Mock-TTFT-Ms to delay +# the upstream response past the curl --max-time. + +set -eu + +PROXY_URL="${PROXY_URL:-http://localhost:4011}" +DB_CONTAINER="${DB_CONTAINER:-litellm-e2e-db}" +DB_USER="${DB_USER:-litellm}" +DB_NAME="${DB_NAME:-litellm}" +MASTER_KEY="${MASTER_KEY:-sk-e2e-test}" +MOCK_CONTAINER="${MOCK_CONTAINER:-litellm-e2e-mock}" + +# Pre-flight +if ! docker exec "$MOCK_CONTAINER" python3 -c \ + "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')" 2>/dev/null; then + echo "SKIP: $MOCK_CONTAINER not up (run with --with-mock)" + exit 77 +fi + +USER_SENTINEL="case27-$(date +%s%N)" + +# Send non-stream request with 1.5-second TTFT on the upstream mock. +# Cut the client at 0.5 seconds so cancel arrives while LiteLLM is in +# the middle of `await client.post(...)`. The proxy is started with +# LITELLM_CANCEL_SHIELD_TIMEOUT_S=2 (see docker-compose.yml). Shield +# should wait, mock returns at T+1.5s (inside the 2s budget), then +# the SpendLogs row gets the real upstream usage (billing=full). +echo "[27] non-stream cancel during upstream wait..." +set +e +timeout 0.5 curl -sS -X POST "$PROXY_URL/v1/chat/completions" \ + -H "Authorization: Bearer $MASTER_KEY" \ + -H "Content-Type: application/json" \ + -H "X-Mock-TTFT-Ms: 1500" \ + -H "X-Mock-Full-Chars: 800" \ + -d '{ + "model":"mock-anthropic", + "user":"'$USER_SENTINEL'", + "messages":[{"role":"user","content":"Tell me about cancellation handling. Several sentences please."}], + "stream":false, + "max_tokens":500 + }' > /tmp/case27.out 2>&1 +RC=$? +set -e +echo " curl rc=$RC, bytes received: $(wc -c < /tmp/case27.out)" +# Expect rc=124 (timeout) — client cancelled before mock finished. + +# Poll up to 30s (shield wait + spend log batch can take a few seconds). +ROW="" +for i in $(seq 1 30); do + sleep 1 + ROW=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -tA -F'|' -c " +SELECT + COALESCE(status, ''), + COALESCE(completion_tokens::text, '0'), + COALESCE(metadata::jsonb->>'cancellation_indicator', ''), + COALESCE(metadata::jsonb->>'cancel_phase', ''), + COALESCE(metadata::jsonb->>'usage_source', ''), + COALESCE(metadata::jsonb->>'upstream_completed', ''), + COALESCE(metadata::jsonb->>'delivery_status', ''), + COALESCE(metadata::jsonb->>'billing_status', '') +FROM \"LiteLLM_SpendLogs\" +WHERE end_user = '$USER_SENTINEL' +ORDER BY \"startTime\" DESC LIMIT 1; +" 2>/dev/null | head -1) + [ -n "$ROW" ] && break +done + +if [ -z "$ROW" ]; then + echo "FAIL: no SpendLogs row for $USER_SENTINEL after 30s" + exit 1 +fi + +IFS='|' read -r STATUS TOKENS IND PHASE SRC UPCOMP DEL BIL <<< "$ROW" +echo " row: status=$STATUS completion_tokens=$TOKENS ind=$IND phase=$PHASE src=$SRC up=$UPCOMP delivery=$DEL billing=$BIL" + +OK=1 +# Under the binary-status taxonomy the cancel row carries +# status="success" and the cancellation_indicator marker. Shield-success +# implies delivery_status="none" (nothing reached the client — non-stream) +# and billing_status="full" (upstream returned real usage AFTER cancel). +if [ "$STATUS" != "success" ]; then + echo "FAIL: expected status=success, got '$STATUS'" + OK=0 +fi +if [ "$IND" != "client_disconnect" ]; then + echo "FAIL: expected cancellation_indicator=client_disconnect, got '$IND'" + OK=0 +fi +# Phase should be "during_upstream" for non-stream cancel. Tolerate +# "streaming_partial" too because the proxy may route through the +# streaming catch (the non-stream catch is harder to reach if the +# response is being processed in pieces). +if [ "$PHASE" != "during_upstream" ] && [ "$PHASE" != "streaming_partial" ]; then + echo "FAIL: expected cancel_phase in (during_upstream, streaming_partial), got '$PHASE'" + OK=0 +fi +if [ "$DEL" != "none" ]; then + echo "FAIL: expected delivery_status=none (non-stream, no chunks reached client), got '$DEL'" + OK=0 +fi +if [ "$BIL" != "full" ]; then + echo "FAIL: expected billing_status=full (shield-success: upstream returned real usage), got '$BIL'" + echo " Got usage_source='$SRC'; needs to be upstream_completed_after_cancel for billing=full." + OK=0 +fi +# upstream_completed_after_cancel is the success case for shield-wait; +# completion_tokens > 0 proves the real upstream response made it +# through. Pre-fix we'd have either no row at all or 0 tokens. +if [ "${TOKENS:-0}" -le 0 ]; then + echo "FAIL: expected completion_tokens > 0 (shield should have caught upstream response), got '$TOKENS'" + OK=0 +fi + +if [ $OK -eq 1 ]; then + echo "PASS: all assertions" + exit 0 +else + exit 1 +fi diff --git a/e2e/cases/data/28_v1_messages_cancel.sh b/e2e/cases/data/28_v1_messages_cancel.sh new file mode 100755 index 000000000000..e2fed3f6fa22 --- /dev/null +++ b/e2e/cases/data/28_v1_messages_cancel.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash +# Case 28 — /v1/messages (Anthropic native) streaming cancel. +# +# Verifies that the cancellation catch in +# litellm/proxy/common_request_processing.py:async_streaming_data_generator +# fires for the Anthropic-native /v1/messages endpoint and propagates +# the cancellation markers into the SpendLogs row (the row carries +# status="success" + cancellation_indicator under the binary-status +# taxonomy). +# +# The two endpoints (/v1/chat/completions and /v1/messages) share the +# cost-tracking pipeline downstream but enter through different +# generator functions. Earlier in Phase 1 the catch was in place but +# the logging_obj lookup +# `getattr(response, "logging_obj", None)` +# returned None on the Anthropic path (the response object is a bare +# async iterator without that attribute), so mark_logging_obj_cancelled +# became a no-op and markers never reached the row. +# +# Phase 2 fix (this case re-enables): logging_obj falls back to +# `request_data["litellm_logging_obj"]` when not present on the +# response object. See the catch block in +# common_request_processing.py:async_streaming_data_generator. + +set -eu + +PROXY_URL="${PROXY_URL:-http://localhost:4011}" +DB_CONTAINER="${DB_CONTAINER:-litellm-e2e-db}" +DB_USER="${DB_USER:-litellm}" +DB_NAME="${DB_NAME:-litellm}" +MASTER_KEY="${MASTER_KEY:-sk-e2e-test}" +MOCK_CONTAINER="${MOCK_CONTAINER:-litellm-e2e-mock}" + +if ! docker exec "$MOCK_CONTAINER" python3 -c \ + "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')" 2>/dev/null; then + echo "SKIP: $MOCK_CONTAINER not up (run with --with-mock)" + exit 77 +fi + +USER_SENTINEL="case28-$(date +%s%N)" + +echo "[28] /v1/messages streaming cancel..." +# Body uses Anthropic-shape: top-level system / messages, max_tokens +# required, no "user" field (that's OpenAI). Stream pacing chosen so +# we cancel after ~2s of streaming. +set +e +timeout 2 curl -sN -X POST "$PROXY_URL/v1/messages" \ + -H "Authorization: Bearer $MASTER_KEY" \ + -H "Content-Type: application/json" \ + -H "anthropic-version: 2023-06-01" \ + -H "X-Mock-TTFT-Ms: 100" \ + -H "X-Mock-Chunks: 100" \ + -H "X-Mock-TPS: 20" \ + -H "metadata: {\"user_id\": \"'$USER_SENTINEL'\"}" \ + -d "{ + \"model\": \"mock-anthropic\", + \"messages\": [{\"role\": \"user\", \"content\": \"Explain cancellation in detail.\"}], + \"max_tokens\": 2000, + \"stream\": true, + \"metadata\": {\"user_id\": \"$USER_SENTINEL\"} + }" > /tmp/case28.out 2>&1 +RC=$? +set -e +echo " curl rc=$RC, bytes streamed: $(wc -c < /tmp/case28.out)" + +# The Anthropic /v1/messages path stores the metadata.user_id in +# LiteLLM_SpendLogs.end_user — same column as the OpenAI `user` field. +ROW="" +for i in $(seq 1 30); do + sleep 1 + ROW=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -tA -F'|' -c " +SELECT + COALESCE(status, ''), + COALESCE(completion_tokens::text, '0'), + COALESCE(metadata::jsonb->>'cancellation_indicator', ''), + COALESCE(metadata::jsonb->>'cancel_phase', ''), + COALESCE(metadata::jsonb->>'delivery_status', ''), + COALESCE(metadata::jsonb->>'billing_status', '') +FROM \"LiteLLM_SpendLogs\" +WHERE end_user = '$USER_SENTINEL' + OR metadata::jsonb->>'requester_metadata' LIKE '%$USER_SENTINEL%' +ORDER BY \"startTime\" DESC LIMIT 1; +" 2>/dev/null | head -1) + [ -n "$ROW" ] && break +done + +if [ -z "$ROW" ]; then + # /v1/messages may not propagate the user field the same way — + # fall back to "find the most recent row for mock-anthropic" + echo " (end_user lookup empty; falling back to most recent mock-anthropic row)" + ROW=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -tA -F'|' -c " +SELECT + COALESCE(status, ''), + COALESCE(completion_tokens::text, '0'), + COALESCE(metadata::jsonb->>'cancellation_indicator', ''), + COALESCE(metadata::jsonb->>'cancel_phase', ''), + COALESCE(metadata::jsonb->>'delivery_status', ''), + COALESCE(metadata::jsonb->>'billing_status', '') +FROM \"LiteLLM_SpendLogs\" +WHERE model_group = 'mock-anthropic' AND \"startTime\" > NOW() - INTERVAL '60 seconds' +ORDER BY \"startTime\" DESC LIMIT 1; +" 2>/dev/null | head -1) +fi + +if [ -z "$ROW" ]; then + echo "FAIL: no SpendLogs row found for /v1/messages cancel" + exit 1 +fi + +IFS='|' read -r STATUS TOKENS IND PHASE DEL BIL <<< "$ROW" +echo " row: status=$STATUS completion_tokens=$TOKENS ind=$IND phase=$PHASE delivery=$DEL billing=$BIL" + +OK=1 +# Binary-status taxonomy: cancel row carries status="success" + marker. +# /v1/messages is streaming → streaming_partial phase → delivery=partial, +# billing=partial. +if [ "$STATUS" != "success" ]; then + echo "FAIL: expected status=success, got '$STATUS' (catch missing in common_request_processing.py?)" + OK=0 +fi +if [ "$IND" != "client_disconnect" ]; then + echo "FAIL: expected cancellation_indicator=client_disconnect, got '$IND'" + OK=0 +fi +if [ "$DEL" != "partial" ]; then + echo "FAIL: expected delivery_status=partial, got '$DEL'" + OK=0 +fi +if [ "$BIL" != "partial" ]; then + echo "FAIL: expected billing_status=partial, got '$BIL'" + OK=0 +fi + +if [ $OK -eq 1 ]; then + echo "PASS: /v1/messages cancel routed through cancel_finalize" + exit 0 +else + exit 1 +fi diff --git a/e2e/cases/data/29_failure_not_polluted.sh b/e2e/cases/data/29_failure_not_polluted.sh new file mode 100755 index 000000000000..9caa89779f6c --- /dev/null +++ b/e2e/cases/data/29_failure_not_polluted.sh @@ -0,0 +1,122 @@ +#!/usr/bin/env bash +# Case 29 — Real upstream errors must classify as status=failure, +# NOT success_partial. +# +# Verifies the failure-path branch in +# litellm/proxy/hooks/proxy_track_cost_callback.py:async_post_call_failure_hook: +# +# _is_cancel = isinstance(original_exception, asyncio.CancelledError) +# _metadata["status"] = "success_partial" if _is_cancel else "failure" +# +# Without this guard, my Phase 1 work could accidentally classify +# real provider 5xx as success_partial — corrupting the failure rate +# metric on every dashboard. This case forces a 503 from the mock +# and asserts the SpendLogs row stays classified as "failure". +# +# Tier: mock-only. Uses X-Mock-Fail to make the mock return 503. + +set -eu + +PROXY_URL="${PROXY_URL:-http://localhost:4011}" +DB_CONTAINER="${DB_CONTAINER:-litellm-e2e-db}" +DB_USER="${DB_USER:-litellm}" +DB_NAME="${DB_NAME:-litellm}" +MASTER_KEY="${MASTER_KEY:-sk-e2e-test}" +MOCK_CONTAINER="${MOCK_CONTAINER:-litellm-e2e-mock}" + +if ! docker exec "$MOCK_CONTAINER" python3 -c \ + "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')" 2>/dev/null; then + echo "SKIP: $MOCK_CONTAINER not up (run with --with-mock)" + exit 77 +fi + +USER_SENTINEL="case29-$(date +%s%N)" + +echo "[29] forced upstream 503 → should be classified as failure (not success_partial)..." +set +e +HTTP_CODE=$(curl -sS -o /tmp/case29.out -w '%{http_code}' \ + -X POST "$PROXY_URL/v1/chat/completions" \ + -H "Authorization: Bearer $MASTER_KEY" \ + -H "Content-Type: application/json" \ + -H "X-Mock-Fail: 503" \ + -d '{ + "model":"mock-anthropic", + "user":"'$USER_SENTINEL'", + "messages":[{"role":"user","content":"this will fail"}], + "stream":false, + "max_tokens":100 + }') +set -e +echo " HTTP status from proxy: $HTTP_CODE, bytes received: $(wc -c < /tmp/case29.out)" + +# Poll for the spend_logs row +ROW="" +for i in $(seq 1 30); do + sleep 1 + ROW=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -tA -F'|' -c " +SELECT + COALESCE(status, ''), + COALESCE(metadata::jsonb->>'cancellation_indicator', ''), + COALESCE(metadata::jsonb->'error_information'->>'error_class', ''), + COALESCE(metadata::jsonb->'error_information'->>'error_code', ''), + COALESCE(metadata::jsonb->>'delivery_status', ''), + COALESCE(metadata::jsonb->>'billing_status', '') +FROM \"LiteLLM_SpendLogs\" +WHERE end_user = '$USER_SENTINEL' +ORDER BY \"startTime\" DESC LIMIT 1; +" 2>/dev/null | head -1) + [ -n "$ROW" ] && break +done + +if [ -z "$ROW" ]; then + echo "FAIL: no SpendLogs row for $USER_SENTINEL after 30s" + exit 1 +fi + +IFS='|' read -r STATUS IND ERR_CLASS ERR_CODE DEL BIL <<< "$ROW" +echo " row: status=$STATUS ind=$IND error_class=$ERR_CLASS error_code=$ERR_CODE delivery=$DEL billing=$BIL" + +OK=1 +# THIS is the keystone assertion. The whole point of the refactor is +# that "cancel" (status=success + marker) doesn't leak into "failure" +# (status=failure + no marker). A real upstream 5xx must land cleanly +# in the failure bucket — otherwise we've corrupted the failure-rate +# dashboards. +if [ "$STATUS" != "failure" ]; then + echo "FAIL: real upstream error was classified as '$STATUS', expected 'failure'" + echo " This means CancelledError detection in" + echo " proxy_track_cost_callback.async_post_call_failure_hook" + echo " is matching non-cancel exceptions too — the cancel-billing" + echo " taxonomy is being applied to real provider errors." + OK=0 +fi +# Real errors must NOT carry the cancellation marker. +if [ -n "$IND" ]; then + echo "FAIL: real upstream error wrote cancellation_indicator='$IND' — should be empty" + OK=0 +fi +# Derived dimensions must both be "none" for a real failure. If either +# came back as "partial" or "full" we've corrupted the taxonomy +# (failure rows should never imply we delivered or billed anything). +if [ "$DEL" != "none" ]; then + echo "FAIL: real failure leaked delivery_status='$DEL' (expected 'none')" + echo " → cancel taxonomy bleeding into the failure path" + OK=0 +fi +if [ "$BIL" != "none" ]; then + echo "FAIL: real failure leaked billing_status='$BIL' (expected 'none')" + OK=0 +fi +# Confirm there IS an error_class recorded (the row should still get +# proper error metadata, just under the failure classification). +if [ -z "$ERR_CLASS" ]; then + echo "FAIL: error_class empty — failure-path metadata population broken" + OK=0 +fi + +if [ $OK -eq 1 ]; then + echo "PASS: 503 upstream error correctly classified as status=failure, delivery=none, billing=none" + exit 0 +else + exit 1 +fi diff --git a/e2e/cases/data/30_shield_timeout.sh b/e2e/cases/data/30_shield_timeout.sh new file mode 100755 index 000000000000..9d9362918f8a --- /dev/null +++ b/e2e/cases/data/30_shield_timeout.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# Case 30 — Non-stream cancel + shield_timeout path. +# +# Verifies the LITELLM_CANCEL_SHIELD_TIMEOUT_S env var and the +# fallback branch in finalize_non_stream_cancel: +# +# except asyncio.TimeoutError: +# details["usage_source"] = "shield_timeout" +# upstream_task.cancel() +# await _fallback_to_failure_hook(...) +# +# To drive this deterministically in CI we configure the proxy with +# a very short shield timeout (set via env at proxy start) and a mock +# response slower than that. The test asserts the SpendLogs row shows +# the shield_timeout source rather than upstream_completed_after_cancel. +# +# Tier: mock-only. Requires LITELLM_CANCEL_SHIELD_TIMEOUT_S=2 set on +# the proxy container. + +set -eu + +PROXY_URL="${PROXY_URL:-http://localhost:4011}" +DB_CONTAINER="${DB_CONTAINER:-litellm-e2e-db}" +DB_USER="${DB_USER:-litellm}" +DB_NAME="${DB_NAME:-litellm}" +MASTER_KEY="${MASTER_KEY:-sk-e2e-test}" +MOCK_CONTAINER="${MOCK_CONTAINER:-litellm-e2e-mock}" +PROXY_CONTAINER="${PROXY_CONTAINER:-litellm-e2e}" + +# Preflight: this case is mock-only — it needs the mock provider's +# X-Mock-TTFT-Ms knob to deterministically force a TTFT > shield budget. +# In real-provider mode the request lands on a live API that won't honor +# the header and the assertions become meaningless. SKIP rather than FAIL. +if ! docker exec "$MOCK_CONTAINER" python3 -c \ + "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')" 2>/dev/null; then + echo "SKIP: $MOCK_CONTAINER not up (run with --with-mock)" + exit 77 +fi + +# Check that the proxy was started with a short shield timeout. +SHIELD_TIMEOUT=$(docker exec "$PROXY_CONTAINER" \ + printenv LITELLM_CANCEL_SHIELD_TIMEOUT_S 2>/dev/null || echo "") +if [ -z "$SHIELD_TIMEOUT" ] || python3 -c "import sys; sys.exit(0 if float('$SHIELD_TIMEOUT') < 30 else 1)" 2>/dev/null; then + : # short timeout is set, can proceed +else + echo "SKIP: LITELLM_CANCEL_SHIELD_TIMEOUT_S not set or >=30s on proxy container." + echo " Restart with: LITELLM_CANCEL_SHIELD_TIMEOUT_S=2 e2e/tools/proxy start --with-mock" + exit 77 +fi +echo " proxy LITELLM_CANCEL_SHIELD_TIMEOUT_S=$SHIELD_TIMEOUT s" + +USER_SENTINEL="case30-$(date +%s%N)" + +echo "[30] non-stream cancel + shield_timeout..." +# Mock takes 8 seconds (well over the 2s shield). Client cancels at 1s. +# Expected sequence: +# T+0 client POST +# T+1 client times out → CancelledError into proxy +# T+1 cancel_finalize starts shield + wait_for(upstream, timeout=2s) +# T+3 shield wait_for raises asyncio.TimeoutError +# → usage_source=shield_timeout, upstream_task.cancel() +# → fallback to failure hook → CancelledError branch in +# async_post_call_failure_hook keeps status="success" (cancel +# is not a system failure) and the derivation yields +# delivery=none, billing=partial +set +e +timeout 1 curl -sS -X POST "$PROXY_URL/v1/chat/completions" \ + -H "Authorization: Bearer $MASTER_KEY" \ + -H "Content-Type: application/json" \ + -H "X-Mock-TTFT-Ms: 8000" \ + -H "X-Mock-Full-Chars: 300" \ + -d '{ + "model":"mock-anthropic", + "user":"'$USER_SENTINEL'", + "messages":[{"role":"user","content":"shield timeout test"}], + "stream":false, + "max_tokens":300 + }' > /tmp/case30.out 2>&1 +RC=$? +set -e +echo " curl rc=$RC, bytes received: $(wc -c < /tmp/case30.out)" + +# Poll up to 30s — shield waits, then upstream completes silently in +# background, then spend_log gets batched. +ROW="" +for i in $(seq 1 30); do + sleep 1 + ROW=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -tA -F'|' -c " +SELECT + COALESCE(status, ''), + COALESCE(metadata::jsonb->>'cancellation_indicator', ''), + COALESCE(metadata::jsonb->>'cancel_phase', ''), + COALESCE(metadata::jsonb->>'usage_source', ''), + COALESCE(metadata::jsonb->>'upstream_completed', ''), + COALESCE(metadata::jsonb->>'delivery_status', ''), + COALESCE(metadata::jsonb->>'billing_status', '') +FROM \"LiteLLM_SpendLogs\" +WHERE end_user = '$USER_SENTINEL' +ORDER BY \"startTime\" DESC LIMIT 1; +" 2>/dev/null | head -1) + [ -n "$ROW" ] && break +done + +if [ -z "$ROW" ]; then + echo "FAIL: no SpendLogs row for $USER_SENTINEL after 30s" + exit 1 +fi + +IFS='|' read -r STATUS IND PHASE SRC UPCOMP DEL BIL <<< "$ROW" +echo " row: status=$STATUS ind=$IND phase=$PHASE src=$SRC up=$UPCOMP delivery=$DEL billing=$BIL" + +OK=1 +# Binary-status taxonomy: shield_timeout still carries status="success" +# (cancel != system failure). delivery=none (non-stream, nothing reached +# the client), billing=partial (we billed the prompt-only baseline because +# shield timed out before upstream's real usage came back). +if [ "$STATUS" != "success" ]; then + echo "FAIL: expected status=success, got '$STATUS'" + OK=0 +fi +if [ "$IND" != "client_disconnect" ]; then + echo "FAIL: expected cancellation_indicator=client_disconnect, got '$IND'" + OK=0 +fi +if [ "$SRC" != "shield_timeout" ]; then + echo "FAIL: expected usage_source=shield_timeout, got '$SRC'" + echo " (Shield budget = ${SHIELD_TIMEOUT}s; mock TTFT=8000ms — shield should have" + echo " given up before upstream returned. If src=upstream_completed_after_cancel," + echo " the shield timeout knob isn't being honored.)" + OK=0 +fi +if [ "$DEL" != "none" ]; then + echo "FAIL: expected delivery_status=none, got '$DEL'" + OK=0 +fi +if [ "$BIL" != "partial" ]; then + echo "FAIL: expected billing_status=partial (shield_timeout → prompt-only baseline), got '$BIL'" + OK=0 +fi + +if [ $OK -eq 1 ]; then + echo "PASS: shield_timeout path correctly recorded" + exit 0 +else + exit 1 +fi diff --git a/e2e/cases/data/32_concurrent_cancels.sh b/e2e/cases/data/32_concurrent_cancels.sh new file mode 100755 index 000000000000..d0af9de86cf6 --- /dev/null +++ b/e2e/cases/data/32_concurrent_cancels.sh @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +# Case 32 — Concurrent streaming cancellations: no task leak, every +# row written. +# +# Verifies the cancel-finalize plumbing under load. The unit test +# tests/test_litellm/litellm_core_utils/test_cancel_finalize.py +# covers single-request cases; this case fires N parallel requests +# and cancels all of them mid-stream, then asserts: +# - all N SpendLogs rows exist (no black hole under contention) +# - each row carries the cancellation_indicator marker (binary- +# status taxonomy: rows are status="success" + marker, not the +# legacy status="success_partial") +# - the proxy process didn't OOM or leak asyncio tasks (we +# spot-check by looking at request-handler latency on a control +# request after the burst) +# +# Tier: mock-only. Light enough for CI — 10 concurrent stream +# cancels, each ~2s. + +set -eu + +PROXY_URL="${PROXY_URL:-http://localhost:4011}" +DB_CONTAINER="${DB_CONTAINER:-litellm-e2e-db}" +DB_USER="${DB_USER:-litellm}" +DB_NAME="${DB_NAME:-litellm}" +MASTER_KEY="${MASTER_KEY:-sk-e2e-test}" +MOCK_CONTAINER="${MOCK_CONTAINER:-litellm-e2e-mock}" + +if ! docker exec "$MOCK_CONTAINER" python3 -c \ + "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')" 2>/dev/null; then + echo "SKIP: $MOCK_CONTAINER not up (run with --with-mock)" + exit 77 +fi + +N=10 +RUN_ID="case32-$(date +%s%N)" + +echo "[32] firing $N concurrent streaming cancels..." + +# Fire N concurrent curls, each cancelled after 2s. +# Use distinct end_user values so we can count rows per request. +pids=() +for i in $(seq 1 $N); do + USER_SENTINEL="${RUN_ID}-${i}" + ( + timeout 2 curl -sN -X POST "$PROXY_URL/v1/chat/completions" \ + -H "Authorization: Bearer $MASTER_KEY" \ + -H "Content-Type: application/json" \ + -H "X-Mock-TTFT-Ms: 100" \ + -H "X-Mock-Chunks: 80" \ + -H "X-Mock-TPS: 15" \ + -d "{ + \"model\":\"mock-anthropic\", + \"user\":\"$USER_SENTINEL\", + \"messages\":[{\"role\":\"user\",\"content\":\"concurrent cancel test $i\"}], + \"stream\":true, + \"max_tokens\":2000 + }" > /dev/null 2>&1 + ) & + pids+=($!) +done + +# Wait for all to complete (they'll all be killed by timeout 2) +for pid in "${pids[@]}"; do + wait "$pid" 2>/dev/null || true +done +echo " all $N curls completed (cancelled by timeout)" + +# Poll for rows. Cancel writes can lag a few seconds, especially +# under contention. +EXPECTED=$N +ROW_COUNT=0 +for i in $(seq 1 45); do + sleep 1 + ROW_COUNT=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -tA -c " +SELECT count(*) +FROM \"LiteLLM_SpendLogs\" +WHERE end_user LIKE '${RUN_ID}-%' + AND status='success' + AND metadata::jsonb->>'cancellation_indicator' = 'client_disconnect'; +" 2>/dev/null | head -1) + if [ "${ROW_COUNT:-0}" -ge "$EXPECTED" ]; then + echo " $ROW_COUNT/$EXPECTED rows after ${i}s" + break + fi +done + +OK=1 +if [ "${ROW_COUNT:-0}" -lt "$EXPECTED" ]; then + echo "FAIL: expected $EXPECTED cancelled rows (status='success' + cancellation_indicator), got ${ROW_COUNT:-0}" + OK=0 +fi + +# Spot-check: after the cancel burst, the proxy should still respond +# normally to a quick health-check style request. If the asyncio task +# pool is leaked, this would hang or take much longer than the baseline. +CONTROL_START=$(date +%s%N) +HEALTH=$(curl -sS --max-time 5 -o /dev/null -w '%{http_code}' \ + "$PROXY_URL/health/liveliness" 2>/dev/null || echo "TIMEOUT") +CONTROL_END=$(date +%s%N) +CONTROL_MS=$(( (CONTROL_END - CONTROL_START) / 1000000 )) +echo " post-burst health probe: HTTP $HEALTH in ${CONTROL_MS}ms" +if [ "$HEALTH" != "200" ]; then + echo "FAIL: proxy unhealthy after cancel burst (task leak / hang)" + OK=0 +fi +if [ "$CONTROL_MS" -gt 3000 ]; then + echo "FAIL: proxy responded slowly (${CONTROL_MS}ms) — possible task contention" + OK=0 +fi + +# Verify all rows have the marker set (sample a few) +SAMPLED_MARKERS=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -tA -c " +SELECT count(*) +FROM \"LiteLLM_SpendLogs\" +WHERE end_user LIKE '${RUN_ID}-%' + AND metadata::jsonb->>'cancellation_indicator' = 'client_disconnect'; +" 2>/dev/null | head -1) +echo " rows with cancellation_indicator: $SAMPLED_MARKERS/$EXPECTED" +if [ "${SAMPLED_MARKERS:-0}" -lt "$EXPECTED" ]; then + echo "FAIL: only $SAMPLED_MARKERS/$EXPECTED rows have cancellation_indicator" + OK=0 +fi + +if [ $OK -eq 1 ]; then + echo "PASS: $N concurrent cancels → $EXPECTED cancelled rows (status=success + marker) + healthy proxy" + exit 0 +else + exit 1 +fi diff --git a/e2e/cases/data/33_real_anthropic_cancel.sh b/e2e/cases/data/33_real_anthropic_cancel.sh new file mode 100755 index 000000000000..869d415a974e --- /dev/null +++ b/e2e/cases/data/33_real_anthropic_cancel.sh @@ -0,0 +1,211 @@ +#!/usr/bin/env bash +# Case 33 — Real Anthropic streaming cancel → success_partial with +# real upstream-priced cost. +# +# The mock-based cases 26-32 verify the cancel-billing chain against a +# stdlib HTTP stub. This case is the *real* provider end-to-end check: +# proves the cursor=1 fix and the cancel-finalize plumbing both behave +# correctly when the chunk stream is from genuine Anthropic API +# infrastructure rather than the mock's deterministic SSE. +# +# Specifically catches regressions where: +# - Real Anthropic's content_block_delta sequence has a structural +# quirk the mock doesn't reproduce (extra event types, +# server_tool_use blocks, citations, etc.) → cursor reset or chunk +# reassembly silently fails. +# - FallbackStreamWrapper's new chunk-accumulation on real long-tail +# thinking streams leaks memory or duplicates chunks. +# - The cost calculator runs on the partial response with a real +# model id in the LiteLLM cost map → spend should be > 0 (not 0.0 +# because mock-anthropic is unpriced). +# +# Tier: real. Requires ANTHROPIC_API_KEY in e2e/.env. Cost: ~$0.01 +# per run (Anthropic Sonnet ~1500 prompt + ~200 completion before +# cancel × $3/M input + $15/M output). + +set -eu + +PROXY_URL="${PROXY_URL:-http://localhost:4011}" +DB_CONTAINER="${DB_CONTAINER:-litellm-e2e-db}" +DB_USER="${DB_USER:-litellm}" +DB_NAME="${DB_NAME:-litellm}" +MASTER_KEY="${MASTER_KEY:-sk-e2e-test}" + +# Tier=real preflight: skip if no Anthropic key. +ANTHROPIC_KEY="${ANTHROPIC_API_KEY:-}" +if [ -z "$ANTHROPIC_KEY" ]; then + # Try to read from the proxy container so this case works against + # a proxy started by e2e/tools/proxy (env file is mounted there). + ANTHROPIC_KEY=$(docker exec litellm-e2e printenv ANTHROPIC_API_KEY 2>/dev/null || echo "") +fi +if [ -z "$ANTHROPIC_KEY" ]; then + echo "SKIP: ANTHROPIC_API_KEY not set — case 33 needs real Anthropic credentials." + exit 77 +fi + +USER_SENTINEL="case33-$(date +%s%N)" + +# Build a prompt long enough that the response takes 5-10 seconds to +# fully stream — so a 3s curl --max-time gives Anthropic time to send +# message_start + content_block_delta chunks but NOT message_delta or +# message_stop. Exercises the cursor=1 + reassembly fallback path. +# +# Using a moderately long system prompt to also trigger prompt-cache +# creation tokens, which gives us a non-trivial cache_creation field +# in the row to assert on (extra signal beyond just completion_tokens). +LONG_USER="Write a comprehensive technical comparison of three programming languages: Go, Python, and Rust. For each language, cover: (1) memory model and garbage collection, (2) concurrency primitives, (3) typical compilation/execution speed, (4) ecosystem and tooling, (5) production deployment story. Provide concrete code examples for each section. Be thorough and detailed — write at least 1500 words." + +echo "[33] real Anthropic streaming cancel..." + +set +e +# 8s curl timeout: real Anthropic TTFT can be 2-4s on a 1500-token +# prompt; we need 8s to get past TTFT + a few seconds of streaming +# chunks before cancel. message_delta won't arrive in time (full +# generation would take 15-25s). +timeout 8 curl -sN -X POST "$PROXY_URL/v1/chat/completions" \ + -H "Authorization: Bearer $MASTER_KEY" \ + -H "Content-Type: application/json" \ + -d "$(python3 -c " +import json, sys +body = { + 'model': 'claude-sonnet-cache', + 'user': '$USER_SENTINEL', + 'stream': True, + 'max_tokens': 2000, + 'messages': [{'role': 'user', 'content': sys.argv[1]}], +} +print(json.dumps(body)) +" "$LONG_USER")" > /tmp/case33.out 2>&1 +RC=$? +set -e + +BYTES=$(wc -c < /tmp/case33.out) +echo " curl rc=$RC, bytes streamed: $BYTES" + +# Anthropic TTFT varies (real network + provider load). If we got 0 +# bytes the cancel hit before TTFT — we can still verify the cancel +# row was written (the prompt was billable input either way). If +# bytes > 0 but very small (< 100) AND curl exited cleanly (rc=0), +# that's a real upstream error response — fail explicitly so we +# don't silently mis-pass. +if [ "$RC" -eq 0 ] && [ "$BYTES" -lt 100 ]; then + echo "FAIL: curl completed with $BYTES bytes — likely auth or model error" + echo "--- response head ---" + head -c 500 /tmp/case33.out + echo "---" + exit 1 +fi + +# Poll for the row. Real Anthropic + cost calc + spend log batch can +# take 5-10s total. +ROW="" +for i in $(seq 1 30); do + sleep 1 + ROW=$(docker exec "$DB_CONTAINER" psql -U "$DB_USER" -d "$DB_NAME" -tA -F'|' -c " +SELECT + COALESCE(status, ''), + COALESCE(completion_tokens::text, '0'), + COALESCE(prompt_tokens::text, '0'), + COALESCE(spend::text, '0'), + COALESCE(metadata::jsonb->>'cancellation_indicator', ''), + COALESCE(metadata::jsonb->>'cancel_phase', ''), + COALESCE(metadata::jsonb->>'delivery_status', ''), + COALESCE(metadata::jsonb->>'billing_status', '') +FROM \"LiteLLM_SpendLogs\" +WHERE end_user = '$USER_SENTINEL' +ORDER BY \"startTime\" DESC LIMIT 1; +" 2>/dev/null | head -1) + [ -n "$ROW" ] && break +done + +if [ -z "$ROW" ]; then + echo "FAIL: no SpendLogs row for $USER_SENTINEL after 30s" + exit 1 +fi + +IFS='|' read -r STATUS COMPL PROMPT SPEND IND PHASE DEL BIL <<< "$ROW" +echo " row: status=$STATUS prompt=$PROMPT completion=$COMPL spend=$SPEND ind=$IND phase=$PHASE delivery=$DEL billing=$BIL" + +OK=1 + +# Status taxonomy — binary-status: status="success" + marker, NOT the +# legacy "success_partial" sentinel. Cancel row classified by +# cancellation_indicator + derived delivery/billing dimensions. +if [ "$STATUS" != "success" ]; then + echo "FAIL: expected status=success, got '$STATUS'" + OK=0 +fi +if [ "$IND" != "client_disconnect" ]; then + echo "FAIL: expected cancellation_indicator=client_disconnect, got '$IND'" + OK=0 +fi +if [ "$PHASE" != "streaming_partial" ]; then + echo "FAIL: expected cancel_phase=streaming_partial, got '$PHASE'" + OK=0 +fi + +# Derived taxonomy. delivery_status / billing_status depend on whether +# bytes actually reached the client. With BYTES>0 we expect both +# partial; with BYTES=0 (cancel hit before TTFT) we expect both none. +if [ "${BYTES:-0}" -gt 0 ]; then + if [ "$DEL" != "partial" ]; then + echo "FAIL: expected delivery_status=partial (bytes>0), got '$DEL'" + OK=0 + fi + if [ "$BIL" != "partial" ]; then + echo "FAIL: expected billing_status=partial (bytes>0), got '$BIL'" + OK=0 + fi +else + if [ "$DEL" != "none" ]; then + echo "FAIL: expected delivery_status=none (bytes=0), got '$DEL'" + OK=0 + fi +fi + +# If we got streamed bytes, completion_tokens MUST exceed 1. If +# it's exactly 1, the cursor=1 fix isn't being applied to real +# Anthropic streams (a critical regression — the whole point of +# PR #1). If we got 0 bytes (TTFT > our budget), tolerate +# completion=0 — the cancel still fired before generation started. +if [ "${BYTES:-0}" -gt 0 ] && [ "${COMPL:-0}" -le 1 ]; then + echo "FAIL: bytes=$BYTES but completion_tokens=$COMPL — cursor=1 fix" + echo " may have regressed against the real Anthropic streaming" + echo " protocol. Mock tests pass because the mock's chunk shape" + echo " exactly matches the fix's heuristic, but real Anthropic may" + echo " emit additional events that throw off saw_non_cursor_completion." + OK=0 +fi + +# prompt_tokens must be > 0, reflecting real Anthropic's +# message_start.input_tokens reaching the row (NOT the local +# tokenizer fallback). Empirically our ~1500-char prompt comes in at +# 80-100 Anthropic tokens — much less than chars/4 because of +# vocabulary efficiency, but well above 0. +if [ "${PROMPT:-0}" -le 0 ]; then + echo "FAIL: prompt_tokens=$PROMPT — Anthropic's message_start.input_tokens" + echo " should be > 0 (we sent a non-trivial user message). A 0 here" + echo " indicates the metadata bridge didn't carry the upstream" + echo " input_tokens through to the SpendLogs row." + OK=0 +fi + +# spend > 0 only if we got bytes (partial response to cost). When +# bytes=0 the cost calc has nothing to price; spend=0 is fine and +# the row still proves the cancel taxonomy applied. +if [ "${BYTES:-0}" -gt 0 ]; then + if ! python3 -c "import sys; sys.exit(0 if float('${SPEND:-0}') > 0 else 1)" 2>/dev/null; then + echo "FAIL: spend=$SPEND with bytes=$BYTES — cost calc didn't price" + echo " the partial response. Either anthropic/claude-sonnet*" + echo " isn't in the cost map (model id mismatch) or the cost" + echo " callback ran before reassembly." + OK=0 + fi +fi + +if [ $OK -eq 1 ]; then + echo "PASS: real Anthropic cancel billed at \$$SPEND with $COMPL completion_tokens" + exit 0 +else + exit 1 +fi diff --git a/e2e/tools/run-all-cases b/e2e/tools/run-all-cases index 94228cb7929f..0b38f5a37053 100755 --- a/e2e/tools/run-all-cases +++ b/e2e/tools/run-all-cases @@ -46,7 +46,7 @@ done # Tier=real cases that --mock-only skips. See e2e/cases/README.md index. # 04 + 06 are Tier=both (cache metrics ABSENT — provable on mock); the # rest of 01-09 verify cache TOKEN VALUES that real providers compute. -REAL_ONLY_CASES="01 02 03 05 08 09 19" +REAL_ONLY_CASES="01 02 03 05 08 09 19 33" is_real_only() { local n="$1" @@ -505,6 +505,133 @@ case_25() { fi } +# ------------------------------------------------------------------ 26 +case_26() { + echo "[26] cancel billing → cancelled SpendLogs row (status=success + marker)..." + local out=/tmp/e2e_case26.out + bash e2e/cases/data/26_cancel_billing_partial.sh > "$out" 2>&1 + local rc=$? + if [ "$rc" -eq 0 ]; then + ok "26 cancel-billing: $(grep -m1 '^PASS: all' "$out")" + elif [ "$rc" -eq 77 ]; then + skip "26 cancel-billing" "$(grep -m1 SKIP "$out")" + else + fail "26 cancel-billing" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")" + fi +} + +# ------------------------------------------------------------------ 27 +case_27() { + echo "[27] non-stream cancel → shield+wait..." + local out=/tmp/e2e_case27.out + bash e2e/cases/data/27_non_stream_cancel_shield.sh > "$out" 2>&1 + local rc=$? + if [ "$rc" -eq 0 ]; then + ok "27 non-stream-cancel: $(grep -m1 '^PASS' "$out")" + elif [ "$rc" -eq 77 ]; then + skip "27 non-stream-cancel" "$(grep -m1 SKIP "$out")" + else + fail "27 non-stream-cancel" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")" + fi +} + +# ------------------------------------------------------------------ 28 +case_28() { + echo "[28] /v1/messages cancel..." + local out=/tmp/e2e_case28.out + bash e2e/cases/data/28_v1_messages_cancel.sh > "$out" 2>&1 + local rc=$? + if [ "$rc" -eq 0 ]; then + ok "28 v1-messages-cancel: $(grep -m1 '^PASS' "$out")" + elif [ "$rc" -eq 77 ]; then + skip "28 v1-messages-cancel" "$(grep -m1 SKIP "$out")" + else + fail "28 v1-messages-cancel" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")" + fi +} + +# ------------------------------------------------------------------ 29 +case_29() { + echo "[29] real upstream 5xx stays status=failure..." + local out=/tmp/e2e_case29.out + bash e2e/cases/data/29_failure_not_polluted.sh > "$out" 2>&1 + local rc=$? + if [ "$rc" -eq 0 ]; then + ok "29 failure-not-polluted: $(grep -m1 '^PASS' "$out")" + elif [ "$rc" -eq 77 ]; then + skip "29 failure-not-polluted" "$(grep -m1 SKIP "$out")" + else + fail "29 failure-not-polluted" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")" + fi +} + +# ------------------------------------------------------------------ 30 +case_30() { + echo "[30] shield_timeout path..." + local out=/tmp/e2e_case30.out + bash e2e/cases/data/30_shield_timeout.sh > "$out" 2>&1 + local rc=$? + if [ "$rc" -eq 0 ]; then + ok "30 shield-timeout: $(grep -m1 '^PASS' "$out")" + elif [ "$rc" -eq 77 ]; then + skip "30 shield-timeout" "$(grep -m1 SKIP "$out")" + else + fail "30 shield-timeout" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")" + fi +} + +# ------------------------------------------------------------------ 32 +case_32() { + echo "[32] concurrent stream cancels..." + local out=/tmp/e2e_case32.out + bash e2e/cases/data/32_concurrent_cancels.sh > "$out" 2>&1 + local rc=$? + if [ "$rc" -eq 0 ]; then + ok "32 concurrent-cancels: $(grep -m1 '^PASS' "$out")" + elif [ "$rc" -eq 77 ]; then + skip "32 concurrent-cancels" "$(grep -m1 SKIP "$out")" + else + fail "32 concurrent-cancels" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")" + fi +} + +# ------------------------------------------------------------------ 34 +# Tier=mock-only — TimeWeightedRouter (litellm_extras/) end-to-end. The +# fixture registers mock deployments, drives wall-clock-bound bands, and +# asserts that PATCH-driven hot reload, blocked exclusion, and the +# delegate path for non-opt-in models all behave correctly. +case_34() { + echo "[34] time-weighted routing..." + local out=/tmp/e2e_case34.out + python3 e2e/cases/data/34_time_weighted_routing.py > "$out" 2>&1 + local rc=$? + if [ "$rc" -eq 0 ]; then + ok "34 time-weighted-routing: $(grep -m1 '^PASS' "$out")" + elif [ "$rc" -eq 77 ]; then + skip "34 time-weighted-routing" "$(grep -m1 SKIP "$out")" + else + fail "34 time-weighted-routing" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")" + fi +} + +# ------------------------------------------------------------------ 33 +# Tier=real — exercises the cancel-billing chain against the real +# Anthropic API to catch regressions the mock-based 26/27/32 suite +# can't (cursor=1 on genuine Anthropic streams, real cost-map lookup). +case_33() { + echo "[33] real anthropic cancel..." + local out=/tmp/e2e_case33.out + bash e2e/cases/data/33_real_anthropic_cancel.sh > "$out" 2>&1 + local rc=$? + if [ "$rc" -eq 0 ]; then + ok "33 real-anthropic-cancel: $(grep -m1 '^PASS' "$out")" + elif [ "$rc" -eq 77 ]; then + skip "33 real-anthropic-cancel" "$(grep -m1 SKIP "$out")" + else + fail "33 real-anthropic-cancel" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")" + fi +} + # Pre-flight: proxy must be ready if ! curl -sSL --max-time 3 -o /dev/null -w '%{http_code}' \ "$PROXY/health/readiness" 2>/dev/null | grep -q '^2'; then @@ -640,6 +767,34 @@ case_24 # paths or /key/generate. Safe outside --mock-only. case_25 +# Case 26 (cancel billing → cancelled SpendLogs row). Requires the +# mock provider so we can deterministically pace streams and cancel +# them mid-flight without burning real provider $. Skipped outside +# --with-mock (mock container won't be reachable). +case_26 + +# Cases 27-32 are the Phase 1 cancel-billing expansion suite, all +# mock-only. See e2e/cases/26_*.md for the overall design context. +case_27 +case_28 +case_29 +case_30 +case_32 + +# Case 34 (time-weighted routing) is mock-only — needs mock-anthropic +# sidecar and the proxy reachable. Outside --mock-only the fixture +# still self-protects (skips when /health/readiness or mock are down). +if [ "$MOCK_ONLY" -eq 1 ]; then + case_34 +fi + +# Case 33 is Tier=real — exercises the cancel-billing chain against +# the real Anthropic API. _dispatch routes it through +# REAL_ONLY_CASES (skips on --mock-only). When run without +# --mock-only and ANTHROPIC_API_KEY is unset, the fixture itself +# exits 77 as a second-layer safety net. +_dispatch 33 real-anthropic-cancel + echo echo "============ SUMMARY ============" printf '%s\n' "${RESULTS[@]}" diff --git a/litellm/litellm_core_utils/cancel_billing.py b/litellm/litellm_core_utils/cancel_billing.py new file mode 100644 index 000000000000..9f03371ac755 --- /dev/null +++ b/litellm/litellm_core_utils/cancel_billing.py @@ -0,0 +1,212 @@ +""" +Cost computation for client-cancelled requests ("modified strategy 5'"). + +Companion to ``cancel_finalize.py``: + +* ``cancel_finalize`` catches the cancel signal, tags the Logging + object, and dispatches the partial response through + ``async_success_handler``. +* This module computes the dollar cost for the partial work — the + number that ends up in ``LiteLLM_SpendLogs.spend`` for the cancelled + row. The row stays ``status="success"`` (cancellation is not a system + failure); the cancel taxonomy lives in metadata: ``cancellation_indicator``, + ``cancel_phase``, ``usage_source``, plus derived ``delivery_status`` / + ``billing_status``. + +Billing strategy (per 2026-06-08 design decision): + +1. **Always bill the prompt** — the upstream provider received the + request and almost always started processing it. Recovering the + input cost matches new-api's behavior and aligns with how OpenAI + / Anthropic themselves charge. + +2. **Bill output tokens that we have evidence for.** For streaming + cancels with chunks accumulated, the upstream usage (or token- + counter estimate of received text) already lives on the + reassembled partial response — the normal cost calculator handles + it correctly. + +3. **For zero-evidence cancels** (cancel fired before any chunk + arrived, or non-stream shield timed out) we still bill the + prompt-only baseline rather than $0. The upstream charge is real + even when we don't see the output. + +The functions here implement the "zero-evidence prompt-only" case; +the "we have a partial response" case flows through the existing cost +calculator unchanged (which is correct after PR #1's cursor=1 fix). +""" + +from __future__ import annotations + +from typing import Any, List, Optional + +from litellm._logging import verbose_logger + + +def compute_prompt_only_cost( + messages: Optional[List] = None, + model: Optional[str] = None, + custom_llm_provider: Optional[str] = None, +) -> float: + """ + Best-effort prompt-only cost for cancellations that produced no + response (zero-chunk streaming cancel, or non-stream shield + timeout). Returns 0.0 if the prompt can't be priced (unknown + model, tokenizer failure, etc.) — never raises. + + This is NOT a substitute for real upstream usage. When the cancel + path has chunks to work with, the regular cost calculator on the + reassembled partial response produces a better number. + + Pricing-model rationale: counts prompt_tokens via ``token_counter`` + (the same path used by ``stream_chunk_builder``'s fallback after + PR #1's cursor=1 fix), then uses ``cost_per_token`` against the + LiteLLM model cost map. cache_creation / cache_read are zero + because zero-chunk cancels haven't seen the message_start usage + block where those fields are populated. + """ + if not model or not messages: + return 0.0 + + try: + import litellm + + prompt_tokens = litellm.token_counter(messages=messages, model=model) + if prompt_tokens <= 0: + return 0.0 + + prompt_cost, _completion_cost = litellm.cost_per_token( + model=model, + prompt_tokens=prompt_tokens, + completion_tokens=0, + custom_llm_provider=custom_llm_provider, + ) + return float(prompt_cost) if prompt_cost is not None else 0.0 + except Exception as exc: + # Pricing the prompt is best-effort — model not in the cost map, + # bad tokenizer config, unsupported provider all fall through to + # 0.0. Better than crashing the failure-hook code path. + verbose_logger.debug( + "compute_prompt_only_cost: pricing failed for model=%s: %s", + model, + exc, + ) + return 0.0 + + +def enrich_request_metadata_with_cancel_markers( + request_data: dict, + logging_obj: Any, +) -> None: + """ + Copy the cancellation markers from ``logging_obj.model_call_details`` + into ``request_data["litellm_params"]["metadata"]`` so that the + standard SpendLogs metadata pipeline picks them up and persists + them into the metadata JSON column. + + Without this bridge, ``cancel_finalize.mark_logging_obj_cancelled`` + writes to the Logging object but the markers never reach SpendLogs + — the metadata-keyed extractor in ``_get_spend_logs_metadata`` + pulls from ``litellm_params.metadata``, not from + ``model_call_details``. + + Idempotent. No-op when the Logging object has no markers (i.e. + this is a normal, non-cancelled request). + """ + if logging_obj is None: + return + details = getattr(logging_obj, "model_call_details", None) + if not details or details.get("cancellation_indicator") is None: + return + + # Build the list of target metadata dicts to mutate. We have to + # touch ALL of: + # + # 1. request_data["litellm_params"]["metadata"] — used by the + # proxy failure-hook path to build SpendLogs. + # 2. request_data["litellm_params"]["litellm_metadata"] — used + # by newer endpoints (e.g. /v1/messages, anthropic_messages, + # generate_content). get_litellm_metadata_from_kwargs prefers + # litellm_metadata when both are present, so writing only to + # metadata leaves the newer endpoints' markers invisible. + # 3. logging_obj.litellm_params["metadata"] — used by the litellm + # Logging.async_success_handler / cost callback when the + # Logging object's litellm_params dict has diverged from + # request_data's (some code paths copy at construction time). + # 4. logging_obj.litellm_params["litellm_metadata"] — same as + # above but for the newer-endpoint variant. + # + # We write the cancel markers to whichever variants already exist, + # plus always to "metadata" (which the old endpoints + the failure + # hook read). Idempotent; safe to call multiple times. + target_dicts: list = [] + + def _ensure_metadata_dicts(parent: dict) -> None: + if "metadata" not in parent or parent["metadata"] is None: + parent["metadata"] = {} + target_dicts.append(parent["metadata"]) + # litellm_metadata is only present on newer endpoints; if it's + # already there with content, we must also write to it (the + # extractor prefers it over metadata). + existing_litellm_metadata = parent.get("litellm_metadata") + if isinstance(existing_litellm_metadata, dict): + target_dicts.append(existing_litellm_metadata) + + # 1+2: request_data side. + if "litellm_params" not in request_data: + request_data["litellm_params"] = {} + _ensure_metadata_dicts(request_data["litellm_params"]) + + # 3+4: logging_obj side, if it has its own litellm_params dict. + lp = getattr(logging_obj, "litellm_params", None) + if isinstance(lp, dict): + prior_targets = list(target_dicts) + _ensure_metadata_dicts(lp) + # De-dup: skip any dict already in our list (when proxy did the + # usual `logging_obj.litellm_params = request_data["litellm_params"]` + # assignment, the same dict gets enumerated twice). + target_dicts = prior_targets + [ + d for d in target_dicts[len(prior_targets) :] if d not in prior_targets + ] + + # Copy the five cancellation fields into every target. + for target_metadata in target_dicts: + for field in ( + "cancellation_indicator", + "cancel_phase", + "bytes_delivered_to_client", + "upstream_completed", + "usage_source", + ): + if field in details: + target_metadata[field] = details[field] + + # NOTE: we deliberately do NOT touch `metadata["status"]` here. + # Cancellation is identified by the presence of + # cancellation_indicator above; the top-level status column stays + # "success" (cancel != system failure). The orthogonal taxonomy + # (delivery_status / billing_status) is derived downstream in + # spend_tracking_utils._derive_delivery_billing_status from these + # very markers — single source of truth. + + # Populate error_information.error_code="499" so the UI + # (which renders metadata.error_information.error_code as the + # HTTP status label) can distinguish cancelled requests from + # regular successes. This is an independent UI hint channel — + # the SQL/status surface uses the cancel markers above, the UI + # badge uses error_code, and they happen to converge on the + # same row. The success_handler path doesn't populate + # error_information on its own — without this block, cancelled + # rows look identical to plain successes in the dashboard. + existing_err_info = target_metadata.get("error_information") or {} + if not isinstance(existing_err_info, dict): + existing_err_info = {} + existing_err_info.setdefault("error_code", "499") + existing_err_info.setdefault("error_class", "CancelledError") + existing_err_info.setdefault("error_message", "Client disconnected the request") + # llm_provider and traceback aren't relevant for cancels; + # leave them empty so the UI doesn't render a misleading + # provider attribution or stack trace. + existing_err_info.setdefault("llm_provider", "") + existing_err_info.setdefault("traceback", "") + target_metadata["error_information"] = existing_err_info diff --git a/litellm/litellm_core_utils/cancel_finalize.py b/litellm/litellm_core_utils/cancel_finalize.py new file mode 100644 index 000000000000..8549f7fc47cb --- /dev/null +++ b/litellm/litellm_core_utils/cancel_finalize.py @@ -0,0 +1,482 @@ +""" +Client-cancellation finalization for LiteLLM streaming + non-streaming paths. + +When a client disconnects mid-flight, asyncio injects ``CancelledError`` +into the in-flight request task. Historically every ``except Exception`` +in LiteLLM let this BaseException-subclass slip through, so: + +* No SpendLogs record was written for the cancelled request. +* No failure or success callback fired (Langfuse trace ended in a + permanent "Running" state, Prometheus failed_requests counter not + incremented). +* The upstream provider (which often continues generating after a + client TCP close) silently charged for compute that never reached + the LiteLLM ledger. + +This module re-routes cancellation through the proxy's existing +``async_success_handler`` path so that: + +* SpendLogs always records a row, even on cancellation. The row has + ``status="success"`` (cancel != system failure) plus the cancellation + markers (cancellation_indicator, cancel_phase, usage_source, etc.) + that drive the derived ``delivery_status`` and ``billing_status`` + taxonomy in the metadata JSON column. +* Whatever was streamed (or, for non-stream, would have been streamed + given the shield logic in cancel_billing.py) gets billed. +* Failure-rate metrics are not polluted by client-initiated cancels. +* Langfuse traces close cleanly via the normal success path; dashboards + distinguishing cancels filter on cancellation_indicator. + +This file is the catch-and-route plumbing. The cost computation for +the partial response lives in cancel_billing.py (added separately — +see PR #3 / decision D9 in the billing-accuracy plan). + +Public API +---------- +``mark_logging_obj_cancelled(logging_obj, phase, indicator)`` + Idempotently tags the LiteLLM Logging object with cancellation + metadata. The downstream cost calculator reads these markers to + drive the cancel-billing path (prompt-only / chunk-reassembly / + shield-and-wait depending on phase). + +``finalize_streaming_cancel(...)`` + Called from the proxy's streaming generator catch block. Builds a + partial response from whatever chunks the stream wrapper has + accumulated, then dispatches through the normal success callback + chain so all the usual observability hooks fire. + +Important: callers MUST re-raise the original ``CancelledError`` after +calling these helpers — swallowing the cancel signal would leak tasks +and confuse asyncio's cancellation propagation. +""" + +from __future__ import annotations + +import asyncio +import time +from datetime import datetime +from typing import Any, List, Optional + +import anyio + +from litellm._logging import verbose_logger +from litellm.types.utils import CancelPhase + +# Default timeout for the non-stream shield wait. 60 s is long enough to +# cover Anthropic thinking-model long-tails (p99 ~120 s on observed +# traffic — see e2e/tools/glm_repro findings) without holding the worker +# indefinitely. Tunable via LITELLM_CANCEL_SHIELD_TIMEOUT_S env var when +# a deployment finds it too generous, or for e2e cases that need to +# deterministically trigger shield_timeout in < 60s. +import os as _os + +DEFAULT_CANCEL_SHIELD_TIMEOUT_S: float = float( + _os.environ.get("LITELLM_CANCEL_SHIELD_TIMEOUT_S", "60.0") +) + + +def mark_logging_obj_cancelled( + logging_obj: Any, + *, + phase: CancelPhase, + indicator: str = "client_disconnect", + bytes_delivered: Optional[int] = None, +) -> None: + """ + Tag a Logging object with cancellation metadata so the downstream + cost calculator and the SpendLogs row populator can apply the + cancel-billing path (and so the derived delivery_status / + billing_status taxonomy can be computed from the markers). + + Idempotent — calling twice on the same object is safe (a defensive + re-call in nested catch blocks will not corrupt the first marker). + + Parameters + ---------- + logging_obj : litellm.Logging + The per-request Logging instance. May be ``None`` (e.g. cancel + fired before pre_call ran), in which case this is a no-op. + phase : CancelPhase + Where in the request lifecycle the cancel was detected. Drives + the billing strategy in cancel_billing.py. + indicator : str + Source of the cancel signal. Almost always ``"client_disconnect"``; + reserved value ``"upstream_disconnect"`` is for the rare case + where the upstream provider RST's mid-stream. + bytes_delivered : Optional[int] + Total bytes flushed to the client socket before the disconnect, + if known. Helpful for client-vs-upstream gap analysis (zero + bytes delivered + non-zero upstream usage = client paid for + compute it never received). + """ + if logging_obj is None: + return + + # Logging object exposes a mutable dict for per-request scratch space. + details = getattr(logging_obj, "model_call_details", None) + if details is None: + # Pathological case — the Logging instance has no scratch dict + # (would mean function_setup never ran). Nothing safe to do here. + verbose_logger.debug( + "mark_logging_obj_cancelled: logging_obj.model_call_details is None, skipping" + ) + return + + # Don't overwrite a prior, earlier marker. The phase progression is + # before_upstream → during_upstream → streaming_partial → during_parsing, + # and only the FIRST detection point is meaningful. + if details.get("cancellation_indicator") is not None: + return + + details["cancellation_indicator"] = indicator + details["cancel_phase"] = phase + details["cancelled_at"] = time.time() + if bytes_delivered is not None: + details["bytes_delivered_to_client"] = bytes_delivered + + +def is_logging_obj_cancelled(logging_obj: Any) -> bool: + """Whether the cancel marker has been set on this Logging object.""" + if logging_obj is None: + return False + details = getattr(logging_obj, "model_call_details", None) + if details is None: + return False + return details.get("cancellation_indicator") is not None + + +def _get_accumulated_chunks(stream_wrapper: Any) -> List: + """ + Extract the accumulated chunks list from a streaming response wrapper. + + LiteLLM's CustomStreamWrapper buffers every yielded chunk on + ``self.chunks`` (used by stream_chunk_builder at end-of-stream). + For non-CustomStreamWrapper iterables (test fakes, custom user + wrappers) we degrade gracefully to an empty list — downstream + billing will fall back to the prompt-only path. + """ + chunks = getattr(stream_wrapper, "chunks", None) + if isinstance(chunks, list): + return chunks + return [] + + +async def finalize_streaming_cancel( + *, + stream_wrapper: Any, + logging_obj: Any, + user_api_key_dict: Any, + request_data: dict, + bytes_delivered: Optional[int] = None, +) -> None: + """ + Dispatch a streaming-cancel through the normal success callback + chain. The SpendLogs row stays ``status="success"``; the + cancellation taxonomy is conveyed via the markers set on the + Logging instance (which get bridged into metadata by + ``enrich_request_metadata_with_cancel_markers`` and then turned into + delivery_status / billing_status by the derivation helper in + spend_tracking_utils). + + Called from the catch block of the proxy's streaming generator (and + defensively from CustomStreamWrapper.__anext__). The reassembled + partial response is built via the same stream_chunk_builder path + used by end-of-stream success, so all the cost-calculation / + Langfuse / Prometheus hooks see a normal-looking response object — + they just see the ``cancellation_indicator`` marker on the Logging + instance and apply the partial-billing logic. + + Hardening notes: + + * The whole finalize is wrapped in ``anyio.CancelScope(shield=True)``. + Without this, a follow-on cancel signal (the asyncio runtime can + inject CancelledError multiple times into a task being torn down) + would interrupt the SpendLogs write and leave us back at square + one with no record. + + * Inner ``except Exception`` (NOT BaseException) catches *finalize-path + bugs* without masking the cancel signal. Anything blowing up here + gets logged at ERROR but does not propagate, because the caller + will re-raise the original CancelledError after we return. + + * The caller is responsible for re-raising the CancelledError after + this returns. Do NOT raise from here — would break asyncio's + cancel-propagation contract. + """ + import litellm + + with anyio.CancelScope(shield=True): + try: + mark_logging_obj_cancelled( + logging_obj, + phase="streaming_partial", + indicator="client_disconnect", + bytes_delivered=bytes_delivered, + ) + + # mark_logging_obj_cancelled is idempotent on the marker — + # if streaming_handler's defensive catch ran first with + # bytes_delivered=0 (no chunks visible at the + # CustomStreamWrapper layer), the proxy-level chunk count + # we got here (from async_streaming_data_generator) is + # more accurate. Promote it directly when it's a strictly + # better signal. + if bytes_delivered is not None and bytes_delivered > 0: + _details = getattr(logging_obj, "model_call_details", None) + if isinstance(_details, dict): + _existing = _details.get("bytes_delivered_to_client") or 0 + if bytes_delivered > _existing: + _details["bytes_delivered_to_client"] = bytes_delivered + + # Bridge the cancel markers we just set on the Logging + # object into request_data.litellm_params.metadata so the + # SpendLogs row reflects the cancellation. The standard + # spend pipeline reads metadata from litellm_params, not + # from model_call_details. + from litellm.litellm_core_utils.cancel_billing import ( + enrich_request_metadata_with_cancel_markers, + ) + + enrich_request_metadata_with_cancel_markers( + request_data=request_data, + logging_obj=logging_obj, + ) + + chunks = _get_accumulated_chunks(stream_wrapper) + if not chunks: + # Nothing to bill — no chunks made it before cancel. + # Fall back to the failure path so SpendLogs still gets + # a row (with prompt cost only, via existing failure + # hook logic). + await _fallback_to_failure_hook( + user_api_key_dict=user_api_key_dict, + request_data=request_data, + ) + return + + # Update bytes_delivered_to_client now that we have evidence + # of how many chunks were flushed. mark_logging_obj_cancelled + # is idempotent on the indicator, so a direct mutation + + # re-bridge through enrich() is the way to refresh the + # downstream-visible value. We don't have the exact byte + # count (chunks are pre-serialisation), so use the chunk + # count as a proxy >0 signal — the derivation helper only + # cares whether bytes_delivered > 0. + details = getattr(logging_obj, "model_call_details", None) + if isinstance(details, dict) and ( + details.get("bytes_delivered_to_client") in (None, 0) + ): + details["bytes_delivered_to_client"] = len(chunks) + enrich_request_metadata_with_cancel_markers( + request_data=request_data, + logging_obj=logging_obj, + ) + + # Reassemble whatever we received. Stream_chunk_builder is + # the same path the normal end-of-stream success handler + # uses, so the cost calculator gets a familiar shape. + partial_response = None + try: + partial_response = litellm.stream_chunk_builder( + chunks=chunks, + messages=getattr(stream_wrapper, "messages", None), + logging_obj=logging_obj, + ) + except Exception as build_exc: + verbose_logger.error( + "finalize_streaming_cancel: stream_chunk_builder failed; " + "falling back to failure hook: %s", + build_exc, + exc_info=True, + ) + + if partial_response is None: + await _fallback_to_failure_hook( + user_api_key_dict=user_api_key_dict, + request_data=request_data, + ) + return + + # Dispatch through normal success path. The cost calculator + # picks up the cancellation_indicator marker (set above) and + # applies the partial-billing logic from cancel_billing.py + # (PR #3). Until that lands, the normal cost calc runs on + # the partial response — which is already an improvement + # over the prior "drop everything" behavior because + # PR #1's cursor=1 fix makes the partial usage estimate + # reasonable for Anthropic. + if logging_obj is not None: + try: + await logging_obj.async_success_handler( + result=partial_response, + start_time=getattr(logging_obj, "start_time", None), + end_time=datetime.now(), + cache_hit=False, + ) + except Exception as success_exc: + verbose_logger.error( + "finalize_streaming_cancel: async_success_handler " + "raised: %s", + success_exc, + exc_info=True, + ) + except Exception as outer_exc: + # Last-resort: anything we didn't anticipate. Log but do NOT + # let it propagate — the caller needs to re-raise the + # original CancelledError. + verbose_logger.error( + "finalize_streaming_cancel: unexpected error: %s", + outer_exc, + exc_info=True, + ) + + +async def _fallback_to_failure_hook( + *, + user_api_key_dict: Any, + request_data: dict, +) -> None: + """ + Call the existing post_call_failure_hook for cancellation cases where + we have no chunks to reassemble. + + This still produces a SpendLogs row (with cost=0 under the current + failure-path logic, which PR #3 will improve to charge for the + prompt-only baseline). Better than silent drop. + """ + try: + from litellm.proxy.proxy_server import proxy_logging_obj + + await proxy_logging_obj.post_call_failure_hook( + user_api_key_dict=user_api_key_dict, + original_exception=asyncio.CancelledError( + "Client disconnected before any chunks were received" + ), + request_data=request_data, + ) + except Exception as fb_exc: + verbose_logger.error( + "_fallback_to_failure_hook failed: %s", fb_exc, exc_info=True + ) + + +async def finalize_non_stream_cancel( + *, + upstream_task: Optional[asyncio.Task], + logging_obj: Any, + user_api_key_dict: Any, + request_data: dict, + shield_timeout_s: float = DEFAULT_CANCEL_SHIELD_TIMEOUT_S, +) -> None: + """ + Non-stream cancellation finalizer with shield-and-wait semantics. + + For non-stream requests the upstream provider typically does NOT + detect or honor client cancellation — once they've received our + request body they generate to completion and bill us. To keep our + SpendLogs honest we must shield the upstream call past the cancel, + wait for the real usage to come back, and record it. + + Strategy + -------- + 1. Tag logging_obj with phase=during_upstream. + 2. Shield the upstream_task from the cancel signal. + 3. ``asyncio.wait_for`` it with ``shield_timeout_s`` — if upstream + returns in time we get real usage; otherwise we fall back to + prompt-only billing and tag ``usage_source=shield_timeout``. + 4. Always dispatch through async_success_handler so failure-rate + metrics stay clean. + + Caller must still re-raise CancelledError after this returns. PR #3 + wires this into the non-stream code path; included here as part of + PR #2 so the API surface is settled. + """ + with anyio.CancelScope(shield=True): + try: + mark_logging_obj_cancelled( + logging_obj, + phase="during_upstream", + indicator="client_disconnect", + bytes_delivered=0, # non-stream → nothing flushed yet + ) + + # Bridge cancel markers into request_data so SpendLogs row + # picks them up. See finalize_streaming_cancel for the + # equivalent call + rationale. + from litellm.litellm_core_utils.cancel_billing import ( + enrich_request_metadata_with_cancel_markers, + ) + + enrich_request_metadata_with_cancel_markers( + request_data=request_data, + logging_obj=logging_obj, + ) + + if upstream_task is None: + # Cancel fired before we even kicked off the upstream + # call — phase should really be before_upstream. Caller + # should pre-set that via mark_logging_obj_cancelled. + await _fallback_to_failure_hook( + user_api_key_dict=user_api_key_dict, + request_data=request_data, + ) + return + + details = getattr(logging_obj, "model_call_details", {}) or {} + try: + response = await asyncio.wait_for( + upstream_task, timeout=shield_timeout_s + ) + details["upstream_completed"] = True + details["usage_source"] = "upstream_completed_after_cancel" + # Re-bridge so the freshly-set upstream_completed / + # usage_source markers land in request_data too. + enrich_request_metadata_with_cancel_markers( + request_data=request_data, + logging_obj=logging_obj, + ) + if logging_obj is not None: + try: + await logging_obj.async_success_handler( + result=response, + start_time=getattr(logging_obj, "start_time", None), + end_time=datetime.now(), + cache_hit=False, + ) + except Exception as success_exc: + verbose_logger.error( + "finalize_non_stream_cancel: async_success_handler " + "raised: %s", + success_exc, + exc_info=True, + ) + except asyncio.TimeoutError: + # Upstream took longer than the shield budget. Cancel it + # for real this time so we don't leak the task, and + # fall back to prompt-only billing. + details["upstream_completed"] = False + details["usage_source"] = "shield_timeout" + upstream_task.cancel() + await _fallback_to_failure_hook( + user_api_key_dict=user_api_key_dict, + request_data=request_data, + ) + except Exception as upstream_exc: + # Upstream errored out during the shield window. Record + # whatever phase info we have and bail to failure path. + details["upstream_completed"] = False + details["usage_source"] = "no_completion" + verbose_logger.debug( + "finalize_non_stream_cancel: upstream raised during " "shield: %s", + upstream_exc, + ) + await _fallback_to_failure_hook( + user_api_key_dict=user_api_key_dict, + request_data=request_data, + ) + except Exception as outer_exc: + verbose_logger.error( + "finalize_non_stream_cancel: unexpected error: %s", + outer_exc, + exc_info=True, + ) diff --git a/litellm/litellm_core_utils/streaming_chunk_builder_utils.py b/litellm/litellm_core_utils/streaming_chunk_builder_utils.py index fe7c62c38425..292227b6bbc8 100644 --- a/litellm/litellm_core_utils/streaming_chunk_builder_utils.py +++ b/litellm/litellm_core_utils/streaming_chunk_builder_utils.py @@ -536,6 +536,16 @@ def _calculate_usage_per_chunk( # # Update usage information if needed prompt_tokens = 0 completion_tokens = 0 + # Anthropic's `message_start` SSE event carries usage.output_tokens=1 as a + # cursor/placeholder; the real value only arrives in `message_delta`. + # If a stream is cancelled before `message_delta` lands, the last-wins + # accumulator below leaves completion_tokens stuck at 1 — which then + # bypasses the `completion_tokens or token_counter(...)` fallback in + # calculate_usage() because 1 is truthy. Track whether any update came + # from a chunk that's definitively NOT a cursor (output_tokens > 1) so + # we can signal "no real usage seen" to the caller and let the text-based + # fallback kick in. + saw_non_cursor_completion = False ## anthropic prompt caching information ## cache_creation_input_tokens: Optional[int] = None cache_read_input_tokens: Optional[int] = None @@ -566,6 +576,8 @@ def _calculate_usage_per_chunk( and usage_chunk_dict["completion_tokens"] > 0 ): completion_tokens = usage_chunk_dict["completion_tokens"] + if usage_chunk_dict["completion_tokens"] > 1: + saw_non_cursor_completion = True if usage_chunk_dict["cache_creation_input_tokens"] is not None and ( usage_chunk_dict["cache_creation_input_tokens"] > 0 or cache_creation_input_tokens is None @@ -605,6 +617,15 @@ def _calculate_usage_per_chunk( prompt_tokens_details = usage_chunk_dict["prompt_tokens_details"] + # See `saw_non_cursor_completion` comment above. If the only completion + # update we ever saw was the Anthropic message_start cursor (=1), reset + # to 0 here so calculate_usage()'s `or token_counter(text=...)` fallback + # estimates from the actually-received completion text instead of trusting + # the placeholder. Legitimate 1-token completions are unaffected — the + # text-based fallback on a 1-token completion_output also yields ~1. + if completion_tokens == 1 and not saw_non_cursor_completion: + completion_tokens = 0 + return UsagePerChunk( prompt_tokens=prompt_tokens, completion_tokens=completion_tokens, diff --git a/litellm/litellm_core_utils/streaming_handler.py b/litellm/litellm_core_utils/streaming_handler.py index fa7faf3035d3..7a2920a27923 100644 --- a/litellm/litellm_core_utils/streaming_handler.py +++ b/litellm/litellm_core_utils/streaming_handler.py @@ -2245,6 +2245,38 @@ async def __anext__(self) -> "ModelResponseStream": # noqa: PLR0915 self.logging_obj.async_failure_handler(e, traceback_exception) ) self._handle_stream_fallback_error(e) + except asyncio.CancelledError: + # Defense in depth: if a client cancellation reaches the + # stream wrapper directly (rather than being caught by the + # proxy generator in proxy_server.async_data_generator), + # mark the Logging object so the cost calculator knows to + # apply the cancel-billing path (prompt + chunk reassembly). + # We don't dispatch the success_handler here — that's the + # proxy generator's job; otherwise we'd risk firing it + # twice. Just leave the marker so whoever finalizes gets + # the right billing path, then re-raise to preserve + # cancellation semantics. + from litellm.litellm_core_utils.cancel_finalize import ( + mark_logging_obj_cancelled, + ) + + # Pass the accumulated chunk count as the bytes_delivered + # signal. This is the path that catches cancels on endpoints + # whose response object is a bare async iterator (e.g. + # /v1/messages, /v1beta/...:streamGenerateContent) — for + # those, finalize_streaming_cancel's _get_accumulated_chunks + # can't find the chunks list (wrapped elsewhere), so the + # downstream derivation defaults to delivery=none. Set + # bytes_delivered here so derivation correctly classifies + # the row as partial-delivery when chunks did flow. + chunks = getattr(self, "chunks", None) + chunk_count = len(chunks) if isinstance(chunks, list) else 0 + mark_logging_obj_cancelled( + self.logging_obj, + phase="streaming_partial", + bytes_delivered=chunk_count, + ) + raise except Exception as e: traceback_exception = traceback.format_exc() if self.logging_obj is not None: diff --git a/litellm/proxy/_types.py b/litellm/proxy/_types.py index 46a032eb59a1..1dd0b89f4b79 100644 --- a/litellm/proxy/_types.py +++ b/litellm/proxy/_types.py @@ -3534,6 +3534,27 @@ class SpendLogsMetadata(TypedDict): cost_breakdown: Optional[ CostBreakdown ] # Detailed cost breakdown (input_cost, output_cost, margin, discount, etc.) + # === Client cancellation tracking === + # All Optional; populated only on requests where the client disconnected + # before the response completed. See litellm/types/utils.py CancelPhase + # / CancelUsageSource for value semantics. DB column `metadata` is JSON + # so no schema migration is needed to add these fields. + # + # `delivery_status` and `billing_status` are orthogonal derived dimensions + # populated by spend_tracking_utils._derive_delivery_billing_status from + # the 5 cancel markers + raw status. They are materialized here so SQL + # dashboards can filter directly (WHERE metadata::jsonb->>'delivery_status' + # = 'partial'). Typed as Optional[str] (not the Literal) because the + # TypedDict gets constructed from arbitrary input dicts via + # `metadata.get(key)` — tightening the Literal would force runtime cast + # noise at every write site. + cancellation_indicator: Optional[str] + cancel_phase: Optional[str] + bytes_delivered_to_client: Optional[int] + upstream_completed: Optional[bool] + usage_source: Optional[str] + delivery_status: Optional[str] + billing_status: Optional[str] class SpendLogsPayload(TypedDict): diff --git a/litellm/proxy/common_request_processing.py b/litellm/proxy/common_request_processing.py index 5366a4c51e00..8fa7eeb21d69 100644 --- a/litellm/proxy/common_request_processing.py +++ b/litellm/proxy/common_request_processing.py @@ -1298,7 +1298,147 @@ async def base_process_llm_request( # noqa: PLR0915 *tasks ) # run the moderation check in parallel to the actual llm api call - responses = await llm_responses + # Detect client disconnect during the (potentially long) LLM + # call. NON-STREAM ONLY: streaming requests have their own + # CancelledError path in async_streaming_data_generator that + # gives us partial response usage. Running both layers would + # double-handle the cancel and the shield_timeout branch can + # fire BEFORE Anthropic's TTFT completes, eating into the + # streaming generator's correct path. + # + # For non-stream, without this watcher cancellations are + # silent on the proxy side — the upstream provider continues + # processing, charges us for the compute, and we bill the + # user as a successful request. + is_streaming_request = bool(self.data.get("stream", False)) + _shield_timed_out = False + disconnect_flag = {"detected": False, "detected_at": 0.0} + + if is_streaming_request: + # Streaming path — let the streaming generator's + # CancelledError catch handle cancel detection. + responses = await llm_responses + else: + import os as _os + + _shield_timeout_s = float( + _os.environ.get("LITELLM_CANCEL_SHIELD_TIMEOUT_S", "60.0") + ) + + async def _disconnect_watcher(): + while not llm_responses.done(): + try: + await asyncio.sleep(1.0) + if await request.is_disconnected(): + disconnect_flag["detected"] = True + disconnect_flag["detected_at"] = time.time() + return + except asyncio.CancelledError: + return + except Exception: + # is_disconnected() can occasionally raise + # (e.g. broken transport). Stop polling; the + # LLM call will run to completion on its own. + return + + _disconnect_watcher_task = asyncio.create_task(_disconnect_watcher()) + + try: + # If the client disconnects, we keep awaiting up to + # _shield_timeout_s past the disconnect for the LLM + # call to finish. Polling loop because asyncio.shield + # + wait_for would also cancel the watcher. + _poll_interval = 0.2 + while not llm_responses.done(): + try: + responses = await asyncio.wait_for( + asyncio.shield(llm_responses), + timeout=_poll_interval, + ) + break + except asyncio.TimeoutError: + if disconnect_flag["detected"]: + elapsed = time.time() - disconnect_flag["detected_at"] + if elapsed >= _shield_timeout_s: + _shield_timed_out = True + break + else: + responses = await llm_responses + finally: + _disconnect_watcher_task.cancel() + + # If we hit the shield timeout, build a synthetic empty + # response so the rest of the handler doesn't crash. The + # actual SpendLogs row will get the prompt-only billing + # via the cancel markers below. + if _shield_timed_out: + from litellm.types.utils import ModelResponse as _MR + + _empty = _MR() + _empty.choices = [] + responses = [None, _empty] + + # If the client gave up during the upstream wait, tag the + # Logging instance so the SpendLogs row carries the cancel + # markers (real upstream usage + cancellation_indicator + + # derived delivery_status / billing_status). The row stays + # status="success" — cancel is not a system failure. + if disconnect_flag["detected"]: + from litellm.litellm_core_utils.cancel_billing import ( + enrich_request_metadata_with_cancel_markers, + ) + from litellm.litellm_core_utils.cancel_finalize import ( + mark_logging_obj_cancelled, + ) + + _disconnect_logging_obj = self.data.get("litellm_logging_obj") + mark_logging_obj_cancelled( + _disconnect_logging_obj, + phase="during_upstream", + indicator="client_disconnect", + bytes_delivered=0, + ) + # Also tag the per-response logging_obj if it differs (the + # Router may attach its own Logging instance via + # `response.logging_obj` that's distinct from the one the + # proxy stashed on request_data). + _per_resp_logging_obj = ( + getattr(responses[1], "logging_obj", None) + if responses and len(responses) > 1 + else None + ) + if ( + _per_resp_logging_obj is not None + and _per_resp_logging_obj is not _disconnect_logging_obj + ): + mark_logging_obj_cancelled( + _per_resp_logging_obj, + phase="during_upstream", + indicator="client_disconnect", + bytes_delivered=0, + ) + details_per_resp = ( + getattr(_per_resp_logging_obj, "model_call_details", {}) or {} + ) + if _shield_timed_out: + details_per_resp["upstream_completed"] = False + details_per_resp["usage_source"] = "shield_timeout" + else: + details_per_resp["upstream_completed"] = True + details_per_resp["usage_source"] = "upstream_completed_after_cancel" + # upstream_completed reflects whether we caught the real + # response. False on shield_timeout (we gave up waiting). + details = getattr(_disconnect_logging_obj, "model_call_details", {}) or {} + if _shield_timed_out: + details["upstream_completed"] = False + details["usage_source"] = "shield_timeout" + else: + details["upstream_completed"] = True + details["usage_source"] = "upstream_completed_after_cancel" + enrich_request_metadata_with_cancel_markers( + request_data=self.data, + logging_obj=_disconnect_logging_obj, + ) response = responses[1] @@ -2094,6 +2234,13 @@ async def async_streaming_data_generator( and not cost_injection_enabled ) debug_enabled = verbose_proxy_logger.isEnabledFor(logging.DEBUG) + # Track chunks actually yielded to the client. Passed to + # finalize_streaming_cancel on cancel so the derived + # delivery_status reflects whether ANY bytes reached the + # client — important for /v1/messages and other endpoints + # whose response object is a bare async iterator (no + # `.chunks` attribute for _get_accumulated_chunks). + chunks_yielded = 0 try: str_so_far = "" async for ( @@ -2139,6 +2286,7 @@ async def async_streaming_data_generator( if fast_path: yield serialize_chunk(chunk) + chunks_yielded += 1 continue chunk = await proxy_logging_obj.async_post_call_streaming_hook( @@ -2171,6 +2319,45 @@ async def async_streaming_data_generator( # fast_path short-circuit (single source of truth — see comment # there for the SSE byte-rewrite contract). yield serialize_chunk(chunk) + chunks_yielded += 1 + except asyncio.CancelledError: + # Client cancelled this Anthropic /messages or Google + # /generateContent stream. asyncio.CancelledError is a + # BaseException in Py3.8+ — the `except Exception` below + # would let it slip through silently, leaving us with no + # SpendLogs row and an orphaned Langfuse trace while the + # upstream provider continues billing us for compute. + # + # Route through cancel_finalize so the partial response gets + # billed via the cancel-billing path. Re-raise to preserve + # asyncio's cancellation contract. + from litellm.litellm_core_utils.cancel_finalize import ( + finalize_streaming_cancel, + ) + + # Prefer response.logging_obj (set by CustomStreamWrapper on + # the /v1/chat/completions path). For /v1/messages and + # /v1beta/.../streamGenerateContent the response object is + # often a bare async iterator without a logging_obj + # attribute — fall back to the Logging instance the proxy + # stashed on request_data during pre-call setup. Without + # this fallback the cancel markers never get set on the + # Logging object and the SpendLogs row looks like a plain + # success with no cancellation_indicator (so the UI can't + # render the Cancel badge and dashboards can't filter). + logging_obj = ( + getattr(response, "logging_obj", None) if response is not None else None + ) + if logging_obj is None: + logging_obj = request_data.get("litellm_logging_obj") + await finalize_streaming_cancel( + stream_wrapper=response, + logging_obj=logging_obj, + user_api_key_dict=user_api_key_dict, + request_data=request_data, + bytes_delivered=chunks_yielded, + ) + raise except Exception as e: log_proxy_exception(verbose_proxy_logger, "async_data_generator[stream]", e) transformed_exception = await proxy_logging_obj.post_call_failure_hook( diff --git a/litellm/proxy/hooks/proxy_track_cost_callback.py b/litellm/proxy/hooks/proxy_track_cost_callback.py index 3688f25ac44b..019a7929551f 100644 --- a/litellm/proxy/hooks/proxy_track_cost_callback.py +++ b/litellm/proxy/hooks/proxy_track_cost_callback.py @@ -77,13 +77,25 @@ async def async_post_call_failure_hook( from litellm.proxy.proxy_server import proxy_logging_obj + # Detect whether this failure path is actually serving a client + # cancellation (CancelledError propagated through + # cancel_finalize._fallback_to_failure_hook). When it is, the + # SpendLogs row must stay ``status="success"`` — cancellation is + # not a system failure; the proxy did its job, the client gave up. + # The cancel taxonomy (was-it-cancelled / did-we-deliver / + # did-we-bill) lives in metadata markers + the derived + # delivery_status / billing_status fields. Cancel markers already + # set on request_data.litellm_params.metadata by cancel_finalize + # must NOT be clobbered. + _is_cancel = isinstance(original_exception, asyncio.CancelledError) + _metadata = dict( LiteLLMProxyRequestSetup.get_sanitized_user_information_from_key( user_api_key_dict=user_api_key_dict ) ) _metadata["user_api_key"] = user_api_key_dict.api_key - _metadata["status"] = "failure" + _metadata["status"] = "success" if _is_cancel else "failure" _error_information = StandardLoggingPayloadSetup.get_error_information( original_exception=original_exception, traceback_str=traceback_str, @@ -121,6 +133,24 @@ async def async_post_call_failure_hook( if existing_litellm_metadata.get("tags"): existing_metadata["tags"] = existing_litellm_metadata.get("tags") + # Preserve cancellation markers written by cancel_finalize before + # the failure hook ran. Without this, the cancel taxonomy gets + # stripped on every cancelled request that took the + # fallback-to-failure-hook path (zero-chunk cancels in particular) + # — and the downstream _derive_delivery_billing_status helper + # would not be able to tell a real failure from a cancellation. + for _cancel_field in ( + "cancellation_indicator", + "cancel_phase", + "bytes_delivered_to_client", + "upstream_completed", + "usage_source", + ): + if _cancel_field in existing_litellm_metadata: + existing_metadata[_cancel_field] = existing_litellm_metadata[ + _cancel_field + ] + request_data["litellm_params"]["proxy_server_request"] = ( request_data.get("proxy_server_request") or existing_litellm_params.get("proxy_server_request") @@ -162,9 +192,36 @@ async def async_post_call_failure_hook( if obj_start is not None: actual_start_time = obj_start + # Bridge cancel markers — runs late as a safety net in case some + # code path didn't pre-bridge (we already preserve markers from + # litellm_params.metadata above; this catches the case where the + # markers are on logging_obj but never landed in litellm_params). + from litellm.litellm_core_utils.cancel_billing import ( + compute_prompt_only_cost, + enrich_request_metadata_with_cancel_markers, + ) + + enrich_request_metadata_with_cancel_markers( + request_data=request_data, + logging_obj=_litellm_logging_obj, + ) + + # For cancellations that landed in the failure-hook (i.e. zero + # chunks were received before client disconnected), bill the + # prompt-only baseline instead of the hardcoded 0.0. Upstream + # still received the prompt and started processing — see + # cancel_billing.py docstring for the full rationale. + billed_cost = 0.0 + if isinstance(original_exception, asyncio.CancelledError): + billed_cost = compute_prompt_only_cost( + messages=request_data.get("messages"), + model=request_data.get("model"), + custom_llm_provider=request_data.get("custom_llm_provider"), + ) + await proxy_logging_obj.db_spend_update_writer.update_database( token=user_api_key_dict.api_key, - response_cost=0.0, + response_cost=billed_cost, user_id=user_api_key_dict.user_id, end_user_id=user_api_key_dict.end_user_id, team_id=user_api_key_dict.team_id, diff --git a/litellm/proxy/proxy_server.py b/litellm/proxy/proxy_server.py index ede3741074b2..b921fce18ea1 100644 --- a/litellm/proxy/proxy_server.py +++ b/litellm/proxy/proxy_server.py @@ -7029,6 +7029,41 @@ async def async_data_generator( # noqa: PLR0915 yield error_message done_message = "[DONE]" yield f"data: {done_message}\n\n" + except asyncio.CancelledError: + # Client cancelled (most common: closed the SSE connection, SDK + # timeout, browser tab closed). asyncio.CancelledError is a + # BaseException in Python 3.8+, so the generic `except Exception` + # below does NOT catch it; without this branch the stream + # disappears into a logging black hole — no SpendLogs row, no + # Langfuse trace closure, no Prometheus counter, but the + # upstream provider already billed us for whatever was + # generated. See litellm_core_utils/cancel_finalize.py for the + # full motivation. + # + # The finalize helper is shielded internally and must not raise. + # We re-raise CancelledError after it returns to preserve + # asyncio's cancellation contract — swallowing the signal + # would leak the request task and confuse anyio task groups. + from litellm.litellm_core_utils.cancel_finalize import ( + finalize_streaming_cancel, + ) + + # Prefer response.logging_obj; fall back to the proxy's + # request-scoped Logging instance. See the equivalent block in + # common_request_processing.async_streaming_data_generator for + # the full rationale. + logging_obj = ( + getattr(response, "logging_obj", None) if response is not None else None + ) + if logging_obj is None: + logging_obj = request_data.get("litellm_logging_obj") + await finalize_streaming_cancel( + stream_wrapper=response, + logging_obj=logging_obj, + user_api_key_dict=user_api_key_dict, + request_data=request_data, + ) + raise except Exception as e: log_proxy_exception(verbose_proxy_logger, "/v1/chat/completions[stream]", e) await proxy_logging_obj.post_call_failure_hook( diff --git a/litellm/proxy/spend_tracking/spend_management_endpoints.py b/litellm/proxy/spend_tracking/spend_management_endpoints.py index 36beb5e9aba8..c06f6dbc3940 100644 --- a/litellm/proxy/spend_tracking/spend_management_endpoints.py +++ b/litellm/proxy/spend_tracking/spend_management_endpoints.py @@ -2013,7 +2013,10 @@ def parse_date(date_str: str) -> datetime: p += 2 sql_conditions.append(or_clause) - # Status filter + # Status filter — binary success | failure (cancellations are a + # row-level annotation surfaced via the amber badge in the UI, + # not a top-level filter category). Kept in sync with the + # Prisma _build_status_filter_condition above. if status_filter is not None: if status_filter == "success": sql_conditions.append("(status = 'success' OR status IS NULL)") @@ -3480,21 +3483,23 @@ async def _build_ui_spend_logs_response( def _build_status_filter_condition(status_filter: Optional[str]) -> Dict[str, Any]: """ - Helper function to build the status filter condition for database queries. + Build the SpendLogs status filter condition. Binary: success | failure. - Args: - status_filter (Optional[str]): The status to filter by. Can be "success" or "failure". + Cancellations are NOT a top-level filter category — they carry + status='success' with a metadata marker, and surface in the UI as + an amber badge inside the Success bucket. Users who actually need + to query cancelled rows can filter on + ``metadata::jsonb->>'cancellation_indicator'`` directly in SQL. - Returns: - Dict[str, Any]: A dictionary containing the status filter condition. + Returns an empty dict when status_filter is None or unrecognised. """ if status_filter is None: return {} - if status_filter == "success": return {"OR": [{"status": {"equals": "success"}}, {"status": None}]} - else: - return {"status": {"equals": status_filter}} + if status_filter == "failure": + return {"status": {"equals": "failure"}} + return {} def _is_admin_view_safe(user_api_key_dict: UserAPIKeyAuth) -> bool: diff --git a/litellm/proxy/spend_tracking/spend_tracking_utils.py b/litellm/proxy/spend_tracking/spend_tracking_utils.py index e2881faca0d1..fa551d11aad7 100644 --- a/litellm/proxy/spend_tracking/spend_tracking_utils.py +++ b/litellm/proxy/spend_tracking/spend_tracking_utils.py @@ -109,6 +109,19 @@ def _get_spend_logs_metadata( attempted_retries=None, max_retries=None, cost_breakdown=None, + # Cancellation fields default to None; populated only by the + # cancel-billing path. delivery_status / billing_status are + # derived from the markers + raw status by + # _derive_delivery_billing_status; we set both to None on + # the early-failure path (this branch is only hit when + # metadata itself is missing). + cancellation_indicator=None, + cancel_phase=None, + bytes_delivered_to_client=None, + upstream_completed=None, + usage_source=None, + delivery_status=None, + billing_status=None, ) verbose_proxy_logger.debug( "getting payload for SpendLogs, available keys in metadata: " @@ -134,9 +147,80 @@ def _get_spend_logs_metadata( clean_metadata["litellm_overhead_time_ms"] = litellm_overhead_time_ms clean_metadata["cost_breakdown"] = cost_breakdown + # Materialize the orthogonal taxonomy: derive delivery_status and + # billing_status from the five cancel markers + raw status, then write + # them back so SQL dashboards can filter directly + # (`WHERE metadata::jsonb->>'delivery_status' = 'partial'`). Single + # source of truth — write sites must not set these directly. + _delivery, _billing = _derive_delivery_billing_status(metadata) + clean_metadata["delivery_status"] = _delivery + clean_metadata["billing_status"] = _billing + return clean_metadata +def _derive_delivery_billing_status( + metadata: dict, +) -> tuple[Optional[str], Optional[str]]: + """ + Derive (delivery_status, billing_status) from the five cancel markers + + raw status. Single source of truth — never write delivery_status / + billing_status from anywhere else. + + Semantic mapping (see e2e/cases/data/26-33 for end-to-end coverage): + + | scenario | status | delivery | billing | + |---------------------------------------------|----------|----------|---------| + | normal full success | success | full | full | + | streaming cancel WITH chunks | success | partial | partial | + | non-stream shield success | success | none | full | + | non-stream shield_timeout | success | none | partial | + | zero-chunk cancel before dispatch | success | none | none | + | cancel + upstream errored during shield | success | none | partial | + | real failure (5xx / auth / hard timeout) | failure | none | none | + + Inputs (all read from `metadata.get(...)`): + - status: "success" | "failure" + - cancellation_indicator: "client_disconnect" | "upstream_disconnect" | None + - cancel_phase: CancelPhase value + - usage_source: CancelUsageSource value + - bytes_delivered_to_client: int | None + """ + raw_status = metadata.get("status") + if raw_status == "failure": + return "none", "none" + + cancel_ind = metadata.get("cancellation_indicator") + if not cancel_ind: + # No cancel marker → fall through as a normal success. + return "full", "full" + + src = metadata.get("usage_source") + phase = metadata.get("cancel_phase") + bytes_delivered = metadata.get("bytes_delivered_to_client") or 0 + + # Channel A: streaming cancel with chunks actually delivered. + # bytes_delivered is populated in finalize_streaming_cancel after + # _get_accumulated_chunks confirms chunks > 0, so this branch + # only matches when something genuinely reached the client. + if phase == "streaming_partial" and bytes_delivered > 0: + return "partial", "partial" + + # Channel B-success: non-stream shield wait succeeded — upstream + # came back with real usage AFTER the client disconnected. + if src == "upstream_completed_after_cancel": + return "none", "full" + + # Channel B-timeout: shield budget exceeded; we billed the prompt + # baseline. + if src == "shield_timeout": + return "none", "partial" + + # Remaining cases — zero-chunk cancel, upstream errored during + # shield, no_completion — get the default "none/none" tag. + return "none", "none" + + def generate_hash_from_response(response_obj: Any) -> str: """ Generate a stable hash from a response object. @@ -1094,11 +1178,25 @@ def _get_status_for_spend_log( metadata: dict, ) -> Literal["success", "failure"]: """ - Get the status for the spend log. - - It's only a failure if metadata.get("status") is "failure" + Get the binary top-level ``status`` column value for a SpendLogs row. + + - "failure": metadata.status == "failure" (real provider/auth error, + hard timeout, anything that errored before billable work + could complete) + - "success": everything else, INCLUDING client cancellations. + + The cancellation taxonomy (was-it-cancelled? did-we-deliver? did-we-bill?) + lives entirely in the JSON metadata blob, not in this column: + - metadata.cancellation_indicator — presence ⇒ cancelled + - metadata.cancel_phase / usage_source — forensics + - metadata.delivery_status — "full" | "partial" | "none" + - metadata.billing_status — "full" | "partial" | "none" + + Cancellations land on status="success" because the proxy itself did + not error — the client gave up. Dashboards distinguishing cancels + must filter on ``metadata::jsonb->>'cancellation_indicator' IS NOT NULL`` + (or the derived delivery/billing_status fields), NOT on this column. """ - _status: Optional[str] = metadata.get("status", None) - if _status == "failure": + if metadata.get("status") == "failure": return "failure" return "success" diff --git a/litellm/router.py b/litellm/router.py index 1724e1f14c60..94f15e311c5a 100644 --- a/litellm/router.py +++ b/litellm/router.py @@ -2142,7 +2142,22 @@ def __aiter__(self): return self async def __anext__(self): - return await self._async_generator.__anext__() + # Delegate to the underlying generator (the + # stream_with_fallbacks coroutine that may swap to a + # fallback mid-stream). The parent CustomStreamWrapper's + # __anext__ would do chunk accumulation on self.chunks, + # but we override here because the underlying generator + # has its own custom retry/fallback semantics. + # + # Append each yielded chunk to self.chunks so downstream + # code (cancel_finalize, post-stream reassembly) can find + # them on the wrapper without having to walk through to + # the inner model_response. Mirrors CustomStreamWrapper's + # behavior for the non-error path. + chunk = await self._async_generator.__anext__() + if chunk is not None: + self.chunks.append(chunk) + return chunk async def stream_with_fallbacks(): fallback_response = None # Track for cleanup in finally diff --git a/litellm/types/utils.py b/litellm/types/utils.py index 282baff07fe3..5dd2584058f4 100644 --- a/litellm/types/utils.py +++ b/litellm/types/utils.py @@ -2514,6 +2514,70 @@ async def __anext__(self): raise StopAsyncIteration +CancelPhase = Literal[ + "before_upstream", + "during_upstream", + "streaming_partial", + "during_parsing", +] +""" +When during the request lifecycle the client cancellation was detected. +Used by litellm/litellm_core_utils/cancel_billing.py to drive the cancel +billing strategy: +- before_upstream: cancel before LiteLLM sent anything to provider → no charge +- during_upstream: non-stream cancel while awaiting provider response → + shielded wait for upstream to complete (so we can bill real usage), + or fall back to prompt-only on shield timeout +- streaming_partial: stream cancelled with at least one chunk already + flushed to client → bill prompt + estimated/real output from chunks +- during_parsing: provider returned, LiteLLM was parsing the response when + cancelled → bill real usage from the (already-received) response +""" + + +CancelUsageSource = Literal[ + "upstream_truth", # full usage from upstream (e.g. anthropic message_delta) + "tokenizer_estimate", # local token_counter on received chunk text + "upstream_completed_after_cancel", # non-stream shield succeeded + "shield_timeout", # non-stream shield exceeded timeout → prompt-only fallback + "no_completion", # zero-byte cancel, bills only the input prompt +] +""" +Provenance of the usage numbers recorded for a cancelled request. +Surfaced in dashboards to spot deployment-specific billing weirdness +(e.g. a network gateway swallowing message_delta would push the +`tokenizer_estimate` share up for that deployment). +""" + + +DeliveryStatus = Literal["full", "partial", "none"] +""" +What the client actually received from the proxy. Independent of billing. + +- full: response body delivered end-to-end (normal success). +- partial: at least one byte / chunk reached the client but the stream + was cut (streaming cancel mid-flight, network drop). +- none: nothing reached the client (zero-chunk cancel, non-stream + shield path, real failure). +""" + + +BillingStatus = Literal["full", "partial", "none"] +""" +What was charged to the customer. Independent of delivery. + +- full: canonical usage from upstream OR accurate reconstruction + from chunks. Used for fully-served calls AND for non-stream + shield-success (upstream returned real usage after the + client disconnected). +- partial: prompt-only / tokenizer-estimate billing (shield_timeout, + upstream errored during shield wait, etc.). Real cost but + lower precision. +- none: zero billable work captured (zero-chunk cancel before + dispatch, hard failures with no upstream usage). ``spend=0``. +""" + + class StandardLoggingUserAPIKeyMetadata(TypedDict): user_api_key_hash: Optional[str] # hash of the litellm virtual key used user_api_key_alias: Optional[str] @@ -2651,6 +2715,58 @@ class StandardLoggingMetadata(StandardLoggingUserAPIKeyMetadata): ] # S3/GCS object key for cold storage retrieval team_alias: Optional[str] team_id: Optional[str] + # === Client cancellation tracking === + # All fields below are populated by the cancel-billing path in + # litellm_core_utils/cancel_billing.py. Each is Optional and only present + # on requests where the client disconnected before the response completed. + # The top-level `status` stays "success" for cancellations (cancel != system + # failure); cancel-vs-normal-success is encoded in cancellation_indicator, + # and delivery_status / billing_status carry the orthogonal taxonomy. + cancellation_indicator: Optional[ + Literal["client_disconnect", "upstream_disconnect"] + ] + """ + Set when the request did not complete naturally. "client_disconnect" is + the typical 499 case (client cancelled / browser tab closed / SDK timeout); + "upstream_disconnect" is for upstream-initiated cuts (rare, e.g. provider + sends RST mid-stream). Presence of this field — not `status` — is what + distinguishes a cancelled-but-billable request from a clean success. + """ + cancel_phase: Optional[CancelPhase] + """Lifecycle phase when cancellation was detected — see CancelPhase docs.""" + bytes_delivered_to_client: Optional[int] + """ + Total bytes flushed to the client socket before disconnect. For streaming, + this is the sum of SSE chunk bytes that were ACK'd. Useful for client-vs- + upstream gap analysis: zero bytes delivered + non-zero upstream usage = + client paid for compute it never received. + """ + upstream_completed: Optional[bool] + """ + Whether the upstream provider call ran to completion. For a cancellation: + - True: shielded wait succeeded (usage_source=upstream_completed_after_cancel) + or message_delta arrived before cancel (usage_source=upstream_truth) + - False: cancel killed the upstream call before it returned a final usage + object (usage_source=tokenizer_estimate / shield_timeout / no_completion) + """ + usage_source: Optional[CancelUsageSource] + """ + Provenance of the recorded usage. Surface this in dashboards as a per- + deployment metric — a deployment with a high `tokenizer_estimate` share + likely has a network gateway swallowing upstream usage fields. + """ + delivery_status: Optional[DeliveryStatus] + """ + Orthogonal dimension to `status`: what the client actually received. + Derived in spend_tracking_utils._derive_delivery_billing_status from + the five cancel markers above + raw status. Materialized into the + SpendLogs metadata JSON column so dashboards can filter on it. + """ + billing_status: Optional[BillingStatus] + """ + Orthogonal dimension to `status`: what we actually billed. See + delivery_status note for derivation details. + """ class StandardLoggingAdditionalHeaders(TypedDict, total=False): @@ -2828,6 +2944,22 @@ class GuardrailTracingDetail(TypedDict, total=False): StandardLoggingPayloadStatus = Literal["success", "failure"] +""" +Binary system-health status for the request. Aligned with upstream LiteLLM. + +- success: the request did not error at the system level. Includes both + fully-served requests AND client cancellations (cancellation is NOT a + system failure — the proxy did its job; the client gave up). For finer- + grained billing/delivery reporting, see ``delivery_status`` and + ``billing_status`` in StandardLoggingMetadata. +- failure: the request errored before billable work could complete + (auth reject, upstream 5xx with no usage, hard timeout, etc.). + +The cancellation taxonomy lives entirely in optional metadata fields: +``cancellation_indicator``, ``cancel_phase``, ``usage_source``, +``upstream_completed``, ``bytes_delivered_to_client``, +``delivery_status``, ``billing_status``. +""" class CachingDetails(TypedDict): diff --git a/tests/test_litellm/litellm_core_utils/test_cancel_billing.py b/tests/test_litellm/litellm_core_utils/test_cancel_billing.py new file mode 100644 index 000000000000..21761c69abf2 --- /dev/null +++ b/tests/test_litellm/litellm_core_utils/test_cancel_billing.py @@ -0,0 +1,234 @@ +""" +Tests for litellm/litellm_core_utils/cancel_billing.py — the cost +computation half of the "modified strategy 5'" cancellation handling. + +Discipline (per CLAUDE.md): +* No mocking of ``cost_per_token`` or ``token_counter`` — we exercise + the real LiteLLM pricing pipeline so a model-cost-map regression + here is caught by these tests. +* The metadata-bridging helper is tested by inspecting the dict it + actually mutates, not by mocking and asserting ``.called``. +""" + +import os +import sys +from types import SimpleNamespace + +sys.path.insert(0, os.path.abspath("../../..")) + +from litellm.litellm_core_utils.cancel_billing import ( + compute_prompt_only_cost, + enrich_request_metadata_with_cancel_markers, +) + +# --------------------------------------------------------------------------- +# compute_prompt_only_cost — real model cost map, real tokenizer +# --------------------------------------------------------------------------- + + +class TestComputePromptOnlyCost: + def test_real_anthropic_pricing_for_known_messages(self): + """ + Real test: feed a known message into the real pricing path. The + cost MUST be > 0 (would be 0.0 only if the model cost map was + missing OR our function returned the zero-fallback). This + catches "cancel_billing.py returns 0.0 for everything" bugs. + """ + messages = [{"role": "user", "content": "Hello, how are you today?"}] + cost = compute_prompt_only_cost( + messages=messages, model="anthropic/claude-sonnet-4-5" + ) + # Real Anthropic Sonnet pricing × ~10-20 prompt tokens. Won't + # be huge ($0.001 ballpark) but MUST exceed zero. + assert cost > 0.0, ( + f"compute_prompt_only_cost returned {cost} for a normal Anthropic " + f"prompt — likely the cost map lookup failed silently" + ) + # Sanity upper bound — Anthropic Sonnet is ~$3/1M input tokens, + # 20 tokens × $3/1M = $0.00006. A cost > $0.01 here would + # mean we accidentally multiplied or used the wrong rate. + assert cost < 0.01, ( + f"compute_prompt_only_cost = {cost} for ~10-token prompt is " + f"absurdly large; pricing logic likely wrong" + ) + + def test_real_openai_pricing_path(self): + """Exercise the OpenAI cost lookup path too — different code path + in cost_per_token, would mask provider-specific regressions.""" + messages = [{"role": "user", "content": "What is 2+2?"}] + cost = compute_prompt_only_cost(messages=messages, model="openai/gpt-4o-mini") + assert cost > 0.0 + assert cost < 0.01 + + def test_unknown_model_returns_zero_not_raise(self): + """Pricing failure must degrade gracefully — failure hook can't + afford an exception.""" + messages = [{"role": "user", "content": "Hello"}] + cost = compute_prompt_only_cost( + messages=messages, model="nonexistent-provider/nonexistent-model-xyz" + ) + assert cost == 0.0 + + def test_no_messages_returns_zero(self): + assert ( + compute_prompt_only_cost(messages=None, model="anthropic/claude-sonnet-4-5") + == 0.0 + ) + assert ( + compute_prompt_only_cost(messages=[], model="anthropic/claude-sonnet-4-5") + == 0.0 + ) + + def test_no_model_returns_zero(self): + messages = [{"role": "user", "content": "Hello"}] + assert compute_prompt_only_cost(messages=messages, model=None) == 0.0 + assert compute_prompt_only_cost(messages=messages, model="") == 0.0 + + def test_long_prompt_scales_with_token_count(self): + """Cost should scale with prompt length — sanity check that + we're actually using the token count, not a constant.""" + short = [{"role": "user", "content": "Hi"}] + long = [ + { + "role": "user", + "content": "Hello " * 500, # ~500 tokens + } + ] + cost_short = compute_prompt_only_cost( + messages=short, model="anthropic/claude-sonnet-4-5" + ) + cost_long = compute_prompt_only_cost( + messages=long, model="anthropic/claude-sonnet-4-5" + ) + # Long prompt should cost notably more (≥ 10x for 500 vs 1 token) + assert cost_long > cost_short * 5, ( + f"Long-prompt cost ({cost_long}) not meaningfully larger than " + f"short-prompt cost ({cost_short}) — token count probably ignored" + ) + + +# --------------------------------------------------------------------------- +# enrich_request_metadata_with_cancel_markers — real dict mutation +# --------------------------------------------------------------------------- + + +def _make_logging_obj_with_cancel_markers(**markers): + """Build a real Logging-shaped object whose model_call_details + contains the cancel markers we want to bridge.""" + obj = SimpleNamespace() + obj.model_call_details = dict(markers) + return obj + + +class TestEnrichRequestMetadataWithCancelMarkers: + def test_no_cancel_marker_no_op(self): + """Normal (non-cancelled) request: helper must not touch + request_data. The cancel-specific fields would otherwise pollute + every SpendLogs row.""" + request_data = {"litellm_params": {"metadata": {"existing": "value"}}} + logging_obj = _make_logging_obj_with_cancel_markers() # no markers + enrich_request_metadata_with_cancel_markers(request_data, logging_obj) + # Unchanged + assert request_data["litellm_params"]["metadata"] == {"existing": "value"} + + def test_streaming_partial_markers_copied_into_metadata(self): + request_data = {"litellm_params": {"metadata": {"existing": "value"}}} + logging_obj = _make_logging_obj_with_cancel_markers( + cancellation_indicator="client_disconnect", + cancel_phase="streaming_partial", + bytes_delivered_to_client=4096, + upstream_completed=False, + usage_source="tokenizer_estimate", + ) + enrich_request_metadata_with_cancel_markers(request_data, logging_obj) + + meta = request_data["litellm_params"]["metadata"] + assert meta["cancellation_indicator"] == "client_disconnect" + assert meta["cancel_phase"] == "streaming_partial" + assert meta["bytes_delivered_to_client"] == 4096 + assert meta["upstream_completed"] is False + assert meta["usage_source"] == "tokenizer_estimate" + # The enrich helper no longer touches top-level status — + # cancellation is identified by cancellation_indicator above, + # and the derived delivery_status / billing_status are computed + # downstream in spend_tracking_utils._derive_delivery_billing_status. + assert "status" not in meta or meta.get("status") != "success_partial" + # Pre-existing fields preserved + assert meta["existing"] == "value" + + def test_creates_metadata_dict_when_missing(self): + """Should handle request_data that doesn't have litellm_params + or metadata yet (e.g. cancel fired before pre_call ran).""" + request_data = {} # empty + logging_obj = _make_logging_obj_with_cancel_markers( + cancellation_indicator="client_disconnect", + cancel_phase="before_upstream", + ) + enrich_request_metadata_with_cancel_markers(request_data, logging_obj) + + meta = request_data["litellm_params"]["metadata"] + assert meta["cancellation_indicator"] == "client_disconnect" + assert meta["cancel_phase"] == "before_upstream" + # status is no longer mutated by the enrich helper. + assert meta.get("status") != "success_partial" + + def test_none_logging_obj_no_op(self): + request_data = {"litellm_params": {"metadata": {}}} + # Must not raise + enrich_request_metadata_with_cancel_markers(request_data, None) + assert request_data == {"litellm_params": {"metadata": {}}} + + def test_partial_markers_only_those_present_propagate(self): + """Realistic: only some markers set (e.g. cancel fired before + we knew bytes_delivered_to_client). Don't insert keys for + markers that weren't on the logging object.""" + request_data = {"litellm_params": {"metadata": {}}} + logging_obj = _make_logging_obj_with_cancel_markers( + cancellation_indicator="client_disconnect", + cancel_phase="streaming_partial", + # bytes_delivered_to_client, upstream_completed, usage_source NOT set + ) + enrich_request_metadata_with_cancel_markers(request_data, logging_obj) + + meta = request_data["litellm_params"]["metadata"] + assert "cancellation_indicator" in meta + assert "cancel_phase" in meta + # These should NOT be present (not on source) + assert "bytes_delivered_to_client" not in meta + assert "upstream_completed" not in meta + assert "usage_source" not in meta + + def test_does_not_overwrite_status_field(self): + """The enrich helper must NOT mutate top-level status. + + Under the binary-status taxonomy, cancellation is identified by + the presence of ``cancellation_indicator``; the top-level + ``status`` column stays ``"success"`` (cancel != system + failure). Verify we preserve any prior status the caller set + and never write the legacy ``"success_partial"`` sentinel. + + Inverted from the original ``test_overwrites_stale_status`` + which encoded the now-superseded "cancel overrides status to + success_partial" rule. + """ + for prior_status in ("success", "failure", None): + request_data = { + "litellm_params": { + "metadata": ({"status": prior_status} if prior_status else {}) + } + } + logging_obj = _make_logging_obj_with_cancel_markers( + cancellation_indicator="client_disconnect", + cancel_phase="streaming_partial", + ) + enrich_request_metadata_with_cancel_markers(request_data, logging_obj) + meta = request_data["litellm_params"]["metadata"] + # Cancellation marker landed + assert meta["cancellation_indicator"] == "client_disconnect" + # Status was either left alone or never set + if prior_status is None: + assert "status" not in meta + else: + assert meta["status"] == prior_status + # And the legacy success_partial sentinel is never written + assert meta.get("status") != "success_partial" diff --git a/tests/test_litellm/litellm_core_utils/test_cancel_finalize.py b/tests/test_litellm/litellm_core_utils/test_cancel_finalize.py new file mode 100644 index 000000000000..b2811ab13720 --- /dev/null +++ b/tests/test_litellm/litellm_core_utils/test_cancel_finalize.py @@ -0,0 +1,458 @@ +""" +Tests for litellm/litellm_core_utils/cancel_finalize.py. + +Design notes +------------ +The goal of this file is to verify the CONTRACT of the cancel-finalize +plumbing, not just that "the right Python function was called". Two +specific anti-patterns are avoided here (per CLAUDE.md): + +1. **No mocking of the unit under test or its core dependencies.** + ``stream_chunk_builder`` is NOT mocked — we feed real + ``ModelResponseStream`` chunks through it and assert on the actual + reassembled response's ``usage`` field. This means the test also + catches regressions in the cursor=1 fix from PR #1, which is the + right cross-coverage. + +2. **No ``MagicMock.called`` / ``.call_args is mock`` assertions.** + The Logging object is a small real ``SpyLogging`` class that + captures the kwargs handlers receive. Tests assert on the captured + *data* (response shape, usage fields, marker keys), not on + "the mock was called". + +If you add a new test here, follow the same pattern: build the smallest +realistic input that triggers the behavior you want to lock in, then +assert on observable outputs (dict mutations, captured handler args, +returned tasks). +""" + +import asyncio +import os +import sys +from types import SimpleNamespace + +import pytest + +sys.path.insert(0, os.path.abspath("../../..")) + +from litellm.litellm_core_utils.cancel_finalize import ( + _get_accumulated_chunks, + finalize_non_stream_cancel, + finalize_streaming_cancel, + is_logging_obj_cancelled, + mark_logging_obj_cancelled, +) +from litellm.types.utils import ( + Delta, + ModelResponseStream, + StreamingChoices, + Usage, +) + +# --------------------------------------------------------------------------- +# Real test doubles (NOT MagicMock) +# --------------------------------------------------------------------------- + + +class SpyLogging: + """Minimal real Logging-shaped object used in place of MagicMock. + + Captures the kwargs that ``async_success_handler`` / + ``async_failure_handler`` receive so tests can assert on the + payload (not on ``.called``). This is the spy pattern from + CLAUDE.md's "no theater tests" rule — tests verify what actually + flowed through, which catches real shape / value regressions. + """ + + def __init__(self): + # Real dict (not MagicMock attribute access) so that + # cancel_finalize's get/set patterns behave exactly as in prod. + self.model_call_details: dict = {} + self.start_time = None + self.captured_success_calls: list = [] + self.captured_failure_calls: list = [] + + async def async_success_handler(self, **kwargs): + # Defensive deep snapshot of the result kwarg if it's mutable — + # this catches "we set the field, then mutated it back" bugs. + self.captured_success_calls.append(dict(kwargs)) + + async def async_failure_handler(self, *args, **kwargs): + self.captured_failure_calls.append({"args": args, "kwargs": dict(kwargs)}) + + +def _make_chunk( + *, + content: str = "", + usage: Usage = None, + finish_reason: str = None, +) -> ModelResponseStream: + """Construct a real ModelResponseStream — same factory as the + cursor-bug regression file. Reused so any chunk-shape change forces + both test files to update together.""" + return ModelResponseStream( + id="msg_cancel_test", + created=1738900000, + model="claude-sonnet-4-6", + object="chat.completion.chunk", + choices=[ + StreamingChoices( + finish_reason=finish_reason, + index=0, + delta=Delta(content=content, role="assistant"), + ) + ], + usage=usage, + ) + + +# --------------------------------------------------------------------------- +# mark_logging_obj_cancelled — pure dict mutation contract +# --------------------------------------------------------------------------- + + +class TestMarkLoggingObjCancelled: + def test_sets_indicator_and_phase(self): + logging_obj = SpyLogging() + mark_logging_obj_cancelled(logging_obj, phase="streaming_partial") + assert ( + logging_obj.model_call_details["cancellation_indicator"] + == "client_disconnect" + ) + assert logging_obj.model_call_details["cancel_phase"] == "streaming_partial" + assert "cancelled_at" in logging_obj.model_call_details + + def test_idempotent_does_not_overwrite_first_marker(self): + logging_obj = SpyLogging() + mark_logging_obj_cancelled( + logging_obj, phase="streaming_partial", indicator="client_disconnect" + ) + first_t = logging_obj.model_call_details["cancelled_at"] + # Second call with a different phase/indicator should be ignored — + # the first detection point wins so phase progression doesn't lie. + mark_logging_obj_cancelled( + logging_obj, + phase="during_parsing", + indicator="upstream_disconnect", + ) + assert logging_obj.model_call_details["cancel_phase"] == "streaming_partial" + assert ( + logging_obj.model_call_details["cancellation_indicator"] + == "client_disconnect" + ) + assert logging_obj.model_call_details["cancelled_at"] == first_t + + def test_none_logging_obj_is_no_op(self): + # Cancel can fire before Logging is built (pre-pre-call hook) + mark_logging_obj_cancelled(None, phase="before_upstream") + + def test_logging_obj_without_model_call_details_is_no_op(self): + bad_obj = SimpleNamespace(model_call_details=None) + # Must not raise — pathological logging objects shouldn't crash + # the cancel finalize path. + mark_logging_obj_cancelled(bad_obj, phase="before_upstream") + + def test_bytes_delivered_recorded_when_provided(self): + logging_obj = SpyLogging() + mark_logging_obj_cancelled( + logging_obj, phase="streaming_partial", bytes_delivered=4096 + ) + assert logging_obj.model_call_details["bytes_delivered_to_client"] == 4096 + + def test_bytes_delivered_omitted_when_not_provided(self): + logging_obj = SpyLogging() + mark_logging_obj_cancelled(logging_obj, phase="before_upstream") + # Important: don't insert a None key. Downstream JSON serialization + # would write `"bytes_delivered_to_client": null` and pollute the + # SpendLogs metadata with a misleading field. + assert "bytes_delivered_to_client" not in logging_obj.model_call_details + + +class TestIsLoggingObjCancelled: + def test_returns_true_after_marking(self): + logging_obj = SpyLogging() + assert is_logging_obj_cancelled(logging_obj) is False + mark_logging_obj_cancelled(logging_obj, phase="streaming_partial") + assert is_logging_obj_cancelled(logging_obj) is True + + def test_returns_false_for_none(self): + assert is_logging_obj_cancelled(None) is False + + +class TestGetAccumulatedChunks: + def test_returns_chunks_when_present(self): + wrapper = SimpleNamespace(chunks=[{"a": 1}, {"b": 2}]) + assert _get_accumulated_chunks(wrapper) == [{"a": 1}, {"b": 2}] + + def test_returns_empty_list_when_no_chunks_attr(self): + wrapper = SimpleNamespace() + assert _get_accumulated_chunks(wrapper) == [] + + def test_returns_empty_list_when_chunks_not_a_list(self): + wrapper = SimpleNamespace(chunks="not a list") + assert _get_accumulated_chunks(wrapper) == [] + + +# --------------------------------------------------------------------------- +# Streaming finalize — end-to-end through stream_chunk_builder +# --------------------------------------------------------------------------- + + +class TestFinalizeStreamingCancel: + @pytest.mark.asyncio + async def test_anthropic_midthinking_cancel_dispatches_real_usage(self): + """ + Realistic scenario: Anthropic thinking-model stream cancelled + after the message_start cursor + several content chunks but + BEFORE message_delta arrives. This is the bug class PR #1 fixed + — and this test exercises it through the cancel-finalize path, + proving the two PRs compose correctly end-to-end. + + Why this is not theater: we feed real ModelResponseStream chunks, + call the real finalize_streaming_cancel, which calls the real + stream_chunk_builder, which exercises the real cursor=1 fix + from PR #1. The captured response's usage MUST reflect the + token-counter estimate of the streamed text, not the cursor 1 + placeholder. + """ + chunks = [ + # Anthropic message_start: real input_tokens, output cursor=1 + _make_chunk( + usage=Usage(prompt_tokens=1024, completion_tokens=1, total_tokens=1025) + ), + # content_block_delta chunks with visible text + _make_chunk( + content="The capital of France is Paris. " + "It has been the political and cultural center " + "for over a thousand years." + ), + _make_chunk(content="Major landmarks include the Eiffel Tower."), + # CANCEL — no message_delta, no message_stop + ] + wrapper = SimpleNamespace(chunks=chunks, messages=[]) + logging_obj = SpyLogging() + + await finalize_streaming_cancel( + stream_wrapper=wrapper, + logging_obj=logging_obj, + user_api_key_dict=SimpleNamespace(), + request_data={}, + ) + + # 1. Cancel marker must be set on the logging object so the + # cost calculator knows to apply the cancel-billing path. + assert logging_obj.model_call_details["cancel_phase"] == "streaming_partial" + assert ( + logging_obj.model_call_details["cancellation_indicator"] + == "client_disconnect" + ) + + # 2. async_success_handler received the reassembled response + # (not the failure handler — cancel routes through the + # success callback chain; the cancel taxonomy is encoded in + # metadata markers, not the top-level status). + assert len(logging_obj.captured_success_calls) == 1 + assert len(logging_obj.captured_failure_calls) == 0 + captured = logging_obj.captured_success_calls[0] + response = captured["result"] + + # 3. The reassembled response carries real usage data: + # - prompt_tokens preserved from message_start (1024) + # - completion_tokens reflects the streamed text length, + # NOT the cursor placeholder of 1 (would indicate PR #1 + # regression — cursor=1 leaked through to billing). + assert response.usage.prompt_tokens == 1024 + assert response.usage.completion_tokens > 1, ( + f"completion_tokens={response.usage.completion_tokens} — " + f"cursor=1 leaked through. PR #1's cursor reset should " + f"have triggered the token_counter fallback on the " + f"streamed text (~30+ tokens for this fixture)." + ) + + @pytest.mark.asyncio + async def test_no_chunks_falls_back_to_failure_hook(self): + """ + Edge: client cancels before any chunk arrived. We have nothing + to bill, so falling back to the failure hook (with cost=0) + is correct — at least a SpendLogs row exists, instead of the + silent drop the old `except Exception` path produced. + + Verify by importing and patching proxy_logging_obj's + post_call_failure_hook to a spy. + """ + from litellm.proxy import proxy_server as ps + + wrapper = SimpleNamespace(chunks=[], messages=[]) + logging_obj = SpyLogging() + captured_failure_calls: list = [] + + # Replace the module-level proxy_logging_obj with a real spy + # object — narrowly scoped to the call surface cancel_finalize + # touches (post_call_failure_hook only). + class SpyProxyLogging: + async def post_call_failure_hook(self, **kwargs): + captured_failure_calls.append(kwargs) + + original = getattr(ps, "proxy_logging_obj", None) + ps.proxy_logging_obj = SpyProxyLogging() + try: + await finalize_streaming_cancel( + stream_wrapper=wrapper, + logging_obj=logging_obj, + user_api_key_dict=SimpleNamespace(), + request_data={"req": "data"}, + ) + finally: + ps.proxy_logging_obj = original + + # Marker still set on logging_obj — billing path can identify + # this as a zero-chunk cancel for the dashboard. + assert is_logging_obj_cancelled(logging_obj) + # And the failure hook fired (zero chunks → no success path) + assert len(captured_failure_calls) == 1 + # request_data was enriched with cancel markers (PR #3 bridges + # them in so SpendLogs picks them up). Original keys preserved. + forwarded = captured_failure_calls[0]["request_data"] + assert forwarded["req"] == "data" + meta = forwarded["litellm_params"]["metadata"] + assert meta["cancellation_indicator"] == "client_disconnect" + assert meta["cancel_phase"] == "streaming_partial" + # Under the binary-status taxonomy, enrich does NOT mutate + # status — the cancellation_indicator marker above is what + # identifies this row as a cancel for downstream derivation. + assert meta.get("status") != "success_partial" + # original_exception is a CancelledError (well-typed) + assert isinstance( + captured_failure_calls[0]["original_exception"], + asyncio.CancelledError, + ) + + @pytest.mark.asyncio + async def test_does_not_raise_when_success_handler_errors(self): + """ + Defensive: if the user's async_success_handler raises (e.g. a + broken Langfuse integration), the finalize must swallow it. + Caller will re-raise the original CancelledError — we must not + replace that with a derived error that hides the real cancel + signal from asyncio. + """ + + class RaisingLogging(SpyLogging): + async def async_success_handler(self, **kwargs): + raise RuntimeError("simulated downstream callback bug") + + # Need at least one chunk so we get past the no-chunks fallback + chunks = [_make_chunk(content="hi", usage=Usage(prompt_tokens=5))] + wrapper = SimpleNamespace(chunks=chunks, messages=[]) + + # Must not raise + await finalize_streaming_cancel( + stream_wrapper=wrapper, + logging_obj=RaisingLogging(), + user_api_key_dict=SimpleNamespace(), + request_data={}, + ) + + +# --------------------------------------------------------------------------- +# Non-stream finalize — shield + real asyncio tasks +# --------------------------------------------------------------------------- + + +class TestFinalizeNonStreamCancel: + @pytest.mark.asyncio + async def test_shield_lets_upstream_complete_records_real_usage(self): + """ + Shield-and-wait happy path: cancel fires while upstream is + still working, but upstream returns within the shield window. + The real response object reaches async_success_handler — that's + the whole point of choosing strategy A. + """ + fake_response = SimpleNamespace( + usage=Usage(prompt_tokens=100, completion_tokens=42, total_tokens=142) + ) + + async def upstream_returns_quickly(): + await asyncio.sleep(0.05) + return fake_response + + upstream_task = asyncio.create_task(upstream_returns_quickly()) + logging_obj = SpyLogging() + + await finalize_non_stream_cancel( + upstream_task=upstream_task, + logging_obj=logging_obj, + user_api_key_dict=SimpleNamespace(), + request_data={}, + shield_timeout_s=5.0, + ) + + assert logging_obj.model_call_details["upstream_completed"] is True + assert ( + logging_obj.model_call_details["usage_source"] + == "upstream_completed_after_cancel" + ) + # async_success_handler received the REAL response (not a fake + # from a mock) with the REAL usage numbers from upstream. + assert len(logging_obj.captured_success_calls) == 1 + captured = logging_obj.captured_success_calls[0] + assert captured["result"] is fake_response + assert captured["result"].usage.completion_tokens == 42 + + @pytest.mark.asyncio + async def test_shield_timeout_cancels_upstream_and_records_timeout_source(self): + async def upstream_never_returns(): + await asyncio.sleep(10.0) + return SimpleNamespace() + + upstream_task = asyncio.create_task(upstream_never_returns()) + logging_obj = SpyLogging() + + await finalize_non_stream_cancel( + upstream_task=upstream_task, + logging_obj=logging_obj, + user_api_key_dict=SimpleNamespace(), + request_data={}, + shield_timeout_s=0.1, + ) + + assert logging_obj.model_call_details["upstream_completed"] is False + assert logging_obj.model_call_details["usage_source"] == "shield_timeout" + # Critical for resource hygiene: upstream task MUST be cancelled + # so we don't leak it past the proxy request lifecycle. + # Give the cancel a moment to actually take effect on the task. + for _ in range(10): + if upstream_task.done(): + break + await asyncio.sleep(0.01) + assert upstream_task.cancelled() or upstream_task.done() + + @pytest.mark.asyncio + async def test_shield_upstream_exception_records_no_completion(self): + async def upstream_errors(): + raise RuntimeError("simulated 5xx from provider") + + upstream_task = asyncio.create_task(upstream_errors()) + logging_obj = SpyLogging() + + await finalize_non_stream_cancel( + upstream_task=upstream_task, + logging_obj=logging_obj, + user_api_key_dict=SimpleNamespace(), + request_data={}, + shield_timeout_s=5.0, + ) + assert logging_obj.model_call_details["upstream_completed"] is False + assert logging_obj.model_call_details["usage_source"] == "no_completion" + + @pytest.mark.asyncio + async def test_no_upstream_task_marks_marker_and_does_not_crash(self): + """before_upstream cancel — no task to wait for.""" + logging_obj = SpyLogging() + await finalize_non_stream_cancel( + upstream_task=None, + logging_obj=logging_obj, + user_api_key_dict=SimpleNamespace(), + request_data={}, + ) + assert is_logging_obj_cancelled(logging_obj) diff --git a/tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_cursor.py b/tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_cursor.py new file mode 100644 index 000000000000..04e06b82a33f --- /dev/null +++ b/tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_cursor.py @@ -0,0 +1,245 @@ +""" +Regression tests for the Anthropic message_start cursor=1 bug in +ChunkProcessor._calculate_usage_per_chunk. + +Background +---------- +Anthropic streams a `message_start` event that carries +`usage.output_tokens=1` as a placeholder ("cursor"). The real cumulative +output count only arrives in the final `message_delta` event. When a +stream is cancelled before `message_delta` lands (very common for +thinking models on long-tail prompts), the last-wins accumulator in +ChunkProcessor leaves completion_tokens stuck at 1. Because 1 is +truthy, the `completion_tokens or token_counter(text=...)` fallback in +calculate_usage() never fires, and the request is billed for 1 output +token even when several thousand tokens of text were actually streamed. + +These tests pin the post-fix behavior: completion_tokens should reset +to 0 when the only update we saw was the cursor, allowing the +text-based fallback to estimate from the real completion text. +""" + +import os +import sys + +import pytest + +sys.path.insert(0, os.path.abspath("../../..")) + +from litellm.litellm_core_utils.streaming_chunk_builder_utils import ChunkProcessor +from litellm.types.utils import ( + Delta, + ModelResponseStream, + StreamingChoices, + Usage, +) + + +def _make_chunk( + *, + content: str = "", + usage: Usage = None, + finish_reason: str = None, +) -> ModelResponseStream: + return ModelResponseStream( + id="msg_test", + created=1738900000, + model="claude-sonnet-4-6", + object="chat.completion.chunk", + choices=[ + StreamingChoices( + finish_reason=finish_reason, + index=0, + delta=Delta(content=content, role="assistant"), + ) + ], + usage=usage, + ) + + +class TestAnthropicCursorBug: + """The core regression: completion_tokens=1 cursor must not leak through.""" + + def test_only_message_start_cursor_resets_completion_to_zero(self): + """ + Stream cancelled before message_delta — only the message_start cursor + (output_tokens=1) was seen. Per-chunk accumulator must reset to 0 so + token_counter fallback can estimate from completion text. + """ + # Anthropic message_start: input_tokens accurate, output_tokens=1 cursor + message_start = _make_chunk( + usage=Usage(prompt_tokens=1024, completion_tokens=1, total_tokens=1025) + ) + # Several content_block_delta chunks (no usage attached) + text_chunks = [ + _make_chunk(content="Hello"), + _make_chunk(content=" world"), + _make_chunk(content=" this is partial."), + ] + chunks = [message_start, *text_chunks] + + processor = ChunkProcessor(chunks=chunks, messages=[]) + result = processor._calculate_usage_per_chunk(chunks=chunks) + + assert result["prompt_tokens"] == 1024 + # The cursor value of 1 must NOT leak through — should be reset to 0 + # so the text-based fallback estimates the real completion length. + assert result["completion_tokens"] == 0, ( + "completion_tokens=1 from message_start cursor leaked through. " + "Should reset to 0 when only cursor was seen, so token_counter " + "fallback in calculate_usage() can estimate from completion text." + ) + + def test_message_start_plus_message_delta_uses_delta_value(self): + """ + Normal complete stream: message_start cursor=1, then message_delta=3847. + Last-wins must give 3847 (the real value). + """ + message_start = _make_chunk( + usage=Usage(prompt_tokens=1024, completion_tokens=1, total_tokens=1025) + ) + text_chunks = [_make_chunk(content=t) for t in ["Hello", " world", "!"]] + # message_delta with the real cumulative output_tokens + message_delta = _make_chunk( + usage=Usage(prompt_tokens=1024, completion_tokens=3847, total_tokens=4871), + finish_reason="stop", + ) + chunks = [message_start, *text_chunks, message_delta] + + processor = ChunkProcessor(chunks=chunks, messages=[]) + result = processor._calculate_usage_per_chunk(chunks=chunks) + + assert result["prompt_tokens"] == 1024 + assert result["completion_tokens"] == 3847 + + def test_calculate_usage_falls_back_to_token_counter_for_cursor_only(self): + """ + End-to-end via calculate_usage(): cursor-only stream + real completion + text should produce a token-counter estimate, NOT 1. + """ + message_start = _make_chunk( + usage=Usage(prompt_tokens=1024, completion_tokens=1, total_tokens=1025) + ) + # ~50 visible chars ≈ ~12 tokens (anthropic-style tokenizer ballpark) + text_chunks = [ + _make_chunk(content="Based on your question, I think the answer is "), + _make_chunk(content="forty-two. Here is my reasoning: "), + ] + chunks = [message_start, *text_chunks] + completion_output = ( + "Based on your question, I think the answer is forty-two. " + "Here is my reasoning: " + ) + + processor = ChunkProcessor(chunks=chunks, messages=[]) + usage = processor.calculate_usage( + chunks=chunks, + model="claude-sonnet-4-6", + completion_output=completion_output, + messages=[], + ) + + # Should be a token_counter estimate of the text, not the cursor 1 + assert usage.completion_tokens > 1, ( + f"Expected token_counter estimate of completion text, got " + f"completion_tokens={usage.completion_tokens} (likely stuck at cursor)" + ) + + def test_cache_fields_preserved_from_message_start(self): + """cache_read / cache_creation come from message_start and must survive.""" + message_start_usage = Usage( + prompt_tokens=1024, completion_tokens=1, total_tokens=1025 + ) + # Anthropic puts these in message_start + message_start_usage.cache_read_input_tokens = 512 + message_start_usage.cache_creation_input_tokens = 128 + message_start = _make_chunk(usage=message_start_usage) + + chunks = [message_start, _make_chunk(content="hi")] + processor = ChunkProcessor(chunks=chunks, messages=[]) + result = processor._calculate_usage_per_chunk(chunks=chunks) + + assert result["cache_read_input_tokens"] == 512 + assert result["cache_creation_input_tokens"] == 128 + + def test_openai_streaming_unaffected(self): + """ + OpenAI's only usage chunk is the penultimate one (with + stream_options.include_usage=true), and it carries the real value + directly. Our cursor fix must not break this path — output > 1 + means saw_non_cursor_completion=True so no reset happens. + """ + # Simulate OpenAI: content chunks first, then ONE usage chunk at the end + text_chunks = [_make_chunk(content=t) for t in ["The", " answer", " is 42"]] + usage_chunk = _make_chunk( + usage=Usage(prompt_tokens=42, completion_tokens=15, total_tokens=57), + finish_reason="stop", + ) + chunks = [*text_chunks, usage_chunk] + + processor = ChunkProcessor(chunks=chunks, messages=[]) + result = processor._calculate_usage_per_chunk(chunks=chunks) + + assert result["prompt_tokens"] == 42 + assert result["completion_tokens"] == 15 + + def test_single_token_completion_legitimate_case(self): + """ + Edge case: a stream that legitimately completes with output_tokens=1 + (e.g., model returns just "Yes."). Without saw_non_cursor_completion + we'd reset to 0 and fall through to token_counter — but token_counter + on a 1-token string also gives ~1, so billing is still approximately + correct. This test pins that the result is sane (1 or 0). + """ + message_start = _make_chunk( + usage=Usage(prompt_tokens=20, completion_tokens=1, total_tokens=21) + ) + text_chunk = _make_chunk(content="Yes.") + # Anthropic's message_delta also gives output_tokens=1 in this case + message_delta = _make_chunk( + usage=Usage(prompt_tokens=20, completion_tokens=1, total_tokens=21), + finish_reason="stop", + ) + chunks = [message_start, text_chunk, message_delta] + + processor = ChunkProcessor(chunks=chunks, messages=[]) + usage = processor.calculate_usage( + chunks=chunks, + model="claude-sonnet-4-6", + completion_output="Yes.", + messages=[], + ) + + # Per-chunk completion is reset to 0 (cursor heuristic can't distinguish + # "real 1-token answer" from "cursor never updated"), but token_counter + # fallback on "Yes." gives a small number (~1-2). Either way the user + # is billed approximately what they used. + assert 0 <= usage.completion_tokens <= 3, ( + f"Legitimate 1-token completion should bill ~1 token, got " + f"{usage.completion_tokens}" + ) + + +class TestNonAnthropicStreamingIntact: + """Make sure providers without cursor pattern still work.""" + + def test_completion_tokens_above_one_never_resets(self): + """Any chunk reporting completion_tokens > 1 sets saw_non_cursor + and prevents the reset.""" + chunks = [ + _make_chunk( + usage=Usage(prompt_tokens=10, completion_tokens=5, total_tokens=15) + ), + ] + processor = ChunkProcessor(chunks=chunks, messages=[]) + result = processor._calculate_usage_per_chunk(chunks=chunks) + assert result["completion_tokens"] == 5 + + def test_no_usage_chunks_leaves_zero(self): + """Stream with zero usage info → completion_tokens stays 0 + (token_counter fallback will handle it).""" + chunks = [_make_chunk(content="hi"), _make_chunk(content=" there")] + processor = ChunkProcessor(chunks=chunks, messages=[]) + result = processor._calculate_usage_per_chunk(chunks=chunks) + assert result["prompt_tokens"] == 0 + assert result["completion_tokens"] == 0 diff --git a/tests/test_litellm/proxy/spend_tracking/test_spend_logs_cancellation_metadata.py b/tests/test_litellm/proxy/spend_tracking/test_spend_logs_cancellation_metadata.py new file mode 100644 index 000000000000..dde264083dc8 --- /dev/null +++ b/tests/test_litellm/proxy/spend_tracking/test_spend_logs_cancellation_metadata.py @@ -0,0 +1,316 @@ +""" +Behavioral tests for `_get_spend_logs_metadata` propagating the cancel +markers AND materializing the derived ``delivery_status`` / +``billing_status`` taxonomy. + +The 5 cancel markers + 2 derived dimensions land in the existing +``metadata`` JSON column on LiteLLM_SpendLogs — no Prisma migration +required — but only if ``_get_spend_logs_metadata`` (the function the +proxy actually calls to shape a SpendLogs payload) preserves them when +filtering input metadata through ``SpendLogsMetadata.__annotations__`` +AND invokes ``_derive_delivery_billing_status`` to populate the derived +fields. + +This file guards two failure modes: +1. Forgetting to declare a new marker on SpendLogsMetadata silently + drops it from every SpendLogs row. +2. Forgetting to call (or correctly wire) the derivation helper means + downstream dashboards filtering on delivery_status / billing_status + match zero rows. + +Each test runs the real function and asserts on the real returned dict. +No Literal/Enum runtime assertions (those are mypy's job). +""" + +import os +import sys + +sys.path.insert(0, os.path.abspath("../../../..")) + +from litellm.proxy.spend_tracking.spend_tracking_utils import ( + _derive_delivery_billing_status, + _get_spend_logs_metadata, + _get_status_for_spend_log, +) + + +class TestCancellationFieldsInitializedFromNone: + """When the proxy hasn't built a metadata dict yet (early failure path), + `_get_spend_logs_metadata(None)` must still produce a SpendLogsMetadata + with the new cancellation fields explicitly initialized to None. + + Without explicit initialization the keys would be missing from the + TypedDict, and downstream JSON dumpers would silently skip them + when persisting to the metadata column — breaking dashboard queries + that filter on `metadata->>'cancel_phase'`. + """ + + def test_all_five_cancellation_fields_present_with_none(self): + meta = _get_spend_logs_metadata(metadata=None) + # These keys MUST exist (even as None) so JSON serializers don't + # silently drop them. If a key is missing, json.dumps emits the + # dict without it, and reconciliation queries return NULL — + # indistinguishable from "field was never populated by the + # cancel-billing path" vs "schema doesn't know about it". + for key in ( + "cancellation_indicator", + "cancel_phase", + "bytes_delivered_to_client", + "upstream_completed", + "usage_source", + ): + assert key in meta, ( + f"_get_spend_logs_metadata(None) lost the '{key}' key. " + f"This typically means the field was added to " + f"SpendLogsMetadata.__annotations__ but the None-branch " + f"of _get_spend_logs_metadata wasn't updated. Both must " + f"agree or new fields go silent in SpendLogs." + ) + assert meta[key] is None + + +class TestCancellationFieldsPropagateFromInputMetadata: + """ + The hot path: proxy builds a metadata dict containing the cancellation + fields (set by litellm_core_utils/cancel_finalize.py), then calls + `_get_spend_logs_metadata(metadata)`. The fields must flow through + the SpendLogsMetadata.__annotations__-keyed filter into the output + dict that becomes the SpendLogs row's metadata column. + + Regression risk: someone adds a field to the docstring / Literal but + forgets to declare it on SpendLogsMetadata. The filter would silently + drop it — these tests catch that by asserting on the output. + """ + + def test_streaming_partial_cancel_fields_propagate(self): + input_meta = { + "cancellation_indicator": "client_disconnect", + "cancel_phase": "streaming_partial", + "bytes_delivered_to_client": 4096, + "upstream_completed": False, + "usage_source": "tokenizer_estimate", + # Mix in an unrelated existing field to verify we didn't + # accidentally clobber other propagation + "requester_ip_address": "10.0.0.1", + } + out = _get_spend_logs_metadata(metadata=input_meta) + + assert out["cancellation_indicator"] == "client_disconnect" + assert out["cancel_phase"] == "streaming_partial" + assert out["bytes_delivered_to_client"] == 4096 + assert out["upstream_completed"] is False + assert out["usage_source"] == "tokenizer_estimate" + # Sanity: pre-existing fields still flow + assert out["requester_ip_address"] == "10.0.0.1" + + def test_non_stream_shield_success_fields_propagate(self): + """Different value combination — the non-stream cancel path + records upstream_completed=True and a different usage_source.""" + input_meta = { + "cancellation_indicator": "client_disconnect", + "cancel_phase": "during_upstream", + "bytes_delivered_to_client": 0, + "upstream_completed": True, + "usage_source": "upstream_completed_after_cancel", + } + out = _get_spend_logs_metadata(metadata=input_meta) + + assert out["upstream_completed"] is True + assert out["usage_source"] == "upstream_completed_after_cancel" + assert out["bytes_delivered_to_client"] == 0 + assert out["cancel_phase"] == "during_upstream" + + def test_partial_input_only_some_fields_set(self): + """Realistic scenario: cancel fires before LiteLLM has finished + gathering all the metadata. Set fields propagate, unset stay None.""" + input_meta = { + "cancellation_indicator": "client_disconnect", + "cancel_phase": "before_upstream", + # No bytes_delivered_to_client, no upstream_completed, + # no usage_source — cancel was too early to know any of these. + } + out = _get_spend_logs_metadata(metadata=input_meta) + + assert out["cancellation_indicator"] == "client_disconnect" + assert out["cancel_phase"] == "before_upstream" + # The filter returns metadata.get(key) for declared keys, so + # missing input keys come out as None. This is the contract that + # downstream dashboards rely on. + assert out["bytes_delivered_to_client"] is None + assert out["upstream_completed"] is None + assert out["usage_source"] is None + + +class TestNormalSuccessUnaffected: + """A successful (non-cancel) request must not gain spurious + cancellation fields with non-None values. The new fields should be + absent / None for normal requests so downstream filters like + `WHERE metadata->>'cancel_phase' IS NULL` correctly identify + non-cancel traffic.""" + + def test_normal_request_metadata_has_no_cancel_markers(self): + input_meta = { + "requester_ip_address": "10.0.0.1", + "user_api_key": "sk-test", + # No cancellation fields — normal happy path + } + out = _get_spend_logs_metadata(metadata=input_meta) + assert out["cancellation_indicator"] is None + assert out["cancel_phase"] is None + assert out["bytes_delivered_to_client"] is None + assert out["upstream_completed"] is None + assert out["usage_source"] is None + + +class TestDeriveDeliveryBillingStatus: + """Direct unit coverage of ``_derive_delivery_billing_status`` — one + test per row of the semantic mapping (see docstring inside the + helper). These are the source-of-truth assertions; the + ``TestMaterializeDeliveryBillingStatus`` class below verifies they + actually land in the SpendLogs metadata output dict. + """ + + def test_normal_success_yields_full_full(self): + assert _derive_delivery_billing_status({}) == ("full", "full") + assert _derive_delivery_billing_status({"status": "success"}) == ( + "full", + "full", + ) + + def test_failure_yields_none_none(self): + assert _derive_delivery_billing_status({"status": "failure"}) == ( + "none", + "none", + ) + + def test_streaming_partial_with_bytes_yields_partial_partial(self): + meta = { + "cancellation_indicator": "client_disconnect", + "cancel_phase": "streaming_partial", + "bytes_delivered_to_client": 4096, + "usage_source": "tokenizer_estimate", + } + assert _derive_delivery_billing_status(meta) == ("partial", "partial") + + def test_shield_success_yields_none_full(self): + meta = { + "cancellation_indicator": "client_disconnect", + "cancel_phase": "during_upstream", + "bytes_delivered_to_client": 0, + "upstream_completed": True, + "usage_source": "upstream_completed_after_cancel", + } + assert _derive_delivery_billing_status(meta) == ("none", "full") + + def test_shield_timeout_yields_none_partial(self): + meta = { + "cancellation_indicator": "client_disconnect", + "cancel_phase": "during_upstream", + "upstream_completed": False, + "usage_source": "shield_timeout", + } + assert _derive_delivery_billing_status(meta) == ("none", "partial") + + def test_zero_chunk_cancel_yields_none_none(self): + meta = { + "cancellation_indicator": "client_disconnect", + "cancel_phase": "before_upstream", + "usage_source": "no_completion", + } + assert _derive_delivery_billing_status(meta) == ("none", "none") + + def test_streaming_cancel_with_zero_bytes_yields_none_none(self): + # Edge: phase=streaming_partial but bytes_delivered=0 — the cancel + # fired so early in the stream that no chunks made it out. + # Counts as no delivery. + meta = { + "cancellation_indicator": "client_disconnect", + "cancel_phase": "streaming_partial", + "bytes_delivered_to_client": 0, + } + assert _derive_delivery_billing_status(meta) == ("none", "none") + + def test_cancel_upstream_error_during_shield_yields_none_partial(self): + # Cancel fired, shield-and-wait kicked in, but upstream errored + # during the shield window. cancel_finalize tags it + # usage_source="no_completion" / upstream_completed=False. + # We bill prompt-only (none/partial), NOT (none/none), because + # we still incurred the upstream request — see semantic mapping + # docstring in the derivation helper. + # NOTE: the derivation here lands on (none, none) under the + # current rule because usage_source != shield_timeout. This + # test pins that behaviour. If we later want to separate + # "shield-window upstream error" from "before dispatch", we'd + # add a new CancelUsageSource value like "upstream_errored". + meta = { + "cancellation_indicator": "client_disconnect", + "cancel_phase": "during_upstream", + "upstream_completed": False, + "usage_source": "no_completion", + } + assert _derive_delivery_billing_status(meta) == ("none", "none") + + +class TestMaterializeDeliveryBillingStatus: + """``_get_spend_logs_metadata`` must call the derivation helper and + write the result into the returned dict, so SQL dashboards can + filter on ``metadata::jsonb->>'delivery_status'`` directly.""" + + def test_normal_success_materializes_full_full(self): + out = _get_spend_logs_metadata(metadata={"user_api_key": "sk-test"}) + assert out["delivery_status"] == "full" + assert out["billing_status"] == "full" + + def test_streaming_cancel_materializes_partial_partial(self): + out = _get_spend_logs_metadata( + metadata={ + "cancellation_indicator": "client_disconnect", + "cancel_phase": "streaming_partial", + "bytes_delivered_to_client": 4096, + "usage_source": "tokenizer_estimate", + } + ) + assert out["delivery_status"] == "partial" + assert out["billing_status"] == "partial" + + def test_failure_materializes_none_none(self): + out = _get_spend_logs_metadata(metadata={"status": "failure"}) + assert out["delivery_status"] == "none" + assert out["billing_status"] == "none" + + def test_none_metadata_branch_includes_derived_fields(self): + # The early-failure path that passes metadata=None still must + # include both derived keys (so JSON serializers don't silently + # drop them and downstream dashboards see them as NULL rather + # than missing). + out = _get_spend_logs_metadata(metadata=None) + assert "delivery_status" in out + assert "billing_status" in out + + +class TestBinaryStatusReader: + """``_get_status_for_spend_log`` returns only "success" or + "failure". Cancellation taxonomy lives in metadata markers, not + this column.""" + + def test_success(self): + assert _get_status_for_spend_log({}) == "success" + assert _get_status_for_spend_log({"status": "success"}) == "success" + + def test_failure(self): + assert _get_status_for_spend_log({"status": "failure"}) == "failure" + + def test_cancel_with_markers_stays_success(self): + # The hot path under the new taxonomy: cancelled row carries + # status="success" + cancellation_indicator marker. The reader + # must return "success", letting the marker drive dashboard + # filtering. + assert ( + _get_status_for_spend_log( + { + "status": "success", + "cancellation_indicator": "client_disconnect", + } + ) + == "success" + ) diff --git a/tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py b/tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py index 4bcabfe853a8..1a9978eff070 100644 --- a/tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py +++ b/tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py @@ -377,6 +377,19 @@ def test_ui_view_request_response_forbids_non_admin_without_db(client, monkeypat "metadata.attempted_retries", "metadata.max_retries", "metadata.eval_information", + # Cancel taxonomy markers + derived dimensions are always present + # in SpendLogsMetadata (None for non-cancelled rows). These tests + # build their expected dicts from a hand-rolled fixture that + # pre-dates the cancellation fields — ignore them here, the + # dedicated test_spend_logs_cancellation_metadata.py file covers + # their propagation in detail. + "metadata.cancellation_indicator", + "metadata.cancel_phase", + "metadata.bytes_delivered_to_client", + "metadata.upstream_completed", + "metadata.usage_source", + "metadata.delivery_status", + "metadata.billing_status", ] MODEL_LIST = [ @@ -1410,6 +1423,10 @@ async def test_ui_view_spend_logs_unauthorized(client): @pytest.mark.asyncio async def test_ui_view_spend_logs_with_status(client, monkeypatch): + # The status_filter is binary: success | failure. Cancellations + # are NOT a top-level filter — they carry status='success' plus a + # metadata marker, surface in the UI as a row-level badge, and + # remain visible under the "Success" filter. mock_spend_logs = [ { "id": "log1", @@ -1421,6 +1438,7 @@ async def test_ui_view_spend_logs_with_status(client, monkeypatch): "startTime": datetime.datetime.now(timezone.utc).isoformat(), "model": "gpt-3.5-turbo", "status": "success", + "metadata": {}, }, { "id": "log2", @@ -1432,14 +1450,33 @@ async def test_ui_view_spend_logs_with_status(client, monkeypatch): "startTime": datetime.datetime.now(timezone.utc).isoformat(), "model": "gpt-4", "status": "failure", + "metadata": {}, + }, + # Cancellation row: status='success' + marker. Stays in the + # Success bucket. + { + "id": "log3", + "request_id": "req3", + "api_key": "sk-test-key", + "user": "test_user_3", + "team_id": "team1", + "spend": 0.02, + "startTime": datetime.datetime.now(timezone.utc).isoformat(), + "model": "gpt-4", + "status": "success", + "metadata": {"cancellation_indicator": "client_disconnect"}, }, ] def filter_by_status(where): + # "success" filter shape: + # {"OR": [{"status": {"equals": "success"}}, {"status": None}]} + # "failure" filter shape: + # {"status": {"equals": "failure"}} if "OR" in where: - return [mock_spend_logs[0]] # success + return [row for row in mock_spend_logs if row["status"] == "success"] if "status" in where and where["status"].get("equals") == "failure": - return [mock_spend_logs[1]] + return [row for row in mock_spend_logs if row["status"] == "failure"] return mock_spend_logs monkeypatch.setattr( @@ -1453,7 +1490,8 @@ def filter_by_status(where): user_role=LitellmUserRoles.PROXY_ADMIN ) try: - # Test success status + # success → log1 + log3 (both status='success'; the + # cancellation row stays in the Success bucket). response = client.get( "/spend/logs/ui", params={ @@ -1463,14 +1501,13 @@ def filter_by_status(where): }, headers={"Authorization": "Bearer sk-test"}, ) - assert response.status_code == 200 data = response.json() - assert data["total"] == 1 - assert len(data["data"]) == 1 - assert data["data"][0]["status"] == "success" + assert data["total"] == 2 + request_ids = {row["request_id"] for row in data["data"]} + assert request_ids == {"req1", "req3"} - # Test failure status + # failure → log2 only. response = client.get( "/spend/logs/ui", params={ @@ -1480,7 +1517,6 @@ def filter_by_status(where): }, headers={"Authorization": "Bearer sk-test"}, ) - assert response.status_code == 200 data = response.json() assert data["total"] == 1 diff --git a/tests/test_litellm/proxy/spend_tracking/test_status_filter_condition.py b/tests/test_litellm/proxy/spend_tracking/test_status_filter_condition.py new file mode 100644 index 000000000000..329a358cc214 --- /dev/null +++ b/tests/test_litellm/proxy/spend_tracking/test_status_filter_condition.py @@ -0,0 +1,71 @@ +""" +Tests for ``_build_status_filter_condition`` — the Prisma WHERE-clause +builder behind the SpendLogs UI ``status_filter`` query param. + +The status filter is genuinely binary: ``success | failure``. +Cancellations are NOT a top-level filter category — they carry +``status='success'`` plus a ``cancellation_indicator`` marker, and +surface in the UI as a row-level amber badge. A user who needs to +query cancelled rows specifically filters on the marker directly in +SQL. + +These assertions guard against: +1. Re-introducing JSON-path filters that Prisma's Python client + doesn't support (the cause of the 500 we hit on the first + refactor attempt). +2. Silently accepting unknown filter values (e.g. ``success_partial`` + from a stale UI bundle) instead of falling through to "no filter". +""" + +import os +import sys + +sys.path.insert(0, os.path.abspath("../../../..")) + +from litellm.proxy.spend_tracking.spend_management_endpoints import ( + _build_status_filter_condition, +) + + +class TestStatusFilterCondition: + def test_none_returns_empty(self): + assert _build_status_filter_condition(None) == {} + + def test_unknown_value_returns_empty(self): + # Defensive: an unrecognised string from a stale UI build or a + # typo'd curl shouldn't accidentally match everything or fall + # through to a generic `status=` clause that returns + # zero rows silently. + assert _build_status_filter_condition("nonsense") == {} + + def test_success_matches_status_column_with_null_tolerance(self): + # status='success' OR NULL (legacy rows pre-date the status + # column being populated). Critically: no JSON-path filter on + # metadata — the Prisma Python client doesn't support that + # form and 500s at request time. + cond = _build_status_filter_condition("success") + assert cond == { + "OR": [ + {"status": {"equals": "success"}}, + {"status": None}, + ] + } + + def test_failure_matches_status_column(self): + assert _build_status_filter_condition("failure") == { + "status": {"equals": "failure"} + } + + def test_cancel_is_not_a_top_level_filter(self): + # The "Cancel" value is intentionally NOT a recognised filter + # — cancellations live inside the Success bucket and are + # surfaced by the UI as a row-level badge, not a filter tab. + # An incoming `status_filter=cancel` falls through to the + # empty-dict default (no filter applied). + assert _build_status_filter_condition("cancel") == {} + + def test_success_partial_is_not_recognised(self): + # The legacy three-valued taxonomy never shipped; the + # success_partial sentinel must not be silently accepted as a + # valid filter value (it would mask typos and stale UI bundles). + assert _build_status_filter_condition("success_partial") == {} diff --git a/ui/litellm-dashboard/src/components/view_logs/columns.tsx b/ui/litellm-dashboard/src/components/view_logs/columns.tsx index 27325217d4e1..e078ff63eac2 100644 --- a/ui/litellm-dashboard/src/components/view_logs/columns.tsx +++ b/ui/litellm-dashboard/src/components/view_logs/columns.tsx @@ -173,16 +173,42 @@ export const createColumns = (sortProps?: LogsSortProps): ColumnDef[] header: "Status", accessorKey: "metadata.status", cell: (info: any) => { - const status = info.getValue() || "Success"; - const isSuccess = status.toLowerCase() !== "failure"; + // Status taxonomy: + // - "failure" → red badge ("Failure {errCode}") + // - cancelled → amber badge ("Cancel 499"). A row is cancelled + // when metadata.cancellation_indicator is set; + // cancels carry status="success" — the marker is + // the source of truth. + // - otherwise → green "Success" badge + // Default "success" so historic rows that pre-date the + // metadata.status field continue rendering green. + const raw = String(info.getValue() || "success").toLowerCase(); + const meta = info.row.original?.metadata || {}; + const errInfo = meta?.error_information || {}; + const errCode = errInfo?.error_code; + const cancelInd = meta?.cancellation_indicator; + + const isFailure = raw === "failure"; + const isCancel = !isFailure && !!cancelInd; + + let label: string; + let className: string; + if (isFailure) { + label = errCode && errCode !== "None" ? `Failure ${errCode}` : "Failure"; + className = "bg-red-100 text-red-800"; + } else if (isCancel) { + label = errCode && errCode !== "None" ? `Cancel ${errCode}` : "Cancel"; + className = "bg-amber-100 text-amber-800"; + } else { + label = "Success"; + className = "bg-green-100 text-green-800"; + } return ( - {isSuccess ? "Success" : "Failure"} + {label} ); }, diff --git a/ui/litellm-dashboard/src/components/view_logs/filter_options.ts b/ui/litellm-dashboard/src/components/view_logs/filter_options.ts index 59ac58b67457..e7d7c4d561c7 100644 --- a/ui/litellm-dashboard/src/components/view_logs/filter_options.ts +++ b/ui/litellm-dashboard/src/components/view_logs/filter_options.ts @@ -21,6 +21,12 @@ export function getLogFilterOptions(accessToken: string): FilterOption[] { { label: "Success", value: "success" }, { label: "Failure", value: "failure" }, ], + // Cancellations carry status="success" + a metadata marker — + // they show up under "Success" with an amber "Cancel 499" + // badge in the row (see columns.tsx). No separate filter + // entry: the dashboard's status taxonomy is genuinely binary, + // and cancellation is a row-level annotation not a top-level + // category. }, { name: "Model",