feat(cancel): billing accuracy + orthogonal delivery/billing status taxonomy

CLAUDE.md

@@ -285,6 +285,34 @@ Tests live in `tests/test_litellm/` (unit), `tests/llm_translation/`
 (per-provider integration), `tests/proxy_unit_tests/` (proxy), and
 `tests/load_tests/`. Full-stack scenarios go in `e2e/cases/`.
 
+- **No theater tests — mock the boundary, not the unit under test.**
+  A test that mocks out the very function or module it is supposed to
+  exercise, then asserts "the mock was called" or "the mock's return
+  value flowed through," verifies nothing real. It looks green in CI
+  but cannot catch a bug in the production code it ostensibly covers.
+  Classic anti-patterns to refuse:
+  - `MyModule.foo = MagicMock(return_value=fake); … ; assert
+    other.bar.call_args.kwargs["x"] is fake` — you tested that
+    Python can pass references.
+  - `obj.handler = AsyncMock(); … ; assert obj.handler.called` —
+    you tested that your code calls a method you told it to call.
+  - All-MagicMock stubs of objects whose real interface might drift —
+    the test passes against a contract that does not exist in prod.
+  - Asserting a Literal/Enum value at runtime (`assert
+    MyLiteral == "value"`) — that is mypy's job, not the test's.
+
+  The fix is to mock the **process / network boundary** (httpx
+  transport, DB cursor, fanout queue, OS clock) and let the unit
+  under test run against real inputs producing real outputs. For
+  observable side effects (a callback firing, a row written), use a
+  **spy fake** — a real class that captures the actual arguments it
+  receives — and assert on the captured *data*, not on `.called`.
+  Example: instead of mocking `stream_chunk_builder`, pass real
+  `ModelResponseStream` chunks in and assert on the reassembled
+  response's `usage.completion_tokens`. The test now catches a
+  cursor-handling regression in `stream_chunk_builder` itself; the
+  theater version cannot.
+
 - **Write assertions from the spec, not the impl.** For new features,
   the `e2e/cases/NN_*.md` runbook IS the spec — write it before the
   fixture and the impl. For bug fixes, the issue's repro steps are the

UPSTREAM_PR_QUEUE.md

@@ -46,6 +46,8 @@ Status legend:
 | PR #52 (Wave 5c) | READY | `fix(anthropic): extend transform_to_anthropic_error to cover broader status code set` | Smallest diff in the queue — builds on upstream's existing helper. Good warm-up after #29748. |
 | PR #60 + #67 (Wave 6e + Layer 1 fix) | READY | `fix(proxy): record TTFT on passthrough streaming path + apply per-deployment overrides before fast_path SSE short-circuit` | The fast_path piece (#67) is a regression introduced by upstream PR #28289 — file directly with that PR's commit referenced. Pair with passthrough TTFT (#60). |
 | PR #48 (Wave 3 small UI fixes) | READY | Three UI fixes: Gemini provider `api_base` field on credential form; credential-form reset on close; Mode badge rendering on model_list | All independent. File as a single PR with 3 commits since they all touch the credential form area. |
+| fix/billing-accuracy-phase-1 (Phase 1 — first commit) | READY | `fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives` | Independent, smallest-blast-radius bug in the cancel-billing series. `stream_chunk_builder_utils.py` cursor=1 escape valve. Pure Tier C. Can be filed BEFORE Phase 3 dogfood completes. |
+| fix/billing-accuracy-phase-1 (Phase 1 — black-hole catch) | READY | `feat(cancel): catch asyncio.CancelledError in streaming + non-stream proxy paths` | The Phase 1 "BaseException slipping through `except Exception`" hole. SpendLogs row + Langfuse trace closure for cancelled requests. Tier C; mechanism only — the billing strategy stays in our metadata derivation (see Phase 3 row below). |
 
 ### Tier D — issue first, then mechanism PR
 
@@ -59,6 +61,7 @@ or `ship/<pin>` config.
 | PR #54 (Wave 5a) | ISSUE-FIRST | Add `cache_ttl` label to `litellm_input_cache_creation_tokens_metric` (Anthropic 5m vs 1h split) | Mechanism = label; opinion = whether `prompt_cache_*_tokens_metric` should be dropped. File issue documenting why per-TTL bucketing matters for cost attribution. |
 | PR #53 (Wave 5b) | ISSUE-FIRST | Per-deployment `returned_model_name` override | Mechanism = the field on `model_list[].litellm_params`. Existing upstream `_override_openai_response_model` is per-call, not per-deployment. Issue should reference this gap. |
 | PR #59 (Wave 6d) | ISSUE-FIRST | Two related opt-ins: thinking-signature retry flag; `anthropic_beta_overrides` per-deployment | Combined Anthropic-features PR. The `anthropic_beta_overrides` part needs the Bedrock-gateway use case in the issue. |
+| fix/billing-accuracy-phase-1 (Phase 3 — cancel taxonomy) | ISSUE-FIRST | Cancel-billing taxonomy: orthogonal `delivery_status` + `billing_status` in metadata, derived from existing cancel markers | **Upstream-friendly: StandardLoggingPayloadStatus stays binary (matches upstream)**; all new fields are additive Optional metadata. Issue should propose the 7-row semantic mapping (normal / streaming-partial / shield-success / shield-timeout / zero-chunk / cancel-upstream-error / failure) + the derivation helper as the single source of truth. Dogfood for one internal release before filing the issue. Tier D because we're shipping our derivation as the default policy. See `e2e/cases/data/26-33_*.sh` + the spend_tracking_utils derivation function for spec. |
 
 ### Dependent PRs (file after a prerequisite lands)
 
@@ -89,4 +92,4 @@ For reference only. Do NOT submit:
 - Last reviewed / refreshed: keep a `_Last reviewed: YYYY-MM-DD_` note
   at the very bottom so quarterly reviews can spot stale entries
 
-_Last reviewed: 2026-06-05 (cut at v1.87.0 bump completion; first upstream PR #29748 OPEN)._
+_Last reviewed: 2026-06-10 (added fix/billing-accuracy-phase-1 candidates after Phase 3 refactor landed + real-Anthropic case 33 verified)._

e2e/_config/docker-compose.yml

@@ -76,6 +76,12 @@ services:
     environment:
       LITELLM_MASTER_KEY: "sk-e2e-test"
       LITELLM_LOG: "WARNING"
+      # Short shield timeout for e2e case 30 (shield_timeout). The
+      # production default is 60s — appropriate for real workloads but
+      # too long to drive in CI. 2s lets case 30 deterministically
+      # trigger the asyncio.TimeoutError branch in
+      # finalize_non_stream_cancel without blocking the suite.
+      LITELLM_CANCEL_SHIELD_TIMEOUT_S: "2"
       # Match the user's production env: force using the locally-bundled
       # model_prices JSON instead of fetching the latest from GitHub.
       LITELLM_LOCAL_MODEL_COST_MAP: "True"

e2e/_config/mock_provider.py

@@ -944,6 +944,13 @@ def _handle_anthropic(self):
                 STATS.request("anthropic", "stream", 499)
             return
 
+        # TTFT delay BEFORE returning non-stream response. Needed by
+        # e2e/cases/27 (non-stream cancel + shield-and-wait) — without
+        # this, the mock answers in ~10ms and a client `--max-time 1`
+        # never has a chance to fire the cancel signal. Mirrors the
+        # stream path's TTFT semantics above.
+        if p["ttft_ms"]:
+            time.sleep(p["ttft_ms"] / 1000.0)
         out_t = max(1, p["full_chars"] // 4)
         resp = {
             "id": msg_id,

e2e/cases/26_cancel_billing_partial.md

@@ -0,0 +1,159 @@
+# Case 26 — Cancel billing: stream + non-stream cancel produces partial-delivery SpendLogs row
+
+## Goal
+
+End-to-end verification of the Phase 1/2/3 billing-accuracy work
+(commits `80127194b5…4dcdd73f1a`):
+
+1. **Streaming cancel** with chunks received → SpendLogs row exists with
+   `status="success"`, `metadata.cancellation_indicator="client_disconnect"`,
+   derived `delivery_status="partial"` + `billing_status="partial"`, and
+   real `spend > 0` (the chunks were reassembled and priced through the
+   regular cost calculator).
+
+2. **Non-streaming cancel** during upstream wait → shield-and-wait
+   succeeds (mock returns quickly), SpendLogs row records
+   `usage_source="upstream_completed_after_cancel"` with `delivery_status="none"`
+   (nothing reached the client) and `billing_status="full"` (real
+   upstream usage).
+
+3. **Zero-chunk cancel** (client gives up before first byte) →
+   SpendLogs row exists with `delivery_status="none"` (or "partial"
+   if the mock managed to emit SSE openers before the cancel),
+   `billing_status="none"`.
+
+Before Phase 1 all three of these were black holes — no SpendLogs row
+at all, an orphaned Langfuse trace, and the upstream provider had
+already billed us. This case proves the bleed is plugged.
+
+## Status taxonomy
+
+The earlier iteration of Phase 1/2 used a tri-valued
+`status="success_partial"` sentinel for these rows. Phase 3
+(2026-06-10) refactored that away: the top-level `status` is now
+binary `"success" | "failure"` (aligned with upstream LiteLLM), and the
+cancellation taxonomy lives in metadata as two orthogonal dimensions:
+
+- `delivery_status: full | partial | none` — what reached the client
+- `billing_status:  full | partial | none` — what we actually charged
+
+Both are derived from the existing 5 cancel markers
+(`cancellation_indicator`, `cancel_phase`, `usage_source`,
+`upstream_completed`, `bytes_delivered_to_client`) by
+`spend_tracking_utils._derive_delivery_billing_status` — single source
+of truth, never written directly.
+
+## Tier
+
+`mock-only` — uses the in-network `mock-anthropic` deployment with
+`X-Mock-TTFT-Ms` + `X-Mock-Chunks` headers to deterministically
+control stream pacing without real provider cost.
+
+## What the fixture does
+
+`e2e/cases/data/26_cancel_billing_partial.sh` runs three probes and
+verifies the resulting SpendLogs rows:
+
+| Probe | Request | Cancel via | Expected `delivery_status` | Expected `billing_status` | Expected `usage_source` |
+|---|---|---|---|---|---|
+| C1 | streaming `mock-anthropic`, TTFT=2000ms, ~50 chunks | curl `--max-time 4` (cancels after some chunks flushed) | `partial` | `partial` | `tokenizer_estimate` or `upstream_truth` |
+| C2 | streaming `mock-anthropic`, more chunks accumulated | curl `--max-time 4` | `partial` | `partial` | as C1 |
+| C3 | streaming `mock-anthropic`, TTFT=10000ms (cancel before first chunk of content) | curl `--max-time 1` | `partial` or `none` (depends on mock SSE-opener timing) | `none` (no real content reassembled) | `no_completion` |
+
+Each probe uses a unique sentinel `end_user` value (`case26-<nanos>-c<N>`)
+so the SpendLogs row is unambiguously identifiable.
+
+For each probe the fixture polls `LiteLLM_SpendLogs` (up to 20s — the
+async spend writer can lag, and the non-stream shield itself can
+intentionally hold the request open for several seconds) and asserts:
+
+- Exactly one row exists for that sentinel
+- `status == 'success'` (NOT failure — cancel is not a system failure)
+- `metadata->>'cancellation_indicator' == 'client_disconnect'`
+- `metadata->>'cancel_phase'` is one of the expected lifecycle values
+- `metadata->>'delivery_status'` and `metadata->>'billing_status'` match
+  the per-probe expectation above
+- `completion_tokens > 0` for C1/C2 (chunks reassembled) — proves the
+  cost calculator ran on the partial response
+
+## Why this is hard to test without a real proxy + DB
+
+The cancel path traverses several independent pieces of plumbing:
+
+1. **asyncio.CancelledError propagation** through the FastAPI request
+   task and the streaming generator. Unit tests can fake this but
+   the actual uvicorn event-loop behavior is what production sees.
+
+2. **The shield+wait timer** in `finalize_non_stream_cancel` only
+   makes sense when there's a real upstream task to await. Mocked
+   tests can prove the shield logic; only a real HTTP roundtrip can
+   prove the FastAPI request-cancel signal actually reaches the
+   handler in time.
+
+3. **The metadata bridge** from `logging_obj.model_call_details` →
+   `request_data.litellm_params.metadata` → `_get_spend_logs_metadata`
+   → derivation helper → `LiteLLM_SpendLogs.metadata` JSON column.
+   Several hops; any one break and the markers (or the derived
+   delivery/billing fields) vanish silently.
+
+Unit tests cover each piece. This case proves the chain is wired.
+
+## Reproducing manually (for debugging a future failure)
+
+```bash
+# Start the proxy with the mock provider
+e2e/tools/proxy start --with-mock
+
+# Probe C1: streaming, cancel mid-stream
+USER_SENTINEL="case26-debug-$(date +%s)"
+timeout 4 curl -N -X POST http://localhost:4011/v1/chat/completions \
+  -H "Authorization: Bearer sk-e2e-test" \
+  -H "Content-Type: application/json" \
+  -H "X-Mock-TTFT-Ms: 500" \
+  -H "X-Mock-Chunks: 50" \
+  -H "X-Mock-TPS: 5" \
+  -d "{
+    \"model\":\"mock-anthropic\",
+    \"user\":\"$USER_SENTINEL\",
+    \"messages\":[{\"role\":\"user\",\"content\":\"Tell me about cancellation handling\"}],
+    \"stream\":true,
+    \"max_tokens\":2000
+  }"
+
+# Verify in DB
+docker exec litellm-e2e-db psql -U litellm -d litellm -c "
+SELECT request_id, status, spend, completion_tokens,
+       metadata::jsonb->>'cancellation_indicator',
+       metadata::jsonb->>'cancel_phase',
+       metadata::jsonb->>'delivery_status',
+       metadata::jsonb->>'billing_status',
+       metadata::jsonb->>'usage_source'
+FROM \"LiteLLM_SpendLogs\"
+WHERE end_user = '$USER_SENTINEL'
+ORDER BY \"startTime\" DESC LIMIT 1;
+"
+```
+
+Expect a row with `status='success'`, `delivery_status='partial'`,
+`billing_status='partial'`, `cancel_phase='streaming_partial'`, and
+`completion_tokens > 0`.
+
+## Cross-references
+
+- `litellm/litellm_core_utils/cancel_finalize.py` — catch + dispatch
+- `litellm/litellm_core_utils/cancel_billing.py` — partial cost + metadata bridge
+- `litellm/proxy/spend_tracking/spend_tracking_utils.py:_derive_delivery_billing_status`
+  — single source of truth for the orthogonal taxonomy
+- `litellm/proxy/hooks/proxy_track_cost_callback.py` — failure-fallback
+  cost compute (was hardcoded 0.0; now uses `compute_prompt_only_cost`)
+- `tests/test_litellm/litellm_core_utils/test_cancel_finalize.py` — unit tests
+- `tests/test_litellm/litellm_core_utils/test_cancel_billing.py` — unit tests
+- `tests/test_litellm/proxy/spend_tracking/test_spend_logs_cancellation_metadata.py`
+  — derivation + materialisation tests
+
+## Tier classification
+
+C (universal bug fix — every LiteLLM proxy operator hits the
+cancellation black hole) + D (the orthogonal `delivery_status` /
+`billing_status` derivation is a billing-policy choice we want carried
+in our fork until upstream agrees on the semantic mapping).

e2e/cases/33_real_anthropic_cancel.md

@@ -0,0 +1,111 @@
+# Case 33 — Real Anthropic streaming cancel → partial-delivery SpendLogs row
+
+## Goal
+
+End-to-end verification of the Phase 1/2/3 cancel-billing chain
+against the *real* Anthropic API, not the in-network mock. Mock-based
+cases 26-32 prove the plumbing wires together correctly; case 33
+proves it survives contact with genuine Anthropic streaming protocol
+quirks and real cost-map pricing.
+
+## Why this can't be a mock case
+
+The mock provider emits a fixed SSE shape:
+`message_start → content_block_start → content_block_delta × N → content_block_stop → message_delta → message_stop`
+
+Real Anthropic streams can include:
+- Multiple parallel `content_block_*` blocks (thinking + text + tool_use interleaved)
+- `signature_delta` events for thinking signatures
+- Per-block `cache_control` markers in message_start
+- Service-tier tags that affect `usage_object` shape
+- Cache creation token details (`ephemeral_5m_input_tokens` vs
+  `ephemeral_1h_input_tokens`)
+
+Any of these can break the cursor=1 reset heuristic (PR #1) or the
+chunk reassembly path (PR #2) in subtle ways that the mock doesn't
+reproduce.
+
+## What the fixture exercises
+
+`e2e/cases/data/33_real_anthropic_cancel.sh`:
+
+1. POST a streaming `/v1/chat/completions` request to the proxy with
+   `model=claude-sonnet-cache` (mapped to real
+   `anthropic/claude-sonnet-4-6` in the rendered config) and a ~1500-
+   token prompt + `max_tokens=2000`.
+2. Kill the client at 3 seconds via `timeout 3 curl ...`. Anthropic's
+   thinking-model long-tail makes it very likely that
+   `message_delta` won't have arrived by then — exercises the cursor
+   reset path on a real chunk stream.
+3. Poll `LiteLLM_SpendLogs` for the row keyed by the per-run sentinel
+   in the OpenAI `user` field (lands in the `end_user` column).
+4. Assert:
+   - `status = 'success'` (Phase 3 taxonomy: cancel is not a system failure)
+   - `metadata.cancellation_indicator = 'client_disconnect'`
+   - `metadata.cancel_phase = 'streaming_partial'`
+   - `metadata.delivery_status = 'partial'` (derived — chunks reached client)
+   - `metadata.billing_status = 'partial'` (derived — partial response was priced)
+   - `completion_tokens > 1` — proves PR #1's cursor reset triggered
+     against the real Anthropic stream and the partial response was
+     reassembled with a real-looking token count, not the cursor=1
+     placeholder
+   - `prompt_tokens > 0` — Anthropic's `message_start.input_tokens`
+     reached the row, not the local tokenizer fallback
+   - `spend > 0` — the cost map lookup found
+     `anthropic/claude-sonnet-*` and the cost callback ran on the
+     partial response
+
+## What a failure means
+
+| Failure mode | Likely cause |
+|---|---|
+| `status` is `failure` not `success` | Cancel was misclassified as failure — recheck `proxy_track_cost_callback._is_cancel` detection of CancelledError |
+| `cancellation_indicator` empty | Cancel markers not propagating — recheck `enrich_request_metadata_with_cancel_markers` for new endpoint variants |
+| `delivery_status` is `none` despite bytes streamed | `bytes_delivered_to_client` not populated — check `async_streaming_data_generator`'s `chunks_yielded` counter and `mark_logging_obj_cancelled` call sites |
+| `billing_status` is `none` despite completion_tokens > 0 | Derivation rule mismatched — check `_derive_delivery_billing_status` against the 7-row semantic mapping |
+| `completion_tokens = 1` | Cursor=1 reset isn't firing for real Anthropic; `saw_non_cursor_completion` heuristic broken |
+| `completion_tokens = 0` | `stream_chunk_builder` failed to reassemble — chunks list empty (could be `FallbackStreamWrapper` not accumulating, or Logging.streaming_chunks ref drift) |
+| `prompt_tokens = 0` | Anthropic `message_start.usage` not reaching the cost calc; check the metadata bridge |
+| `spend = 0` | Either the model id leaked through to a cost-map miss, or the cost calc fired before reassembly |
+
+## Tier
+
+`real` — requires `ANTHROPIC_API_KEY` set in `e2e/.env`. Costs roughly
+$0.01 per run (Anthropic Sonnet pricing: 1500 input × $3/M + ~200
+output × $15/M ≈ $0.0075).
+
+When `ANTHROPIC_API_KEY` is unset the fixture exits 77 (SKIP) so it's
+safe in `--mock-only` runs.
+
+## How to run
+
+```bash
+# 1. Make sure ANTHROPIC_API_KEY is in e2e/.env
+# 2. Start the proxy WITHOUT --with-mock (real provider mode)
+e2e/tools/proxy stop
+e2e/tools/proxy start
+
+# 3. Run case 33 directly (or the full suite)
+bash e2e/cases/data/33_real_anthropic_cancel.sh
+# OR
+e2e/tools/run-all-cases     # runs everything; case 33 sits at the end
+```
+
+## Cross-references
+
+- `litellm/litellm_core_utils/streaming_chunk_builder_utils.py` —
+  cursor=1 reset (PR #1)
+- `litellm/litellm_core_utils/cancel_finalize.py` — catch + dispatch
+- `litellm/litellm_core_utils/cancel_billing.py` — markers bridge
+- `litellm/proxy/common_request_processing.py` — `/v1/messages` catch
+  + non-stream disconnect watcher (Phase 2)
+- `litellm/proxy/spend_tracking/spend_tracking_utils.py:_derive_delivery_billing_status`
+  — single source of truth for the orthogonal taxonomy (Phase 3)
+- `e2e/cases/26_cancel_billing_partial.md` — mock-only companion that
+  covers the same three sub-scenarios
+
+## Tier classification
+
+Operates against a real provider for the cancel-billing PRs (Tier C
+universal bug fix + Tier D opinionated billing mechanism). Catches
+real-provider regressions the mock suite cannot.