fix(anthropic): buffer streamed responses carrying server-fulfilled tools so retrieval tool calls never reach the client

[mateo-berri]

TLDR

Problem this solves:

• Headroom compresses a streaming /v1/messages turn and injects headroom_retrieve
• The model's headroom_retrieve tool_use streams straight to the client
• Claude Code fails the turn with No such tool available: headroom_retrieve
• The server-side retrieval rerun lands after message_stop, where no client reads
• The user pays for that rerun and never sees its answer

How it solves it:

• Callbacks declare the tools they answer server-side via server_fulfilled_tool_names
• Streams carrying such a tool are buffered until the agentic loop finishes
• ping events every 15s keep client stall watchdogs quiet meanwhile
• Only the follow-up answer streams; the buffered tool_use message is dropped
• With no interception fired, the buffer replays byte-identical
• Retrieval leaving a server-fulfilled tool_use buffered ends on an SSE error, never the tool call
• Requests without such a tool keep true live streaming
• Ping-only disconnects release the budget reservation instead of holding worst case
• The router forwards that leading ping live and passes the retrieval error through instead of falling back to another deployment
• The withheld-output flag rides through the router, the streaming wrappers, and the stream cache writer, so disconnect accounting holds in production, not just on the bare iterator

User Flow

Before: the flow fails at step 4, where the session errors on a tool the user never configured

1. A Claude Code user points ANTHROPIC_BASE_URL at a LiteLLM gateway that has the Headroom compression guardrail enabled; every turn is POST https://litellm-domain/v1/messages with "stream": true
2. Once the conversation grows, the gateway compresses older history into markers; turns keep streaming normally
3. On a later turn the assistant says it will recover compressed context, and the stream ends with stop_reason: "tool_use" naming headroom_retrieve
4. The session shows Error: No such tool available: headroom_retrieve, the turn is wasted, and the model re-derives the lost context by re-reading files
5. The user is also billed for a follow-up model call whose corrected answer never appears anywhere in their session

After: the flow succeeds at step 3 with one clean streamed answer

1. A Claude Code user points ANTHROPIC_BASE_URL at a LiteLLM gateway that has the Headroom compression guardrail enabled; every turn is POST https://litellm-domain/v1/messages with "stream": true
2. Once the conversation grows, the gateway compresses older history into markers; turns keep streaming normally
3. On a later turn the connection carries only ping events for a few seconds while the gateway recovers the compressed content server-side
4. The reply arrives as a single normal message that already incorporates the recovered context; no unknown-tool error and nothing billed that the user cannot see

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Real Claude Code 2.1.246 driven interactively in tmux against a local proxy at each commit (2 uvicorn workers, no DB), real Anthropic API (claude-haiku-4-5-20251001), and a stub standing in for the private Headroom compression service. The stub answers POST /v1/compress by replacing unprotected user and assistant rows with [COMPRESSED by Headroom. Recover the original text by calling the headroom_retrieve tool with hash=<sha1[:24]>] (same row count) and GET /v1/retrieve/<hash> with {"original_content": <the original row text>}

Proxy config, identical on both sides:

model_list:
  - model_name: claude-*
    litellm_params:
      model: anthropic/claude-*
      api_key: os.environ/ANTHROPIC_API_KEY

guardrails:
  - guardrail_name: headroom-compress
    litellm_params:
      guardrail: headroom
      mode: pre_call
      default_on: true
      api_base: http://127.0.0.1:<stub-port>

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Steps, identical on both sides:

1. python litellm/proxy/proxy_cli.py --config config.yaml --port <proxy-port> --num_workers 2 --detailed_debug
2. In a fresh HOME and an empty project directory: ANTHROPIC_BASE_URL=http://127.0.0.1:<proxy-port> ANTHROPIC_AUTH_TOKEN=<master-key> claude --model claude-haiku-4-5-20251001
3. Turn 1: For later in this conversation: the launch code is OCTOPUS-7734.
4. Turn 2: What is the launch code? Reply with only the code. If the earlier message shows a compression marker instead of the original text, call the headroom_retrieve tool with its hash to recover it first.

On turn 2 the guardrail compresses the first user row (the stub log shows compress: 1 rows in, replaced 1 followed by GET /v1/retrieve/<hash> on both sides), so the model asks for headroom_retrieve and the proxy fulfils it server-side either way. What differs is what the user sees

Before (04818a3)

Claude Code screen after turn 2:

❯ What is the launch code? Reply with only the code. If the earlier message shows a compression marker instead of the original text, call the headroom_retrieve tool with its hash to recover it first.

  Thought for 2s (ctrl+o to expand)
  ⎿  Error: No such tool available: headroom_retrieve

⏺ OCTOPUS-7734

  Thought for 1s (ctrl+o to expand)

⏺ I'm ready to help! What would you like to work on?

✻ Cogitated for 5s · done 4:11 PM

• Client sees the server-fulfilled tool call and errors
• The answer arrives as a stray second message
• A third, unprompted assistant message follows
• Leaked tool rows persist: next stub call had 5 rows

After (8d87798)

Claude Code screen after turn 2:

❯ What is the launch code? Reply with only the code. If the earlier message shows a compression marker instead of the original text, call the headroom_retrieve tool with its hash to recover it first.

⏺ OCTOPUS-7734

✻ Cogitated for 3s · done 5:32 PM

• One clean assistant message, no error line
• No tool call visible to the client
• Turn took 4s wall clock, no visible stall
• Stub log shows the same compress then retrieve pair

Controls on the same after proxy: the same three-row conversation sent with x-headroom-bypass: true streams live (first byte at 0.80s, text deltas spread across the stream), and a request with nothing to compress (no hash, so no tool injected) streams live with the same event sequence as the base proxy

Hold-back pacing, measured on the same after proxy with a raw SSE client on a long held-back turn (a 2500 word story request declaring headroom_retrieve): headers and the first ping arrive at 0.55s, keepalive pings continue every 15s while the turn is held, and the buffered turn replays in a burst ending with message_stop at 38.4s (133 events, largest quiet gap 15.0s), so nothing in the path buffers past the ping interval

Disconnect during hold-back still bills: the same held-back request cut client-side at 8s (one ping received, zero content) lands in /spend/logs as a single row charging the key $0.00345 for the 575 prompt and 575 completion tokens the drain finished upstream after the client left

Type

🐛 Bug Fix

Caveats (if any)

Medium

• Hold-back covers streaming /v1/messages only
  • Streaming /chat/completions still leaks the tool call; fix(headroom): resolve CCR retrieval on streaming /chat/completions #35017 tracks that surface
• A turn carrying a server-fulfilled tool loses time-to-first-token
  • Nothing but ping events until the upstream message and the rerun both finish

Low

• A hook crash or a rerun that changes nothing ends the turn on an SSE error, so the user retries
• Nested reruns re-enter the same hold-back; a depth or fingerprint guard trip takes the SSE error path
• The follow-up turn shares the first turn's request id, so /spend/logs keeps one row per request (the first turn's tokens) while key and team totals include both turns
  • Pre-existing agentic rerun behavior, also on the base commit; the server-generated spend-log key in [Bug]: Reused x-litellm-call-id silently drops spend-log rows #35563 is the fix
• A proxy that also injects keepalive pings (anthropic_sse_ping_interval_seconds) shows two ping events per 15s interval during a hold-back, which clients ignore

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/a11f814e7c18431fad858ea4e7953f0f
Requested by: @mateo-berri

• 8d87798 passes /live-pr-risk

---

Note

Medium Risk
Changes Anthropic streaming, agentic reruns, router fallback, and proxy spend/disconnect accounting on a hot path; mitigated by broad tests but hold-back defers time-to-first-token and only covers streaming /v1/messages.

Overview
Fixes streaming /v1/messages turns where server-side hooks (Headroom, compression interception) inject tools like headroom_retrieve or litellm_content_retrieve—the model’s tool_use was reaching clients that cannot run them, while the corrected answer only arrived after message_stop.

Callbacks now declare server_fulfilled_tool_names; when those tools are in the request, AgenticAnthropicStreamingIterator runs in hold-back mode: provider SSE is buffered, 15s SSE ping keepalives keep the connection alive, agentic hooks run, then the client gets the follow-up stream only (dropping the buffered tool message), a byte-identical replay if nothing intercepted, or an SSE error if a server-fulfilled tool_use would still leak (hook failure or no rerun). Requests without those tools keep live streaming.

Proxy and router behavior is aligned: shared keepalive ping constants; has_buffered_provider_output propagates through stream wrappers so disconnect handling treats pings as non-delivery (budget refund on ping-only cancel) but keeps reservations when real output was withheld; the router forwards a leading ping during hold-back and does not retry on the gateway’s retrieval-failure error frame.

^{Reviewed by Cursor Bugbot for commit 8d87798. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

The PR prevents callback-owned Anthropic tool calls from escaping through streamed Messages responses and preserves the corrected response path through routing and proxy accounting.

• Declares callback-owned tools and enables selective stream hold-back only when those tools are present.
• Emits keepalive pings while draining the initial response, running agentic hooks, and waiting for follow-up chunks.
• Propagates withheld-output state through wrappers for disconnect accounting.
• Treats retrieval-failure frames as terminal gateway results rather than fallback candidates.
• Adds focused coverage for buffering, hook failures, keepalives, routing, caching, and budget reservations.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported keepalive gaps, hidden-tool replay, and terminal-error fallback behavior are addressed by the current code.

Important Files Changed

Filename	Overview
litellm/llms/anthropic/experimental_pass_through/messages/agentic_streaming_iterator.py	Implements held-back streaming, periodic keepalives, safe hook/follow-up task cleanup, and terminal handling that prevents server-owned tool calls from reaching clients.
litellm/llms/custom_httpx/llm_http_handler.py	Detects callback-owned tools in Anthropic requests and enables hold-back only for affected streams.
litellm/router.py	Forwards leading keepalives, preserves buffered-output state across fallback wrappers, and delivers retrieval failures without launching another deployment.
litellm/proxy/common_request_processing.py	Distinguishes keepalive-only delivery from withheld provider output when reconciling reservations after disconnects.
litellm/integrations/custom_logger.py	Adds the callback contract for declaring tools fulfilled by server-side agentic processing.
litellm/proxy/guardrails/guardrail_hooks/headroom/headroom.py	Declares Headroom retrieval as server-fulfilled so affected Anthropic streams use hold-back.
litellm/integrations/compression_interception/handler.py	Declares compression retrieval as server-fulfilled for the same protected streaming path.

_{Reviews (13): Last reviewed commit: "fix(caching): carry has_buffered_provide..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 99.28058% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ass_through/messages/agentic_streaming_iterator.py	98.90%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_headroom_stream_leak (8d87798) with litellm_internal_staging (ba8d8b6)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (99e1eaa) during the generation of this report, so ba8d8b6 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mateo-berri]

@greptileai

[devin-ai-integration]

@greptileai keepalives now cover the hooks, failures emit an SSE error instead of replaying the tool_use, and the docstring is trimmed

[devin-ai-integration]

Summary's second concern was legit too: f994068 now emits an SSE error instead of replaying a buffered server-fulfilled tool_use, so a hook crash cannot recreate the unknown-tool failure. A client-owned tool_use still replays verbatim.

@greptileai

[greptile-apps]

Option 1 (error on the whole message) is what's currently implemented. _buffer_holds_server_fulfilled_tool_use finds any server-fulfilled tool_use block in the buffer and _anext_held_back emits the error SSE unconditionally — it does not check whether client-owned tool_use blocks are also present.

Here's a concrete suggestion for option 2. It needs two changes:

1. Add a helper that rebuilds a filtered iterator (add after _buffer_holds_server_fulfilled_tool_use):

def _make_filtered_response_iterator(self) -> "AsyncIterator | None":
    """
    Rebuild the buffered response with server-fulfilled tool_use blocks stripped.
    Returns a FakeAnthropicMessagesStreamIterator over the filtered content,
    or None if stripping leaves no content blocks (pure server-fulfilled message).
    """
    rebuilt = self._rebuild_anthropic_response_from_sse(self._collected_bytes)
    if rebuilt is None:
        return None

    filtered_content = [
        block for block in rebuilt.get("content", [])
        if not (
            block.get("type") == "tool_use"
            and block.get("name") in self._server_fulfilled_tool_names
        )
    ]
    if not filtered_content:
        return None

    rebuilt["content"] = filtered_content
    # If every tool_use was stripped, the stop_reason is no longer "tool_use"
    if rebuilt.get("stop_reason") == "tool_use" and not any(
        b.get("type") == "tool_use" for b in filtered_content
    ):
        rebuilt["stop_reason"] = "end_turn"

    from litellm.llms.anthropic.experimental_pass_through.messages.fake_stream_iterator import (
        FakeAnthropicMessagesStreamIterator,
    )
    from litellm.types.llms.anthropic_messages.anthropic_response import AnthropicMessagesResponse

    return FakeAnthropicMessagesStreamIterator(
        response=cast(AnthropicMessagesResponse, rebuilt)
    ).__aiter__()

2. Replace the error block in _anext_held_back to try filtered replay first:

The existing block is:

        if self._buffer_holds_server_fulfilled_tool_use():
            if self._error_emitted:
                raise StopAsyncIteration
            self._error_emitted = True
            verbose_logger.error(
                "AgenticStreamingIterator: hooks did not replace a message containing a server-fulfilled "
                "tool_use [model=%s]; emitting an SSE error instead of leaking the tool call to the client",
                self._model,
            )
            return SERVER_FULFILLED_TOOL_LEAK_ERROR_SSE_BYTES

Replace with:

        if self._buffer_holds_server_fulfilled_tool_use():
            # Try to salvage the message by stripping server-fulfilled blocks
            # and replaying only client-owned content (mixed-tool case).
            filtered = self._make_filtered_response_iterator()
            if filtered is not None:
                self._follow_up_iterator = filtered
                return await self._follow_up_iterator.__anext__()
            # Nothing survives stripping — emit an error rather than leaking
            # the server-fulfilled tool_use to a client that cannot run it.
            if self._error_emitted:
                raise StopAsyncIteration
            self._error_emitted = True
            verbose_logger.error(
                "AgenticStreamingIterator: hooks did not replace a message containing a server-fulfilled "
                "tool_use [model=%s]; emitting an SSE error instead of leaking the tool call to the client",
                self._model,
            )
            return SERVER_FULFILLED_TOOL_LEAK_ERROR_SSE_BYTES

The key behavior changes:

• Mixed-tool message (both headroom_retrieve and a client tool_use): strips the server-fulfilled block, fixes stop_reason if needed, and re-streams via FakeAnthropicMessagesStreamIterator. Subsequent __anext__ calls fall into the existing if self._follow_up_iterator is not None branch.
• Pure server-fulfilled message: behavior unchanged — emits the SSE error.

The stop_reason handling covers both sub-cases correctly: if the server-fulfilled tool was the only tool_use but a text block remains, stop_reason becomes "end_turn"; if client-owned tool_use blocks remain, stop_reason stays "tool_use".

[devin-ai-integration]

Staying with option 1: rewriting stop_reason and dropping a block the model emitted silently changes semantics on an already degraded path, and headroom's loop covers the mixed case before we get there.

@greptileai keepalives now also wrap the follow-up stream in cbefb1c, with tests for slow follow-up chunks, follow-up errors, and cancellation on close

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

[devin-ai-integration]

@greptileai please review the latest commit cbefb1c, which extends the keepalive loop over the follow-up stream

[devin-ai-integration]

Good catch, fixed in bb0bb48: a keepalive ping no longer counts as a delivered chunk, so a ping-only disconnect still reconciles the reservation.

@greptileai please review bb0bb48

[devin-ai-integration]

Also fair, fixed in 2d1ee3a: a disconnect while the buffer already holds provider output keeps the reservation instead of reconciling to input cost.

@greptileai please review 2d1ee3a

[devin-ai-integration]

🚅 Hi, thanks for the PR! I'm Agent Shin, the automated triage bot for this repository. What's this and why am I getting it?

I read the description against our contribution rubric. Here's how it lined up:

What you got right:

• ✅ Clear problem description
• ✅ Expected vs. actual behavior

What's still missing:

• linked GitHub issue or explicit issue reference
• end-to-end QA proof with real screenshot/video or commands plus real output

The body clearly explains the bug and expected vs. actual behavior, so context is present. However, the only QA evidence is pasted command text and narrative output without a real screenshot/video or verifiable command output, and there is no linked GitHub issue.

If the description isn't updated in the next 24 hours, I'll auto-close this PR. That's not us saying we don't care about the change; we want the open-PR list to mirror what a maintainer can act on right now, so contributors don't get lost in a backlog. A closed PR is a soft "park this for later," not a rejection. Take your time; everything below still works after the close.

During the grace period: just update the PR description with the missing pieces. No need to ping me; I'll re-check on the next sweep and skip the auto-close if it now passes. See what counts as QA proof for the full rubric (a linked issue alone isn't enough; it covers context, not proof).

If the PR does get auto-closed in 24 hours, you still have easy recovery paths:

• Comment @agent-shin reconsider after updating the description. I'll re-evaluate and reopen the PR if it now passes.
• Comment @greptileai to request a fresh Greptile review; that still works even after the PR is closed, and a stronger score is one of the signals that lifts the PR back into the queue. So a low Greptile score isn't a blocker either.

Internal BerriAI contributors: this rubric doesn't apply to you; ping a maintainer.

(I'm an LLM, so I'm not infallible. If you think I got this wrong, ping a maintainer; they'll override me.)

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Withheld check misses wrapper type
  • Updated _withheld_provider_output to unwrap AnthropicMessagesStreamingResponse.completion_stream before the AgenticAnthropicStreamingIterator isinstance check, and updated the regression test to wrap the iterator like production so it fails without the fix.

Or push these changes by commenting:

@cursor push 817c38842a

Preview (817c38842a)

diff --git a/litellm/proxy/common_request_processing.py b/litellm/proxy/common_request_processing.py
--- a/litellm/proxy/common_request_processing.py
+++ b/litellm/proxy/common_request_processing.py
@@ -51,6 +51,9 @@
 from litellm.llms.anthropic.experimental_pass_through.messages.agentic_streaming_iterator import (
     AgenticAnthropicStreamingIterator,
 )
+from litellm.llms.anthropic.experimental_pass_through.messages.streaming_iterator import (
+    AnthropicMessagesStreamingResponse,
+)
 from litellm.proxy._types import ProxyException, UserAPIKeyAuth
 from litellm.proxy.auth.auth_checks import can_key_call_resolved_model
 from litellm.proxy.auth.auth_utils import check_response_size_is_safe
@@ -207,7 +210,10 @@
 
 
 def _withheld_provider_output(response: object) -> bool:
-    return isinstance(response, AgenticAnthropicStreamingIterator) and response.has_buffered_provider_output
+    inner: Final = (
+        response.completion_stream if isinstance(response, AnthropicMessagesStreamingResponse) else response
+    )
+    return isinstance(inner, AgenticAnthropicStreamingIterator) and inner.has_buffered_provider_output
 
 
 def _should_return_raw_model_name(request_data: dict[str, object]) -> bool:

diff --git a/tests/test_litellm/proxy/test_budget_reservation.py b/tests/test_litellm/proxy/test_budget_reservation.py
--- a/tests/test_litellm/proxy/test_budget_reservation.py
+++ b/tests/test_litellm/proxy/test_budget_reservation.py
@@ -13,6 +13,10 @@
 from litellm.llms.anthropic.experimental_pass_through.messages.agentic_streaming_iterator import (
     AgenticAnthropicStreamingIterator,
 )
+from litellm.llms.anthropic.experimental_pass_through.messages.streaming_iterator import (
+    AnthropicMessagesStreamHiddenParams,
+    AnthropicMessagesStreamingResponse,
+)
 from litellm.proxy._types import (
     LiteLLM_BudgetTable,
     LiteLLM_EndUserTable,
@@ -2527,10 +2531,17 @@
         server_fulfilled_tool_names=frozenset({"headroom_retrieve"}),
         ping_interval_seconds=0.01,
     )
+    # Match production wiring: llm_http_handler wraps the agentic iterator in
+    # AnthropicMessagesStreamingResponse before handing it to the proxy stream
+    # generator. The withheld-output check must unwrap that to see the buffer.
+    wrapped_held_back = AnthropicMessagesStreamingResponse(
+        completion_stream=held_back,
+        hidden_params=AnthropicMessagesStreamHiddenParams(additional_headers={}),
+    )
 
     async def ping_then_cancel(user_api_key_dict, response, request_data):
         yield await response.__anext__()
-        while not response.has_buffered_provider_output:
+        while not response.completion_stream.has_buffered_provider_output:
             yield await response.__anext__()
         raise asyncio.CancelledError()
 
@@ -2538,7 +2549,7 @@
     streaming_logging_obj.async_post_call_streaming_iterator_hook = ping_then_cancel
     streaming_logging_obj._arelease_max_parallel_requests_on_disconnect = AsyncMock()
     generator = ProxyBaseLLMRequestProcessing.async_streaming_data_generator(
-        response=held_back,
+        response=wrapped_held_back,
         user_api_key_dict=valid_token,
         request_data=_request_body(),
         proxy_logging_obj=streaming_logging_obj,

_{You can send follow-ups to the cloud agent here.}

[devin-ai-integration]

@greptileai please review 1bc441c

bugbot run

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[devin-ai-integration]

@greptileai please review 1aca5a9

bugbot run

[mateo-berri]

@greptileai

[devin-ai-integration]

@greptileai please review 64d8b24

bugbot run

[mateo-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Cache writer drops withheld flag
  • Added a has_buffered_provider_output property on AnthropicMessagesStreamCacheWriter that forwards the flag from its wrapped stream, so the router property no longer falls through to False when Anthropic Messages caching wraps the stream.

Or push these changes by commenting:

@cursor push af8f25fdc4

Preview (af8f25fdc4)

diff --git a/litellm/llms/anthropic/experimental_pass_through/messages/response_cache.py b/litellm/llms/anthropic/experimental_pass_through/messages/response_cache.py
--- a/litellm/llms/anthropic/experimental_pass_through/messages/response_cache.py
+++ b/litellm/llms/anthropic/experimental_pass_through/messages/response_cache.py
@@ -46,6 +46,10 @@
             stream._hidden_params if isinstance(stream, AnthropicMessagesStreamingResponse) else _EMPTY_MAPPING
         )
 
+    @property
+    def has_buffered_provider_output(self) -> bool:
+        return getattr(self.stream, "has_buffered_provider_output", False) is True
+
     def __aiter__(self) -> "AnthropicMessagesStreamCacheWriter":
         return self
 

diff --git a/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_response_cache.py b/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_response_cache.py
--- a/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_response_cache.py
+++ b/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_response_cache.py
@@ -262,3 +262,37 @@
         await asyncio.sleep(0)
 
     mock_route.assert_called_once()
+
+
+def test_cache_writer_forwards_has_buffered_provider_output():
+    """The cache writer sits between the provider stream and the router, so
+    it must forward `has_buffered_provider_output` or disconnect billing on
+    ping-only streams incorrectly refunds withheld provider tokens."""
+    from litellm.llms.anthropic.experimental_pass_through.messages.response_cache import (
+        AnthropicMessagesStreamCacheWriter,
+    )
+
+    class _HoldingBack:
+        has_buffered_provider_output = True
+
+        def __aiter__(self):
+            return self
+
+        async def __anext__(self) -> bytes:
+            raise StopAsyncIteration
+
+    class _NotHoldingBack:
+        def __aiter__(self):
+            return self
+
+        async def __anext__(self) -> bytes:
+            raise StopAsyncIteration
+
+    assert (
+        AnthropicMessagesStreamCacheWriter(stream=_HoldingBack(), caching_handler=None).has_buffered_provider_output
+        is True
+    )
+    assert (
+        AnthropicMessagesStreamCacheWriter(stream=_NotHoldingBack(), caching_handler=None).has_buffered_provider_output
+        is False
+    )

_{You can send follow-ups to the cloud agent here.}

[mateo-berri]

@greptileai

[devin-ai-integration]

@greptileai please review 8d87798

bugbot run

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8d87798. Configure here.}