Handle OpenRouter SSE control noise on Anthropic /v1/messages - #27666
Conversation
[Infra] Promote Internal Staging to main
[Infra] Promote Internal Staging to main
Two-layer fix for Anthropic-format streaming through OpenRouter, which injects non-spec SSE control events (`: OPENROUTER PROCESSING`, `data: [DONE]`, empty / `:`-prefixed `data:` payloads): 1. Logging-path tolerance — `ModelResponseIterator.convert_str_chunk_to_generic_chunk` short-circuits on the noise shapes before `json.loads`, so the proxy's spend-logging reassembly no longer crashes mid-stream. 2. Egress filtering — new `AnthropicSSENoiseFilter` strips the same noise from client-bound bytes in `PassThroughStreamingHandler.chunk_processor`, while leaving `raw_bytes` (used by the internal logging path) unfiltered. Filter is per-request and only active for `EndpointType.ANTHROPIC`. Noise predicate is identical in both sites and must be kept in sync. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR adds a two-layer defence against non-Anthropic SSE control noise (OpenRouter keep-alives,
Confidence Score: 4/5Safe to merge; the filter and logging-path guard are well-isolated and thoroughly tested, and raw bytes continue to reach the logging path unchanged. The core filtering logic in _is_noise_event silently drops any SSE event with no data: lines, relying on a spec invariant that is true today but undocumented in the code. The per-chunk identity test for the Anthropic endpoint was updated in a way that constrains the test data rather than exercising split-chunk rebundling at the integration layer. Neither is a current defect, but both are worth addressing before the pattern expands. anthropic_sse_filter.py (_is_noise_event spec assumption) and test_unit_test_streaming.py (weakened per-chunk identity assertion for Anthropic)
|
| Filename | Overview |
|---|---|
| litellm/proxy/pass_through_endpoints/anthropic_sse_filter.py | New AnthropicSSENoiseFilter class — chunk-boundary-safe SSE filter. Logic is sound with good test coverage; one semantic edge case in _is_noise_event noted. |
| litellm/proxy/pass_through_endpoints/streaming_handler.py | Wires AnthropicSSENoiseFilter into chunk_processor for ANTHROPIC endpoint; raw_bytes are captured before filtering, correctly preserving the logging path. |
| litellm/llms/anthropic/chat/handler.py | Adds early-exit guards in convert_str_chunk_to_generic_chunk for empty data, [DONE], and comment payloads before json.loads — straightforward defensive fix. |
| tests/pass_through_unit_tests/test_unit_test_streaming.py | Modified existing test to use per-endpoint raw_chunks; the Anthropic case now uses valid SSE events because the filter re-bundles at event boundaries. The change is justified but the per-chunk identity invariant was weakened. |
| tests/test_litellm/proxy/pass_through_endpoints/test_anthropic_sse_filter.py | 13 new unit tests for AnthropicSSENoiseFilter covering clean streams, each noise shape, chunk-boundary splits, CRLF, flush, and UTF-8 multibyte edge cases. |
| tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_sse_wrapper.py | New test verifies logging path correctly reassembles usage from a realistic OpenRouter-style stream with injected noise events. |
Reviews (1): Last reviewed commit: "Handle OpenRouter SSE control noise on A..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @staticmethod | ||
| def _is_noise_event(event: str) -> bool: | ||
| for ln in event.splitlines(): | ||
| stripped = ln.strip() | ||
| if not stripped.startswith("data:"): | ||
| continue | ||
| payload = stripped[5:].strip() | ||
| if payload and payload != "[DONE]" and not payload.startswith(":"): | ||
| return False | ||
| return True |
There was a problem hiding this comment.
_is_noise_event returns True for any SSE event that contains no data: lines at all — including event:-only blocks like a bare event: ping\n or any future Anthropic event type that only carries event: and id: fields with no accompanying data: line. For the current Anthropic SSE spec every real event includes data:, so this is unlikely to bite today, but a future spec extension without a data: line would be silently dropped with no log or error. At minimum, a short comment at the end of the method spelling out the spec invariant it relies on (# Relies on Anthropic SSE spec invariant: every meaningful event carries a data: line) would make this assumption explicit and easier to catch in a future review.
| ), | ||
| # Anthropic pass-through enables an SSE noise filter that re-bundles | ||
| # bytes at event boundaries, so each input chunk must be a complete | ||
| # SSE event for the per-chunk identity assertion to hold. | ||
| ( | ||
| EndpointType.ANTHROPIC, | ||
| "/v1/messages", | ||
| [ | ||
| b'event: message_start\ndata: {"type":"message_start","message":{"id":"m_1"}}\n\n', | ||
| b'event: content_block_delta\ndata: {"type":"content_block_delta","delta":{"text":"hi"}}\n\n', | ||
| b'event: message_stop\ndata: {"type":"message_stop"}\n\n', | ||
| ], | ||
| ), |
There was a problem hiding this comment.
The original test asserted that every chunk yielded by chunk_processor is present in the original raw_chunks list (chunk in raw_chunks). This held because the old ANTHROPIC parameter used the same non-SSE byte blobs as VERTEX_AI. The change is justified — the noise filter re-bundles at event boundaries, so non-SSE inputs produce different output. However, the updated test now requires the author to supply chunks that are already complete SSE events, which implicitly constrains the test data rather than the production code. A complementary test that feeds mid-event split inputs and confirms the reassembled bytes match the total stream (like test_buffers_across_chunk_boundary_mid_event in test_anthropic_sse_filter.py) would close the gap for the chunk_processor integration layer.
Rule Used: What: Flag any modifications to existing tests and... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
7471728
into
BerriAI:litellm_agent_oss_staging_05_11_2026
|
🤖 litellm-agent: Squash-merged into staging branch Triage Summary 281 lines across 6 files (+272 / -9) Merge Confidence: 5/5 ✅ READY All checks green. Greptile 4/5, no blocking pattern findings, no CircleCI runs (OSS-typical). |
3 participants
Relevant issues
None filed.
Pre-Submission checklist
tests/test_litellm/(and updated one existing test intests/pass_through_unit_tests/whose invariant the new filter changes).make test-unitpasses for the touched suites (test_anthropic_sse_filter,test_unit_test_streaming::test_chunk_processor_yields_raw_bytes,test_sse_wrapper).Type
🐛 Bug Fix
Changes
Certain providers (e.g. OpenRouter) implementing "Anthropic-compatible Messages API" inject non-spec SSE control events into a
/v1/messagesstream, such as:: OPENROUTER PROCESSINGcomment keep-alivesdata: [DONE]terminatordata:lines whose payload is empty or itself a:-prefixed commentThese cause two failure modes in LiteLLM
ModelResponseIterator.convert_str_chunk_to_generic_chunk(used by the proxy's pass-through spend-logging reassembly) triesjson.loadson everydata:payload, blowing up mid-stream. Result: request processed but not logged to SpendLogsTwo-Pronged Fix
litellm/llms/anthropic/chat/handler.py) — short-circuitconvert_str_chunk_to_generic_chunkon the noise shapes beforejson.loads.litellm/proxy/pass_through_endpoints/anthropic_sse_filter.py) — newAnthropicSSENoiseFilteris a per-request, chunk-boundary-safe filter wired intoPassThroughStreamingHandler.chunk_processorforEndpointType.ANTHROPIC. It strips the noise from client-bound bytes while leavingraw_bytes(the internal logging path) unfiltered.The noise predicate is intentionally duplicated at the two sites (proxy and
llms/anthropic) rather than imported across the provider/proxy boundary; the in-code comment flags this so the two stay in sync.Tests
tests/test_litellm/proxy/pass_through_endpoints/test_anthropic_sse_filter.py— 13 unit tests covering: clean streams, each noise shape, mid-event and mid-separator chunk boundaries, CRLF separators, ping preservation, flush behavior, and split UTF-8 multibyte sequences across chunk boundaries.tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_sse_wrapper.py— added a logging-path test that feeds a realistic OpenRouter-style stream and asserts_build_complete_streaming_responsereassembles usage correctly.tests/pass_through_unit_tests/test_unit_test_streaming.py::test_chunk_processor_yields_raw_bytes— the Anthropic parametrization is updated to use proper SSE-bounded chunks; the filter (correctly) re-bundles at event boundaries, so the per-chunk identity invariant required real SSE data.