Skip to content

docs(release): expand Anthropic streaming perf depth-item write-up (v1.87.0rc1) #225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yassin-berriai merged 2 commits into from
May 25, 2026
Merged

docs(release): expand Anthropic streaming perf depth-item write-up (v1.87.0rc1) #225

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dbb30a6
docs(release): expand Anthropic streaming perf depth-item write-up
claude May 25, 2026
1b9531f
docs(release): use internal real-deployment perf numbers for #28289
claude May 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions release_notes/v1.87.0rc1/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ pip install litellm==1.87.0rc1
- **Gemini 3.5 Flash Day-0 support** — `gemini-3.5-flash` and `gemini-3.1-flash-lite` ship on Vertex AI, Google AI Studio, and OpenRouter with full pricing, function calling, web search, code execution, and managed-agents support.
- **MCP UI for OAuth tool calls** — the dashboard now resolves tool list and tool call against OAuth-protected MCP servers directly, plus native MCP OAuth support for Cursor and clearer OAuth error messages.
- **Codex CLI auth hardening** — JWT-derived team aliases and SSO form-URL flow for the OpenAI Codex CLI, plus allowlisted OIDC-claim persistence across the CLI SSO poll.
- **Anthropic streaming hot-path perf** — measurable per-request and per-chunk overhead reduction across the `/v1/messages` SSE path; lazy-loaded response streaming for Bedrock SageMaker.
- **Anthropic streaming hot-path perf** — ~90% lower TTFT overhead and higher sustained throughput on the proxy's Anthropic `/v1/messages` SSE path, measured on a real 4-pod deployment against both Anthropic and Bedrock Invoke (wire output is parity-tested); plus lazy-loaded response streaming for Bedrock SageMaker.

## New Providers and Endpoints

Expand Down Expand Up @@ -209,7 +209,11 @@ Plus a Vertex / Anthropic `supports_output_config` flag flip on all `claude-opus

## Performance / Loadbalancing / Reliability improvements

- **Anthropic streaming hot path** — Measurable per-request and per-chunk overhead reduction across `/v1/messages` SSE - [PR #28289](https://github.com/BerriAI/litellm/pull/28289)
- **Anthropic `/v1/messages` streaming hot path** — cut per-request and per-chunk overhead on the proxy's Anthropic streaming path, with byte-identical wire output guaranteed by parity tests that diff the logged and billed payloads between the fast and legacy paths. Measured on a real 4-pod `m7i.xlarge` deployment (no HPA) streaming 256 `text_delta` chunks per request, against both Anthropic and Bedrock Invoke: **TTFT overhead ~90% lower** (p50 2220% → 165%, p95 3057% → 316%, p99 3111% → 328%) with **higher sustained throughput** (TPM +12% / +6% / +4% at p50 / p95 / p99) - [PR #28289](https://github.com/BerriAI/litellm/pull/28289)
- Skip work that's a no-op in the default config: the per-chunk Datadog span when tracing is off, the per-chunk streaming hook when no callback / guardrail / cost-injection is active, and the agentic post-processing wrapper when no callback overrides its hook (it otherwise buffers every chunk and rebuilds the response from SSE just to call hooks that all return `(False, {})`).
- Stop doing the same work twice per request: serialize the request body once and reuse it for the pre-call log and the wire, memoize the optional-params type-hint resolution (~80µs/request), and skip the redundant `strip_empty_text_blocks` scan when the async wrapper already sanitized.
- Cheaper end-of-stream reconstruction: collapse the homogeneous run of `content_block_delta` text events into a single equivalent SSE event before `stream_chunk_builder`, removing O(output-token) `ModelResponseStream` constructions; tool-use / thinking / citations streams fall back to the unchanged legacy path.
- Cheaper hot-path logging: gate debug f-string evaluation behind `isEnabledFor(DEBUG)`, hoist `cost_injection_active` out of the per-chunk loop, and drop one async-generator layer per chunk in `async_sse_data_generator`.
- **Bedrock / SageMaker** — Switch to lazy loading for response streaming - [PR #28189](https://github.com/BerriAI/litellm/pull/28189)
- **Granian ASGI** — Add Granian as a supported ASGI server for better throughput stability - [PR #26027](https://github.com/BerriAI/litellm/pull/26027)
- **Prisma** — Expose Prisma idle/connect timeout + extra DB URL params so production deployments can tune connection pools - [PR #28395](https://github.com/BerriAI/litellm/pull/28395)
Expand Down