Skip to content

Litellm dev 04 11 2026 p1 #25585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ishaan-berri merged 3 commits into from
Apr 12, 2026
Merged

Litellm dev 04 11 2026 p1 #25585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
12bca64
docs: refactor benchmarking docs to be clearer
krrish-berri-2 Apr 12, 2026
e08e3bf
docs: clarify how to get benchmarking script
krrish-berri-2 Apr 12, 2026
973986a
docs: readme tweak
krrish-berri-2 Apr 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
</a>
</p>
</p>
<h4 align="center"><a href="https://docs.litellm.ai/docs/simple_proxy" target="_blank">LiteLLM Proxy Server (AI Gateway)</a> | <a href="https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy" target="_blank"> Hosted Proxy</a> | <a href="https://litellm.ai/enterprise"target="_blank">Enterprise Tier</a> | <a href="https://litellm.ai/" target="_blank">Website</a></h4>
<h4 align="center"><a href="https://docs.litellm.ai/docs/simple_proxy" target="_blank">LiteLLM Proxy Server (AI Gateway)</a> | <a href="https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy" target="_blank"> Hosted Proxy</a> | <a href="https://litellm.ai/enterprise"target="_blank">Enterprise Tier</a> | <a href="https://www.litellm.ai/ai-gateway" target="_blank">Website</a></h4>

@greptile-apps greptile-apps Bot Apr 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 "Website" link points to a product page, not the site root

The link label is Website but the destination is now https://www.litellm.ai/ai-gateway — a specific product page. Visitors clicking "Website" typically expect the homepage. Consider either updating the label to AI Gateway or keeping the URL as the site root (https://www.litellm.ai/).

<h4 align="center">
<a href="https://pypi.org/project/litellm/" target="_blank">
<img src="https://img.shields.io/pypi/v/litellm.svg" alt="PyPI Version">
Expand Down
94 changes: 51 additions & 43 deletions docs/my-website/docs/benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,55 @@ import Image from '@theme/IdealImage';

Benchmarks for LiteLLM Gateway (Proxy Server) tested against a fake OpenAI endpoint.


LiteLLM Gateway has **8ms P95 latency** at 1k RPS (See benchmarks [here](#4-instances))

## Machine Spec used for testing

Each machine deploying LiteLLM had the following specs:

- 4 CPU
- 8GB RAM

## Configuration

- Database: PostgreSQL
- Redis: Not used


### 2 Instance LiteLLM Proxy

In these tests the baseline latency characteristics are measured against a fake-openai-endpoint.

#### Performance Metrics

| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 200 | 630 | 1200 | 262.46 | 1035.7 |
| Custom | LiteLLM Overhead Duration (ms) | 12 | 29 | 43 | 14.74 | 1035.7 |
| | Aggregated | 100 | 430 | 930 | 138.6 | 2071.4 |

<!-- <Image img={require('../img/1_instance_proxy.png')} /> -->

<!-- ## **Horizontal Scaling - 10K RPS**

<Image img={require('../img/instances_vs_rps.png')} /> -->


### 4 Instances

| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 100 | 150 | 240 | 111.73 | 1170 |
| Custom | LiteLLM Overhead Duration (ms) | 2 | 8 | 13 | 3.32 | 1170 |
| | Aggregated | 77 | 130 | 180 | 57.53 | 2340 |

#### Key Findings
- Doubling from 2 to 4 LiteLLM instances halves median latency: 200 ms → 100 ms.
- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms.
- Setting workers equal to CPU count gives optimal performance.


## Setting Up Benchmarking with Network Mock

The fastest way to benchmark proxy overhead is using `network_mock` mode. This intercepts outbound requests at the httpx transport layer and returns canned responses, no need for setting up a mock provider.
Expand Down Expand Up @@ -41,6 +90,8 @@ litellm --config benchmark_config.yaml --port 4000 --num_workers 8
python scripts/benchmark_mock.py --requests 2000 --max-concurrent 200 --runs 3
```

Get the benchmarking script [here](https://github.com/BerriAI/litellm/blob/main/scripts/benchmark_mock.py)

This measures pure proxy overhead on the hot path without any network latency to a real or fake provider.

## Setting Up a Fake OpenAI Endpoint
Expand All @@ -61,38 +112,6 @@ model_list:
api_key: "test"
```

### 2 Instance LiteLLM Proxy

In these tests the baseline latency characteristics are measured against a fake-openai-endpoint.

#### Performance Metrics

| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 200 | 630 | 1200 | 262.46 | 1035.7 |
| Custom | LiteLLM Overhead Duration (ms) | 12 | 29 | 43 | 14.74 | 1035.7 |
| | Aggregated | 100 | 430 | 930 | 138.6 | 2071.4 |

<!-- <Image img={require('../img/1_instance_proxy.png')} /> -->

<!-- ## **Horizontal Scaling - 10K RPS**

<Image img={require('../img/instances_vs_rps.png')} /> -->


### 4 Instances

| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 100 | 150 | 240 | 111.73 | 1170 |
| Custom | LiteLLM Overhead Duration (ms) | 2 | 8 | 13 | 3.32 | 1170 |
| | Aggregated | 77 | 130 | 180 | 57.53 | 2340 |

#### Key Findings
- Doubling from 2 to 4 LiteLLM instances halves median latency: 200 ms → 100 ms.
- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms.
- Setting workers equal to CPU count gives optimal performance.

## `/realtime` API Benchmarks

End-to-end latency benchmarks for the `/realtime` endpoint tested against a fake realtime endpoint.
Expand All @@ -115,17 +134,6 @@ End-to-end latency benchmarks for the `/realtime` endpoint tested against a fake
| **System** | 4 vCPUs, 8 GB RAM, 4 workers, 4 instances |
| **Database** | PostgreSQL (Redis unused) |

## Machine Spec used for testing

Each machine deploying LiteLLM had the following specs:

- 4 CPU
- 8GB RAM

## Configuration

- Database: PostgreSQL
- Redis: Not used

## Infrastructure Recommendations

Expand Down
14 changes: 4 additions & 10 deletions docs/my-website/sidebars.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1059,16 +1059,7 @@ const sidebars = {
"proxy/health_check_routing"
],
},
{
type: "category",
label: "Load Testing",
items: [
"benchmarks",
"load_test_advanced",
"load_test_sdk",
"load_test_rpm",
]
},
"benchmarks",
{
type: "category",
label: "Contributing",
Expand Down Expand Up @@ -1097,6 +1088,9 @@ const sidebars = {
"data_retention",
"proxy/security_encryption_faq",
"migration_policy",
"load_test_advanced",
"load_test_sdk",
"load_test_rpm",
Comment on lines 1062 to +1093

@greptile-apps greptile-apps Bot Apr 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Load-testing docs split across two unrelated locations

benchmarks is now a standalone top-level item while load_test_advanced, load_test_sdk, and load_test_rpm land in the "Extras" section alongside data-retention and security pages. Users looking for load-testing material will have to search in two different parts of the sidebar. Keeping them together (either as a flat group at the top level or as siblings in "Extras") would avoid the split.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

{
type: "category",
label: "❤️ 🚅 Projects built on LiteLLM",
Expand Down
Loading