Releases: BerriAI/litellm
Releases · BerriAI/litellm
v1.65.2.dev1
What's Changed
- Openrouter streaming fixes + Anthropic 'file' message support by @krrishdholakia in #9667
- fix(cost_calculator.py): allows checking received + sent model name w… by @krrishdholakia in #9669
- Revert "Revert "Correct Databricks llama3.3-70b endpoint and add databricks c…" by @krrishdholakia in #9676
- Update model_prices_and_context_window.json add gemini-2.5-pro-exp-03-25 by @superpoussin22 in #9650
- fix(proxy_server.py): Fix "Circular reference detected" error when max_parallel_requests = 0 by @krrishdholakia in #9671
- UI (new_usage.tsx): Report 'total_tokens' + report success/failure calls by @krrishdholakia in #9675
- [Reliability] - Ensure new Redis + DB architecture tracks spend accurately by @ishaan-jaff in #9673
- [Bug fix] - Service accounts - only apply
service_account_settings.enforced_paramson service accounts by @ishaan-jaff in #9683 - UI - New Usage Tab fixes by @krrishdholakia in #9696
- [Reliability Fixes] - Ensure no deadlocks occur when updating
DailyUserSpendTransactionby @ishaan-jaff in #9690 - Virtual key based policies in Aim Guardrails by @hxtomer in #9499
- fix(streaming_handler.py): fix completion start time tracking + Anthropic 'reasoning_effort' param mapping by @krrishdholakia in #9688
Full Changelog: v1.65.1-nightly...v1.65.2.dev1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.65.2.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 200.0 | 228.05634387180868 | 6.2817559363541715 | 0.0 | 1880 | 0 | 183.1938070000092 | 4938.761445000011 |
| Aggregated | Passed ✅ | 200.0 | 228.05634387180868 | 6.2817559363541715 | 0.0 | 1880 | 0 | 183.1938070000092 | 4938.761445000011 |
Contributors
krrishdholakia, superpoussin22, and 2 other contributors
Assets 4
v1.65.1-nightly
What's Changed
- Litellm fix db testing by @krrishdholakia in #9593
- Litellm new UI build by @krrishdholakia in #9601
- Support max_completion_tokens on Mistral by @Cmancuso in #9589
- Revert "Support max_completion_tokens on Mistral" by @krrishdholakia in #9604
- fix(mistral_chat_transformation.py): add missing comma by @krrishdholakia in #9606
- Support discovering gemini, anthropic, xai models by calling their
/v1/modelendpoint by @krrishdholakia in #9530 - Connect UI to "LiteLLM_DailyUserSpend" spend table - enables usage tab to work at 1m+ spend logs by @krrishdholakia in #9603
- Update README.md by @krrishdholakia in #9616
- Add support to Vertex AI transformation for anyOf union type with null fields by @NickGrab in #9618
- fix(proxy_server.py): get master key from environment, if not set in … by @krrishdholakia in #9617
- fix(logging): add json formatting for uncaught exceptions (#9615) by @krrishdholakia in #9619
- fix: wrong indentation of ttlSecondsAfterFinished in chart by @Dbzman in #9611
- Fix anthropic thinking + response_format by @krrishdholakia in #9594
- Add support to Vertex AI transformation for anyOf union type with null fields by @NickGrab in #9625
- fix(openrouter/chat/transformation.py): raise informative message for openrouter key error by @krrishdholakia in #9626
- [Reliability] - Reduce DB Deadlocks by storing spend updates in Redis and then committing to DB by @ishaan-jaff in #9608
- Add bedrock latency optimized inference support + Vertex AI Multimodal embedding cost tracking by @krrishdholakia in #9623
- build(pyproject.toml): add new dev dependencies - for type checking by @krrishdholakia in #9631
- install prisma migration files - connects litellm proxy to litellm's prisma migration files by @krrishdholakia in #9637
- update docs for openwebui by @tan-yong-sheng in #9636
- Add gemini audio input support + handle special tokens in sagemaker response by @krrishdholakia in #9640
- [Docs - Release notes v0] v1.65.0-stable by @ishaan-jaff in #9643
- [Feat] - MCP improvements, add support for using SSE MCP servers by @ishaan-jaff in #9642
- [FIX] - Add password to sync sentinel client by @jmarshall-medallia in #9622
- fix: Anthropic prompt caching on GCP Vertex AI by @sammcj in #9605
- Fixes Databricks llama3.3-70b endpoint and add databricks claude 3.7 sonnet endpoint by @anton164 in #9661
- fix(docs): update xAI Grok vision model reference by @colesmcintosh in #9286
- docs(gemini): fix typo by @GabrielLoiseau in #9581
- Update all_caches.md by @KPCOFGS in #9562
- [Bug fix] - Sagemaker endpoint with inference component streaming by @ishaan-jaff in #9515
- Revert "Correct Databricks llama3.3-70b endpoint and add databricks c… by @krrishdholakia in #9668
- Revert "fix: Anthropic prompt caching on GCP Vertex AI" by @krrishdholakia in #9670
- [Refactor] - Expose litellm.messages.acreate() and litellm.messages.create() to make LLM API calls in Anthropic API spec by @ishaan-jaff in #9567
New Contributors
- @Cmancuso made their first contribution in #9589
- @Dbzman made their first contribution in #9611
- @tan-yong-sheng made their first contribution in #9636
- @jmarshall-medallia made their first contribution in #9622
- @GabrielLoiseau made their first contribution in #9581
- @KPCOFGS made their first contribution in #9562
Full Changelog: v1.64.1.dev1...v1.65.1-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.65.1-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 220.0 | 261.03979166611845 | 6.112143157921839 | 0.0 | 1827 | 0 | 196.8891020000001 | 5075.201525000011 |
| Aggregated | Passed ✅ | 220.0 | 261.03979166611845 | 6.112143157921839 | 0.0 | 1827 | 0 | 196.8891020000001 | 5075.201525000011 |
Contributors
Dbzman, sammcj, and 10 other contributors
Assets 4
v1.65.0-stable
What's Changed
- Fix route check for non-proxy admins on jwt auth by @krrishdholakia in #9454
- docs(predibase): fix typo by @luisegarduno in #9464
- build(deps): bump next from 14.2.21 to 14.2.25 in /ui/litellm-dashboard by @dependabot in #9458
- [Feat] Add OpenAI Web Search Tool Call Support - Initial support by @ishaan-jaff in #9465
- Refactor vertex ai passthrough routes - fixes unpredictable behaviour w/ auto-setting default_vertex_region on router model add by @krrishdholakia in #9467
- [Feat] Add testing for
litellm.supports_web_search()and render supports_web_search on model hub by @ishaan-jaff in #9469 - Litellm dev 03 22 2025 release note by @krrishdholakia in #9475
- build: add new vertex text embedding model by @krrishdholakia in #9476
- enables viewing all wildcard models on /model/info by @krrishdholakia in #9473
- Litellm redis semantic caching by @tylerhutcherson in #9356
- Log 'api_base' on spend logs by @krrishdholakia in #9509
- [Fix] Use StandardLoggingPayload for GCS Pub Sub Logging Integration by @ishaan-jaff in #9508
- [Feat] Support for exposing MCP tools on litellm proxy by @ishaan-jaff in #9426
- fix(invoke_handler.py): remove hard coded final usage chunk on bedrock streaming usage by @krrishdholakia in #9512
- Add vertexai topLogprobs support by @krrishdholakia in #9518
- Update model_prices_and_context_window.json by @superpoussin22 in #9459
- fix vertex ai multimodal embedding translation by @krrishdholakia in #9471
- ci(publish-migrations.yml): add action for publishing prisma db migrations by @krrishdholakia in #9537
- [Feat - New Model] Add VertexAI
gemini-2.0-flash-liteand Google AI Studiogemini-2.0-flash-liteby @ishaan-jaff in #9523 - Support
litellm.api_basefor vertex_ai + gemini/ across completion, embedding, image_generation by @krrishdholakia in #9516 - Nova Canvas complete image generation tasks (#9177) by @krrishdholakia in #9525
- [Feature]: Support for Fine-Tuned Vertex AI LLMs by @ishaan-jaff in #9542
- feat(prisma-migrations): add baseline db migration file by @krrishdholakia in #9565
- Add Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows by @krrishdholakia in #9538
- Support Gemini audio token cost tracking + fix openai audio input token cost tracking by @krrishdholakia in #9535
- [Reliability Fixes] - Gracefully handle exceptions when DB is having an outage by @ishaan-jaff in #9533
- [Reliability Fix] - Allow Pods to startup + passing /health/readiness when
allow_requests_on_db_unavailable: Trueand DB is down by @ishaan-jaff in #9569 - Add OpenAI gpt-4o-transcribe support by @krrishdholakia in #9517
- Allow viewing keyinfo on request logs by @krrishdholakia in #9568
- Allow team admins to add/update/delete models on UI + show api base and model id on request logs by @krrishdholakia in #9572
- Litellm fix db testing by @krrishdholakia in #9593
- Litellm new UI build by @krrishdholakia in #9601
- Support max_completion_tokens on Mistral by @Cmancuso in #9589
- Revert "Support max_completion_tokens on Mistral" by @krrishdholakia in #9604
- fix(mistral_chat_transformation.py): add missing comma by @krrishdholakia in #9606
- Support discovering gemini, anthropic, xai models by calling their
/v1/modelendpoint by @krrishdholakia in #9530 - Connect UI to "LiteLLM_DailyUserSpend" spend table - enables usage tab to work at 1m+ spend logs by @krrishdholakia in #9603
- Update README.md by @krrishdholakia in #9616
- fix(proxy_server.py): get master key from environment, if not set in … by @krrishdholakia in #9617
New Contributors
- @luisegarduno made their first contribution in #9464
- @Cmancuso made their first contribution in #9589
Full Changelog: v1.63.14-stable.patch1...v1.65.0-stable
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.65.0-stable
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 200.0 | 233.43193575834258 | 6.214443976298119 | 0.0 | 1858 | 0 | 180.17820199997914 | 4614.819022000006 |
| Aggregated | Passed ✅ | 200.0 | 233.43193575834258 | 6.214443976298119 | 0.0 | 1858 | 0 | 180.17820199997914 | 4614.819022000006 |
Contributors
Cmancuso, krrishdholakia, and 5 other contributors
Assets 4
v1.65.0.rc
What's Changed
- Nova Canvas complete image generation tasks (#9177) by @krrishdholakia in #9525
- [Feature]: Support for Fine-Tuned Vertex AI LLMs by @ishaan-jaff in #9542
- feat(prisma-migrations): add baseline db migration file by @krrishdholakia in #9565
- Add Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows by @krrishdholakia in #9538
- Support Gemini audio token cost tracking + fix openai audio input token cost tracking by @krrishdholakia in #9535
- [Reliability Fixes] - Gracefully handle exceptions when DB is having an outage by @ishaan-jaff in #9533
- [Reliability Fix] - Allow Pods to startup + passing /health/readiness when
allow_requests_on_db_unavailable: Trueand DB is down by @ishaan-jaff in #9569 - Add OpenAI gpt-4o-transcribe support by @krrishdholakia in #9517
- Allow viewing keyinfo on request logs by @krrishdholakia in #9568
- Allow team admins to add/update/delete models on UI + show api base and model id on request logs by @krrishdholakia in #9572
- Litellm fix db testing by @krrishdholakia in #9593
- Litellm new UI build by @krrishdholakia in #9601
- Support max_completion_tokens on Mistral by @Cmancuso in #9589
- Revert "Support max_completion_tokens on Mistral" by @krrishdholakia in #9604
- fix(mistral_chat_transformation.py): add missing comma by @krrishdholakia in #9606
- Support discovering gemini, anthropic, xai models by calling their
/v1/modelendpoint by @krrishdholakia in #9530 - Connect UI to "LiteLLM_DailyUserSpend" spend table - enables usage tab to work at 1m+ spend logs by @krrishdholakia in #9603
- Update README.md by @krrishdholakia in #9616
- fix(proxy_server.py): get master key from environment, if not set in … by @krrishdholakia in #9617
New Contributors
Full Changelog: v1.64.1-nightly...v1.65.0.rc
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.65.0.rc
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 540.0 | 624.3702232376279 | 5.474375767460597 | 0.0 | 1637 | 0 | 487.16235000006236 | 2776.3750889999983 |
| Aggregated | Failed ❌ | 540.0 | 624.3702232376279 | 5.474375767460597 | 0.0 | 1637 | 0 | 487.16235000006236 | 2776.3750889999983 |
Contributors
Cmancuso, krrishdholakia, and ishaan-jaff
Assets 4
v1.65.0-nightly
What's Changed
- Nova Canvas complete image generation tasks (#9177) by @krrishdholakia in #9525
- [Feature]: Support for Fine-Tuned Vertex AI LLMs by @ishaan-jaff in #9542
- feat(prisma-migrations): add baseline db migration file by @krrishdholakia in #9565
- Add Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows by @krrishdholakia in #9538
- Support Gemini audio token cost tracking + fix openai audio input token cost tracking by @krrishdholakia in #9535
- [Reliability Fixes] - Gracefully handle exceptions when DB is having an outage by @ishaan-jaff in #9533
- [Reliability Fix] - Allow Pods to startup + passing /health/readiness when
allow_requests_on_db_unavailable: Trueand DB is down by @ishaan-jaff in #9569 - Add OpenAI gpt-4o-transcribe support by @krrishdholakia in #9517
- Allow viewing keyinfo on request logs by @krrishdholakia in #9568
- Allow team admins to add/update/delete models on UI + show api base and model id on request logs by @krrishdholakia in #9572
- Litellm fix db testing by @krrishdholakia in #9593
- Litellm new UI build by @krrishdholakia in #9601
- Support max_completion_tokens on Mistral by @Cmancuso in #9589
- Revert "Support max_completion_tokens on Mistral" by @krrishdholakia in #9604
- fix(mistral_chat_transformation.py): add missing comma by @krrishdholakia in #9606
- Support discovering gemini, anthropic, xai models by calling their
/v1/modelendpoint by @krrishdholakia in #9530 - Connect UI to "LiteLLM_DailyUserSpend" spend table - enables usage tab to work at 1m+ spend logs by @krrishdholakia in #9603
- Update README.md by @krrishdholakia in #9616
- fix(proxy_server.py): get master key from environment, if not set in … by @krrishdholakia in #9617
New Contributors
Full Changelog: v1.64.1-nightly...v1.65.0-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.65.0-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 530.0 | 578.5163737856723 | 5.6521867286945735 | 0.0 | 1689 | 0 | 483.77581299996564 | 4841.4478800000325 |
| Aggregated | Failed ❌ | 530.0 | 578.5163737856723 | 5.6521867286945735 | 0.0 | 1689 | 0 | 483.77581299996564 | 4841.4478800000325 |
Contributors
Cmancuso, krrishdholakia, and ishaan-jaff
Assets 4
v1.64.1.dev1
What's Changed
- Nova Canvas complete image generation tasks (#9177) by @krrishdholakia in #9525
- [Feature]: Support for Fine-Tuned Vertex AI LLMs by @ishaan-jaff in #9542
- feat(prisma-migrations): add baseline db migration file by @krrishdholakia in #9565
- Add Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows by @krrishdholakia in #9538
- Support Gemini audio token cost tracking + fix openai audio input token cost tracking by @krrishdholakia in #9535
- [Reliability Fixes] - Gracefully handle exceptions when DB is having an outage by @ishaan-jaff in #9533
- [Reliability Fix] - Allow Pods to startup + passing /health/readiness when
allow_requests_on_db_unavailable: Trueand DB is down by @ishaan-jaff in #9569 - Add OpenAI gpt-4o-transcribe support by @krrishdholakia in #9517
- Allow viewing keyinfo on request logs by @krrishdholakia in #9568
- Allow team admins to add/update/delete models on UI + show api base and model id on request logs by @krrishdholakia in #9572
Full Changelog: v1.64.1-nightly...v1.64.1.dev1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.64.1.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.64.1.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.64.1.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 530.0 | 588.3618842036032 | 5.564625107213408 | 0.0 | 1665 | 0 | 469.31838699998707 | 4872.995692000018 |
| Aggregated | Failed ❌ | 530.0 | 588.3618842036032 | 5.564625107213408 | 0.0 | 1665 | 0 | 469.31838699998707 | 4872.995692000018 |
Contributors
krrishdholakia and ishaan-jaff
Assets 4
v1.64.1-nightly
What's Changed
- Log 'api_base' on spend logs by @krrishdholakia in #9509
- [Fix] Use StandardLoggingPayload for GCS Pub Sub Logging Integration by @ishaan-jaff in #9508
- [Feat] Support for exposing MCP tools on litellm proxy by @ishaan-jaff in #9426
- fix(invoke_handler.py): remove hard coded final usage chunk on bedrock streaming usage by @krrishdholakia in #9512
- Add vertexai topLogprobs support by @krrishdholakia in #9518
- Update model_prices_and_context_window.json by @superpoussin22 in #9459
- fix vertex ai multimodal embedding translation by @krrishdholakia in #9471
- ci(publish-migrations.yml): add action for publishing prisma db migrations by @krrishdholakia in #9537
- [Feat - New Model] Add VertexAI
gemini-2.0-flash-liteand Google AI Studiogemini-2.0-flash-liteby @ishaan-jaff in #9523 - Support
litellm.api_basefor vertex_ai + gemini/ across completion, embedding, image_generation by @krrishdholakia in #9516
Full Changelog: 1.64.0.dev1...v1.64.1-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.64.1-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 530.0 | 583.0955007522234 | 5.641309914749418 | 0.0 | 1687 | 0 | 483.3096179999643 | 5048.277267999993 |
| Aggregated | Failed ❌ | 530.0 | 583.0955007522234 | 5.641309914749418 | 0.0 | 1687 | 0 | 483.3096179999643 | 5048.277267999993 |
Contributors
krrishdholakia, superpoussin22, and ishaan-jaff
Assets 4
1.64.0.dev1
What's Changed
- Fix route check for non-proxy admins on jwt auth by @krrishdholakia in #9454
- docs(predibase): fix typo by @luisegarduno in #9464
- build(deps): bump next from 14.2.21 to 14.2.25 in /ui/litellm-dashboard by @dependabot in #9458
- [Feat] Add OpenAI Web Search Tool Call Support - Initial support by @ishaan-jaff in #9465
- Refactor vertex ai passthrough routes - fixes unpredictable behaviour w/ auto-setting default_vertex_region on router model add by @krrishdholakia in #9467
- [Feat] Add testing for
litellm.supports_web_search()and render supports_web_search on model hub by @ishaan-jaff in #9469 - Litellm dev 03 22 2025 release note by @krrishdholakia in #9475
- build: add new vertex text embedding model by @krrishdholakia in #9476
- enables viewing all wildcard models on /model/info by @krrishdholakia in #9473
- Litellm redis semantic caching by @tylerhutcherson in #9356
New Contributors
- @luisegarduno made their first contribution in #9464
Full Changelog: v1.63.14-nightly...1.64.0.dev1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-1.64.0.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 530.0 | 595.2345953515114 | 5.639441403576418 | 0.0 | 1687 | 0 | 488.5956009999859 | 4882.238139000037 |
| Aggregated | Failed ❌ | 530.0 | 595.2345953515114 | 5.639441403576418 | 0.0 | 1687 | 0 | 488.5956009999859 | 4882.238139000037 |
Contributors
krrishdholakia, tylerhutcherson, and 3 other contributors
Assets 4
v1.63.14-stable.patch1
Full Changelog: v1.63.14-stable...v1.63.14-stable.patch1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.63.14-stable.patch1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 240.0 | 255.95973818329688 | 6.163881077009807 | 0.0 | 1844 | 0 | 213.38151600002675 | 2969.717206000041 |
| Aggregated | Passed ✅ | 240.0 | 255.95973818329688 | 6.163881077009807 | 0.0 | 1844 | 0 | 213.38151600002675 | 2969.717206000041 |
v1.63.14-stable
Full Changelog: v1.63.14-nightly...v1.63.14-stable
🚨 Issue identified on Bedrock streaming usage calculation - fixed in v1.63.14-stable.patch 1 - https://github.com/BerriAI/litellm/releases/tag/v1.63.14-stable.patch1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.63.14-stable
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 190.0 | 207.58892887694674 | 6.219076098228175 | 0.0 | 1861 | 0 | 165.81997500003354 | 2278.4240120000163 |
| Aggregated | Passed ✅ | 190.0 | 207.58892887694674 | 6.219076098228175 | 0.0 | 1861 | 0 | 165.81997500003354 | 2278.4240120000163 |