Repository navigation
docs(blog): how we cut LiteLLM's Redis round trips per request by 64% - #1927
Merged
Merged
Conversation
… 22 to 8 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
yassin-berriai
changed the title
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Engineering blog post covering the request-scoped Redis pipeline series as one change: #43320, #43369, #43407, #43779 and #43776. Same voice and diagram style as the usage-page bounded-loading post, with three React diagrams (
PerformanceResults,RoundTripTimeline,BatchLifecycle) indiagrams.jsand the CSS module copied from that post and extended with the timeline lanesEvery number comes from one fresh local run, pre-series commit
27c110cbvs main at13d004fc(all five PRs in): governed chat 22 -> 8 Redis round trips, streaming 24 -> 8,/v1/messages21 -> 8,/v1/responses26 -> 9, response-cache hit 18 -> 7, and the once-a-minute auth refresh request 46 -> 16 with the refresh itself down from 16 serial calls to 3 trips. The results sections lead with the steady-state reduction from 22 to 8 round trips and the auth-refresh request reduction from 46 to 16. Benchmark setup appears in a separate section.npm run buildpasses; rendered page belowLink to Devin session: https://app.devin.ai/sessions/f732394a6aca4bc197a2f4b1d54d0e7e
Open in Devin Desktop: https://app.devin.ai/desktop/session/f732394a6aca4bc197a2f4b1d54d0e7e?variant=devin
Requested by: @yassin-berriai