Repository navigation
Perf: serve ready local DRAM sources without the core state lock - #148
Merged
Merged
Conversation
mkhazraee
marked this pull request as draft
October 28, 2026 05:43
mkhazraee
force-pushed
the
moein/lock_opt_bk
branch
2 times, most recently
from
October 28, 2026 07:06
1055af5 to
3bd2443
Compare
mkhazraee
marked this pull request as ready for review
October 28, 2026 18:02
mkhazraee
force-pushed
the
moein/lock_opt_bk
branch
from
October 28, 2026 22:43
af242d8 to
8996451
Compare
mkhazraee
force-pushed
the
moein/lock_opt_bk
branch
5 times, most recently
from
October 29, 2026 04:15
25ca174 to
31b2d8f
Compare
Publish complete local reservations through a separate source lock and reuse their assembled descriptors. Hold reader protection until native transfer completion; leave access accounting, reclamation, and framework callbacks on the caller. Keep partial and mixed-source requests on the existing fallback path, and retry source-lock contention without reporting misses. Signed-off-by: Moein Khazraee <moein@nvidia.com>
…ity waiters Signed-off-by: Moein Khazraee <moein@nvidia.com>
mkhazraee
force-pushed
the
moein/lock_opt_bk
branch
from
October 29, 2026 18:37
31b2d8f to
5ec8308
Compare
Collaborator
Author
|
/ok to test 5ec8308 |
@mkhazraee, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
Contributor
|
/ok to test 5ec8308 |
Signed-off-by: Moein Khazraee <moein@nvidia.com>
mkhazraee
force-pushed
the
moein/lock_opt_bk
branch
from
October 29, 2026 22:54
c9057f3 to
8bcae8c
Compare
ziqifan617
approved these changes
Oct 29, 2026
…d clean up Signed-off-by: Moein Khazraee <moein@nvidia.com>
mkhazraee
force-pushed
the
moein/lock_opt_bk
branch
from
October 30, 2026 00:00
8bcae8c to
563f215
Compare
karya0
approved these changes
Oct 30, 2026
ziqifan617
added a commit
that referenced
this pull request
Oct 30, 2026
## Summary Add a public-source SGLang HiCache/KVCR quick start with a separate Dockerfile, alongside the vLLM guide and image. - Build KVCR from the local Docker build-context checkout, including local source changes, rather than fetching a separate KVCR revision. - Pin SGLang [#45647](sgl-project/sglang#45647) at `b0506add3eac49bbb2c303e83169ea19c907fb98` and Dynamo at `83ea37e849e883c27d7ed848e0f329961e1fb173`. - Run the state agent and publish KVCR-owned inventory for router-guided peer delivery; verify completed delivery, destination reuse, and matching output tokens. No KVCR production-code changes. - After this PR merges, the guide clones the default branch, so the KVCR build context includes changes already merged to main, including #148. ## Validation - Built the SGLang quick-start Dockerfile from KVCR PR checkout `d186752ce78092aa2bd0cc4c1d48d24a1c247354`, SGLang `b0506add3eac49bbb2c303e83169ea19c907fb98`, and Dynamo `83ea37e849e883c27d7ed848e0f329961e1fb173`. KVCR 0.2.0 was built from the checkout and the image's import/version checks passed. The unchanged Dockerfile was built with BuildKit; this was not a GPU inference test. - With the earlier SGLang `0d37a2bf4db3521a876d1b450a884b91c0e334c9` and KVCR `7b7743fe0104393ac56c6fe83ae58387268dc72b` source pair, a custom two-worker Qwen3-32B GPU test verified cross-node peer delivery, 115,968 restored tokens, and matching output IDs. This was not the guide-as-written Qwen3-0.6B test. - The earlier SGLang `4a083003f9393b758077afe9bbce3f6b1c64f790` and KVCR `7186e7c20ba6e4d71ac3aff507fac9bfd83e8a07` pair passed the public-source Qwen3-0.6B two-worker check: 1,984/2,048 peer-reused tokens, matching 16 output token IDs, and 31 blocks / 62 spans with no failed keys. - GPU end-to-end validation of the current `b0506add` SGLang pin together with the post-merge KVCR main containing #148 remains pending. Do not infer that validation from the successful image build or earlier source-pair runs. --------- Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remote requests for ready local DRAM can wait behind unrelated caller work holding the core state lock. Allow the progress thread to acquire fully ready local reservations through a separate source lock and reuse cached descriptor tuples.
Protect source slots until native transfer completion, including cancellation and timeout paths. Release readers on the progress thread while keeping access accounting, reclamation, and framework callbacks on the caller. Drain queued completions before admission decisions.
Preserve the existing caller path for partial, missing, or framework-backed sources. Retry source-lock contention without reporting missing data. Recovery rebuilds source references for the receiving agent.
Extend existing lifecycle tests to cover reader protection, eviction races, completion ordering, and recovery.