[3/N][Core][KV Connector] Support reliable partial-tail KV offload for sub-block prompts - #49502
Merged
Merged
Conversation
Enable fine-grained prefix lookup and MooncakeStore offload when a prompt ends before the shared block boundary. Handle lazy hash sequences and local-only partial hits correctly so sub-block requests can reuse cached state without triggering an invalid remote load. Flush partial-tail handoffs in the scheduling step that creates them, carry the exact snapshot boundary, and pin off-table Mamba CoW blocks until connector-delayed free completes. Persist every cache group from its durable offset through the boundary, including smaller-group full blocks that normal LCM-aligned saves omit. Gate producer-only work on connector availability and fail closed under pressure, null source blocks, lookup errors, or failed puts. Keep partial-tail progress separate from normal-save progress so failed offloads remain cache misses instead of publishing corrupt data or advancing the durable watermark. Consolidate the partial-hit feature predicate and computed-block truncation used by reconciliation. Tests: partial-prefix cache, scheduler, MooncakeStore scheduler/worker/coordinator, and hybrid-memory end-to-end suites; Ruff and git diff checks. AI assistance was used to implement this change. Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Dao Le <daole@inferact.ai> Signed-off-by: Dao Le <Dao007forever@gmail.com>
Dao007forever
requested review from
ApostaC,
NickLucche,
WoosukKwon,
alexm-redhat,
heheda12345,
ivanium,
njhill,
orozery,
robertgshaw2-redhat,
xuechendi and
ywang96
as code owners
July 23, 2026 01:28
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
ivanium
changed the title
ivanium
changed the title
1 task
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
process_tokens bounded iteration by the compact chunk-hash list, whose length only counts complete physical blocks. Fine-grained cache hits ending at a hash boundary inside a physical block were therefore omitted entirely for sub-block hits or lost their trailing KV for cross-block hits. Make the caller's hash-alignment and hash-coverage contracts explicit, enumerate the token range directly with ceiling division, and key every emitted chunk by the fine-grained hash at its ending boundary. Extend the existing HMA test to cover sub-block and cross-block partial tails while retaining the aligned case. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Dao Le <daole@inferact.ai> Signed-off-by: Dao Le <Dao007forever@gmail.com>
ivanium
approved these changes
Jul 25, 2026
# Conflicts: # vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/data.py # vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/scheduler.py Signed-off-by: Dao Le <Dao007forever@gmail.com>
Align lookup bounds to the hash block size when partial hits are enabled so persisted sub-block tails remain discoverable. Retain LCM alignment for regular block-granular lookups. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Dao Le <daole@inferact.ai> Signed-off-by: Dao Le <Dao007forever@gmail.com>
Signed-off-by: Dao Le <daole@inferact.ai> Signed-off-by: Dao Le <Dao007forever@gmail.com>
Dao007forever
force-pushed
the
dao/partial-hit
branch
from
July 25, 2026 07:52
0a21339 to
09e62a9
Compare
Signed-off-by: Dao Le <Dao007forever@gmail.com>
puririshi98
added a commit
to puririshi98/vllm
that referenced
this pull request
Jul 27, 2026
Resolve conflict in the Mooncake store coordinator: vllm-project#49502 independently landed the MambaSpec eagle gate upstream and extended it with a fine-grained hash margin. Keep upstream's superset and retain the comment explaining why a recurrent group gets no peek margin (vllm-project#43559). Signed-off-by: Rishi Puri <riship@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 tasks
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
see #45702
Enable fine-grained prefix lookup and MooncakeStore offload when a prompt ends before the shared block boundary. Handle lazy hash sequences and local-only partial hits correctly so sub-block requests can reuse cached state without triggering an invalid remote load.
max_tokens=1failure modeA producer's partial-tail marker previously became an offload handoff only when a later allocation triggered copy-on-write. Requests that finished after their first sample never allocated again, so freeing the request discarded the marker and forced later consumers to recompute that otherwise shareable tail. As a result, the tail was not offloaded when
max_tokens=1.Flush partial-tail handoffs in the scheduling step that creates them, carry the exact snapshot boundary, and pin off-table Mamba CoW blocks until connector-delayed free completes. Persist every cache group from its durable offset through the boundary, including smaller-group full blocks that normal LCM-aligned saves omit.
Gate producer-only work on connector availability and fail closed under pressure, null source blocks, lookup errors, or failed puts. Keep partial-tail progress separate from normal-save progress so failed offloads remain cache misses instead of publishing corrupt data or advancing the durable watermark.
Consolidate the partial-hit feature predicate and computed-block truncation used by reconciliation.
AI assistance was used to implement this change.
Test Plan
Tests: partial-prefix cache, scheduler, MooncakeStore scheduler/worker/coordinator, and hybrid-memory end-to-end suites; Ruff and git diff checks.
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.