Repository navigation
DeepSeek-V4.1-Flash: LMCache tier offload through the MP server (vLLM plugin) #2598
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 52 commits
Commits
Show all changes
67 commits
Select commit
Hold shift + click to select a range
f0610f5
DeepSeek-V4.1-Flash: LMCache tier offload through the MP server (vLLM…
zgplvyou 26f5c0c
dsv41 offload: serve checkpoint-backed local hits; drop debug instrum…
zgplvyou d10a698
dsv41 offload: address PR review (preemption fence, tail saves, ride …
zgplvyou 9c04c16
recipe(dsv41): HBM-then-tier results, DSpark LMCache on/off, theory v…
zgplvyou 96927f7
test(dsv41): sort the decode-keep test's imports (ruff I001)
zgplvyou 95c687d
dsv41 offload: look up only the extent a load can end on (#2642)
linsun12 08d6753
style(dsv41): black after rebasing onto main
zgplvyou 807d20b
mp worker: release finished transfers on the step thread, while its s…
zgplvyou 13bae6f
dsv41: restore checkpoints when the cursor read is deferred
zgplvyou 6fdb823
dsv41 checkpoints: store decode keeps at the commit, from the build's…
zgplvyou a7d323e
mp worker: destroy IPC resources on the step thread once nobody else …
zgplvyou bece476
Merge main 481c2bfd4 into the V4.1 LMCache offload branch
zgplvyou 0dcb360
test(lmcache-mp): skip the submit_request fallback test without its p…
zgplvyou b845e89
dsv41: keep the plugin's checkpoint restore and warmup exemption out …
zgplvyou ac8bcbd
dsv41 offload: address review (vLLM 0.33 hit cap, save retirement, fe…
zgplvyou a6ed491
dsv41 offload: address follow-up review (MP stalled saves, layout ove…
zgplvyou 428fd67
dsv41 offload: give the plugin test fixtures the connector's new fields
zgplvyou dd4a59a
deepseek_v41: keep vLLM plugin rules out of the model validator
zgplvyou 52cbd1e
deepseek_v41: drop the plugin check from the model validator
zgplvyou 65cd127
dsv41 offload: keep vLLM's knowledge out of the shared offload code
zgplvyou 6e7ba5f
docs: describe what the V4.1 plugin admits now
zgplvyou fee2de2
dsv41 offload: address review round 9
zgplvyou 9b2cdf9
tests: let the index-plane parity test's QuantizationConfig stub take…
zgplvyou 4c07290
Address round 10 review: V4.1 plugin guards and per-step lookup budget
zgplvyou 57f8757
plugin state checkpoints: one defaults helper and one runner wrapper
zgplvyou 69e2ebc
mp lookup: check the client once instead of adapting every request
zgplvyou 7512caf
mp worker: take one last look at pending loads in get_finished
zgplvyou 22021a6
mp worker: reuse step events instead of creating and destroying one p…
zgplvyou 831f5c8
plugin V2 runner: read checkpoint block ids from host rows, not a tab…
zgplvyou 71b69db
V4.1 plugin: one owner for the prefill tail and one prefill-chunk cli…
zgplvyou 0763613
V4.1 plugin: let V2 profile CUDA-graph memory instead of skipping by …
zgplvyou 9cc401a
mp worker: one native-state registration and block-id path for every …
zgplvyou c4721c2
V4.1 plugin: carry the state checkpoint as a native-state image
zgplvyou d8e827d
V4.1 plugin: a finished request saves to its last checkpointed boundary
zgplvyou c4569ad
V4.1 plugin: say which side enforces the tier hit grain
zgplvyou 8eaf726
vLLM plugin offload: one BoundaryPlanner base for the KDA and V4.1 pl…
zgplvyou 69fbd74
vLLM plugin offload: drive every boundary planner through one loop pe…
zgplvyou 9da531f
vLLM plugin offload: build each planner next to its class, and refuse…
zgplvyou de2c4dd
recipe: DeepSeek-V4.1 nvfp4 does not start on vLLM 0.31.1
zgplvyou ee65d81
Merge origin/main into the DeepSeek-V4.1 LMCache offload branch
zgplvyou 414de75
plugin V2 host rows: replace a resumed row, arm only when kept; CPU-s…
zgplvyou 65e2907
tests: use list for the empty runner-class stub (ruff PIE807)
zgplvyou 105730d
mp worker: record the step event only for a step that reaches the server
zgplvyou 0687f4b
V4.1 plugin: store decode keeps at the next build, before it stages
zgplvyou 6da7026
plugin V2 host rows: keep them only on runners that read them
zgplvyou 3587050
test: give the proxy factory contract the decode-keep store site
zgplvyou 7cb4bf9
V4.1 plugin: build the step from the host where the host is exact
zgplvyou f5d9788
V4.1 plugin: defer the cursor read only while serving synchronously
zgplvyou b6f6685
offload: address the core review of 11-02
zgplvyou 53e32bd
offload: report native saves that land during the forward that step
zgplvyou 413df1b
V4.1: build the pool geometry beside the model config
zgplvyou 3593a34
offload: run native ATOM's MP transfers on the step thread again
zgplvyou 7081137
offload: address the second pass of the 11-02 core review
zgplvyou 9e70cbc
Merge main 498133ed9 into the V4.1 LMCache offload branch
zgplvyou 97c8dde
offload: native ATOM drops finished MP futures as main does
zgplvyou 85d55d8
offload: keep native V4.1's LMCache namespace as main computes it
zgplvyou 0132de1
offload: one MP transfer thread for both frontends, and scheduler fixes
zgplvyou 4b589d8
offload: drop handed-back MP futures while the stream is idle
zgplvyou 45355d2
offload: the plugin's polling worker as a constructor flag, and revie…
zgplvyou a278334
offload: swap the transfer thread's error under its lock
zgplvyou 6a08af2
offload: native MP paths as main, one floor record, and the 11-05 review
zgplvyou d7619fe
tests: wire two plugin tests to the plugin's dense scheduler
zgplvyou ba157dd
offload: settle the polling thread's shutdown, failure and stalls
zgplvyou ce7dd45
offload: deadlines from the forward thread, and the 11-06 review
zgplvyou 4b0dc75
offload: never fail-stop a finished transfer from the forward thread
zgplvyou ce1187d
offload: remove the forward thread's deadline check
zgplvyou f259bbb
Merge origin/main into perzhang/dsv41-lmcache-031
zgplvyou File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_unbuilt_save_is_finaldefaults to True in the base class, the opposite of the pre-PR behaviour (keep the tracker and retry next step). In core, the only scheduler whose_build_save_requestreturns None is NativeState, which has to override it back to False (native_state_scheduler.py:388); the only one that needs True is the plugin's V4.1. Please flip the default to False and let V4.1 opt in. Also,_retire_finished_save(L706-735) is a fourth hand-written "pop the tracker, return the lease" — and it deductsstill_read, while_retry_or_retire_failed_save's finished branch (L1441-1455) returns the whole lease; one concept, two rules — fold them into one helper. And_save_source_is_backed(L737-750) is unreachable in core (core'sblock_tableis an int array, sois not Noneis always true) yet runs for every model, with a WARNING and stats.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in b6f6685: the default is False and V4.1 opts in. One helper retires a finished save on every path and keeps blocks another operation still reads. The unbacked-save check moved to the plugin's schedulers.