[Fast Start] Cache the MTP draft model in a separate daemon group - #57312
Merged
Merged
Conversation
liusy58
requested review from
22quinn,
MatthewBonanni,
ProExpertProg,
WoosukKwon,
benchislett,
houseroad,
luccafong,
mgoin,
njhill,
robertgshaw2-redhat,
tlrmchlsmth,
yewentao256 and
youkaichao
as code owners
September 17, 2026 07:13
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
14 tasks
Isotr0py
reviewed
Sep 19, 2026
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Isotr0py
approved these changes
Sep 21, 2026
Isotr0py
enabled auto-merge (squash)
September 21, 2026 14:29
github-actions
Bot
added
the
ready
Member
|
/ci run --allow-stale |
|
✅ Triggered Buildkite CI #90226 for commit |
Member
|
/ci run --allow-stale |
|
✅ Triggered Buildkite CI #90251 for commit
|
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Member
|
/ci retry |
|
✅ Triggered Buildkite CI #90327 for commit |
3 tasks
khluu
added a commit
that referenced
this pull request
Sep 23, 2026
…ture (#58107) Signed-off-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com> Co-authored-by: kimi-no-na-wa <kimi-no-na-wa@users.noreply.github.com> Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Under
--load-format ipc_cache, the MTP draft model was not served by the weight cache daemon: the engine either reloaded it from disk or sent it to the target daemon with a mismatching fingerprint.The launcher now starts a second daemon group for the MTP draft, with its own cache key, sockets (
*_draft0.sock) and rendezvous port (--weight-cache-draft-master-port). On the engine side, get_draft_load_config routes an MTP draft to that group and lets other draft types (EAGLE3, DFlash, DSpark) fall back to disk. Only MTP drafts are cached; the draft must use the target's TP size.Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.