Conversation
Build cross-layer draft buffers after layerwise restoration and refresh their storage in place. Opt migrated model hooks into reload without replaying unrelated cold-start transforms. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen524 <aoshen524@gmail.com>
Refresh derived buffers on their owning backbones after cold load and reload. Remove speculative MoE changes and root hook wrappers, and consolidate buffer updates and regression coverage. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen524 <aoshen524@gmail.com>
aoshen02
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Partial draft reloads can build fused KV/norm tensors while untouched weights or RoPE buffers are still meta placeholders. Refresh cross-layer derived buffers only after all layers are ready, and copy into existing nonpersistent buffers to preserve captured storage addresses.
The owning backbone exposes
refresh_derived_buffers(). Cold-load postprocessing and reload finalization both invoke it after their existing processing. This operation only refreshes derived buffers; it does not rerun model-level weight conversion. Qwen DFlash/DSpark, DFlash2, Gemma DSpark, Laguna DFlash and Kimi K3 MLA DSpark share this lifecycle through existing inheritance. Production code has a net increase of 25 lines.Scope: no DeepSeek MoE changes or reload-capability flags. This does not establish general reload safety for arbitrary model-level PWAL hooks.
Related work: upstream vllm-project#55076 guards the same meta-placeholder failure; this changes the lifecycle and preserves derived-buffer storage. vllm-project#52487 also rebuilds DSpark fused state after finalize, so there is partial overlap; its broader scope is disk-backed sleep/wake recovery, whereas this covers the shared checkpoint reload finalizer. vllm-project#56706 / fork #46 remove HPC QK-norm mirrors. Fork #47 and #48 address iteration and reload API organization. This draft targets aoshen02/vllm:main only.
Validation:
pre-commit run <hook> --files <changed Python files>passed for ruff-check, ruff-format, check-spdx-header, check-root-lazy-imports, check-forbidden-imports, check-torch-cuda-call and check-boolean-context-manager.git diff --checkpassed..venv/Scripts/python.exe ../run_cpu_checks.py: 12 CPU source-isolated checks passed. The local harness executes actual loader/reload functions, builder methods and committed test bodies with lightweight model shells and isolated accelerator imports. This is not a full pytest run..venv/bin/python -m pytest tests/model_executor/model_loader/test_reload.py tests/model_executor/model_loader/test_draft_post_load.py, then draft-model inference/evaluations.AI assistance: implemented and self-audited with Codex. Draft for the owner's line-by-line review and accelerator validation; no human review or model evaluation is claimed.