[Model Loader] Run the model-level post-load hook once at the end of every load and reload by hao-aaron · Pull Request #64357 · vllm-project/vllm

hao-aaron · 2026-10-28T20:04:48Z

Thanks to Tony Wang (@new-TonyWang) for design and review contributions.

PR 1 of the direct weight reload stack (RFC #54477). It replaces #57733. The code is the same, plus the Llama4 DP-vision fix that #57733 dropped when it was narrowed.

Problem

RL weight updates do not load a model in one load_weights call. The weight-transfer engines (NCCL, IPC, M2N, sharded RDT) and reload_weights call it once per bucket, or once per tensor, between initialize_layerwise_reload and finalize_layerwise_reload. On main, three things go wrong with that:

The model-level hook never runs after a reload. finalize_layerwise_reload only finishes the layers. The model's process_weights_after_loading() runs only from the cold-load path. So any state a model derives from its weights in that hook, such as fused buffers or broadcast copies, keeps the old weights' values after an update.
Some models do that work inside load_weights instead. It then runs once per bucket, on partial weights. Under layerwise reload it runs while the layers are still on meta, which fails with Cannot copy out of meta tensor. This affects DeepSeek-V4/V4.1, the DFlash/DSpark drafters and Muse Spark.
Checks on "what this call loaded" fire on every bucket. A bucket that happens not to contain a layer or head raises "MTP layer N weights missing", or silently sets the DSpark confidence_head to None.

Fix

One finish step. New finalize_model_weights(model) calls the model's process_weights_after_loading(). It is called once at the end of a cold load (as before, after per-layer quant processing) and once at the end of every finalize_layerwise_reload. That covers reload_weights and the NCCL, IPC, M2N and sharded-RDT engines.
- The sparse NCCL engine patches weights in place without layerwise reload; its finish_weight_update now calls finalize_model_weights once.
- An M2N update whose plan is all direct does not call finalize_layerwise_reload, so it still skips the hook, as on main.
- The torchao reload decorator and the expert-substitution error path finish the layers in the middle of a load. They now call finalize_layerwise_processing, so they do not run the model hook early.
Post-load work moves from load_weights into the hook. Each load_weights below now only loads, and the hook runs at the end:
- DeepSeek-V4 NVIDIA, V4.1 NVIDIA/AMD: the mHC broadcast (V4.1 AMD's finalize_mega_moe_weights is a no-op).
- DeepSeek-V4.1 NVIDIA: the Engram checkpoint page-cache drop and THP collapse. They now also run after every reload and after dummy loads; both are harmless to repeat.
- DeepSeek-V4 XPU target, MTP and DSpark: finalize_mega_moe_weights. The lazy call in MegaMoE forward() is removed, since dummy loads run the hook too.
- Qwen3 DFlash / Qwen3 DSpark / Gemma4 DSpark / Muse Spark DSpark: _build_fused_kv_buffers.
  - The hook now runs after quant processing.
  - In Qwen3 DFlash/DSpark and Gemma4 DSpark, the fused path that reads .weight directly is used only when the projections are unquantized and not CPU-repacked (_reads_raw_weights). Otherwise each projection runs through its quant method.
  - Muse Spark DSpark still slices qkv_proj.weight directly, now after quant processing, so a quantized or CPU-repacked Muse Spark drafter would read kernel-layout weights.
  - Qwen3 DFlash's MXFP8 context K/V projection copied the K/V rows of each layer's QKV weight and scales, which is only valid before quant processing. It is now created at construction and loaded from the checkpoint's K/V tensors as they stream through load_weights, so its own post-load step runs and a reload refills it in place.
- Muse Spark AMD and Muse Spark DSpark: materialize_residual_path_constants.
The DeepSeek-V4/V4.1 VL wrappers drop their workaround. They had reordered the weight stream so that the language model's in-load_weights finalization saw all its weights (stream_language_model_first, finalizes_weights_during_load, _weights_finalized). That is no longer needed and is removed.
"Complete load" checks are gated on is_mtp_completeness_check_enabled(), which the weight-transfer engines turn off during an update:
- The MTP layer checks in DeepSeek-V3.2 AMD, GLM5-next and Kimi-K3 (AMD/NVIDIA).
- The DSpark confidence_head drop in Qwen3, Gemma4 and DeepSeek-V4/V4.1 (NVIDIA/AMD).
- The Qwen3 DSpark vocab checks.
- AutoWeightsLoader._check_skipped_aliases.
dots3 vision: the block compile runs once. The vision hook now reruns after every reload, and main's compile_block_modules would wrap the blocks in another torch.compile each time. It now compiles in place with Module.compile (parameter names keep no _orig_mod) and only once. The vision MoE is unchanged from main: its hook fuses the experts into FP8 once, deletes the checkpoint experts and returns early afterwards, so a reload leaves the vision experts as they are.
Llama4 DP vision (mm_encoder_tp_mode="data").
- Before: the vision attention used ReplicatedLinear, and load_weights gathered q_proj/k_proj/v_proj and called torch.cat once all three had arrived. An update that sends them in separate calls failed on torch.cat of None.
- Now: it uses QKVParallelLinear / RowParallelLinear with disable_tp=True in DP mode, so q/k/v go through the normal stacked mapping and each part is written into its rows as it arrives. _consolidate_qkv_weights is removed.
Raw Tensorizer loads use the shared loader path. Loading a non-vLLM-tensorized checkpoint used its own _load_model_serialized_cpu, which ran the model hook but skipped per-layer process_weights_after_loading. It now goes through BaseModelLoader.load_model.

Not in this PR

Weights combined from several checkpoint tensors inside load_weights. A weight plus its scale, or a fused FP8 qkv, still need every part in the same call, as on main:
- the FP8 indexer wk dequant;
- GLM5-next MLA projections;
- MiMo-V2 fused qkv;
- Molmo/Molmo2 split embedding.
These are later PRs in the stack (PR 6).
Drafters that still finish inside load_weights, unchanged from main:
- LiLiCorr: the coverage check and materialize_inference_buffers.
- Laguna DFlash: the fused KV build.
- Qwen3 XPress: the mix.L check, fold_from_raw_ and the fused buffers. It also inherits the new DFlash hook, so at cold start its fused KV buffers are built twice, the second time after quant processing.
A bucketed update of these drafters behaves as on main. LiLiCorr and Laguna are in the F9 follow-up.
Hooks that are not safe to rerun. Every model-level hook reachable from a top-level model was checked for a second run on reloaded weights. These are not fixed here:
- DeepSeek-V4 on ROCm (model, MTP, DSpark): a reload now fails instead of leaving wrong weights.
  - On main this hook ran only at cold start, so a reload left the mega-MoE experts and the preshuffled attention and gate/up weights unshuffled or stale.
  - With this PR it reruns at the end of a reload. With fuse_shared_expert, the first run sets shared_experts = None (finalize_mega_moe_weights), so the second run raises AttributeError in load_shared_expert.
  - Without it, the mega-MoE finalize_weights returns early (stale experts), and prepare_attn_preshuffle / prepare_gateup_preshuffle shuffle in place again, so layers that the update did not resend are shuffled twice.
- dots3 vision MoE: stale, as on main. Making it reloadable without keeping the BF16 experts resident (about 11 GiB per GPU on dots3-note-prev-fp8 TP4) needs a loader that quantizes each expert into the fused FP8 buffers as it arrives; that is a follow-up.
- Muse Spark NVIDIA: stale, as on main. The router multiplier is folded once and then dropped, and the mega-MoE finalize_weights returns early after dropping w13_weight.
- DeepSeek-V4 XPU mega-MoE: guarded, as on main. finalize_weights returns early once the weights are transformed. The XPU runtime currently rejects mega-MoE, so the hook is a no-op.
- Rebuilt as new tensors. Kimi-K3 AMD's MLA _w_fold and Muse Spark's residual alpha/beta are re-created by the hook, so a captured CUDA graph keeps reading the old ones.
  - On main Kimi-K3's hook did not run on reload, and Muse Spark rebuilt them the same way from load_weights, so neither is worse than main.
  - The DFlash/DSpark (Qwen3, Gemma4, Muse Spark) and Beam DFlash2 context K/V buffers, and Kimi-K3 NVIDIA DSpark's _context_kv_norm_weights, are also re-created. The V2 speculator's FULL CUDA graph reads the context K/V buffers, so after a reload it reads freed memory and draft acceptance drops (Beam DFlash2 TP4, split reload: 7.03 → 4.89; MiniMax-M3.1 MXFP8 DSpark TP4: 5.70 → 4.07). Outputs stay correct because the target verifies the drafts. Writing these buffers back in place is PR 5 (refill_derived); with it, acceptance after the reload matches the cold start.

Tests

Unit tests:
- tests/model_executor/model_loader/test_reload.py::TestModelHookOnSplitLoad: the model hook runs once, after all weights, for a layerwise reload split into two buckets, a partial layerwise update, and a sparse NCCL update in two chunks.
- tests/models/test_qwen3_dflash_mxfp8_context_kv.py: the MXFP8 context projection matches the QKV projection, following the real load order (post-load steps, then the context buffers).
- test_expert_substitution.py: updated for the Tensorizer path.
- tests/models/test_deepseek_v4_vl.py: replaces test_deepseek_v4_vl_stream_weights.py and test_deepseek_v4_vl_rocm.py. It checks that the VL wrappers leave finalization to the hook.
- test_tensorizer.py::test_deserialized_hf_weights_are_ready_for_forward.
Run on GB200 (slurm 41251), nightly dc45a5f552 (this PR's merge base) with this branch at 4df1d673d overlaid:
- Passed: test_reload.py::TestModelHookOnSplitLoad 2, test_qwen3_dflash_mxfp8_context_kv.py 6, test_deepseek_v4_vl.py 3, test_expert_substitution.py 13.
- tests/v1/spec_decode/test_lilicorr.py: 33 passed after 7e0898bdc (its coverage test builds the model without __init__ and needed _context_kv_proj).
- test_tensorizer.py::test_deserialized_hf_weights_are_ready_for_forward was not run here: the venv has no tensorizer package.
End to end, Llama-4-Scout BF16 (RedHatAI mirror) on GB200, slurm 41251 (same build):
- Setup: TP=4, mm_encoder_tp_mode="data", prefix caching off.
- The vision q/k/v are reloaded in separate load_weights calls in v, k, q order, through initialize_layerwise_reload / finalize_layerwise_reload.
- main fails with expected Tensor as element 0 ... got NoneType (the torch.cat in _consolidate_qkv_weights).
- This PR: 1047/1047 tensors byte-identical to cold start, and the greedy image+text sample is identical before and after the reload.
DeepSeek-V4-Flash-DSpark on GB200, head 0403d305a, nightly dbcf568821 with this branch overlaid:
- Cold start with the DSpark drafter (slurm 40659): TP=4, 7 speculative tokens, GSM8K 1319 questions, 5-shot, greedy.
  
  GSM8K draft acceptance
  
  main 0.9477 0.4938
  
  this PR 0.9484 0.4938
- Weight update through NCCL (slurm 40658), with the sharded-trainer benchmark:
  - Setup: server TP=2, trainer fsdp=2, layerwise reload, 2 syncs of a checkpoint in which every tensor changes.
  - The audit compares every device tensor with a fresh server on that checkpoint.
  - main: the update fails with Cannot copy out of meta tensor (the post-load step inside load_weights).
  - This PR: 3440 tensors compared, 0 mismatched, no memory growth.
  - 86 derived tensors are reallocated: MoE kernel scale configs and permute caches, gemm1_clamp_limit, tid2eid, the RoPE cache and topk_indices_buffer. These are rebuilt by layerwise reload's per-layer processing, not by this PR's hook; keeping their addresses is the job of direct reload ([RL][Model Loader] Direct weight reload: write checkpoint tensors into the live model #58382).
Cold-start A/B against the same nightly, on GB300, H200 and MI355X (comment): DeepSeek-V4.1-Flash, Qwen3-8B + DFlash, Llama-4-Scout FP8 with DP vision, dots3-note-prev-fp8 and DeepSeek-V4-Flash on MI355X. GSM8K, image accuracy, DFlash acceptance length, throughput and KV cache size match within run-to-run noise. The dots3 KV cache loss it found came from the earlier dots3 vision MoE change, which is reverted here.
Not covered: DeepSeek-V4.1 reload (needs two nodes), the drafter weights during an update (the benchmark updates only the target), Muse Spark, dots3 reload, XPU and ROCm reload.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SeFSSrpw7yTpLz2C2sHDKA

… hook load_weights must not assume all weights arrive in one call. RL weight updates call it once per bucket (verl: 512MB buckets, tensor by tensor; the NCCL engine: once per tensor), so a weight and its scale, or the halves of a merged tensor, can land in different calls, and post-load work inside load_weights ran on partial weights, once per bucket. One finish step, finalize_model_weights(model), now runs after every full load and every reload: model_loader/utils.py::process_weights_after_loading (cold load, including dummy loads) and finalize_layerwise_reload (reload_weights and the NCCL/IPC/M2N/sharded-RDT finish_weight_update). It raises on multi-tensor weights whose parts never all arrived, then calls the model's process_weights_after_loading once. No per-model opt-ins. This mirrors SGLang's single post-processing at end_weight_update. Cross-call pairing: stash_weight_part(module, key, part, tensor) parks a part on the module (cloned, since transfer buckets reuse their buffers) and returns all parts once complete. It replaces every call-local pairing buffer, which dropped or raised on a pair split across calls: the FP8 indexer wk dequant (DeepSeek-V2/V3.2 nvidia+amd model and MTP, deepseek_mtp, HY-V4 amd/nvidia model and nvidia MTP, dots3), GLM5-next indexer wk and BF16-kept MLA projections (model and MTP), MiMo-V2 and MiMo-V2 MTP fp8 qkv (their own persisted dicts), the molmo/molmo2 split embedding, the Llama4 DP-vision q/k/v concat and the Jina-v3 LoRA merge. Moved to process_weights_after_loading (out of load_weights): - DFlash family fused context-KV buffers (Qwen3 DFlash, Qwen3/Gemma4 DSpark, LiLiCorr, Laguna). The fused raw-weight path is used only when the projections are unquantized and not CPU-repacked (_reads_raw_weights); otherwise each projection runs through its quant method. - BERT MLM tied decoder copy; DeepSeek-V4/V4.1 mHC broadcast; LiLiCorr inference buffers (a dummy load now builds them); the OpenPanguModel sweep; Gemma4 MTP lm-head cache invalidation and suppress-token index. - DeepSeek-V4 XPU target/MTP/DSpark finalize_mega_moe_weights (the lazy call in MegaMoE forward() is gone); MiniCPM-V/O resampler device move (a dummy load crashed); OpenPangu text/MTP sweeps (VL delegates); DeepSeek-V4.1 NVIDIA Engram page-cache drop and THP collapse. StaticSinkAttention.update_sink_kv clears sink_populated so a re-run hook rewrites the cached sink KV. No opt-in handshakes: the DeepSeek-V4/V4.1 VL wrappers drop finalizes_weights_during_load / _weights_finalized and the language-model- first regrouping and always delegate the hook; stream_language_model_first is removed. No load_weights calls the model hook any more. Decisions on one call's loaded set run only for a complete load (is_mtp_completeness_check_enabled, which the weight-transfer engines turn off): AutoWeightsLoader's skipped-alias guard, DSpark confidence-head drops, Qwen3 DSpark, Eagle3, AliceAI and LiLiCorr checkpoint checks, the Phi-3-V embedding alias, the NemotronVL seq-cls head init, and the MTP layer checks that were still ungated. TensorizerLoader: a raw (not vLLM-serialized) checkpoint loads through BaseModelLoader.load_model instead of a private helper that skipped per-layer quant processing; block-FP8 checkpoints crashed in their first forward. Same change and test as vllm-project#55318. Co-authored-by: nicole-lihui <44200120+nicole-lihui@users.noreply.github.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: aoshen02 <aoshen@inferact.ai> # Conflicts: # .buildkite/test_areas/models_basic.yaml # tests/models/test_deepseek_v4_vl_stream_weights.py # vllm/model_executor/models/mimo_v2.py # vllm/models/deepseek_v41/amd/vl_model.py

- Leave the raw tensorizer path, the MiniCPM-V/O resampler move and the DeepSeek-V4 XPU lazy finalize in forward() as on main; none is needed for loads split across calls (the tensorizer change is vllm-project#55318). - torchao reload and the expert-substitution error path run inside load_weights, so they restore layers only (finalize_layerwise_processing) and leave the model hook to the load finish. - dots3 vision: the hook now re-runs at every reload finish; compile once. - Drop the OpenPangu delegate test; one-line docstrings. - Fix a mis-indented CI path from the main merge. Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Once tied, `embed_tokens` was the canonical name of the shared embedding, so the checkpoint's `model.embed_tokens.weight` was skipped as its alias and every reload left the embedding stale. Re-register the tie last so the language model's name stays canonical. Also update the MiMo-V2 split-qkv test to the shared pending-parts store. Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: aoshen02 <aoshen@inferact.ai> # Conflicts: # .buildkite/test_areas/models_basic.yaml

… reload Model Runner V2 records the DFlash context-KV step in its FULL CUDA graph, so reallocating the fused buffers on a drafter reload left the graph reading the old weights. After a W0->W1 update the acceptance rate fell from 0.310 to 0.013 on Qwen3-4B + DFlash-b16 (GB300). Copy the rebuilt tensors into the existing ones when the shape is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>

LiLiCorr scores candidates inside the DFlash2 FULL CUDA graph, so its fused edge weights and factor splits must keep their addresses across a reload, like the DFlash context-KV buffers. Extend `_rebuild_in_place` to tuples of tensors and use it for `materialize_inference_buffers`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>

…tream Signed-off-by: aoshen02 <aoshen@inferact.ai>

…remaining models Bring the PR in line with the direct-reload branch it is split from. - Weight parts: `discard_weight_parts` drops parts parked by an update that never finished when the next reload starts; a parked part carries an event recorded on the stream that parked it, so pairing across the packed NCCL receive streams is safe. `finalize_model_weights` checks the parts, then runs the model-level hook. - DeepSeek V3.2 / DeepSeek MTP: the FP8 indexer wk pair goes through `stash_weight_part` (it keeps vllm-project#59223's cross-call pairing and adds the discard and parts check). - Tensorizer runs the loader hook; MiniCPM-O/V, Phi-3-V, DeepSeek V4 XPU, Muse Spark (AMD model, DSpark) build their caches in `process_weights_after_loading`; BERT MLM derives the decoder only when the checkpoint omits it. - DFlash/LiLiCorr: drop `_rebuild_in_place` and its test; refilling the derived buffers in place moves to the follow-up that adds `set_cached_tensor`. The dots3 vision compile guard moves to the dots3 vision follow-up. - Tests: weight parts (discard, streams, BERT decoder), packed tensor, tensorizer, expert substitution, OpenPangu post-load; CI lists the merged `test_deepseek_v4_vl.py` instead of the removed `test_deepseek_v4_vl_stream_weights.py`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>

…g rerunnable Weight reloads rerun process_weights_after_loading. dots3 NOTE vision: - the FP8 vision MoE returned early once its fused buffers existed, so a reload left them built from the startup experts; it also deleted the checkpoint experts, so a reload had nowhere to write them. Rebuild the fused experts on every call, refill the buffers in place, and keep the checkpoint experts registered. - compile_block_modules replaced attn/norm modules with torch.compile wrappers, renaming their parameters under `_orig_mod` (a reload could not find `visual.blocks.N.attn.*`) and wrapping them again on the second call. Compile the modules in place with `Module.compile`, once. Static sink attention (OpenPangu): `update_sink_kv` only reset `sink_populated`; the population runs in the `maybe_populate_sink` op, recorded as a no-op when CUDA graphs were captured, so graph replay never wrote the new sink K/V into the KV cache. Write them into the KV cache at once when it exists (every reload); cold start keeps the lazy population in forward, which runs during graph warmup before capture. Tested on the real attention class, before and after `bind_kv_cache`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>

Multi-part checkpoint weights no longer park parts across load_weights calls. finalize_model_weights keeps only the model-level post-load hook. - Molmo/Molmo2: wte.embedding and wte.new_embedding are written straight into their rows of this rank's embed_tokens shard, in any order or call. - Llama4 DP vision: q/k/v load into their rows of the replicated qkv_proj through the stacked mapping; _consolidate_qkv_weights is gone. - The FP8 multi-part cases (indexer wk dequant for DeepSeek-V2/V3.2, dots3, GLM5-next; GLM5-next FP8 MLA projections; MiMo-V2 fused qkv) and the Jina-v3 LoRA merge go back to main's handling; they move to checkpoint-layout parameters on top of direct reload (vllm-project#58382). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeFSSrpw7yTpLz2C2sHDKA Signed-off-by: Aaron Hao <aaronhao@inferact.ai>

…exer-wk revert Follow-up to 5abd769 (review): - `_try_load_fp8_indexer_wk` went back to main (a dict buffer), but deepseek_mtp, HY-V4 (amd/nvidia model, nvidia mtp) and two GLM5-next tests in test_quark.py still passed the module. Revert those to main too. - Molmo/Molmo2 wrote each vocab half straight into `embed_tokens.weight.data`, bypassing the parameter's weight loader. Layerwise reload restores a layer on meta and wraps its loaders to buffer and count what arrives, so a direct write landed on the meta tensor and counted nothing: the layer would be reported as failed to load and keep its old embedding. The half now goes through `param.weight_loader(param, weight, shard_id)`, an offset-aware loader installed on `embed_tokens.weight` at construction (whole-weight loads still use the default vocab-parallel loader). Molmo2 shares the helpers with Molmo. - A checkpoint with only one half is rejected again, except under `disable_mtp_completeness_check` (partial updates, where the other half may come in a later call). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeFSSrpw7yTpLz2C2sHDKA Signed-off-by: Aaron Hao <aaronhao@inferact.ai>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeFSSrpw7yTpLz2C2sHDKA Signed-off-by: Aaron Hao <aaronhao@inferact.ai>

Split the other models out into a follow-up PR. Revert to main: Molmo, Molmo2, Llama4 DP vision, BERT MLM, Phi-3-V, OpenPangu (and StaticSinkAttention), MiniCPM-V/O, Gemma4 MTP, Laguna DFlash, LiLiCorr, Eagle3 (Llama, Qwen3), NemotronVL, AliceAI and Bailing V3 MTP, with their tests. Qwen3 DFlash/DSpark stay: the DeepSeek-V4/V4.1 and Muse Spark DSpark drafters build on them. Gemma4 DSpark stays too, since it inherits their post-load hook and its fused K build must handle processed weights. Signed-off-by: Aaron Hao <hao-aaron@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

With mm_encoder_tp_mode="data" the vision qkv_proj is replicated, and load_weights gathered q, k and v and concatenated them once all three had arrived. A weight update that sends them in separate load_weights calls hit torch.cat on None. Give the replicated qkv_proj weight and bias a loader that writes each part into its rows, and route q/k/v through the normal stacked mapping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeFSSrpw7yTpLz2C2sHDKA Signed-off-by: Aaron Hao <aaronhao@inferact.ai>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

aoshen02 · 2026-10-30T10:55:15Z

/ci run

github-actions · 2026-10-30T10:55:45Z

✅ Triggered Buildkite CI #98695 for commit 97b171c135ff.

… in place The V2 speculator captures the context K/V step in its FULL CUDA graph. Beam's DFlash2 drafter rebuilt its stacked K/V and K-norm weights, and Kimi-K3's DSpark drafter its stacked KV-norm weights and KV scales, as new tensors on every load, so after a draft weight update the graph read freed memory (Beam: draft acceptance 7.03 -> 4.89). Refill them in place. Originally aoshen02's 1d14daf on vllm-project#64357, moved here. Co-authored-by: aoshen02 <aoshen@inferact.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Aaron Hao <hao-aaron@users.noreply.github.com>

aoshen02 · 2026-10-30T12:18:46Z

/ci retry

github-actions · 2026-10-30T12:19:05Z

✅ Queued 2 failed job(s) for retry in Buildkite CI #98695.

hao-aaron · 2026-10-30T22:07:49Z

/ci retry

github-actions · 2026-10-30T22:08:10Z

✅ Triggered Buildkite CI #98826 for commit e9e1d3331d4e, running 1 failed step(s) from Buildkite CI #98695.

vllm-project#64357 runs the model's process_weights_after_loading at the end of every reload, and vllm-project#64336's direct mode reruns it too, so the block-FP8 shards a reload writes are requantized in place. Remove the refusal added until that landed. Test both modes through start_reload / finish_reload. Signed-off-by: Canlin <canlinguosdu@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The earlier revision (run the model-level hook on every load and reload) landed in reworked form as vllm-project#64357. This PR now carries the follow-up: one post-load hook signature, with the loader running every module's hook. Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…g hooks vllm-project#64357 and vllm-project#63440 each added process_weights_after_loading to MuseSpark1xForCausalLM, and the second definition replaced the first. On ROCm that skipped materialize_residual_path_constants, so the monokernel refused to attach, and mypy fails on main with no-redef. Run both steps in one hook: materialize the constants first, then attach the monokernel, which needs the folded router weights. Co-authored-by: Claude Signed-off-by: Alex-Wengg <hanweng9@gmail.com> Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

vllm-project#64357 and vllm-project#63440 each added a process_weights_after_loading to MuseSpark1xForCausalLM, so the second definition silently replaced the first and broke pre-commit on main (ruff F811, mypy no-redef). Run both steps in one method: materialize the residual-path constants first, then attach the monokernel, which snapshots the final weights. Signed-off-by: zydo <zydo.git@gmail.com>

DarkLight1337 · 2026-11-02T06:58:37Z

/ci run all

github-actions · 2026-11-02T06:58:56Z

❌ CI commands require an open PR.

aoshen02 and others added 21 commits October 19, 2026 04:30

Merge remote-tracking branch 'origin/main' into p57733-goal

db13b10

Signed-off-by: aoshen02 <aoshen@inferact.ai> # Conflicts: # .buildkite/test_areas/models_basic.yaml

Merge branch 'main' into wu/pwal-placement-upstream

2169106

Merge remote-tracking branch 'origin/main' into wu/pwal-placement-ups…

c465a50

…tream Signed-off-by: aoshen02 <aoshen@inferact.ai>

Merge branch 'main' into wu/pwal-placement-upstream

6fc163b

[Test] Fix the return annotation of the Molmo embedding test helper

f769b5e

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeFSSrpw7yTpLz2C2sHDKA Signed-off-by: Aaron Hao <aaronhao@inferact.ai>

Merge branch 'main' into wu/pwal-placement-upstream

7402a62

Merge branch 'main' into wu/pwal-placement-upstream

3255a85

Merge branch 'main' into wu/pwal-placement-upstream

c78eb82

Merge branch 'main' into wu/pwal-placement-upstream

29f1e69

hao-aaron requested a review from a team October 28, 2026 20:04

hao-aaron requested a review from tjtanaa as a code owner October 28, 2026 20:04

claude Bot reviewed Oct 28, 2026

View reviewed changes

hao-aaron requested review from AndreasKaratzas, DarkLight1337, dllehr-amd, hongxiayang, vadiklyutiy and zyongye as code owners October 28, 2026 20:04

ywang96 approved these changes Oct 30, 2026

View reviewed changes

Merge branch 'main' into direct-reload-pr1

e9e1d33

khluu disabled auto-merge October 30, 2026 22:52

khluu merged commit 14a5bb4 into vllm-project:main Oct 30, 2026
22 checks passed

github-project-automation Bot moved this from Backlog to Done in Sprint - DFlash Oct 30, 2026

aoshen02 mentioned this pull request Oct 31, 2026

[ROCm][Model] DeepSeek-V4: keep weight-derived tensors in place across reloads aoshen02/vllm#226

Open

okorzh-amd mentioned this pull request Oct 31, 2026

[Bugfix][ROCm][Kimi-K3] Refresh derived weight copies after in-place weight writes #64886

Open

subhashpolisetti mentioned this pull request Nov 1, 2026

[Perf][Model] DeepSeek V4/V4.1 Vision: encode a call's images in packed passes #65030

Open

5 of 6 tasks

xiaohong42 mentioned this pull request Nov 2, 2026

[ROCm][Bugfix] DeepSeek-V4: B-preshuffle the FP8 linears in their kernel so weight reloads redo it xiaohong42/vllm#3

Open

6 tasks done

aoshen02 mentioned this pull request Nov 2, 2026

[Model Loader] One post-load hook signature; the loader runs every module's hook #57678

Open

This was referenced Nov 2, 2026

[Bugfix][ROCm] Muse Spark: merge the two process_weights_after_loading hooks #65080

Merged

[CI Failure]: pre-commit mypy on main - process_weights_after_loading redefined in vllm/models/muse_spark/amd/model.py #65081

Closed

gcanlin mentioned this pull request Nov 2, 2026

[Perf][GLM-5.3-Flash] DeepGEMM FP8 MegaMoE with fused shared experts #60895

Open

6 tasks done

zydo mentioned this pull request Nov 2, 2026

[Frontend] Validate JSON body and request_ids in /inference/v1/abort_requests #63302

Open

6 tasks done

hyukjlee mentioned this pull request Nov 2, 2026

[ROCm][Perf][Kimi-K3] DSpark Markov sampling sharded over the vocabulary, one launch #64031

Open

fanxingran mentioned this pull request Nov 2, 2026

[Bugfix][Spec Decode] Muse Spark DSpark: initialize _context_kv_proj so the draft loads #65125

Open

6 tasks done

apex-mochen mentioned this pull request Nov 2, 2026

[Bugfix][Prefix] Skip EAGLE block drop for KV-shared drafters + widen periodic SWA tail #64875

Open

Navigation Menu

Repository navigation

[Model Loader] Run the model-level post-load hook once at the end of every load and reload - #64357

[Model Loader] Run the model-level post-load hook once at the end of every load and reload#64357
khluu merged 49 commits into
vllm-project:mainfrom
hao-aaron:direct-reload-pr1

hao-aaron commented Oct 28, 2026 •

edited by aoshen02

Loading

Uh oh!

claude Bot left a comment

Uh oh!

aoshen02 commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

aoshen02 commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

hao-aaron commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

Uh oh!

DarkLight1337 commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

	GSM8K	draft acceptance
main	0.9477	0.4938
this PR	0.9484	0.4938

Conversation

hao-aaron commented Oct 28, 2026 • edited by aoshen02 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Not in this PR

Tests

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

aoshen02 commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

aoshen02 commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

hao-aaron commented Oct 30, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

Uh oh!

DarkLight1337 commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

hao-aaron commented Oct 28, 2026 •

edited by aoshen02

Loading