[aoshen02]

Problem

VLLM_RELOAD_MODE=direct (vllm-project#58382) writes checkpoint tensors into the live tensors, viewed back to their checkpoint layout. Online quantization (a BF16 checkpoint quantized at load, quant_method.uses_meta_device) is the one case that cannot work: the FP8 kernel storage cannot hold the BF16 checkpoint tensor, so there is nothing to view back to.

Fix

Those modules reload the way they cold start. At cold start they are created on the meta device and the layerwise loaders stage their checkpoint tensors until the module is complete, then quantize. Direct reload reuses exactly that:

• begin stages every online-quantized module (stage_layer_for_reload: keep the kernel tensors aside, restore the module on meta, wrap its loaders).
• finish runs finalize_layerwise_processing first: a complete module is quantized and the result copied into its kernel tensors (addresses kept for CUDA graphs); a module the update left out gets its kernel tensors back. Their one-shot post-load guard stays, so the post-load rerun for the other modules does not quantize the kernel bytes again.

stage_layer_for_reload is the loop body of initialize_layerwise_reload, factored out; layerwise mode is unchanged.

Tests

• tests/model_executor/model_loader/test_direct_reload.py: the toy model gets an online-quantized layer (meta BF16 checkpoint tensor, int8 kernel tensor + scale built at load). A reload equals a cold start on the new weights, at the startup addresses; an unsent module keeps its bytes. 13 passed (GB300, slurm 25132).

E2E on GB300 (verda), Qwen3.5-35B-A3B (BF16 MoE + GDN), TP2, online quantization at load, 2 same-weight updates per run, stacked on vllm-project#57733 + vllm-project#62407 + vllm-project#58382 + #179 + #180. Direct and layerwise run side by side; the oracle is a batch-1 fixed-token generation compared before/after.

Online quant (--quantization)	Mode	Job	GSM8K pre → post	Oracle
mxfp8 (linear + MoE)	direct	25477	0.785 → 0.792	equal
mxfp8	layerwise	25478	0.792 → 0.797	equal
mxfp4 (linear + MoE)	direct	25514 / 25534	0.842 → 0.823 / 0.829 → 0.825	equal
mxfp4	layerwise	25515	0.827 → 0.829	equal
nvfp4_per_token (MoE)	direct	25442	0.753 → 0.751	equal
nvfp4_per_token	layerwise	25443	0.755 → 0.757	equal
fp8_per_channel (CUTLASS MoE)	direct	25451 / 25535	0.644 → 0.623 / 0.616 → 0.619	equal

• The vision tower is left unquantized (--quantization-config '{"ignore":["*visual*"]}'): its 4304-wide projections split to 2152 per TP2 rank, which MX formats refuse at startup.
• mxfp4 runs on nightly 00b6c4c5 with the Python half of [Perf][MXFP4] Optimize activation quant for mxfp4. 3x kernel speedup, 5% throughput improvement vllm-project/vllm#63101 reverted in the overlay, because the image's get_cutlass_moe_mm_data predates its new argument.
• fp8_per_channel needs [Model Loader] Write weight-derived tensors back in place on reload #180's CUTLASS stride constants: without them a reload leaves the CUDA graphs on freed strides (illegal memory access, GSM8K 0.618 → 0.0).
• Dense: Qwen3-8B online fp8_per_tensor, direct, 0.892 → 0.892 (0 flips of 1319), oracle equal (25144).
• Repeated runs of the same configuration differ by up to 1.5 GSM8K points before any update; the two drops of about 2 points did not reproduce on rerun.

Known limitation

Online fp8_per_tensor on Qwen3.5-35B-A3B: GSM8K holds and no tensor changes, but the batch-1 fixed-token oracle differs after a reload. Plain nightly layerwise does the same, and it persists with no post-load rerun at all, so it is not specific to this PR; the per-tensor dynamic activation scale of the MoE is the suspect. Not investigated further here.

Dependencies

Stacked on vllm-project#58382 (wu/stack/direct-g). fp8_per_channel (CUTLASS MoE) also needs #180. Follow-up F8 of vllm-project#54477.

🤖 Generated with Claude Code