[Perf][Model Loader] Reload layout-preserving weights directly

[aoshen02]

Summary

This PR removes redundant reconstruction work from online weight reload while
preserving the runtime tensor storage used by kernels and CUDA graphs.

It adds two fail-closed fast paths:

1. Layout-identical direct reload: when checkpoint and runtime tensors have
   exactly the same shape, stride, and dtype, the original weight loader writes
   directly into the existing runtime storage.
2. Runtime-backed checkpoint views: an explicitly opted-in quantization
   method may expose checkpoint-layout views backed by already transformed
   runtime storage. The loader writes through those views, and finalization only
   restores the original Parameter objects.

The initial runtime-view implementation covers compressed-tensors WNA16 Triton
MoE weights and scales. Unsupported methods, unverified layouts, and failed atomic bindings keep
using the existing layerwise materialize/process/copy fallback. Attention
wrappers may use direct/runtime-view loading when their layouts pass the same
contract, but all attention variants still share one deferred post-load
finalization stage.

The important result is that this PR does not skip the original weight
loader, change the inference backend, replace Triton with Marlin, or rebuild
CUDA-graph-visible kernel tensors. It removes the temporary layer lifecycle
that is unnecessary when the final storage can be safely targeted directly.

Motivation

The existing layerwise reload path is necessary when
process_weights_after_loading() changes the checkpoint representation into a
different runtime representation. However, it applies the same reconstruction
lifecycle to layers whose final storage is already known and safely writable.

Before this PR, each arriving checkpoint tensor could go through:

checkpoint tensor
  -> recompute layer size and wrap late-registered loaders
  -> bind and normalize loader arguments
  -> retain BoundArguments and count loaded numel
  -> update and scan the set of concurrently loading layers
  -> materialize a temporary checkpoint-layout layer
  -> replay the original loaders
  -> rerun post-load quantization / packing / transpose
  -> copy processed tensors back into the original runtime storage
  -> restore the original Parameter / Buffer objects

This is correct, but redundant for two common cases:

• the checkpoint tensor already has the same layout as the runtime tensor; or
• the runtime tensor can safely expose a checkpoint-layout view sharing the
  same storage.

After this PR:

layout-identical tensor
  -> original loader -> existing runtime storage

WNA16 runtime-view tensor
  -> original loader -> checkpoint-layout view of existing packed storage
  -> restore original Parameter object at finalize (no data movement)

unverified / unsupported tensor
  -> unchanged layerwise fallback

Design

1. One non-mutating format-owner hook

Quantization methods expose one opt-in contract:

get_runtime_weight_reload_mapping(
    layer,
    checkpoint_params,
    runtime_params,
) -> Mapping[str, Parameter] | None

The default returns None. An implementation must not mutate layer; it either
returns a complete checkpoint-name-to-runtime-storage mapping or declines the
fast path. This replaces the earlier boolean supports_direct_weight_reload()
plus mutating bind_runtime_weight_reload() pair. One mapping describes both
identity and nontrivial runtime-view cases, so capability and destination cannot
disagree.

Modules without a quantization method may use the generic identity mapping when
the complete tensor layout matches. A quantized method must opt in explicitly;
generic code never guesses that a packed or transformed quantized layout is
safe.

2. Complete-set direct-layout validation

Identity/direct reload requires the complete checkpoint-facing parameter and
buffer key sets to match the reloadable runtime key sets. Every tensor must
match in shape, stride, and dtype. Lazy/uninitialized tensors fail closed. A
matching name, shape alone, or equal byte count is insufficient. Attention, MLAAttention, and MMEncoderAttention use the same deferred
post-load stage because their scales and runtime state have ordering requirements
beyond a local identity copy. Deferring finalization no longer forces their
checkpoint-compatible tensors onto the layerwise reconstruction path.

3. Explicit reusable reload plans

Every layer receives exactly one cached mode:

• DIRECT: the original loader targets the identical runtime tensor;
• RUNTIME_VIEW: the format owner supplies a checkpoint-layout view over the
  existing runtime storage;
• LAYERWISE: the existing materialize/process/copy-back fallback.

The plan records signatures for all runtime parameters and buffers: shape,
stride, dtype, device, data pointer, storage pointer, storage offset, and storage
size. Before every update, a stale plan is discarded if any signature changed.
DIRECT and RUNTIME_VIEW also re-confirm the current quant-method mapping on every
update, so replacing a quant method cannot silently reuse an old capability.
Finalization verifies the signature again and raises if reload changed runtime
storage or layout.

4. Atomic runtime-view binding validation

A custom mapping is accepted only after the entire mapping passes validation:

• every checkpoint parameter is present;
• the mapped Parameter class matches the checkpoint-facing class;
• the view remains on the same device as its runtime tensor;
• storage pointer and storage byte size are identical;
• byte offset and covered byte length are identical;
• the view has no internal overlap;
• checkpoint-facing buffers require exact-layout handling and are not inferred
  through the custom mapping.

Only after all entries pass does generic code bind the complete mapping to the
layer. A detached clone, partial mapping, mismatched view, or unsupported method
falls back without partially rebinding the live layer. This atomicity applies to
structural binding; it does not claim rollback of weight bytes after a transport
failure has already partially written an update.

5. Preserve loader metadata without retaining layer cycles

Direct/runtime-view loading still invokes the original weight loader, preserving
its slicing, TP/PP/EP sharding, stacked-parameter, and copy semantics. Loader
metadata is restored onto the live destination from a fresh meta copy.
Layer-bound references are restored only on that copy: the stored metadata
remains sanitized, avoids a layer-reference cycle, and cannot leave a loader
bound to the sanitizer sentinel. A real H200 run exposed this sentinel-bound
method case; the implementation and a bound-loader regression test cover it.

6. WNA16 Triton runtime views

Compressed-tensors WNA16 uses different checkpoint and runtime representations:

• packed runtime weights are transposed contiguous uint8 storage, while the
  checkpoint-facing loader expects an int32 view with checkpoint axis order;
• runtime scales are already transposed relative to checkpoint metadata.

The WNA16 method owns this transform. It returns int32 views over the complete
packed uint8 storage and checkpoint-facing views of the transposed scales,
copies loader metadata to temporary Parameters, and clears is_transposed for
the loader-facing view. Generic code proves that these views cover the exact
live storage before binding them. Finalization restores the original runtime
Parameter objects without transposing, repacking, or copying weight data.

7. Conservative fallback

The existing layerwise path remains for:

• FNUZ and unproven dtype conversions;
• Marlin WNA16;
• methods with buffers or derived state not fully represented by the mapping;
• unknown quantization methods;
• incomplete key sets, lazy tensors, detached views, internal overlap, or any
  shape/stride/dtype/storage mismatch.

Initial opt-ins

• UnquantizedLinearMethod on every platform; generic key/shape/stride/dtype
  validation rejects CPU optimized layouts that removed or replaced weight;
• block-FP8 Fp8LinearMethod with CutlassFp8BlockScaledMMKernel;
• block-FP8 Fp8MoEMethod with Triton or Batched Triton;
• compressed-tensors CompressedTensorsWNA16MoEMethod checkpoint-layout runtime
  views.

Unified post-load and CPU layout update

The latest head removes two remaining policy special cases.

Attention finalizes once, independently of load mode

Attention, MLAAttention, and MMEncoderAttention are classified by one
_POST_LOAD_ATTENTION_TYPES tuple. Their checkpoint-compatible tensors may now
use the same direct/runtime-view planner as other layers. Regardless of whether
a layer selected direct, runtime-view, or legacy layerwise loading, it is queued
for the same final stage after all incoming tensors have arrived:

1. validate and restore the original runtime storage for a direct/runtime-view
   layer, or restore/reload attention scales for a layerwise layer;
2. invoke layer.process_weights_after_loading(model_config.dtype) exactly once;
3. reset the reload state.

This separates two independent questions: where bytes can safely be written, and
when module-wide derived state must be finalized.

CPU Linear uses the generic contract

UnquantizedLinearMethod.get_runtime_weight_reload_mapping() now returns the
runtime parameters on every platform. That is only a candidate mapping. Generic
code still requires the complete checkpoint-facing and runtime key sets to match
in shape, stride, and dtype.

Therefore:

• ordinary CPU functional Linear retains a matching weight and can reload
  directly;
• CPU optimized implementations such as paths that remove or replace
  weight no longer match and automatically fall back;
• the reload path contains no CPU backend-name allowlist.

The same simplification is deliberately not applied to arbitrary quantized
methods. Equal shape/dtype cannot prove that packed or transformed runtime
storage is checkpoint-compatible; the format owner must explicitly opt in.

Broad local A/B validation

A second validation layer was run on exact vLLM v0.25.1 using the same container
and the same prerequisite model-support/backend fixes on both sides:

• baseline port: 48c2ba006277e50331580c789a56ba623aa52494;
• candidate port: eda369098c21eaea99fe0e4329815b0f6c1f85eb;
• PR head: 55da21d9604334f6ff418630d8aef8a153b44740;
• image: sha256:267dab8accfcdbbb31aa09ce8b74a31e13a9aaec3d7968164b3dad7ae865dd2c;
• hosts: h200-0 and h200-1, each with 8 H200 GPUs.

Each paired process used eager execution, prefix caching disabled, a fixed
prompt, greedy generation of eight tokens, and three same-checkpoint reloads.
The table reports the median of reloads 2 and 3. A and B ran concurrently on
separate GPUs where possible.

Model / format	A steady	B steady	Delta	B mapped / layerwise
Pythia-14M BF16	0.050 s	0.031 s	+60.5%	68 / 2
OPT-125M FP16	0.268 s	0.259 s	+3.4%	116 / 1
Qwen2.5-0.5B BF16	0.316 s	0.235 s	+34.6%	271 / 1
Qwen3-0.6B BF16	0.352 s	0.332 s	+6.3%	371 / 1
Qwen3-4B BF16	2.073 s	1.890 s	+9.7%	475 / 1
Qwen3-4B INT4	0.961 s	0.985 s	-2.5%	295 / 181
Qwen3-4B NVFP4	1.031 s	1.055 s	-2.2%	295 / 181
Qwen3.5-0.8B GDN	0.550 s	0.458 s	+20.2%	484 / 1
Qwen3-VL-2B, MM encoder	1.231 s	1.085 s	+13.5%	665 / 1
Qwen2 1.5B FP8 dynamic	0.694 s	0.690 s	+0.6%	175 / 142
Llama-3.1-8B FP8	2.312 s	2.406 s	-3.9%	199 / 162
Gemma4-12B BF16	6.099 s	5.577 s	+9.4%	746 / 2
MiMo-7B-RL BF16	3.325 s	3.549 s	-6.3%	403 / 2
GLM-Z1-9B BF16	4.509 s	4.381 s	+2.9%	527 / 2
PowerMoE-3B	3.754 s	3.748 s	+0.2%	359 / 34
GLM-4.7-Flash INT4 AWQ	19.968 s	20.319 s	-1.7%	806 / 279
Qwen3-30B-A3B FP8	14.075 s	11.765 s	+19.6%	487 / 146
Qwen3-30B-A3B INT4	26.281 s	26.491 s	-0.8%	535 / 98
Moonlight-16B-A3B FP8, MLA	5.855 s	5.474 s	+6.9%	407 / 191
Qwen3.6-35B-A3B INT4	30.704 s	30.940 s	-0.8%	747 / 52
Qwen3-Coder-Next FP8, TP2	49.709 s	39.706 s	+25.2%	751 / 206

All 21 pairs passed. For every model, all three post-reload outputs matched the
pre-reload token IDs, text, and selected-token logprobs exactly. Baseline and
candidate token IDs and selected-token logprobs also matched exactly (maximum
cross-variant selected-logprob absolute difference: 0.0).

Small negative timing deltas on mostly-layerwise formats are within the observed
NFS/page-cache variance and have no corresponding output difference. This disk
matrix is a breadth/correctness regression test; the VIME/NCCL results below
remain the end-to-end transport performance evidence.

Three attempted fixtures failed identically on A and B and are not counted:
DeepSeek-V2-Lite had an incomplete offline remote-code cache; GPT-OSS MXFP4 hit
the pre-existing w13_weight meta-restore collision; and the local GPT-OSS FP8
checkpoint did not match the model's w2_bias parameter set.

Benchmark methodology

The benchmark was designed to isolate this PR from both sender-side
optimizations and inference-backend changes.

Fixed environment

• Hardware: two independent nodes, h200-0 and h200-1, each with 8 NVIDIA
  H200 141 GB GPUs.
• Container image digest:
  sha256:267dab8accfcdbbb31aa09ce8b74a31e13a9aaec3d7968164b3dad7ae865dd2c.
• Image vLLM: 0.25.1, commit
  752a3a504485790a2e8491cacbb35c137339ad34.
• VIME validation head:
  b41b854964a162ada035f1875e055b949fc82796.
• Sender buffer: 512 MiB.
• vLLM sleep mode: Level 2.
• Eager execution for all three model integrations.
• Both A and B include the same VIME sender behavior, including
  vllm-project/vime#340; sender optimization is therefore held constant.

Same-backend requirement and #49065

Both A and B explicitly use --vllm-moe-backend triton.

Both sides also include the behavior from #49065: WNA16 may select Marlin only
when moe_backend in ("auto", "marlin"); an explicit Triton selection remains
on CompressedTensorsWNA16MoEMethod.

This is important because an earlier Qwen3.6 comparison accidentally used
Marlin for A and Triton for B. That number was invalid as a reload-only A/B and
was removed. The results below rerun both sides on Triton. Logs from every Qwen
run contain Using CompressedTensorsWNA16MoEMethod and contain no Marlin
selection.

#49065 is not part of this PR. It only holds backend selection constant so that
the measured delta belongs to this reload change.

A/B definition

• A: vLLM v0.25.1 plus the [Bugfix][Quantization] Respect explicit WNA16 MoE backend #49065 backend-selection behavior, without this
  PR.
• B: the exact same environment plus this PR's functional changes.

The profiling code, transport instrumentation, model checkpoint, VIME code,
container, topology, sender settings, and backend selection are identical
between A and B.

Topologies

Model	Trainer	Rollout	Transport/runtime
Qwen3.6-35B-A3B INT4	TP1 / PP4 on GPUs 0-3	2 engines x TP2 on GPUs 4-7	non-colocated, WNA16 Triton
Moonlight-16B-A3B FP8	TP4 / EP2 on 8 GPUs	4 engines x TP2 / EP2	colocated IPC, block-FP8 Triton
Qwen3-Coder-Next FP8	PP6 on GPUs 0-5	1 engine x TP2 / EP2 on GPUs 6-7	non-colocated packed NCCL, FP8 Triton

Timing definitions

Three nested timing levels are reported and are never added across parent/child
boundaries:

1. Outer update: VIME's trainer-side Timer update_weights; this is the
   user-visible end-to-end time.
2. Receiver critical path: the single vLLM worker with the largest complete
   start_weight_update() to finish_weight_update() interval. All additive
   stages in a row come from that same worker.
3. Nested receiver stages: transport and loader blocks inside the receiver.

The receiver hierarchy is:

receiver total
├── transfer.initialize
├── lifecycle uncovered / inter-chunk gap
├── transfer.receive
│   ├── IPC tensor reconstruction, or
│   └── NCCL broadcast + unpack + model.load_weights callback
└── transfer.finalize

transfer.receive contains model.load_weights; those values must not be
added. Similarly, loader.trigger_process contains the six
_layerwise_process() sub-blocks.

For attribution runs, transport and callback boundaries synchronize CUDA so
the callback values are CUDA-complete. The per-weight direct-loader timer is
CPU-only to avoid a synchronization per tensor.

The matrix measures the initial update and stops only after:

• the outer timer ends;
• the update endpoint returns successfully; and
• the expected profile JSON is present for every worker.

The final KeyboardInterrupt in these controlled logs is the orchestrator
stopping the job after the measured update, not an update failure.

End-to-end performance

The low-instrumentation/reference outer results are the appropriate production
performance comparison:

Model	A initial	B initial	Speedup	A post-train	B post-train	B direct / runtime-view / layerwise
Moonlight-16B-A3B FP8	5.4604 s	3.4695 s	1.57x	4.5 s	2.9 s	569 / 0 / 29
Qwen3-Coder-Next FP8	49.4392 s	31.9505 s	1.55x	50.0 s	32.8 s	943 / 0 / 14
Qwen3.6-35B-A3B INT4	6075.7 s	20.6 s	294.9x	not run	20.9 s	747 / 40 / 12

The detailed profiler adds synchronization and Python timing overhead, so its
absolute outer values are not substituted for the table above. Its purpose is
to identify where the time went:

Model	Profiled outer A / B	Receiver critical A / B	model.load_weights A / B
Moonlight FP8	5.5 / 4.0 s	4.468 / 3.144 s	2.119 / 0.878 s
Qwen3-Coder-Next FP8	67.2 / 42.4 s	67.067 / 42.303 s	37.549 / 19.885 s
Qwen3.6 INT4, h200-0 same-host	6608.2 / 23.2 s	6608.095 / 23.091 s	6478.328 / 8.621 s
Qwen3.6 INT4, h200-1 repeat	5853.9 / 21.5 s	5853.846 / 21.460 s	5664.862 / 8.579 s

Detailed breakdown: Moonlight-16B-A3B FP8

Receiver critical path

Stage	A	B	B - A
Receiver total	4.468 s	3.144 s	-1.324 s
transfer.initialize	0.032 s	0.019 s	-0.013 s
Lifecycle uncovered	1.918 s	1.835 s	-0.083 s
transfer.receive (60 calls)	2.508 s	1.279 s	-1.229 s
transfer.finalize	0.010 s	0.011 s	+0.001 s
Outer minus receiver	1.032 s	0.856 s	-0.176 s

Moonlight uses colocated IPC:

Nested receive stage	A	B	Interpretation
IPC handle-to-tensor reconstruction	0.387 s	0.398 s	unchanged; not optimized by this PR
model.load_weights	2.119 s	0.878 s	-1.241 s; explains almost all receiver improvement

A: old callback blocks

Mutually exclusive top-level callback block	Calls	Time
refresh layer size and wrap late parameters	10,472	0.090 s
inspect.Signature.bind and defaults	10,472	0.089 s
retain BoundArguments and count numel	10,472	0.823 s
device-buffer accounting	10,472	0.052 s
trigger completed-layer processing	298	0.459 s

These blocks total approximately 1.513 s. The remaining approximately 0.606 s
inside model.load_weights is model traversal, original-loader work outside
the wrapper, and CUDA completion.

The 0.459 s trigger block contains:

_layerwise_process() sub-block	Time
materialize	0.041 s
reset quantization flag	0.0010 s
unwrap deferred loaders	0.0014 s
replay original loaders	0.354 s
quantize/process after loading	0.016 s
copy back and restore runtime objects	0.019 s

B: direct and fallback work

• 569 layers use direct reload; 29 remain on fallback.
• 10,496 direct loader calls use 0.446 s of CPU dispatch time.
• Fallback reaches the buffered processing path only twice; buffer time is
  0.0008 s and trigger time is approximately 0.0016 s.
• Direct-layout checking for all 598 layers is 0.0062 s.
• Restoring loader metadata for 569 direct layers is 0.0034 s.
• Only the 29 fallback layers capture runtime tensors, restore meta tensors,
  and wrap loaders: 0.0001 / 0.0005 / 0.0021 s.

An independent h200-1 repeat reproduced outer 5.0 -> 3.4 s, receiver
4.145 -> 2.683 s, and model.load_weights 2.088 -> 0.921 s. The A-side
buffer_and_count hotspot remained 0.826 s.

Detailed breakdown: Qwen3-Coder-Next FP8

Receiver critical path

Stage	A	B	B - A
Receiver total	67.067 s	42.303 s	-24.764 s
transfer.initialize	0.057 s	0.028 s	-0.029 s
Lifecycle uncovered	27.634 s	20.899 s	-6.735 s
transfer.receive (56 calls)	39.370 s	21.371 s	-17.999 s
transfer.finalize	0.006 s	0.006 s	approximately unchanged
Outer minus receiver	0.133 s	0.097 s	-0.036 s

Coder uses packed NCCL. broadcast_and_unpack includes the callback:

Nested receive stage	A	B	B - A
NCCL broadcast + unpack + callback	39.369 s	21.369 s	-18.000 s
104 model.load_weights callbacks	37.549 s	19.885 s	-17.664 s

A: old callback blocks

123,807 wrapped loader calls are measured; 122,880 come from
RoutedExperts.

Mutually exclusive top-level callback block	Calls	Time	RoutedExperts contribution
refresh and wrap	123,807	0.993 s	dominant call count
bind arguments	123,807	1.041 s	dominant call count
retain arguments and count numel	123,807	10.986 s	10.907 s
device-buffer accounting	123,807	0.627 s	0.620 s
trigger completed-layer processing	603	4.600 s	4.381 s

The five blocks total approximately 18.247 s. The remaining approximately
19.302 s is model traversal and original-loader/CUDA work outside the wrapper;
it is reported as residual rather than assigned to a function that was not
directly timed.

The trigger block contains:

_layerwise_process() sub-block	Time
materialize	1.046 s
reset quantization flag	0.0017 s
unwrap deferred loaders	0.0027 s
replay original loaders	3.302 s
quantize/process after loading	0.026 s
copy back and restore runtime objects	0.042 s

B: direct and fallback work

• 943 layers use direct reload; 14 remain on fallback.
• 148,381 direct loader calls use 4.070 s of CPU dispatch time.
• 147,456 calls / 4.043 s are RoutedExperts.
• Fallback reaches the buffered processing path only twice, taking about
  0.020 s.
• Direct checking for 957 layers is 0.0106 s.
• Metadata restoration for 943 direct layers is 0.0056 s.
• Capture / restore-meta / wrap for the 14 fallback layers is
  0.00004 / 0.00047 / 0.00177 s.

The direct and old wrapper call counts are not expected to match exactly. The
old wrapper returns early after a layer has already been processed, while the
direct timer records every original-loader invocation. The authoritative
comparison is the CUDA-complete callback boundary: 37.549 -> 19.885 s.

An independent h200-1 repeat reproduced outer 50.0 -> 34.0 s, receiver
49.946 -> 33.988 s, and model.load_weights 34.008 -> 19.338 s. Of the
15.958 s receiver reduction, 14.670 s is again inside the callback.

Detailed breakdown: Qwen3.6-35B-A3B INT4

Qwen is the most important same-backend result because the old path exposes a
pathological interaction between checkpoint ordering and layerwise buffer
accounting.

h200-0 same-host receiver critical path

Stage	A	B	B - A
Receiver total	6608.095 s	23.091 s	-6585.004 s
transfer.initialize	0.048 s	0.026 s	-0.022 s
Lifecycle uncovered	126.173 s	13.993 s	-112.180 s
transfer.receive (50 calls)	6479.109 s	9.065 s	-6470.045 s
transfer.finalize	2.765 s	0.0075 s	-2.758 s
Outer minus receiver	0.105 s	0.109 s	+0.004 s

NCCL broadcast/unpack is 6479.108 -> 9.064 s and includes 50 CUDA-complete
model.load_weights callbacks totaling 6478.328 -> 8.621 s.

The four B workers are tightly grouped at 23.0900-23.0914 s, so the critical
worker is not an outlier.

A: exact old-path hotspot

The critical worker enters 92,773 wrapped loader calls; 92,160 come from
RoutedExperts.

Mutually exclusive top-level callback block	Calls	Time	RoutedExperts contribution
refresh layer size and wrap late parameters	92,773	1.906 s	92,160 calls
bind and normalize loader arguments	92,773	1.735 s	92,160 calls
retain BoundArguments and count numel	92,773	24.519 s	24.434 s
device-buffer accounting and warning	92,773	6431.107 s	6399.295 s
trigger completed-layer processing	463	0.203 s	not dominant

The device-buffer accounting block alone is 97.32% of the profiled outer time;
its RoutedExperts contribution alone is 96.84%.

The five mutually exclusive blocks total 6459.470 s. The remaining 18.857 s
inside model.load_weights contains traversal, can_load/early-return paths,
original-loader work outside the timed wrapper blocks, and CUDA completion.

463 layers process immediately. Another 40 partially loaded WNA16
RoutedExperts process during finalization in 2.755 s. Across both paths,
503 _layerwise_process() calls contain:

_layerwise_process() sub-block	Calls	Time
materialize temporary layer	503	0.102 s
reset quantization flag	503	0.0019 s
unwrap deferred loaders	503	0.0032 s
replay original loaders	503	2.689 s
quantize / repack	503	0.0202 s
copy back and restore runtime objects	503	0.0268 s

These six sub-blocks total approximately 2.843 s. Therefore the 6000+ second
result is not caused by Triton quantization or repacking; it is caused by the
per-weight Python accounting/warning block.

Why accounting becomes pathological

For every arriving device weight, the old wrapper:

1. inserts the parent layer into LOADING_LAYERS;
2. constructs and sorts the class-name list for all currently loading layers;
3. traverses all those layers and sums get_info_size(...);
4. calls logger.warning_once with the computed memory and layer-name list.

The list, sort, and sum execute before entering the logger. In addition,
warning_once is backed by @lru_cache with (logger, message, *args) as the
key. Both mem_used and the layer-name list change as more RoutedExperts
accumulate, producing new cache keys and actual warning writes.

The h200-0 attribution run emits exactly 179,828 such warnings (about 111 MB of
log); B emits zero. During the slow buckets, all eight GPUs show 0% compute
utilization while the four vLLM TP workers use approximately 90-98% CPU. The
GPU is waiting for Python accounting and logging, not executing a slow Triton
kernel.

B: WNA16 direct/runtime-view path

Initialization sees 799 layers:

• 747 are layout-identical direct layers;
• 52 attempt runtime binding;
• 40 RoutedExperts bind successfully;
• 12 layers conservatively remain on fallback.

Measured initialization/finalization cost:

B code block	Calls	Time
direct-layout check	799	0.00755 s
restore direct-loader metadata	747	0.00335 s
runtime-bind attempts	52	0.00469 s
restore original runtime-bound parameters	40	0.00182 s

The loader side contains:

• 92,771 direct loader calls in 2.052 s of CPU dispatch time;
• 92,160 RoutedExperts calls in 2.036 s;
• two fallback loads (embedding and LM head);
• 0.00084 s fallback buffer time;
• 0.00365 s fallback trigger time.

The fallback trigger contains 0.00221 s materialize, 0.00066 s replay, and
0.00055 s copy/restore; all other sub-blocks total less than 0.00006 s.

No WNA16 RoutedExperts layer is materialized, repacked, and copied back in B.
The original loader writes through the runtime-backed checkpoint view into the
existing kernel storage.

Independent Qwen repeat

h200-1 independently reproduces the same result:

Metric	A	B
Outer update	5853.9 s	21.5 s
Receiver critical	5853.846 s	21.460 s
model.load_weights	5664.862 s	8.579 s
Device-buffer accounting	5623.209 s	removed from the direct path
buffer_and_count	21.537 s	0.00091 s fallback only
Direct loader CPU	not applicable	2.115 s
Runtime bind	not applicable	0.00426 s

This is a 272.3x same-host speedup. The h200-1 A run also emits exactly 179,828
accounting warnings, and 5595.712 s of its accounting time is specifically
attributed to RoutedExperts.

The two absolute profiled A times differ because both long runs overlap on a
shared NFS log path and profiling changes synchronization/interleaving. Their
call counts, warning counts, hotspot, and B results agree. Production speedup is
therefore reported from the lower-instrumentation 6075.7 -> 20.6 s result, while
the synchronized profiles are used only for code-block attribution.

What remains after the fast path

The following table locates remaining B cost. Columns are nested; they must not
be added horizontally.

Model	B receiver	Lifecycle uncovered	Receive	model.load_weights	Direct-loader CPU	Callback residual
Moonlight	3.144 s	1.835 s	1.279 s	0.878 s	0.446 s	approximately 0.430 s
Coder-Next	42.303 s	20.899 s	21.371 s	19.885 s	4.070 s	approximately 15.796 s
Qwen3.6 INT4	23.091 s	13.993 s	9.065 s	8.621 s	2.052 s	approximately 6.565 s

The callback residual is inclusive callback time minus direct-loader CPU and
the tiny fallback trigger. It includes model traversal, original-loader CUDA
work, and callback-end CUDA completion; it is not labeled as pure NCCL or pure
kernel time.

The largest remaining top-level component is generally lifecycle uncovered
time: sender/chunk arrival, HTTP/Ray scheduling, and inter-chunk gaps. This PR
does not change transport scheduling.

Profiling caveats

• Transfer chunk and callback boundaries synchronize CUDA for attribution.
• A records multiple perf_counter blocks per wrapped loader call.
• B times direct loaders on CPU only; synchronizing every tensor materially
  perturbs the result and was excluded.
• Parent and child timings are never added.
• A callback residuals are reported explicitly rather than assigned to an
  unmeasured function.
• Production performance comes from low-instrumentation outer timers; detailed
  profiles are used for attribution.
• The two long Qwen A profiles overlapped on shared NFS to keep both H200 nodes
  occupied, so their absolute logging cost is intentionally treated as
  profiling-disturbed.

Despite the instrumentation overhead, the direction and attribution are
stable. For example, Coder's profiled speedup is 67.2 / 42.4 = 1.58x versus
49.4 / 32.0 = 1.54x without detailed profiling.

Validation

Static and unit validation

• final PR head: ce0cf2e2a944cb366fb43de37aa7642f676a786d;
• exact vLLM v0.25.1 validation port: 14c2444aed42705873fc8b614e702c38ed9218a6;
• focused direct/runtime-view reload tests: 9 passed;
• full final-fix pre-commit passed, including ruff, format, mypy, SPDX,
  forbidden-import, configuration, and sign-off checks;
• exact container source passed py_compile.

Tests cover exact key/layout acceptance and rejection, lazy parameters and
buffers, detached mapping rejection before mutation, bound-method metadata,
repeated runtime-view validation, quant-method replacement, storage replacement
detection, model cleanup, alias/non-persistent buffer preservation, WNA16
checkpoint views, and identity mappings in the presence of runtime-only MoE
tensors.

Final-head three-model validation

Only the PR side was rerun; the pre-change measurements above were reused. All
three runs used the same H200 host, pinned image, explicit Triton MoE backend,
and exact final code.

Model	Initial update	Post-train update	Train/rollout logprob abs diff	Result
Qwen3.6-35B-A3B INT4	20.5 s	20.8 s	0.0159767	PR48382_CASE_DONE
Moonlight-16B-A3B FP8	3.5 s	3.5 s	0.0507528	PR48382_CASE_DONE
Qwen3-Coder-Next FP8	33.7 s	31.5 s	0.0144532	PR48382_CASE_DONE

The final revalidation caught and fixed an intermediate fail-closed regression:
a runtime-only MoE tensor made an otherwise identity mapping look non-identity,
which skipped restoration of FP8 scale loader metadata. Identity is now checked
against the checkpoint-reloadable runtime tensor set, and the focused test
includes that exact boundary. No final log contains a weight-update traceback,
endpoint failure, storage/layout validation error, NaN, or invalid MoE scale
quant method.

Cross-framework design validation

The design was checked against 12 locally cloned inference or inference-adjacent
frameworks: SGLang, LMDeploy, TensorRT-LLM, xLLM, LightLLM, TGI, DeepSpeed,
Megatron-LM, llama.cpp, MLC-LLM, FlexFlow, and Dynamo. The common requirements
are: quiesce inference, plan destinations before mutation, let the format owner
define transforms, preserve runtime storage, finalize exactly once, and fail
closed when support cannot be proved.

CUDA graph and transaction boundary

This PR preserves and verifies registered Parameter/Buffer storage used by the
fast path. It deliberately does not recursively walk arbitrary backend object
graphs. RFC #48478 owns explicit registration and a ModelRunner-level completion
gate for graph-visible backend tensors; draft #48902 is useful as an audit
prototype, not the production discovery contract. PR #48908 owns the common
prepare/finish/abort lifecycle.

This is a zero-copy refit, not a full value rollback. If transport fails after
some bytes have been written, correctness requires the worker to remain
paused/unhealthy until full reload or restart; successful binding validation
cannot reconstruct the previous model without retaining a second copy.

Scope and non-goals

• This PR does not choose between Marlin and Triton. [Bugfix][Quantization] Respect explicit WNA16 MoE backend #49065 owns that backend
  selection correction.
• This PR does not optimize sender collectives; VIME Decode error while inferencing a batch of prompts #340 is held constant.
• This PR does not remove the layerwise fallback.
• This PR does not claim the remaining callback residual is one specific
  kernel or transport cost.
• This PR does not opt in unknown quantization methods based on tensor names.
• This PR does not recursively discover unregistered backend tensors; [RFC] Fail-Closed Graph Storage Contract for Weight Reload #48478 owns the explicit graph-storage contract.
• This PR does not promise value rollback after a partially written transfer; failed workers must remain paused.

Related work

• [Bugfix][Quantization] Respect explicit WNA16 MoE backend #49065 fixes WNA16 backend selection so explicit Triton is respected.
• vllm-project/vime#340 optimizes sender collectives and packed transfer; it is
  present on both benchmark sides.
• [Bugfix][Attention] Preserve post-load tensors across weight reloads #48251 fixes graph-captured derived-tensor identity; this PR preserves kernel
  storage while removing redundant reload work.
• Skip layerwise reload for non-quantized models #48284 skips reload for an entirely non-quantized model; this PR selects safe
  layers inside quantized models.
• [Bugfix][Model Loader] Make model.load_weights safe to invoke on already-initialized model #42823 routes repeated raw loading through layerwise reload; this PR optimizes
  that pipeline without changing its entry points.
• [Refactor][Model Loader] Unify the weight loading lifecycle #48908 unifies the prepare/finish/abort lifecycle; this PR supplies safe destination plans within that lifecycle.
• [RFC] Fail-Closed Graph Storage Contract for Weight Reload #48478 defines the production fail-closed graph-storage registry and completion gate.
• [Bugfix][Reload] Preserve unmanaged tensor addresses across weight reload #48902 audits unmanaged backend tensors; production support still requires explicit registration.

AI assistance was used. The submitter reviewed every changed line and ran the
listed validations.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @aoshen02.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.