[Online Quantization] Support memory-efficient online quantization via layerwise loading

[kylesayrs]

Purpose

• Support online quantization in a more maintainable way by integrating with existing layerwise processing functionality

Changes

• Change layerwise logic to only copy and re-place into kernel tensors if reloading

Weights with initialized values

Handling weights which require values placed at init time is a little tricky. One example are rotary embeddings, whose values are created at init time, and are not loaded from disk. In order to avoid overwriting these values with materialized empty tensors, we explicitly exclude these modules from our restore/materialize process. However, handling a weight which initializes some values and loads values would be more complex, although not at all impossible to handle. One way of doing this would be to load values directly into weights, and count those, rather than using get_numel_loaded.

TODO

• Rename "reload" utilities to just "layerwise"
• Break out attention processing into a separate for loop
• Check that there are no side effects from loading and processing the model in device context

Testing

• Quantized reloading regression tests pass (test_reload.py)
• Smoke tested online quantization

from vllm import LLM; llm = LLM("Qwen/Qwen3-0.6B", quantization="fp8", tensor_parallel_size=2)
llm.generate("Hello there")

[gemini-code-assist]

Code Review

This pull request introduces a memory-efficient online quantization method by implementing layerwise loading. This is a significant improvement that refactors specialized loading logic from Fp8OnlineLinearMethod into a more generic and reusable layerwise loading mechanism. The changes are well-structured and adapt the existing reloading infrastructure for initial model loading.

However, I've identified a critical issue where the new generic loading mechanism is enabled for all online FP8 quantization paths, but the necessary refactoring was not applied to Fp8OnlineMoEMethod. This will likely cause issues for MoE models. My detailed comment addresses this with a recommendation for a fix.