[CI] Fix online FP8 quantization materializing tensors on CPU

[haosdent]

Purpose

Address CI failures:

• https://buildkite.com/vllm/ci/builds/58606/steps/canvas?sid=019d36b8-7870-4d4e-abce-ba03d2561896&tab=output
• https://buildkite.com/vllm/ci/builds/58628/steps/canvas?sid=019d382e-c763-4230-a089-84d52dd42e6a&tab=output

After #38426 moved load_weights() outside the with target_device: context (to fix OOM), online FP8 quantization broke in two ways:

1. materialize_meta_tensor() uses torch.empty_strided() without a device= arg, relying on the ambient device context. Without it, tensors are created on CPU and process_weights_after_loading fails with NotImplementedError: Could not run '_C::dynamic_scaled_fp8_quant' with arguments from the 'CPU' backend.

2. Fp8OnlineMoEMethod.process_weights_after_loading() creates scale tensors with torch.ones(...) without device=, causing them to land on CPU and crash the Triton fused MoE kernel with ValueError: Pointer argument (at 5) cannot be accessed from Triton (cpu tensor?).

Test Plan

pytest tests/quantization/test_fp8.py::test_online_quantization \
  tests/quantization/test_fp8.py::test_online_quant_peak_mem \
  tests/quantization/test_fp8.py::test_online_quant_load_format_dummy \
  tests/compile/fullgraph/test_full_graph.py::test_fp8_kv_scale_compile[Qwen/Qwen2-0.5B-None-0] \
  tests/compile/fullgraph/test_full_graph.py::test_fp8_kv_scale_compile[Qwen/Qwen2-0.5B-None-3] -v

Test Result

All 8 tests pass:

tests/quantization/test_fp8.py::test_online_quantization[False-False-auto] PASSED
tests/quantization/test_fp8.py::test_online_quantization[False-False-fp8] PASSED
tests/quantization/test_fp8.py::test_online_quantization[False-True-auto] PASSED
tests/quantization/test_fp8.py::test_online_quantization[False-True-fp8] PASSED
tests/quantization/test_fp8.py::test_online_quant_peak_mem PASSED
tests/quantization/test_fp8.py::test_online_quant_load_format_dummy PASSED
tests/compile/fullgraph/test_full_graph.py::test_fp8_kv_scale_compile[Qwen/Qwen2-0.5B-None-0] PASSED
tests/compile/fullgraph/test_full_graph.py::test_fp8_kv_scale_compile[Qwen/Qwen2-0.5B-None-3] PASSED

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[gemini-code-assist]

Code Review

This pull request modifies the model loading logic to support explicit device targeting during tensor materialization. By updating materialize_meta_tensor and materialize_layer to accept a device parameter, the code now ensures tensors are created on the correct platform device. The review feedback identifies the use of a private PyTorch API for type hinting and recommends using public types to ensure long-term stability.

[haosdent]

@jikunshang Can you help to take a look when you are available? Many thanks!

[haosdent]

Ah, didn't notice #38442 could address the CI failure mentioned in this PR , but I think #38442 is indeed better since it addresses the issue systematically.
While #38442 may take longer to review since it changes more files, I still think this PR is necessary since it could unblock the CI failure before #38442 is merged.

[jikunshang]

thanks for quick fixing! #38442 is approved now. let's wait for CI result. hope it can be merged soon.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @haosdent.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork