[chaojun-zhang]

Purpose

This PR fixes an issue following #43645 and routes block-quantized weights to the W8A8 kernel unconditionally (no opt-in flag --linear-backend xpu required), while keeping the existing opt-in behavior for per-tensor/per-channel weights.

Test Plan

vllm serve RedHatAI/Qwen3-32B-FP8-block --port 8072 \
    --trust-remote-code --gpu-memory-util=0.9 \
    --max-num-batched-tokens=8192 --max-model-len=6000 \
    --no-enable-prefix-caching --block-size 64 -tp=2

Test Result

Without this PR: server fails to start, worker crashes during load_model:

Traceback

(Worker_TP0 pid=319002) INFO 08-03 00:56:48 [gpu_model_runner.py:5308] Starting to load model RedHatAI/Qwen3-32B-FP8-block...
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912] WorkerProc failed to start.
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912] Traceback (most recent call last):
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/v1/executor/multiproc_executor.py", line 879, in worker_main
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     worker = WorkerProc(*args, **kwargs)
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/v1/executor/multiproc_executor.py", line 648, in __init__
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     self.worker.load_model()
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/v1/worker/gpu_worker.py", line 442, in load_model
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/v1/worker/gpu_model_runner.py", line 5324, in load_model
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     self.model = model_loader.load_model(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     model = initialize_model(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/model_loader/utils.py", line 62, in initialize_model
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     model = model_class(vllm_config=vllm_config, prefix=prefix)
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/models/qwen3.py", line 293, in __init__
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     self.model = Qwen3Model(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/models/qwen3.py", line 266, in __init__
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     super().__init__(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/models/qwen2.py", line 375, in __init__
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     self.start_layer, self.end_layer, self.layers = make_layers(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/models/utils.py", line 824, in make_layers
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     + get_offloader().wrap_modules(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/offloader/base.py", line 104, in wrap_modules
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     return list(modules_generator)
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/models/utils.py", line 825, in <genexpr>
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     layer_fn(prefix=f"{prefix}.{idx}") for idx in range(start_layer, end_layer)
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/models/qwen2.py", line 377, in <lambda>
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     lambda prefix: decoder_layer_type(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/models/qwen3.py", line 198, in __init__
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     self.self_attn = Qwen3Attention(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/models/qwen3.py", line 105, in __init__
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     self.qkv_proj = QKVParallelLinear(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/layers/linear.py", line 1090, in __init__
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     super().__init__(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/layers/linear.py", line 504, in __init__
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     self.quant_method.create_weights(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 1001, in create_weights
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     layer.scheme.create_weights(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a16_fp8.py", line 112, in create_weights
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     self.linear_kernel = init_wfp8_a16_linear_kernel(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/kernels/linear/__init__.py", line 959, in init_wfp8_a16_linear_kernel
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     kernel_type = choose_scaled_mm_linear_kernel(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]   File "/work/vllm/vllm/model_executor/kernels/linear/__init__.py", line 604, in choose_scaled_mm_linear_kernel
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912]     raise ValueError(
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912] ValueError: Failed to find a kernel that can implement the ScaledMM linear layer. Reasons:
(Worker_TP0 pid=319002) ERROR 08-03 00:56:49 [multiproc_executor.py:912] XPUW8A16FP8LinearKernel XPUW8A16FP8Linear only support per-channel and per-tensor quantization.

Root cause: block-quantized weights were still routed to CompressedTensorsW8A16Fp8 by default (no --linear-backend xpu flag), whose kernel selection (XPUW8A16FP8LinearKernel.can_implement) explicitly rejects the BLOCK strategy.

With this PR: block-quant weights are routed to CompressedTensorsW8A8Fp8 / XPUFp8BlockScaledMMKernel instead, which does support the BLOCK strategy, and the model loads successfully.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[jikunshang]

cc @zufangzhu PTAL

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaojun-zhang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[zufangzhu]

Hi @chaojun-zhang please aware this pr: #49664
we have a torch backend which also support w8a8 per-tensor and w8a8 per-channel/per-token. And block-wise kernel will be enabled as well when #50826 merged.

[chaojun-zhang]

Hi @chaojun-zhang please aware this pr: #49664 we have a torch backend which also support w8a8 per-tensor and w8a8 per-channel/per-token. And block-wise kernel will be enabled as well when #50826 merged.

Thanks @zufangzhu , I added logic that if w8a8 is not enabled and the policy is block_quant, then I set it to true .

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaojun-zhang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jikunshang]

/ci run

[github-actions]

✅ Triggered Buildkite CI #83523 for commit eb35a945d032.