[XPU] Add W8A8 FP8 linear kernel with multi-granularity quant support by chaojun-zhang · Pull Request #43645 · vllm-project/vllm

chaojun-zhang · 2026-05-26T05:05:44Z

Purpose

This PR adds XPUW8A8FP8LinearKernel and XPUW8A16FP8LinearKernel for XPU,
replacing the previous XPUFP8ScaledMMLinearKernel with proper support for
multiple weight/activation quantization granularities.

Changes

Add XPUW8A8FP8LinearKernel and XPUW8A16FP8LinearKernel, replacing XPUFP8ScaledMMLinearKernel
Select W8A8 or W8A16 kernel based on whether activation quantization is present
Ensure dynamic activation quantization uses per-token granularity (kFp8DynamicTokenSym) on XPU

Accuracy Tests

Benchmark: gsm8k, 250 samples, 5-shot, eager mode

Test script (acc.py):

import argparse
from lm_eval import evaluator
from lm_eval.models.vllm_causallms import VLLM

parser = argparse.ArgumentParser()
parser.add_argument("--model", required=True)
parser.add_argument("--dtype", default="bfloat16")
parser.add_argument("--tp", type=int, default=1)
parser.add_argument("--quantization", default=None)
parser.add_argument("--gpu-memory-util", type=float, default=0.9)
parser.add_argument("--max-model-len", type=int, default=None)
args = parser.parse_args()

model_kwargs = {
    "pretrained": args.model,
    "dtype": args.dtype,
    "tensor_parallel_size": args.tp,
    "gpu_memory_utilization": args.gpu_memory_util,
    "add_bos_token": True,
    "trust_remote_code": True,
    "enforce_eager": True,
    "linear_backend": "xpu",
}
if args.quantization:
    model_kwargs["quantization"] = args.quantization
if args.max_model_len:
    model_kwargs["max_model_len"] = args.max_model_len

results = evaluator.simple_evaluate(
    model=VLLM(**model_kwargs),
    tasks=["gsm8k"],
    num_fewshot=5,
    limit=250,
    batch_size=20,
)
print(results["results"])

Case Definitions

Case	Weight Quant Key	Act Quant Key	Model	TP	Extra Args
W-tensor/A-static	`kFp8StaticTensorSym`	`kFp8StaticTensorSym`	`RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8`	2	—
W-channel/A-dyn-token	`kFp8StaticChannelSym`	`kFp8DynamicTokenSym`	`RedHatAI/Qwen3-32B-FP8-dynamic`	2	`--max-model-len 30720`
W-tensor/A-dyn-token	`kFp8StaticTensorSym`	`kFp8DynamicTokenSym`	`neuralmagic/Llama-3.2-1B-Instruct-FP8-dynamic`	1	—
W-tensor/A-dyn-tensor	`kFp8StaticTensorSym`	`kFp8DynamicTensorSym`	`meta-llama/Llama-3.2-1B-Instruct`	1	`--quantization fp8`

Each case is run with:

 VLLM_USE_V1=1 python3 acc.py \
    --model <model> --dtype bfloat16 --tp <tp> \
    --block-size 64 --gpu-memory-util 0.9 --eager [extra args]

Results

Case	exact_match,strict-match	exact_match_stderr,strict-match	exact_match,flexible-extract	exact_match_stderr,flexible-extract
W-tensor/A-static	0.724	0.0283	0.776	0.0264
W-channel/A-dyn-token	0.764	0.0269	0.640	0.0304
W-tensor/A-dyn-token	0.288	0.0287	0.284	0.0286
W-tensor/A-dyn-tensor	0.324	0.0297	0.328	0.0298

gemini-code-assist

Code Review

This pull request introduces separate kernels for W8A8 and W8A16 FP8 quantization on XPU (XPUW8A8FP8LinearKernel and XPUW8A16FP8LinearKernel), replacing the single XPUFP8ScaledMMLinearKernel. It also enables dynamic token and tensor quantization support for XPU platforms. A review comment suggests that XPUW8A16FP8LinearKernel should inherit directly from FP8ScaledMMLinearKernel instead of XPUW8A8FP8LinearKernel to prevent a confusing inheritance hierarchy.

mergify · 2026-05-26T05:14:33Z

Hi @chaojun-zhang, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

mergify · 2026-05-26T05:23:25Z

Hi @chaojun-zhang, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

jikunshang · 2026-05-26T09:13:55Z

cc @zufangzhu

zufangzhu · 2026-05-27T04:39:48Z

@@ -108,7 +108,7 @@ def __init__(self):

        self.weight_quant_key = kFp8StaticTensorSym
        # Use per-token quantization for better perf if dynamic and cutlass
-        if cutlass_fp8_supported():
+        if cutlass_fp8_supported() or current_platform.is_xpu():


Have you compare per-token and per-tensor performance on xpu?

Yes, I ran a systematic comparison (TP=2, meta-llama/Llama-3.2-1B-Instruct).

W-tensor / A-dyn-token

Workload out-tok/s TTFT mean(ms) TTFT median(ms) TTFT P99(ms) TPOT mean(ms) TPOT median(ms) TPOT P99(ms)

np=1, in=128, out=512 31.1 4733.9 4733.9 4733.9 22.96 22.96 22.96

np=4, in=128, out=512 334.0 267.1 334.9 335.6 11.46 11.34 11.79

np=13, in=128, out=512 985.4 132.7 122.6 156.8 12.94 12.95 13.02

np=1, in=1024, out=512 91.6 91.4 91.4 91.4 10.75 10.75 10.75

np=4, in=1024, out=512 336.9 197.8 231.5 232.8 11.49 11.44 11.66

np=13, in=1024, out=512 1083.4 410.3 435.7 599.7 11.19 11.14 11.71

np=1, in=2048, out=128 58.9 148.8 148.8 148.8 15.94 15.94 15.94

np=4, in=2048, out=128 224.5 330.2 396.4 397.2 15.31 14.83 16.70

np=13, in=2048, out=128 682.7 817.4 817.1 1150.5 12.59 12.60 17.21

np=32, in=512, out=256 2410.1 557.3 578.6 754.9 11.08 11.01 12.52

np=32, in=256, out=512 2842.9 354.9 431.0 433.5 10.56 10.42 10.98

W-tensor / A-dyn-tensor

Workload out-tok/s TTFT mean(ms) TTFT median(ms) TTFT P99(ms) TPOT mean(ms) TPOT median(ms) TPOT P99(ms)

np=1, in=128, out=512 34.7 3703.5 3703.5 3703.5 21.60 21.60 21.60

np=4, in=128, out=512 278.7 216.5 273.9 274.9 13.92 13.84 14.14

np=13, in=128, out=512 1099.8 140.7 129.3 165.9 11.54 11.56 11.62

np=1, in=1024, out=512 74.1 103.6 103.6 103.6 13.32 13.32 13.32

np=4, in=1024, out=512 348.0 208.8 243.3 244.5 11.09 11.04 11.26

np=13, in=1024, out=512 1064.1 501.2 464.8 631.6 11.23 11.30 11.88

np=1, in=2048, out=128 83.6 164.0 164.0 164.0 10.75 10.75 10.75

np=4, in=2048, out=128 280.5 353.4 423.0 424.5 11.53 11.02 13.00

np=13, in=2048, out=128 634.4 878.8 878.6 1235.3 13.54 13.57 18.45

np=32, in=512, out=256 2226.1 586.9 617.9 797.1 12.05 11.93 13.47

np=32, in=256, out=512 2281.2 366.5 455.9 461.3 13.30 13.14 13.74

Conclusion

Neither granularity consistently wins — the optimal choice is workload-dependent:

Per-token is better for decode-heavy + high-concurrency scenarios (e.g. np=32, in=256, out=512: +24% out-tok/s, TPOT 10.56 vs 13.30 ms)

Per-tensor is better for large-batch long-prefill scenarios (e.g. np=4, in=2048, out=128: +25% out-tok/s, TPOT 11.53 vs 15.31 ms)

For moderate batch sizes and mixed workloads, differences are within ±5%

Since there is no universally better choice , I preserve the existing default (per-token)

Our current tests don't really use small inputs or outputs, so maybe per-tensor is better choice.
@jikunshang what do you think?

updated and use default per-tensor

zufangzhu · 2026-05-27T04:40:34Z

@@ -308,7 +308,7 @@ def __init__(self, quant_config: Fp8Config):
            # Use per-token quantization for better perf if dynamic and cutlass
            if self.act_q_static:
                self.activation_quant_key = kFp8StaticTensorSym
-            elif cutlass_fp8_supported():
+            elif cutlass_fp8_supported() or current_platform.is_xpu():


Same with above, compare performance then choose better one

see #43645 (comment)

zufangzhu · 2026-05-27T04:44:34Z

Is reasonable for the accuracy result of :
W-tensor/A-dyn-token
W-tensor/A-dyn-tensor

chaojun-zhang · 2026-05-28T02:17:17Z

Is reasonable for the accuracy result of :
W-tensor/A-dyn-token
W-tensor/A-dyn-tensor

Yes, the accuracy results are reasonable. Validated with GSM8k (5-shot, 250 samples) for meta-llama/Llama-3.2-1B-Instruct:

Configuration	Weight	Activation	GSM8k strict-match
BF16 baseline (no quantization)	—	—	0.316 ± 0.029
online FP8 (`--quantization fp8`)	static-tensor	dyn-token	0.344 ± 0.030
online FP8 (`--quantization fp8`)	static-tensor	dyn-tensor	0.368 ± 0.031
pre-quantized FP8-dynamic	static-tensor	dyn-token	0.300 ± 0.029
pre-quantized FP8-dynamic	static-tensor	dyn-tensor	0.300 ± 0.029

zufangzhu

Just one small suggestion; LGTM

zufangzhu · 2026-05-29T08:49:59Z

+                    self._check_scheme_supported(
+                        CompressedTensorsW8A8Fp8.get_min_capability(), error=False
+                    )
+                    or current_platform.is_xpu()


Since this function _check_scheme_supported is to check capability, can we directly return true inner it.

jikunshang · 2026-06-01T02:05:09Z

@@ -335,6 +335,9 @@ def get_config_filenames(cls) -> list[str]:
    def _check_scheme_supported(
        min_capability: int, error: bool = True, match_exact: bool = False
    ) -> bool:
+        if current_platform.is_xpu():


we should be careful about adding such change...

So I was using _check_scheme_supported or current_platform.is_xpu() at the external call site for this check — do you think that works?

jikunshang · 2026-06-01T02:06:51Z

+    _SUPPORTED_ACT_QUANT_KEYS = {
+        kFp8DynamicTensorSym,
+        kFp8DynamicTokenSym,
+        kFp8StaticTensorSym,
+        kFp8StaticTokenSym,
+    }
+    _SUPPORTED_WEIGHT_QUANT_KEYS = {
+        kFp8StaticChannelSym,
+        kFp8StaticTensorSym,
+    }


do we support all _SUPPORTED_WEIGHT_QUANT_KEYS x _SUPPORTED_ACT_QUANT_KEYS ?

Yes, all 8 combinations (2 weight types × 4 activation types) are supported

mergify · 2026-06-03T15:35:25Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaojun-zhang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2026-07-01T05:54:13Z

Documentation preview: https://vllm--43645.org.readthedocs.build/en/43645/

jikunshang · 2026-07-01T06:30:54Z

+    commands:
+      - >-
+        bash .buildkite/scripts/hardware_ci/run-intel-test.sh
+        'python3 examples/basic/offline_inference/generate.py --linear-backend xpu --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 --dtype bfloat16 --block-size 64 --gpu-memory-utilization 0.9 --enforce-eager --max-model-len 4096 &&


we can remove --dtype bfloat16 --block-size 64 --gpu-memory-utilization 0.9

jikunshang · 2026-07-01T06:31:56Z

@@ -316,28 +319,51 @@ def check_model(model):
        assert output


-def test_compressed_tensors_fp8(vllm_runner):
+@pytest.mark.parametrize(


emmm, necessary to change these uts?

zufangzhu · 2026-07-02T07:39:21Z

+                    config = get_current_vllm_config_or_none()
+                    is_fp8_w8a8_supported = (
+                        config is not None
+                        and config.kernel_config.linear_backend == "xpu"


--linear-backend xpu-w8a8 ?

offline talked with @jikunshang , we will add two linear-backend options ('xpu' and 'xpu_woq' ).

--linear-backend xpu: Use the default XPU kernels. (e.g. w8a8 kernel for fp8) .

--linear-backend xpu_woq: Use weight only quant kernels

Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com>

Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>

Replace the xpu_w8a8_fp8 linear backend with two clearer options: - xpu: W8A8 and block-scaled quantization kernels - xpu_woq: weight-only quantization kernels (W8A16) Update CI commands, documentation, and kernel selection logic. Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…vllm-project#43645) Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com> Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

chaojun-zhang requested review from mgoin, pavanimajety, robertgshaw2-redhat, tlrmchlsmth, yewentao256 and zyongye as code owners May 26, 2026 05:05

mergify Bot added the intel-gpu Related to Intel GPU label May 26, 2026

gemini-code-assist Bot reviewed May 26, 2026

View reviewed changes

Comment thread vllm/model_executor/kernels/linear/scaled_mm/xpu.py Resolved

chaojun-zhang force-pushed the xpu_w8a8_fp8 branch from eeaa914 to 7333788 Compare May 26, 2026 05:18

chaojun-zhang force-pushed the xpu_w8a8_fp8 branch 3 times, most recently from 5cc2270 to feae6eb Compare May 26, 2026 05:44

zufangzhu reviewed May 27, 2026

View reviewed changes

yuwenzho mentioned this pull request May 27, 2026

[XPU] Enable compressed-tensors FP8 block-scaled quantization #43657

Closed

4 tasks

chaojun-zhang force-pushed the xpu_w8a8_fp8 branch 2 times, most recently from 842f839 to 9c63545 Compare May 29, 2026 02:22

chaojun-zhang requested a review from zufangzhu May 29, 2026 08:44

zufangzhu reviewed May 29, 2026

View reviewed changes

zufangzhu approved these changes May 29, 2026

View reviewed changes

chaojun-zhang force-pushed the xpu_w8a8_fp8 branch from 9c63545 to e461379 Compare May 29, 2026 09:20

jikunshang reviewed Jun 1, 2026

View reviewed changes

mergify Bot added the needs-rebase label Jun 3, 2026

chaojun-zhang force-pushed the xpu_w8a8_fp8 branch from 9c5b346 to 09807e7 Compare July 1, 2026 01:56

chaojun-zhang requested a review from AndreasKaratzas as a code owner July 1, 2026 05:53

mergify Bot added the documentation Improvements or additions to documentation label Jul 1, 2026

jikunshang reviewed Jul 1, 2026

View reviewed changes

chaojun-zhang force-pushed the xpu_w8a8_fp8 branch from 223eb3a to ae1419b Compare July 1, 2026 07:39

jikunshang approved these changes Jul 2, 2026

View reviewed changes

jikunshang added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 2, 2026

zufangzhu reviewed Jul 2, 2026

View reviewed changes

chaojun-zhang force-pushed the xpu_w8a8_fp8 branch 2 times, most recently from b454531 to 945bbad Compare July 2, 2026 12:47

chaojun-zhang and others added 3 commits July 3, 2026 09:53

[XPU] Add W8A8 FP8 linear kernel with multi-granularity quant support

ad39ee2

Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com>

[XPU] Make FP8 W8A8 selection opt-in

36963b2

Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>

chaojun-zhang force-pushed the xpu_w8a8_fp8 branch from 13ac077 to 64bb275 Compare July 3, 2026 09:54

Merge branch 'main' into xpu_w8a8_fp8

bd49c64

jikunshang approved these changes Jul 4, 2026

View reviewed changes

jikunshang merged commit 1a308c4 into vllm-project:main Jul 4, 2026
115 checks passed

Jrojas90 mentioned this pull request Jul 9, 2026

[Bug][XPU] compressed-tensors FP8 W8A8 (dynamic) generates garbage output on Intel Arc Pro B70 (Battlemage) #48058

Closed

chaojun-zhang mentioned this pull request Aug 3, 2026

[XPU] Route block-quantized FP8 weights to the W8A8 kernel #50787

Merged

peymanr mentioned this pull request Nov 4, 2026

[ROCm][MiniMax-M3.1] MXFP8 linear on AITER's tuned FlyDSL GEMM #65479

Open

Workload	out-tok/s	TTFT mean(ms)	TTFT median(ms)	TTFT P99(ms)	TPOT mean(ms)	TPOT median(ms)	TPOT P99(ms)
np=1, in=128, out=512	31.1	4733.9	4733.9	4733.9	22.96	22.96	22.96
np=4, in=128, out=512	334.0	267.1	334.9	335.6	11.46	11.34	11.79
np=13, in=128, out=512	985.4	132.7	122.6	156.8	12.94	12.95	13.02
np=1, in=1024, out=512	91.6	91.4	91.4	91.4	10.75	10.75	10.75
np=4, in=1024, out=512	336.9	197.8	231.5	232.8	11.49	11.44	11.66
np=13, in=1024, out=512	1083.4	410.3	435.7	599.7	11.19	11.14	11.71
np=1, in=2048, out=128	58.9	148.8	148.8	148.8	15.94	15.94	15.94
np=4, in=2048, out=128	224.5	330.2	396.4	397.2	15.31	14.83	16.70
np=13, in=2048, out=128	682.7	817.4	817.1	1150.5	12.59	12.60	17.21
np=32, in=512, out=256	2410.1	557.3	578.6	754.9	11.08	11.01	12.52
np=32, in=256, out=512	2842.9	354.9	431.0	433.5	10.56	10.42	10.98

Workload	out-tok/s	TTFT mean(ms)	TTFT median(ms)	TTFT P99(ms)	TPOT mean(ms)	TPOT median(ms)	TPOT P99(ms)
np=1, in=128, out=512	34.7	3703.5	3703.5	3703.5	21.60	21.60	21.60
np=4, in=128, out=512	278.7	216.5	273.9	274.9	13.92	13.84	14.14
np=13, in=128, out=512	1099.8	140.7	129.3	165.9	11.54	11.56	11.62
np=1, in=1024, out=512	74.1	103.6	103.6	103.6	13.32	13.32	13.32
np=4, in=1024, out=512	348.0	208.8	243.3	244.5	11.09	11.04	11.26
np=13, in=1024, out=512	1064.1	501.2	464.8	631.6	11.23	11.30	11.88
np=1, in=2048, out=128	83.6	164.0	164.0	164.0	10.75	10.75	10.75
np=4, in=2048, out=128	280.5	353.4	423.0	424.5	11.53	11.02	13.00
np=13, in=2048, out=128	634.4	878.8	878.6	1235.3	13.54	13.57	18.45
np=32, in=512, out=256	2226.1	586.9	617.9	797.1	12.05	11.93	13.47
np=32, in=256, out=512	2281.2	366.5	455.9	461.3	13.30	13.14	13.74

Conversation

chaojun-zhang commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Changes

Accuracy Tests

Case Definitions

Results

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

mergify Bot commented May 26, 2026

Uh oh!

mergify Bot commented May 26, 2026

Uh oh!

jikunshang commented May 26, 2026

Uh oh!

zufangzhu May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

W-tensor / A-dyn-token

W-tensor / A-dyn-tensor

Conclusion

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chaojun-zhang May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zufangzhu commented May 27, 2026

Uh oh!

chaojun-zhang commented May 28, 2026

Uh oh!

zufangzhu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jun 3, 2026

Uh oh!

mergify Bot commented Jul 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

chaojun-zhang commented May 26, 2026 •

edited

Loading

zufangzhu May 27, 2026 •

edited

Loading

chaojun-zhang May 29, 2026 •

edited

Loading