[CompressedTensors] FP4 Qutlass Integration

[kylesayrs]

Summary

This PR integrates the NVFP4 Qutlass kernel for compressed-tensors models with NVFP4 weights and online transforms. It supports any block-diagonal transform, and is not limited to deterministic hadamard transforms like hadacore. It is selected as the default, over hadacore, for any applicable configuration. To reduce memory usage, online-transform weights are shared where applicable.

MXFP4 Qutlass integration will be added in follow-up:

• [CompressedTensors] MXFP4 Qutlass Integration neuralmagic/vllm#167

Test Plan:

• added smoke model to unit tests
• Recovery/throughput benchmarks can be viewed in comment below -- [CompressedTensors] FP4 Qutlass Integration #43229 (comment)

Suggested Reviewers

• @mgoin @robertgshaw2-redhat @LopezCastroRoberto

[gemini-code-assist]

Code Review

This pull request implements the process_weights_after_loading and apply methods for the QutlassNvFP4LinearMethod class, facilitating quantized matrix multiplication with CUTLASS and Hadamard matrix normalization. The feedback identifies a potential AttributeError in the weight processing logic, suggesting that self.input_transform.weight should be indexed before accessing its partitions.

[brian-dellabetta]

I added a smoke test for nm-testing/Llama-3.2-1B-Instruct-quipv16-nvfp4 and ran lm-evals for Llama 3 8B. Wikitext PPL is a good amount better, and I am reproducing the results in for NVFP RTN vs. RTN+Had in https://arxiv.org/pdf/2509.23202 Table 5 -- the gsm8k/mmlu score accuracy slightly decreases. QUTLASS results match hadacore

NVFP4

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc		0.6450	±	0.0038
gsm8k	3	flexible-extract	5	exact_match	↑	0.7286	±	0.0122
		strict-match	5	exact_match	↑	0.6126	±	0.0134
gsm8k_llama	3	flexible_extract	5	exact_match	↑	0.7362	±	0.0121
		strict_match	5	exact_match	↑	0.5542	±	0.0137
wikitext	2	none	5	bits_per_byte	↓	0.6978	±	N/A
		none	5	byte_perplexity	↓	1.6221	±	N/A
		none	5	word_perplexity	↓	13.2839	±	N/A

Had16+NVFP4 (hadacore achieves 11.3593 PPL)

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc		0.6299	±	0.0039
gsm8k	3	flexible-extract	5	exact_match	↑	0.7422	±	0.0120
		strict-match	5	exact_match	↑	0.5732	±	0.0136
gsm8k_llama	3	flexible_extract	5	exact_match	↑	0.7218	±	0.0123
		strict_match	5	exact_match	↑	0.5557	±	0.0137
wikitext	2	none	5	bits_per_byte	↓	0.6560	±	N/A
		none	5	byte_perplexity	↓	1.5757	±	N/A
		none	5	word_perplexity	↓	11.3741	±	N/A

Had128+NVFP4 (hadacore achieves 11.3938 PPL)

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
wikitext	2	none	5	bits_per_byte	↓	0.6581	±	N/A
		none	5	byte_perplexity	↓	1.5780	±	N/A
		none	5	word_perplexity	↓	11.4653	±	N/A

[brian-dellabetta]

On this branch, after the changes to always use the fused Nv pathway for fused layers (like qkv and gate_up), I am getting the following results. Unsafe means the old custom op (which will error in torch 2.12+), just to confirm operating in-place and returning modified tensors doesn't affect throughput.

Throughput overhead is about 5% for Had16 and 5.6% for Had128. The default hadacore pathway is typically 3-4% overhead:
vllm bench serve --backend openai-chat --base-url http://localhost:8081 --endpoint /v1/chat/completions --model ./Meta-Llama-3-8B-Instruct-quipv16-nvfp4 --dataset-name random --num-prompts 500 --input-len 2048 --output-len 4096 --max-concurrency 128

Benchmark Metric	Had16+NVFP4	Unsafe Had16+NVFP4	Had128+NVFP4	NVFP4
General Stats
Successful / Failed Requests	500 / 0	500 / 0	500 / 0	500 / 0
Max Request Concurrency	128	128	128	128
Benchmark Duration (s)	215.81	216.18	217.91	205.70
Total Input Tokens	1,028,662	1,028,662	1,028,662	1,028,662
Total Generated Tokens	2,048,000	2,048,000	2,048,000	2,048,000
Throughput
Request Throughput (req/s)	2.32	2.31	2.29	2.43
Output Token Throughput (tok/s)	9,489.78	9,473.68	9,398.43	9,956.15
Peak Output Token Throughput (tok/s)	15,274.00	15,177.00	15,104.00	16,316.00
Peak Concurrent Requests	198.00	190.00	192.00	220.00
Total Token Throughput (tok/s)	14,256.28	14,232.09	14,119.03	14,956.89
Time to First Token (TTFT)
Mean TTFT (ms)	547.33	512.58	607.02	456.39
Median TTFT (ms)	298.96	300.50	307.96	260.40
P99 TTFT (ms)	2,126.56	2,044.50	2,390.69	1,719.46
Time per Output Token (TPOT)
Mean TPOT (ms)	13.06	13.09	13.17	12.46
Median TPOT (ms)	13.27	13.30	13.37	12.66
P99 TPOT (ms)	13.31	13.36	13.43	12.72
Inter-token Latency (ITL)
Mean ITL (ms)	13.05	13.08	13.17	12.46
Median ITL (ms)	12.81	12.81	12.88	12.27
P99 ITL (ms)	17.86	17.91	17.87	17.27

[LopezCastroRoberto]

I have one question left: in our case, the way our transforms are defined, no transposition takes place at runtime, correct? @brian-dellabetta

[brian-dellabetta]

Yeah this only happens for non-hadamard transforms. What we've added on this PR doesn't go through HadamardTransforms's forward method, and if it does it goes through the hadacore transform. this path with the transpose only occurs if dense

[LopezCastroRoberto]

Similar with this type casts. Is this here for safety, or is it actually casting data at runtime?

[brian-dellabetta]

yeah likely just safety. this precedes my PR, we are just removing the scale factor here, absorbing it into the transform weight matrix earlier on

[brian-dellabetta]

Added smoke model test runs successfully locally:

pytest -s tests/quantization/test_compressed_tensors.py -k test_compressed_tensors_transforms_perplexity

but model is failing to initialize in buildkite here

UPDATE: this is because i am testing on a b200 but the buildkite test runs on a device with incorrect compute capability. we want both paths to work. i have updated the method selector to only choose QutlassNvFP4LinearMethod if current_platform.has_device_capability(90)

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @kylesayrs.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[brian-dellabetta]

The test I added is succeeding now, buildkite is failing here on an unrelated error.

I am seeing the same errors in my unrelated 2-liner PR -- #49483

[mgoin]

This should probably be is_device_capability_family(100) if this only works on Blackwell? has is >=