[xuebwang]

Purpose

Fixes background issue #52833

VLLM_ROCM_USE_AITER_FP8BMM and VLLM_ROCM_USE_AITER_FP4BMM both default to True. Once VLLM_ROCM_USE_AITER=1 is set, MLA kv_b_proj weights are re-quantized online regardless of the precision the checkpoint asked for, and there is no per-model opt-out.

kv_b_proj holds [W_UK; W_UV], which decompress the latent KV cache on every read, so the perturbation applies to the whole KV history rather than to a single GEMM. That is why the damage shows up first in long context — exactly the
regime where these defaults have not been validated.

Two data points:

• [ROCm] Don't re-quantize bf16 MLA kv_b_proj to fp4/fp8 for GLM MoE DSA #48051 measures ~11% error per attention layer from this path, accumulating into repetition loops.
• On amd/GLM-5.2-MXFP4, MTP acceptance rate goes 0% -> 58.5% when the re-quantization is skipped ([Bug]: GLM-5.2 MTP accepts 0% of drafts on MI355X (gfx950); disabling expert parallelism hits hipErrorIllegalAddress #52833).

This PR flips both defaults to False. Accuracy over speed until the kernels are validated in long context. Anyone who wants them still gets them with VLLM_ROCM_USE_AITER_FP8BMM=1 / VLLM_ROCM_USE_AITER_FP4BMM=1.

Changes

• vllm/envs.py: both dataclass annotations and both os.getenv defaults flip True -> False; the two "By default is enabled" comments now say what the flags do and why they are off.
• tests/kernels/quantization/test_rocm_fp8.py: ENV_DEFAULTS pins the new VLLM_ROCM_USE_AITER_FP8BMM default.
• tests/kernels/quantization/test_rocm_mxfp4.py: test_fp4_env_defaults likewise.

Test Plan

• tests/kernels/quantization/test_rocm_fp8.py
• tests/kernels/quantization/test_rocm_mxfp4.py

Test Result

Passed.

--end--

=======================================
History for ones who may be interested

Purpose

Fixes issue #52833

This PR replaces #48051’s model-name guard with the generic rule:

kv_b_proj.quant_config is not None
and isinstance(kv_b_proj.quant_method, UnquantizedLinearMethod)

When that condition is true, both AITER FP4BMM and FP8BMM re-quantization are disabled for the projection and MLA uses the existing BF16 BMM path.

The behavior remains unchanged for:

• fully BF16 models, where quant_config is null;
• projections with a real quantized method; and
• DeepSeek-V3-family Quark checkpoints whose intentional dynamic MXFP4
  attention path selects QuarkLinearMethod.

The current dispatch treats a BF16 weight as sufficient evidence that this online re-quantization is safe. That is not true for quantized checkpoints which explicitly exclude kv_b_proj to preserve it in high precision.

Test Plan

focused unit tests

python -m pytest tests/v1/attention/test_mla_backends.py

gfx950 model validation

export VLLM_USE_V2_MODEL_RUNNER=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FP4BMM=1
export VLLM_ROCM_USE_AITER_FP8BMM=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1

serve_args=(
  /home/xuebwang/models/amd/GLM-5.2-MXFP4
  --port 8000
  --kv-cache-dtype fp8_e4m3
  --trust-remote-code
  --tensor-parallel-size 8
  --enable-expert-parallel
  --linear-backend aiter
  --moe-backend aiter
  --tool-call-parser glm47
  --enable-auto-tool-choice
  --reasoning-parser glm45
  --max-model-len 262144
  --max-num-seqs 2
  --gpu-memory-utilization 0.80
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'
)
vllm serve "${serve_args[@]}"

Test Result

Mean acceptance length: 3.92
Accepted: 193
Drafted: 330
Avg Draft acceptance rate: 58.5%
Per-position acceptance rate: 0.909, 0.727, 0.545, 0.424, 0.318

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

Documentation preview: https://vllm--53123.org.readthedocs.build/en/53123/

[fxmarty-amd]

I think these self.is_aiter_triton_fp8_bmm_enabled / self.is_aiter_triton_fp4_bmm_enabled flags were precisely meant to bypass any exclusion rule, e.g. with https://huggingface.co/amd/DeepSeek-R1-0528-MXFP4 - see my offline message

See #52167 => these flags are enabled by default, silently turning on online quantization while users are unaware. This should be moved to use --quantization-config.targets.

Enabling silent online quantization that may have an impact on accuracy is not great IMO

[xuebwang]

@fxmarty-amd :

• amd/DeepSeek-R1-0528-MXFP4 isn't affected by this PR. It keeps re-quantization on every layer, only checkpoints that never opted in are denied. The guard looks at dynamic_mxfp4_quant, and that flag is the opt-in you mean: it's the thing that turns the exclusion into a bypass. maybe_update_config only switches it on for DeepSeek-V3-family fp4 checkpoints, so the two models go different ways:

	amd/DeepSeek-R1-0528-MXFP4	amd/GLM-5.2-MXFP4 (model in #52833)
model_type	deepseek_v3	glm_moe_dsa
kv_b_proj excluded	yes, all 61 layers	yes, 79 entries
dynamic_mxfp4_quant	True	False
get_quant_method	QuarkLinearMethod	UnquantizedLinearMethod
this PR	allows, unchanged	denies — the fix

• --quantization-config.targets is complementary: an override only helps who already know a checkpoint is misquantized, while this fixes the default that GLM-5.2-MXFP4 MTP goes 0% → 58.5% with no flag.
• On silence: agree. The test now derives dynamic_mxfp4_quant from maybe_update_config instead of setting it by hand.

[xuebwang]

cc @BowenBao @ColinZ22

[fxmarty-amd]

dynamic_mxfp4_quant has nothing to do with this PR afaik?

[xuebwang]

Correct, it's not in the diff. The guard only checks isinstance(kv_b_proj.quant_method, UnquantizedLinearMethod)

[fxmarty-amd]

I fail to understand

vllm/vllm/model_executor/layers/quantization/quark/quark.py

Lines 87 to 112 in a60c66e

	def maybe_update_config(
	self,
	model_name: str,
	hf_config: PretrainedConfig | None = None,
	revision: str | None = None,
	):
	"""Enable dynamic MXFP4 only for DeepSeek-V3-family fp4 checkpoints."""
	if hf_config is None:
	return
	if (
	getattr(hf_config, "model_type", None)
	not in _DEEPSEEK_V3_FAMILY_MODEL_TYPES
	):
	return
	quant_config = getattr(hf_config, "quantization_config", None)
	if isinstance(quant_config, dict):
	quant_dtype = (
	quant_config.get("global_quant_config", {})
	.get("weight", {})
	.get("dtype")
	)
	if quant_dtype == "fp4":
	self.dynamic_mxfp4_quant = True

that is FP4-specific,

vllm/vllm/model_executor/layers/attention/mla_attention.py

Lines 1112 to 1124 in a60c66e

	if self.is_aiter_triton_fp4_bmm_enabled:
	from vllm.model_executor.layers.quantization.quark.utils import (
	quark_quantize_weight_to_mxfp4,
	)
	self.W_K, self.W_K_scale = quark_quantize_weight_to_mxfp4(W_UK)
	# Convert from (L, N, P) to (N, L, P)
	self.W_K = self.W_K.transpose(0, 1)
	self.W_K_scale = self.W_K_scale.transpose(0, 1)
	self.W_V, self.W_V_scale = quark_quantize_weight_to_mxfp4(
	W_UV.permute(1, 2, 0)
	)

Completely unrelated to:

vllm/vllm/model_executor/layers/quantization/quark/schemes/quark_ocp_mx.py

Line 145 in a60c66e

w_q, w_s = dynamic_mxfp4_quant(layer.weight)

I also fail to see the connection with FP8:

self.is_aiter_triton_fp8_bmm_enabled -> _is_mla_bmm_requantization_allowed(kv_b_proj) and how it would relate to dynamic_mxfp4_quant

[fxmarty-amd]

--quantization-config.targets is complementary

It is not complementary, it should be the proper long-term solution, removing all this aiter-specific logic. The current vllm behavior of silently quantizing layers by default without users opting in is the cause of issues as #48051.

[tjtanaa]

@xuebwang-amd I would prefer for now, we just change the env variables of VLLM_ROCM_USE_AITER_FP4BMM=0 and VLLM_ROCM_USE_AITER_FP8BMM=0 as when these two are enabled by default, they have not been validated well in long context. The part that they are quantizing seems to be very sensitive to accuracy changes.

[xuebwang]

Thanks @tjtanaa
Yes, quantize kv_b_proj would probably impact accuracy, it perturbs the whole KV history on every read, which is exactly why it would surface in long context first. This can be seen from two measurements: #48051 reports ~11% error per attention layer accumulating into repetition loops, and on GLM-5.2-MXFP4 the MTP acceptance rate goes 0% → 58.5% once the requantization is skipped.

[fxmarty-amd]

@xuebwang-amd TJ's point is to switch:

vllm/vllm/envs.py

Lines 147 to 148 in 7a99938

	VLLM_ROCM_USE_AITER_FP8BMM: bool = True
	VLLM_ROCM_USE_AITER_FP4BMM: bool = True

to False by default.

[xuebwang]

I understand the solution. This is also what @vllmellm and I pointed out in the background issue #52833.

[tjtanaa]

Let's do this first, accuracy over speed for now.

@xuebwang-amd TJ's point is to switch:

vllm/vllm/envs.py

Lines 147 to 148 in 7a99938

	VLLM_ROCM_USE_AITER_FP8BMM: bool = True
	VLLM_ROCM_USE_AITER_FP4BMM: bool = True

to False by default.

[xuebwang]

OK, re-purpose of this PR to be turning off VLLM_ROCM_USE_AITER_*BMM by default for MLA accuracy first. PR title and description are updated as well.
Please have a review, thanks! @tjtanaa @vllmellm @fxmarty-amd @AndreasKaratzas

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @xuebwang-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[tjtanaa]

LGTM

[tjtanaa]

/ci run

[github-actions]

✅ @xuebwang, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[github-actions]

✅ Triggered Buildkite CI #94742 for commit 84e0d2aa4dc7.

[fxmarty-amd]

Hey TJ, feel free to review #55741 as well that moves these VLLM_ROCM_USE_AITER_FP8BMM and VLLM_ROCM_USE_AITER_FP4BMM to https://docs.vllm.ai/en/stable/features/quantization/online/. I will update the PR.

[xuebwang]

Glad to see this PR landing. Thanks @tjtanaa ! and @fxmarty-amd as well.

[fxmarty-amd]

thanks @xuebwang :D