[Online quantization] Support activation quant key override

[fxmarty-amd]

Purpose

Support activation quant key override in online quantization as e.g.:

--quantization online --quantization-config.targets '{"*self_attn*":{"weight":"mxfp4","activation":"mxfp8"}}'

Currently, activation override for online quantization raises a NotImplementedError, but overriding the activation quant key is truly needed to dispatch on preferred MOE backends (e.g. AITER_MXFP4_BF16, AITER_MXFP4_MXFP4, AITER_MXFP4_FP8) that can not be selected in any other way.

More context at #52167

TODO in an other PR:

• Add __str__ method to QuantKey directly.

Test Plan

• pytest tests/quantization/test_quantization_config_args.py -s -vvvvv
• pytest tests/quantization/test_online.py -s -vvvvv

Example usage on MI350:

VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3.5-35B-A3B \
    --quantization online \
    --quantization-config.targets '{"*experts*":{"weight":"mxfp4","activation":null}}' \
    --tensor-parallel-size 1

showing:

[mxfp4.py:606] Using 'AITER_MXFP4_BF16' Mxfp4 MoE backend.
[base_loader.py:111] Quantized 40 layers of types: mlp.experts: 40 (from targets: *experts*, {"weight":"mxfp4","activation":null})

Test Result

Passing on MI350

[mergify]

Documentation preview: https://vllm--51808.org.readthedocs.build/en/51808/

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[BowenBao]

Nice. Left some questions / comments. Overall looks good

[BowenBao]

Probably offtopic, does this only support gpt-oss for now? if not let's extend the description here with other mxfp4 models (kimi k3, deepseek v4) and other quantizers (quark, etc)

[fxmarty-amd]

Updated, this should read as "weight quantization dtype specific" (weight quant key), this previous doc is outdated.

Note however that some linear/MOE oracles do NOT support activation quant key, see e.g.

vllm/vllm/model_executor/layers/fused_moe/oracle/mxfp8.py

Line 107 in 402547d

def select_mxfp8_moe_backend(

[fxmarty-amd]

Fixed in 7846c2061c808d9757b550d3d2358ff9694fb40e

[BowenBao]

Is it sound to use None as no change to default? Not sure if other places use None to represent BF16.

[BowenBao]

also how is the default activation quant key for each quant method documented / decided?

[fxmarty-amd]

This is a good point. Historically, None represents "no quantization"/BF16, see:

vllm/vllm/config/quantization.py

Lines 25 to 36 in 75dde08

	# User-facing names addressable from quantization_config.
	QUANT_KEY_NAMES: dict[str, QuantKey] = {
	"fp8_per_tensor_static": kFp8StaticTensorSym,
	"fp8_per_tensor_dynamic": kFp8DynamicTensorSym,
	"fp8_per_token": kFp8DynamicTokenSym,
	"fp8_per_channel_static": kFp8StaticChannelSym,
	"fp8_per_block_static": kFp8Static128BlockSym,
	"fp8_per_block_dynamic": kFp8Dynamic128Sym,
	"mxfp8": kMxfp8Dynamic,
	"mxfp4": kMxfp4Dynamic,
	"int8_per_channel_static": kInt8StaticChannelSym,
	}

However we do need to distinguish between a default None (currently on main):

vllm/vllm/config/quantization.py

Lines 65 to 73 in 75dde08

	@config
	class QuantSpec:
	"""Quantization spec for one layer kind (linear or MoE).
	`None` on either side means the method class falls back to its own default
	(typically inherited from the checkpoint, or unquantized for online).
	"""
	weight: QuantKeyField = None

and user-set --quantization-config {"linear": {"activation": "null"}}, this is why weight: QuantKeyField = _UNSET, activation: QuantKeyField = _UNSET is introduced in this PR, and used in https://github.com/fxmarty-amd/vllm/blob/7846c2061c808d9757b550d3d2358ff9694fb40e/vllm/config/quantization.py#L97-L110

[BowenBao]

any reason not to create a dedicated enum for unset?

[fxmarty-amd]

Do you mean why does None represent unquantized activation? Not certain about this, there has historically been no QuantKey as kUnquantized for example.

This would indeed allow to represent "unset" as None, rather. But I think it'd be a broader refactor as current vLLM codebase considers None to be unquantized.

[BowenBao]

oh I was thinking to add the enum kUnset, for kernel to use for their default behavior. So they don't clobber with None. Not sure if that's good idea. Let's raise this in next sig meeting.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[github-actions]

✅ @fxmarty-amd, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[fxmarty-amd]

will wait for #51392 first.

[mergify]

Hi @fxmarty-amd, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[BowenBao]

LGTM, missed the approval stamp last time.