[Online quantization] Add targeted online quantization configuration based on user patterns by fxmarty-amd · Pull Request #51285 · vllm-project/vllm

fxmarty-amd · 2026-08-06T16:58:14Z

Motivation

Enable targeted online quantization on specific layers. This includes:

(future PR) preventing silent layer-specific online quantization:

vllm/vllm/model_executor/layers/attention/mla_attention.py

Line 547 in dc9f845

self.is_aiter_triton_fp8_bmm_enabled = rocm_aiter_ops.is_fp8bmm_enabled()

==> should use this API instead (as this hard-coding causes issues, see [ROCm] Don't re-quantize bf16 MLA kv_b_proj to fp4/fp8 for GLM MoE DSA #48051)
(future PR probably) enable shared expert quantization on checkpoints shipping unquantized shared experts (e.g. useful for evaluation purposes) ==> this is already possible combining linear + ignore excluding everything else, but e.g. FSE is not possible at the moment
Simplify the online quantization API (see e.g. complex pattern --quantization-config '{"linear":"fp8_per_channel","ignore":["lm_head","re:.*embed_tokens.*","re:.*mlp\\.gate$","re:.*expert.*","re:.*kv_b_proj.*"]}' to achieve this targets feature (e.g. in [Quark] Online-requant unquantized layers via quantization_config ove… #49313))

Purpose

Implements per-layer online quantization through quantization_config.targets, which follows the same spec as existing quantization_config.ignore.

This addresses the per-layer online quantization RFC.

Exact layer names, re: regex or fnmatch-style patterns allow to select different online shorthands per linear or MoE layer names patterns.

Implementation details:

unmatched layers remain unquantized,
conflicting matches with ignore raise errors,
fused-layer matching is shared by ignore and targets.

Additionally, an info log is added so that users are aware of what online quantization actually does, as:

INFO 08-06 16:35:33 [base_loader.py:106] Quantized 48 layers of types: block_sparse_moe.experts: 24 (from targets: re:.*block_sparse_moe\.experts, fp8_per_tensor); self_attn.o_proj: 24 (from targets: re:.*self_attn\.o_proj, fp8_per_block)

Additionally, Fused projections can now be targeted either by their fused module name (for example, re:.*self_attn\.qkv_proj) or by patterns that match every original shard (for example, re:.*self_attn\.[qkv]_proj). This was a TODO:

vllm/tests/quantization/test_online.py

Lines 64 to 65 in 81be2e0

    
                       # TODO(future PR): also make 're:.*qkv_proj.*' work 
        
                       {"ignore": ["model.layers.1.self_attn.o_proj", "re:.*[qkv]_proj"]},

Note: this behavior is now aligned for quark/compressed-tensors/online.

Note: previous ambiguous ignore regex as re:.*v_proj that did NOT resolve to qkv_proj now do. This is a "breaking" change. These kind of ambiguous patterns should probably not be used.

Left to do:

Probably left to an other PR: implement online quantization compatibility with other quantization methods (done in [Quantization] Support online quantization with partially pre-quantized checkpoints #51392)
End-to-end test on real model
Other PR: Shared experts fusion support

Test Plan

pytest tests/quantization/test_online.py -s -vvvvv (includes new tests)
pytest tests/quantization/test_config_utils.py -s -vvvvv (includes new tests for targets case)
pytest tests/quantization/test_config_utils.py -s -vvvvv (includes tests for ignore/targets parity and should_ignore_layer refactor correctness)

Test Result

Tests pass.

Disclosure

AI assistance was used to develop this PR.

Co-authored-by: Codex Signed-off-by: Felix Marty <Felix.Marty@amd.com>

mergify · 2026-08-06T16:58:49Z

Documentation preview: https://vllm--51285.org.readthedocs.build/en/51285/

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

Co-authored-by: Codex Signed-off-by: Felix Marty <Felix.Marty@amd.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

BowenBao

overall looks good, haven't finished reviewing all the files

BowenBao · 2026-08-06T22:37:20Z

+!!! info
+
+    - `targets` is mutually exclusive with online `linear` and `moe`: set one or the other, not both.
+    - A layer that matches no `targets` pattern is left unquantized.


Suggested change

- A layer that matches no `targets` pattern is left unquantized.

- A layer that matches no `targets` pattern is left unchanged from its checkpoint state.

suggestion since it may be pre-quantized?

yes, updating, although the compatibility of this PR with pre-quantized checkpoints is not tested.

I suggest to leave online + pre-quantized checkpoint compatibility to an other PR.

BowenBao · 2026-08-06T22:37:46Z

+    - A layer name may not match both `targets` and `ignore`.
+    - A layer may not match more than one `targets` pattern.


what's the behavior when these happen?

error is raised in both cases, see vllm/model_executor/layers/quantization/online/base.py -> _dispatch_target

BowenBao · 2026-08-06T22:42:21Z

+                str(quant_key),
+            )
+
+        return quant_key_str(self.weight)


not including activation?

BowenBao · 2026-08-06T22:48:11Z

@mgoin changes here look reasonable but would love to have another look from your team.

For context, this is to avoid duplicate strictly identical regex / ignore processing between quark/compressed-tensors/online.

Please refer to the comment in the thread below #51285 (comment)

BowenBao · 2026-08-07T00:53:55Z

+import regex as re
+
+
+def check_equal_or_regex_match(layer_name: str, targets: Iterable[str]) -> bool:


seems not needed anymore?

This was still used in quark/utils.py. Removed it there as well in favor of find_matching_patterns there.

Please note the following changes:

On main:

Ignore pattern Quark should_ignore_layer behavior Compressed-tensors should_ignore_layer behavior Online should_ignore_layer behavior

"model.layers.0.self_attn.qkv_proj" Does NOT ignore Ignores Ignores

"re:.*qkv_proj.*" Does NOT ignore Does NOT ignore Does NOT ignore

On this PR:

Ignore pattern Quark should_ignore_layer behavior Compressed-tensors should_ignore_layer behavior Online should_ignore_layer behavior

"model.layers.0.self_attn.qkv_proj" Ignores Ignores Ignores

"re:.*qkv_proj.*" Ignores Ignores Ignores

This fixes a mismatching behavior in compressed-tensors, aligns the three, and simplifies the duplicate in the codebase. The layer_name not in ignore in

vllm/vllm/model_executor/layers/quantization/compressed_tensors/utils.py

Line 66 in 4f76c8a

if proj_name in fused_mapping and layer_name not in ignore:

used to accept model.layers.0.self_attn.qkv_proj ignore pattern, but NOT "re:.*qkv_proj.*" on compressed-tensors/online side.

On Quark side on main, should_ignore_layer does NOT have this layer_name not in ignore check:

vllm/vllm/model_executor/layers/quantization/quark/utils.py

Line 58 in 4f76c8a

if proj_name in fused_mapping:

I believe this alignment is safe on Quark, cc @mgoin @kylesayrs please confirm on compressed-tensors side

mergify · 2026-08-07T04:37:06Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

fxmarty-amd · 2026-08-18T09:56:39Z

@mgoin as discussed, 2d99778 adds support for e.g. fnmatch-style ignore/target detection in online quantization only:

--quantization-config '{"ignore":"*mlp.experts*"}'
--quantization-config '{"targets":"*mlp.experts*":"mxfp4"}'

doc is updated in this regard. This does not impact Quark/compressed-tensors.

mergify · 2026-08-18T19:09:06Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

mergify · 2026-09-01T14:04:35Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

mgoin · 2026-09-01T16:25:56Z

/ci run

github-actions · 2026-09-01T16:26:23Z

✅ Triggered Buildkite CI #86647 for commit 82f60a7e8e89.

Signed-off-by: mgoin <mgoin64@gmail.com>

mgoin · 2026-09-01T18:53:43Z

/ci run

github-actions · 2026-09-01T18:54:01Z

✅ Triggered Buildkite CI #86679 for commit 84b57d27bae5.

Signed-off-by: mgoin <mgoin64@gmail.com>

mgoin · 2026-09-01T19:48:34Z

/ci run

github-actions · 2026-09-01T19:48:52Z

✅ Triggered Buildkite CI #86692 for commit de5da770def5.

…based on user patterns (vllm-project#51285)

Add targeted online quantization configuration

494e2fb

Co-authored-by: Codex Signed-off-by: Felix Marty <Felix.Marty@amd.com>

mergify Bot added documentation Improvements or additions to documentation quantization labels Aug 6, 2026

fxmarty-amd added 3 commits August 6, 2026 12:16

precise doc and extend tests

60ef235

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

test: cover fused quantization target matching

d2bdef8

Co-authored-by: Codex Signed-off-by: Felix Marty <Felix.Marty@amd.com>

validate quantization target keys

0d62eba

Co-authored-by: Codex Signed-off-by: Felix Marty <Felix.Marty@amd.com>

fxmarty-amd marked this pull request as ready for review August 6, 2026 17:36

fxmarty-amd requested review from 22quinn, AndreasKaratzas, ProExpertProg, WoosukKwon, houseroad, mgoin, pavanimajety, robertgshaw2-redhat, tjtanaa, tlrmchlsmth, yewentao256, youkaichao and zyongye as code owners August 6, 2026 17:36

claude Bot reviewed Aug 6, 2026

View reviewed changes

BowenBao reviewed Aug 7, 2026

View reviewed changes

mergify Bot added the needs-rebase label Aug 7, 2026

Merge branch 'main' into per-layer-online-quantization

8c1d40e

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

mergify Bot removed the needs-rebase label Aug 7, 2026

use find_matching_patterns in quark's should_ignore_layer

6e261ce

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

fxmarty-amd requested a review from BowenBao August 7, 2026 09:29

fxmarty-amd mentioned this pull request Aug 7, 2026

[Quantization] Support online quantization with partially pre-quantized checkpoints #51392

Merged

3 tasks

fxmarty-amd added 2 commits August 18, 2026 04:53

reset file

45f01bd

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

Merge branch 'main' into per-layer-online-quantization

e41a132

mergify Bot added the needs-rebase label Aug 18, 2026

Merge branch 'main' into per-layer-online-quantization

a16f942

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

mergify Bot removed the needs-rebase label Aug 19, 2026

explicitely reject unsupported layers from targets

9a14c70

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

mergify Bot added the needs-rebase label Sep 1, 2026

Merge branch 'main' into per-layer-online-quantization

82f60a7

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

mergify Bot removed the needs-rebase label Sep 1, 2026

Merge main into per-layer-online-quantization

84b57d2

Signed-off-by: mgoin <mgoin64@gmail.com>

mgoin force-pushed the per-layer-online-quantization branch from 0f47edf to 84b57d2 Compare September 1, 2026 18:52

Fix online quantization parameter IDs

de5da77

Signed-off-by: mgoin <mgoin64@gmail.com>

fxmarty-amd mentioned this pull request Sep 2, 2026

[Cleanup] Remove online quantization support in fp8.py in favor of online shorthands #53585

Merged

mgoin approved these changes Sep 2, 2026

View reviewed changes

mgoin merged commit 7894394 into vllm-project:main Sep 2, 2026
131 checks passed

mylibrar pushed a commit to tanyuqian/vllm that referenced this pull request Sep 3, 2026

[Online quantization] Add targeted online quantization configuration …

221c683

…based on user patterns (vllm-project#51285)

This was referenced Sep 3, 2026

[RFC]: Per-layer online quantization configuration #50281

Closed

[ROCm][Perf][DeepSeek V4] FP8 shared-expert load-time requant for AITER fused MoE on gfx942 #53847

Draft

stecasta mentioned this pull request Oct 5, 2026

[MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head #59973

Merged

6 tasks done

michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026

[Online quantization] Add targeted online quantization configuration …

c53e9bb

…based on user patterns (vllm-project#51285)

	# TODO(future PR): also make 're:.qkv_proj.' work
	{"ignore": ["model.layers.1.self_attn.o_proj", "re:.*[qkv]_proj"]},

	- A layer that matches no `targets` pattern is left unquantized.
	- A layer that matches no `targets` pattern is left unchanged from its checkpoint state.

		- A layer name may not match both `targets` and `ignore`.
		- A layer may not match more than one `targets` pattern.

		import regex as re


		def check_equal_or_regex_match(layer_name: str, targets: Iterable[str]) -> bool:

Ignore pattern	Quark `should_ignore_layer` behavior	Compressed-tensors `should_ignore_layer` behavior	Online `should_ignore_layer` behavior
`"model.layers.0.self_attn.qkv_proj"`	Does NOT ignore	Ignores	Ignores
`"re:.qkv_proj."`	Does NOT ignore	Does NOT ignore	Does NOT ignore

Conversation

fxmarty-amd commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Purpose

Test Plan

Test Result

Disclosure

Uh oh!

mergify Bot commented Aug 6, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

BowenBao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fxmarty-amd Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 7, 2026

Uh oh!

fxmarty-amd commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Aug 18, 2026

Uh oh!

mergify Bot commented Sep 1, 2026

Uh oh!

mgoin commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

mgoin commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

mgoin commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fxmarty-amd commented Aug 6, 2026 •

edited

Loading

fxmarty-amd Aug 7, 2026 •

edited

Loading

fxmarty-amd commented Aug 18, 2026 •

edited

Loading