Skip to content

Deprecate old FP8 online MoE quantization class - #44514

Merged
vllm-bot merged 2 commits into
vllm-project:mainfrom
yma11:deprecate-fp8
Jun 23, 2026
Merged

Deprecate old FP8 online MoE quantization class#44514
vllm-bot merged 2 commits into
vllm-project:mainfrom
yma11:deprecate-fp8

Conversation

@yma11

@yma11 yma11 commented Jun 4, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Purpose

Deprecate old FP8 online quantization classes Fp8OnlineLinearMethod and Fp8OnlineMoEMethod, as it can be replaced by new online quantization frontend.

Test Plan

pytest -s -v tests/quantization/test_fp8.py

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@mergify

mergify Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--44514.org.readthedocs.build/en/44514/

@mergify mergify Bot added the documentation Improvements or additions to documentation label Jun 4, 2026
@mergify

mergify Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Hi @yma11, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?
mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally: 
# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

AndreasKaratzas
AndreasKaratzas reviewed Jun 4, 2026
View reviewed changes
Comment thread tests/quantization/test_fp8.py
if kv_cache_dtype == "fp8":
attn = model.model.decoder.layers[0].self_attn.attn
assert isinstance(attn.quant_method, Fp8KVCacheMethod)
assert attn._k_scale == 1.0
assert attn._v_scale == 1.0

if current_platform.is_cuda():
if current_platform.is_cuda() or current_platform.is_xpu():

@AndreasKaratzas AndreasKaratzas Jun 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@divakar-amd could you please check this part if it should be cuda_alike?

@divakar-amd divakar-amd Jun 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine. The rocm case is handled below at line 182/194 and is unaffected.

divakar-amd
divakar-amd reviewed Jun 5, 2026
View reviewed changes
Comment thread tests/quantization/test_fp8.py
if kv_cache_dtype == "fp8":
attn = model.model.decoder.layers[0].self_attn.attn
assert isinstance(attn.quant_method, Fp8KVCacheMethod)
assert attn._k_scale == 1.0
assert attn._v_scale == 1.0

if current_platform.is_cuda():
if current_platform.is_cuda() or current_platform.is_xpu():

@divakar-amd divakar-amd Jun 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine. The rocm case is handled below at line 182/194 and is unaffected.

Comment thread tests/quantization/utils.py Outdated
@yma11
yma11 force-pushed the deprecate-fp8 branch 2 times, most recently from ce9f545 to 735ee3e Compare June 8, 2026 07:00
@yma11

yma11 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

@AndreasKaratzas Can you take a look again to see whether okay to merge?

@mergify

mergify Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @yma11.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 8, 2026
@AndreasKaratzas

AndreasKaratzas commented Jun 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

You got a merge conflict. But also I am not codeowner for this. Tests look alright, but source needs to be reviewed by codeowners

@mergify mergify Bot removed the needs-rebase label Jun 9, 2026
@yma11

yma11 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

@mgoin @yewentao256 Can you PTAL? Thanks.

@mergify

mergify Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @yma11.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 16, 2026
@mergify mergify Bot removed the needs-rebase label Jun 17, 2026
@yma11
yma11 force-pushed the deprecate-fp8 branch 2 times, most recently from b2b70d7 to 8c088b2 Compare June 17, 2026 01:51
@yma11 yma11 changed the title Deprecate old FP8 online quantization classes Deprecate old FP8 online MoE quantization class Jun 17, 2026
@yma11

yma11 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@yewentao256 I found your PR #45463 has removed Fp8OnlineLinearMethod so I update my PR to remove the MoE class, can you take a look?

yewentao256
yewentao256 reviewed Jun 17, 2026
View reviewed changes

@yewentao256 yewentao256 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work!

Comment thread vllm/model_executor/model_loader/base_loader.py
@@ -65,7 +65,7 @@ def load_model(

# Log peak GPU memory after loading weights. This is needed
# to have test coverage on peak memory for online quantization.
if current_platform.is_cuda_alike():
if current_platform.is_cuda_alike() or current_platform.is_xpu():

@yewentao256 yewentao256 Jun 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to update here, and related XPU config?

@yma11 yma11 Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. This change is to make XPU pass on test_fp8.py.

yewentao256
yewentao256 approved these changes Jun 18, 2026
View reviewed changes

@yewentao256 yewentao256 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the work!

@yewentao256 yewentao256 added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 18, 2026
yma11 added 2 commits June 23, 2026 03:22
@yma11
Deprecate old FP8 online quantization classes
382cf39 
Signed-off-by: Yan Ma <yan.ma@intel.com>
@yma11
update
e923ea9 
Signed-off-by: Yan Ma <yan.ma@intel.com>
@vllm-bot
vllm-bot merged commit acce57d into vllm-project:main Jun 23, 2026
85 of 87 checks passed
nkzhenhua pushed a commit to nkzhenhua/vllm that referenced this pull request Jun 24, 2026
@yma11 @nkzhenhua
Deprecate old FP8 online MoE quantization class (vllm-project#44514)
fed57f2 
Signed-off-by: Yan Ma <yan.ma@intel.com>
wincent8 pushed a commit to wincent8/vllm that referenced this pull request Jun 29, 2026
@yma11 @wincent8
Deprecate old FP8 online MoE quantization class (vllm-project#44514)
71a54f5 
Signed-off-by: Yan Ma <yan.ma@intel.com>
efschu pushed a commit to efschu/shvllm that referenced this pull request Jul 18, 2026
@yma11
Deprecate old FP8 online MoE quantization class (vllm-project#44514)
19c832c 
Signed-off-by: Yan Ma <yan.ma@intel.com>
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
@yma11
Deprecate old FP8 online MoE quantization class (vllm-project#44514)
128837c 
Signed-off-by: Yan Ma <yan.ma@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AndreasKaratzas AndreasKaratzas AndreasKaratzas left review comments
@divakar-amd divakar-amd divakar-amd left review comments
@mgoin mgoin mgoin approved these changes
@yewentao256 yewentao256 yewentao256 approved these changes
@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner
@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety is a code owner
@zyongye zyongye Awaiting requested review from zyongye zyongye is a code owner
@22quinn 22quinn Awaiting requested review from 22quinn
@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@yma11 @AndreasKaratzas @mgoin @yewentao256 @divakar-amd @vllm-bot