Skip to content

[CT] Support Humming for WNA16 MoE - #48918

Merged
mgoin merged 24 commits into
vllm-project:mainfrom
yiliu30:yiliu/ct-humming-w2a16
Aug 19, 2026
Merged

mgoin merged 24 commits into
vllm-project:mainfrom
yiliu30:yiliu/ct-humming-w2a16

Conversation

@yiliu30

@yiliu30 yiliu30 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Enable Humming for compressed-tensors WNA16 MoE checkpoints.

This change:

  • extends the WNA16 MoE Humming bridge to accept compressed-tensors QuantizationArgs
  • allows the Humming path to initialize for sub-byte WNA16 checkpoints instead of hard-failing in the Marlin-only path
  • uses the checkpoint layout Humming expects for MoE weights
  • adds regression coverage for the W2A16 Humming route
  • adds a small helper script for the local offline inference repro command

Why This Is Not Duplicating An Existing PR

I checked for open PRs in the same area and did not find an existing one covering this fix.

Model Evaluation Results

vllm ({'pretrained': 'INC4AI/Qwen3-30B-A3B-Instruct-2507-Attn4bits-Mlp2bits-AutoRound-200iters-TestOnly', 'tensor_parallel_size': 1, 'max_model_len': 8192, 'max_num_batched_tokens': 32768, 'max_num_seqs': 128, 'add_bos_token': True, 'gpu_memory_utilization': 0.4, 'dtype': 'bfloat16', 'max_gen_toks': 2048, 'enable_prefix_caching': False}), gen_kwargs: ({}), limit: 1000.0, num_fewshot: None, batch_size: 128
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match||0.909|±  |0.0091|
|     |       |strict-match    |     5|exact_match||0.909|±  |0.0091|

AI Assistance

This PR was created with AI assistance. The submitting human is responsible for reviewing every changed line and validating the change end-to-end.

cc @mgoin @bradhilton @hshen14 @thuang6

yiliu30 and others added 3 commits July 15, 2026 11:45
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify

mergify Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--48918.org.readthedocs.build/en/48918/

@mergify mergify Bot added the documentation Improvements or additions to documentation label Jul 17, 2026
@yiliu30 yiliu30 changed the title Support Humming for compressed-tensors WNA16 MoE [CT] Support Humming for WNA16 MoE Jul 17, 2026
yiliu30 and others added 6 commits July 17, 2026 06:22
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
yiliu30 and others added 2 commits July 17, 2026 13:29
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
@mgoin mgoin added ready ONLY add when PR is ready to merge/full CI is needed quantization labels Jul 21, 2026
@mergify

mergify Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @yiliu30.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: yiliu30 <yi4.liu@intel.com>
@mergify

mergify Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Hi @yiliu30, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Co-authored-by: Codex <codex@openai.com>

Signed-off-by: yiliu30 <yi4.liu@intel.com>
@mergify

mergify Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Hi @yiliu30, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

@yiliu30

yiliu30 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #84356 for commit 0377ea651d8f.

@mergify

mergify Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Hi @yiliu30, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com>
@yiliu30
yiliu30 force-pushed the yiliu/ct-humming-w2a16 branch from b888898 to 3894612 Compare August 18, 2026 08:03
@mergify

mergify Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Hi @yiliu30, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Comment thread vllm/model_executor/layers/quantization/utils/quant_utils.py
@mergify

mergify Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Hi @yiliu30, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

@mgoin

mgoin commented Aug 18, 2026

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #84461 for commit b55b88dc4cb9.

@yiliu30

yiliu30 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #84511 for commit e9f894ecc274.

@mgoin
mgoin merged commit 54dd98b into vllm-project:main Aug 19, 2026
125 checks passed
zyp2014 pushed a commit to zyp2014/vllm that referenced this pull request Aug 21, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
wyettzeng pushed a commit to wyettzeng/vllm that referenced this pull request Aug 21, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Wyett <wyettzeng@gmail.com>
zufangzhu pushed a commit to zufangzhu/vllm that referenced this pull request Aug 24, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>
khushali9 pushed a commit to khushali9/vllm that referenced this pull request Aug 29, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: khushali9 <khushali.desai9@gmail.com>
dsikka pushed a commit to vllm-project/llm-compressor that referenced this pull request Sep 1, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>

SUMMARY:
"please provide a brief summary"


- model
https://huggingface.co/INC4AI/Qwen3-30B-A3B-Instruct-2507-Attn4bits-Mlp2bits-AutoRound-200iters-TestOnly

```bash
# vllm ({'pretrained': '/home/yiliu7/workspace/llmc-ds/examples/autoround/quantization_wNa16/Qwen3-30B-A3B-Instruct-2507-Attn4bits-Mlp2bits-AutoRound-200iters', 'tensor_parallel_size': 1, 'max_model_len': 8192, 'max_num_batched_tokens': 32768, 'max_num_seqs': 128, 'add_bos_token': True, 'gpu_memory_utilization': 0.4, 'dtype': 'bfloat16', 'max_gen_toks': 2048, 'enable_prefix_caching': False}), gen_kwargs: ({}), limit: 1000.0, num_fewshot: None, batch_size: 128
# |Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
# |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
# |gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.909|±  |0.0091|
# |     |       |strict-match    |     5|exact_match|↑  |0.909|±  |0.0091|
```

The evluation is depends on vllm's PR
vllm-project/vllm#48918

---------

Signed-off-by: yiliu30 <yi4.liu@intel.com>
Signed-off-by: Yi Liu <yi4.liu@intel.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
am-cohere pushed a commit to am-cohere/vllm that referenced this pull request Sep 1, 2026
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation quantization ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants