[CT] Support Humming for WNA16 MoE - #48918
Conversation
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: yiliu30 <yi4.liu@intel.com>
|
Documentation preview: https://vllm--48918.org.readthedocs.build/en/48918/ |
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: yiliu30 <yi4.liu@intel.com>
…llm-fork into yiliu/ct-humming-w2a16
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: yiliu30 <yi4.liu@intel.com>
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: yiliu30 <yi4.liu@intel.com>
|
This pull request has merge conflicts that must be resolved before it can be |
|
Hi @yiliu30, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
Co-authored-by: Codex <codex@openai.com> Signed-off-by: yiliu30 <yi4.liu@intel.com>
|
Hi @yiliu30, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
|
/ci run |
|
✅ Triggered Buildkite CI #84356 for commit |
|
Hi @yiliu30, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: yiliu30 <yi4.liu@intel.com>
b888898 to
3894612
Compare
|
Hi @yiliu30, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
|
Hi @yiliu30, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
|
/ci run |
|
✅ Triggered Buildkite CI #84461 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #84511 for commit |
Signed-off-by: yiliu30 <yi4.liu@intel.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Misha Goin <mgoin64@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Misha Goin <mgoin64@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: Wyett <wyettzeng@gmail.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Misha Goin <mgoin64@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Misha Goin <mgoin64@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: khushali9 <khushali.desai9@gmail.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com> SUMMARY: "please provide a brief summary" - model https://huggingface.co/INC4AI/Qwen3-30B-A3B-Instruct-2507-Attn4bits-Mlp2bits-AutoRound-200iters-TestOnly ```bash # vllm ({'pretrained': '/home/yiliu7/workspace/llmc-ds/examples/autoround/quantization_wNa16/Qwen3-30B-A3B-Instruct-2507-Attn4bits-Mlp2bits-AutoRound-200iters', 'tensor_parallel_size': 1, 'max_model_len': 8192, 'max_num_batched_tokens': 32768, 'max_num_seqs': 128, 'add_bos_token': True, 'gpu_memory_utilization': 0.4, 'dtype': 'bfloat16', 'max_gen_toks': 2048, 'enable_prefix_caching': False}), gen_kwargs: ({}), limit: 1000.0, num_fewshot: None, batch_size: 128 # |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr| # |-----|------:|----------------|-----:|-----------|---|----:|---|-----:| # |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.909|± |0.0091| # | | |strict-match | 5|exact_match|↑ |0.909|± |0.0091| ``` The evluation is depends on vllm's PR vllm-project/vllm#48918 --------- Signed-off-by: yiliu30 <yi4.liu@intel.com> Signed-off-by: Yi Liu <yi4.liu@intel.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: yiliu30 <yi4.liu@intel.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Misha Goin <mgoin64@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Summary
Enable Humming for compressed-tensors WNA16 MoE checkpoints.
This change:
Hummingbridge to accept compressed-tensorsQuantizationArgsHummingpath to initialize for sub-byte WNA16 checkpoints instead of hard-failing in the Marlin-only pathWhy This Is Not Duplicating An Existing PR
I checked for open PRs in the same area and did not find an existing one covering this fix.
Model Evaluation Results
vllm ({'pretrained': 'INC4AI/Qwen3-30B-A3B-Instruct-2507-Attn4bits-Mlp2bits-AutoRound-200iters-TestOnly', 'tensor_parallel_size': 1, 'max_model_len': 8192, 'max_num_batched_tokens': 32768, 'max_num_seqs': 128, 'add_bos_token': True, 'gpu_memory_utilization': 0.4, 'dtype': 'bfloat16', 'max_gen_toks': 2048, 'enable_prefix_caching': False}), gen_kwargs: ({}), limit: 1000.0, num_fewshot: None, batch_size: 128 |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr| |-----|------:|----------------|-----:|-----------|---|----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.909|± |0.0091| | | |strict-match | 5|exact_match|↑ |0.909|± |0.0091|AI Assistance
This PR was created with AI assistance. The submitting human is responsible for reviewing every changed line and validating the change end-to-end.
cc @mgoin @bradhilton @hshen14 @thuang6