Skip to content

[XPU] Fix inc int4 model - #50209

Closed
mayuyuace wants to merge 5 commits into
vllm-project:mainfrom
mayuyuace:qiming/fix-inc-int4
Closed

[XPU] Fix inc int4 model#50209
mayuyuace wants to merge 5 commits into
vllm-project:mainfrom
mayuyuace:qiming/fix-inc-int4

Conversation

@mayuyuace

@mayuyuace mayuyuace commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Refer to #48555 and patch from the comment.

  1. Dispatch INC W4 model on XPU
  2. The shape of Inc moe model is [E, N, K] which is different with normal moe model, so Inc moe model does not need _process_weights_xpu.

Verified with Qwen/Qwen3-30B-A3B-GPTQ-Int4, Intel/Qwen3-30B-A3B-Instruct-2507-int4-AutoRound.

gurwinderintel and others added 4 commits July 29, 2026 01:51
gurwinderintel @mayuyuace
[XPU] Route INC WNA16 MoE to oracle backend instead of bf16 dequant
402293a 
INCWna16Scheme.get_moe_method returned UnquantizedFusedMoEMethod for XPU,
materializing every expert as full bf16. On a 24GB card an int4 AutoRound MoE
(e.g. Qwen3-30B-A3B) OOMs at load. XPU already has a working quantized MoE path
(the WNA16 oracle -> XPUExpertsWNA16, added in vllm-project#41426), reached via
_resolve_gptq_moe -> MoeWNA16Method, so the dequant fallback is both wasteful
and broken on memory-constrained cards.

- Drop XPU from the dequant early-return (keep CPU, which lacks a quantized MoE
  path) so sym-int4 GPTQ/AutoRound flows to _resolve_gptq_moe.
- Gate use_marlin off for XPU in _resolve_gptq_moe: check_moe_marlin_supports_layer
  is platform-agnostic, but Marlin is CUDA-only (torch.ops._C), so XPU must take
  the MoeWNA16Method path, not AutoGPTQMoEMethod.

Verified on Arc Pro B60 with Intel/Qwen3-30B-A3B-Instruct-2507-int4-AutoRound:
- Before: OOM at load ('not enough GPU memory', UnquantizedFusedMoEMethod).
- After: loads and generates correctly via the XPU WNA16 backend.

Fixes vllm-project#47937.

Signed-off-by: gurwinderintel <gurwinder.singh@intel.com>
@mayuyuace
fix INC shape
c355687 
Signed-off-by: mayuyuace <qiming1.zhang@intel.com>
@mayuyuace
file mode
0811c16 
Signed-off-by: mayuyuace <qiming1.zhang@intel.com>
@mayuyuace
remove useless code
94e806e 
Signed-off-by: mayuyuace <qiming1.zhang@intel.com>
claude[bot]
claude Bot reviewed Jul 29, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added quantization intel-gpu Related to Intel GPU labels Jul 29, 2026
@urakozz urakozz mentioned this pull request Jul 29, 2026
Closed
@mergify

mergify Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mayuyuace.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jul 30, 2026
@mayuyuace
Merge branch 'main' into qiming/fix-inc-int4
ef8783b 
Signed-off-by: Qiming Zhang <qiming1.zhang@intel.com>
@mayuyuace mayuyuace closed this Jul 30, 2026
@sgurwinderr

sgurwinderr commented Jul 31, 2026

Copy link
Copy Markdown

@mayuyuace could we reconsider closing this? I don't think #47124 fully closes the bug — I think it only makes it much harder to hit.

#47124 routes INC to AutoGPTQConfig/AutoAWQConfig only when check_moe_marlin_supports_layer passes. When it fails, _resolve_gptq_moe still falls through to MoeWNA16Method (inc_wna16_scheme.py, the use_marlin branch), and since XPU no longer takes the bf16 dequant early return, that now reaches the oracle.

At that point nothing stops it: _backend_incompatibility_reason rejects MoeWNA16Config for MARLIN / BATCHED_MARLIN / EMULATION, but not XPU (int_wna16.py:152-156). So it goes into _process_weights_xpu, which is exactly the mismatch you documented in this PR.

Concrete case — Intel/Qwen3-30B-A3B-Instruct-2507-int4-AutoRound at TP=4: moe_intermediate_size 768 shards to 192, and 192 % 128 != 0 fails the layer-shape check. TP=1/2 (what you verified) passes it, which is why it looks fixed. Caveat: I traced this statically, I don't have XPU hardware — could you sanity-check at TP=4?

If it reproduces, two options:

  1. Reopen this PR — skipping _process_weights_xpu for N-first sources is the better fix, since it keeps the config working.
  2. A one-liner: add WNA16MoEBackend.XPU to the tuple at int_wna16.py:153-155. Smaller and covers every caller, but it turns the case into a clear NotImplementedError instead of making it work.

I'd prefer 1 if you're up for it; happy to send 2 as a stopgap otherwise. Either way it beats silently transposing the weights.

@mayuyuace

mayuyuace commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

@mayuyuace could we reconsider closing this? I don't think #47124 fully closes the bug — I think it only makes it much harder to hit.

#47124 routes INC to AutoGPTQConfig/AutoAWQConfig only when check_moe_marlin_supports_layer passes. When it fails, _resolve_gptq_moe still falls through to MoeWNA16Method (inc_wna16_scheme.py, the use_marlin branch), and since XPU no longer takes the bf16 dequant early return, that now reaches the oracle.

At that point nothing stops it: _backend_incompatibility_reason rejects MoeWNA16Config for MARLIN / BATCHED_MARLIN / EMULATION, but not XPU (int_wna16.py:152-156). So it goes into _process_weights_xpu, which is exactly the mismatch you documented in this PR.

Concrete case — Intel/Qwen3-30B-A3B-Instruct-2507-int4-AutoRound at TP=4: moe_intermediate_size 768 shards to 192, and 192 % 128 != 0 fails the layer-shape check. TP=1/2 (what you verified) passes it, which is why it looks fixed. Caveat: I traced this statically, I don't have XPU hardware — could you sanity-check at TP=4?

If it reproduces, two options:

  1. Reopen this PR — skipping _process_weights_xpu for N-first sources is the better fix, since it keeps the config working.
  2. A one-liner: add WNA16MoEBackend.XPU to the tuple at int_wna16.py:153-155. Smaller and covers every caller, but it turns the case into a clear NotImplementedError instead of making it work.

I'd prefer 1 if you're up for it; happy to send 2 as a stopgap otherwise. Either way it beats silently transposing the weights.

192 % 128 != 0 means that N=192 but group size =128.
So this model cannot run with tp=4. In this case you can use tp+ep=4 to launch this case to avoid the error.
I think it is expected.

@mayuyuace
mayuyuace deleted the qiming/fix-inc-int4 branch August 5, 2026 06:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments
@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner
@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner
@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner
@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner
@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety is a code owner
@zyongye zyongye Awaiting requested review from zyongye zyongye is a code owner

Assignees

No one assigned

Labels

intel-gpu Related to Intel GPU needs-rebase quantization

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mayuyuace @sgurwinderr