Skip to content

[Hardware][Power] Unqualized MoE Backend for Power (VSX) - #51624

Merged
bigPYJ1151 merged 25 commits into
vllm-project:mainfrom
Akashcodes732:feat/unquant_moe_power
Aug 13, 2026
Merged

[Hardware][Power] Unqualized MoE Backend for Power (VSX)#51624
bigPYJ1151 merged 25 commits into
vllm-project:mainfrom
Akashcodes732:feat/unquant_moe_power

Conversation

@Akashcodes732

@Akashcodes732 Akashcodes732 commented Aug 10, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

This PR adds PowerPC specific unquantized backend support for fused MoE using Power10 VSX MMA instructions.

Purpose

Currently, grouped GEMM is not supported for Power architecture in vLLM. This PR introduces a Power/VSX specific unquantized CPU backend for Fused MoE.
Key features include:

  • Addition of csrc/cpu/micro_gemm/cpu_micro_gemm_vsx.hpp to support grouped GEMM on Power architectures.
  • Use of Power10 MMA instructions for optimized BF16 matrix multiplication.

Test Plan

  • Verified mathematical correctness across all unaligned and aligned MoE kernel test cases using: 
    pytest tests/kernels/moe/test_cpu_fused_moe.py -v
  • Performance benchmarking using vllm bench with google/gemma-4-26B-A4B-it to compare against the main branch.

Test Result

The unquantized Power10 MMA implementation yields massive speedups in the prefill phase (TTFT), and provides strong throughput gains at higher batch sizes compared to the main branch.

Concurrency TTFT Mean (Prefill) vs Main Total Tok/s (Throughput) vs Main TPOT Mean (Decode) vs Main
C=1 35.5% faster 1.1% faster same as main
C=4 35.6% faster same as main same as main
C=8 37.0% faster same as main same as main
C=64 35.4% faster 15.7% faster 9.9% faster
C=128 35.1% faster 13.9% faster 10.2% faster
Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Akash kaothalkar and others added 22 commits August 3, 2026 16:33
Akash kaothalkar @shimonenator
Add Power10 MMA Grouped GEMM integration for Fused MoE
5269ddb 
Co-authored-by: Antigravity <antigravity@google.com>
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar @shimonenator
Fix __builtin_mma_build_acc signature
64985bd 
Co-authored-by: Antigravity <antigravity@google.com>
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Fix CPU MoE python frontend to select VSX kernels on PowerPC
440c7a4 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Fix KeyError for vsx in tests
83870b3 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Fix Load-Hit-Store memory stalls in VSX kernel
78cab83 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Pack A matrix ahead of time to eliminate LHS penalties completely in VSX
3ca7bcc 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar @shimonenator
Remove dead union VecBF16 code
dae0f99 
Co-authored-by: Antigravity <antigravity@google.com>
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar @shimonenator
Fix TPOT memory inflation by replacing AOT packing with explicit scal…
b272cf4 
…ar loads

Use explicit unrolled constexpr branches for scalar vector initialization
to eliminate zero-padding memory overhead during decode (M=1) while
maintaining maximum L1 pipeline utilization during prefill (M=8).

Co-authored-by: Antigravity <antigravity@google.com>
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar @shimonenator
Optimize M=1 MoE decode by bypassing unnecessary memory copy
6707d46 
When actual_token_num == 1 (Decode phase) and PackA == false, the
input row is already contiguous. Bypass the unnecessary INT8Vec64
gather phase and pass the global input pointer directly to the
GEMM kernel to eliminate 28KB of RAM-to-RAM copying per token.

Co-authored-by: Antigravity <antigravity@google.com>
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Optimize PackA=false math loop using vector merges
8ce6756 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Optimize VSX vector transposition and packing
26b4c60 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Revert GCC unrolls on GER loops to fix pipeline stalls
9a3882f 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Optimize TPOT Decode: Revert MoE memory bypass and remove heavy vec_m…
06baac2 
…erge tree in favor of scalar loads

Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Revert scalar loads (hurt M=4) and remove GCC unrolls on init/disasse…
8653f6b 
…mbly loops to fully restore main TPOT performance

Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Revert unnecessary Little Endian vec_merge changes to fully restore m…
2768257 
…ain TPOT

Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Implement native VSX vector fallback for M<=4 to fix TPOT regression …
08d336d 
…on Power10

Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Keep NSize=16 but interleave weights to eliminate L1 cache thrashing …
33dba47 
…and fix Gemma 4 26B crash

Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Fix missing braces in cpu_micro_gemm_vsx.hpp
4c2f28b 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Merge remote-tracking branch 'vllm/main' into feat/unquant_moe_power
46c12b7
Akash kaothalkar
pre-commit fix
5f66b7f 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Fix unrolled loop MMA data loading in VSX GEMM kernel
9f38360 
The unrolled M>=4 loop incorrectly used 16-bit vec_merge instructions which scrambled the A matrix rows and columns before feeding them to xvbf16ger2pp, causing math errors for K>=8. This fixes it to use 32-bit and 64-bit merges to construct the correct 4x4 transposed layout for the A registers.

Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Akash kaothalkar
Merge remote-tracking branch 'vllm/main' into feat/unquant_moe_power
1c7c821
claude[bot]
claude Bot reviewed Aug 10, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the cpu Related to CPU backends label Aug 10, 2026
Akash kaothalkar
pre-commit fix
c0794a9 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
@Akashcodes732

Akashcodes732 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Hi @bigPYJ1151, can you please take a look at the changes made in this PR ?

bigPYJ1151
bigPYJ1151 reviewed Aug 11, 2026
View reviewed changes
Comment thread vllm/model_executor/layers/fused_moe/experts/cpu_moe.py
Akash kaothalkar added 2 commits August 11, 2026 12:58
@bigPYJ1151

bigPYJ1151 commented Aug 11, 2026

Copy link
Copy Markdown
Member

/ci run

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #83342 for commit 8298754ac9bd.

@Akashcodes732

Akashcodes732 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Hi @bigPYJ1151 , all the checks have passed, can we go ahead with the merge ?

@bigPYJ1151
bigPYJ1151 merged commit 61826c1 into vllm-project:main Aug 13, 2026
119 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments
@bigPYJ1151 bigPYJ1151 bigPYJ1151 approved these changes
@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner
@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner
@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner
@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner
@zyongye zyongye Awaiting requested review from zyongye zyongye is a code owner
@AndreasKaratzas AndreasKaratzas Awaiting requested review from AndreasKaratzas AndreasKaratzas is a code owner
@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety is a code owner

Assignees

No one assigned

Labels

cpu Related to CPU backends

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Akashcodes732 @bigPYJ1151