Skip to content

[Doc] Add benckmark_moe doc #24860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

[Doc] Add benckmark_moe doc #24860

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bae725c
Init
jeejeelee Sep 15, 2025
82e6b05
Merge branch 'main' into add-benchmark-moe-docs
jeejeelee Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions ..._executor/layers/fused_moe/configs/README → ...ecutor/layers/fused_moe/configs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,21 @@
# Fused MoE Triton Kernel Configurations

## Introduce

This directory contains tuned configurations for different settings of the fused_moe kernel.
For different settings of

- E (number of experts)
- N (intermediate size)
- device_name (torch.cuda.get_device_name())

the JSON file contains a mapping from M (batch size) to the chosen configuration.

The example configurations provided are for the Mixtral model for TP2 on H100
and TP4 on A100. Mixtral has intermediate size N = 14336, i.e. for TP2 we have
N = 7168 and for TP4 we have N = 3584.

See `benchmark/kernels/benchmark_moe.py` on how to generate these config files.


Check failure on line 20 in vllm/model_executor/layers/fused_moe/configs/README.md

View workflow job for this annotation

GitHub Actions / pre-commit 

Multiple consecutive blank lines [Expected: 1; Actual: 2]
### Tune MoE kernel

Check failure on line 21 in vllm/model_executor/layers/fused_moe/configs/README.md

View workflow job for this annotation

GitHub Actions / pre-commit 

Files should end with a single newline character

Check failure on line 21 in vllm/model_executor/layers/fused_moe/configs/README.md

View workflow job for this annotation

GitHub Actions / pre-commit 

Trailing spaces [Expected: 0 or 2; Actual: 1]
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The new section ### Tune MoE kernel is empty. This could be confusing for readers and makes the documentation feel incomplete. Please either add the relevant content for tuning the MoE kernel or remove this section if it's not yet ready.

Loading