[RFC]: Improve MoE triton kernel tuning

[jeejeelee]

Motivation.

Different triton versions produce different MoE tuned configurations. For example, the triton 3.4.0 tuned config differs from the config on version 0.10.1.1. These differences may impact model performance. This RFC mainly discusses how to handle different versions of triton tuned config.

Proposed Change.

We want to:

• Split the MoE tuned config into N folders (currently only contains legacy_configs and triton_3_4_0). legacy_configs represents the folder for configurations that cannot be traced back to the committed triton version, while triton_3_4_0 represents the folder for the corresponding triton version, which is currently 3.4.0. As triton versions are updated, there may be more triton version folders. See: [Kernel] Split moe tuned configs #24113
• Add documentation related to benchmark_moe, including how to benchmark, how to pass local tuned configs through VLLM_TUNED_CONFIG_FOLDER, etc. This will encourage more users in tuning kernels on their hardware. see: [Doc] Add benckmark_moe doc #24860

Feedback Period.

No response

CC List.

@mgoin @simon-mo @WoosukKwon @youkaichao

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ahengljh]

My commits are NOT ready yet, just a draft.

[jeejeelee]

My commits are NOT ready yet, just a draft.

I have sumbitted a PR: #24113

[cadedaniel]

How much performance impact is there?

[jeejeelee]

The comparative experiment we conducted before

BTW, sgl-project/sglang#5955 also has similar conclusions

[simon-mo]

VLLM_TUNED_CONFIG_FOLDER and docs changes SGTM. However, I don't think we need one folder per version, rather, the version should be part of config json and updated over time. Because for each commit on vLLM main, there's a pinned Triton version via pinned PyTorch conference for the given SHA.

[jeejeelee]

@simon-mo Thank you for your reply and suggestions. IIUC:

• For the previous config, still use legacy_configs
• For the newly added config, use a name like E=1,N=1792,device_name=NVIDIA_A100-SXM4-80GB,dtype=int8_w8a16_triton340.json,triton version included in json name, is that correct?

[simon-mo]

No I think the Triton version should be in the config json itself. Because it doesn't make sense to have

E=1,N=1792,device_name=NVIDIA_A100-SXM4-80GB,dtype=int8_w8a16_triton340.json

and

E=1,N=1792,device_name=NVIDIA_A100-SXM4-80GB,dtype=int8_w8a16_triton350.json

Both in vLLM as given a commit, we will only work with one pinned Triton verison.

So it sound be like

{
   "triton_version": "3.4.0"
   ...
}

[gau-nernst]

However, I don't think we need one folder per version, rather, the version should be part of config json and updated over time. Because for each commit on vLLM main, there's a pinned Triton version via pinned PyTorch conference for the given SHA.

I get this point. But I think maybe it's also useful to document which triton a particular config was tuned for, so another person knows whether re-running the tuning is necessary. It can be a simple extra key in the config json...

[jeejeelee]

However, I don't think we need one folder per version, rather, the version should be part of config json and updated over time. Because for each commit on vLLM main, there's a pinned Triton version via pinned PyTorch conference for the given SHA.

I get this point. But I think maybe it's also useful to document which triton a particular config was tuned for, so another person knows whether re-running the tuning is necessary. It can be a simple extra key in the config json...

We have already added the triton version in the configuration ,see #24769