[ROCm] Promote tuned Kimi-K2.5 MXFP4 MoE env vars into extra_env - #703
Merged
dllehr-amd merged 2 commits intoJul 30, 2026
Merged
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the configuration for the Kimi-K2.5 model in models/moonshotai/Kimi-K2.5.yaml by adding two environment variables, VLLM_ROCM_USE_SKINNY_GEMM: "0" and AITER_MXFP4_INTERMEDIATE: "1", to the extra_env section. These variables configure kernel selection for the tuned AITER MXFP4 MoE path to ensure the faster RadeonFlow MXFP4 intermediate GEMM path is selected. I have no feedback to provide.
dllehr-amd
approved these changes
Jul 30, 2026
dllehr-amd
left a comment
dllehr-amd
left a comment
There was a problem hiding this comment.
Thanks for snagging these @Fangzhou-Ai !
The tuned AITER MXFP4 MoE kernel-selection env vars were documented in the guide prose (vllm-project#661) but only two of them (VLLM_ROCM_USE_AITER_MOE, VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS) were promoted into the structured mxfp4 extra_env (vllm-project#680). The two behavior-changing vars that flip the kernel away from vLLM defaults were missing from the applied config: - AITER_MXFP4_INTERMEDIATE=1 selects the RadeonFlow MXFP4 intermediate GEMM path; default is "0", so without it the MoE oracle falls back to a slower kernel. - VLLM_ROCM_USE_SKINNY_GEMM=0 disables skinny GEMM; default is on. AITER_BYPASS_TUNE_CONFIG and AITER_MOE_SORT_BACKEND are left out because the values used in the guide (0 / auto) already match the AITER defaults. Signed-off-by: Fangzhou-Ai <Fangzhou-Ai@users.noreply.github.com>
Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
Fangzhou-Ai
force-pushed
the
rocm-kimi-k25-extra-env-tuned-moe
branch
from
July 30, 2026 16:03
bb5a423 to
174c261
Compare
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The tuned AITER MXFP4 MoE configuration for
amd/Kimi-K2.5-MXFP4on MI355X was documented as prose in theguideblock by #661, but the machine-readableextra_envfor themxfp4variant only carries the two vars added by #680 (VLLM_ROCM_USE_AITER_MOE,VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS).The two behavior-changing kernel-selection vars from that guide were never promoted into the structured
extra_env, so anything consuming the structured recipe (rather than reading the guide by hand) silently gets a different, slower MoE kernel than the validated configuration:AITER_MXFP4_INTERMEDIATE=1— selects the RadeonFlow MXFP4 intermediate GEMM path. Default is"0"; without it (withVLLM_ROCM_USE_AITER=1) the MoE oracle falls back to a different kernel. This is the var the guide itself calls out as the one that selects the path.VLLM_ROCM_USE_SKINNY_GEMM=0— disables skinny GEMM. Default isTrue.This PR adds those two to the
mxfp4extra_envso the applied config matches the documented tuned path.Intentionally not added, because their guide values already equal the upstream defaults (so they'd be no-ops):
AITER_BYPASS_TUNE_CONFIG— aiter default is0.AITER_MOE_SORT_BACKEND— aiter default isauto.VLLM_ROCM_USE_AITER_RMSNORM=0is guide-documented as TP<8-only; themxfp4variant runstp: 8(RMSNorm left at its default on), so it is not added as a static env.OMP_NUM_THREADS=1is a host/CPU-threading knob rather than a model kernel-selection var and is left to the guide.Test plan
models/moonshotai/Kimi-K2.5.yamlparses as YAML.git diff --checkclean (no whitespace errors).envs.pydefaults (VLLM_ROCM_USE_SKINNY_GEMMdefaultTrue) and aiterfused_moe.pydefaults (AITER_MXFP4_INTERMEDIATEdefault"0").Made with Cursor