[ROCm] Add Kimi-K2.5 AMD AITER MoE env vars - #680
Conversation
Add the AITER MoE and fused shared-experts environment variables to the AMD Kimi-K2.5 recipe override. Signed-off-by: haic0 <haichzha@amd.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds the environment variables VLLM_ROCM_USE_AITER_MOE and VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS to the AMD hardware overrides for the Kimi-K2.5 model. Feedback suggests that applying VLLM_ROCM_USE_AITER_MOE globally under AMD hardware overrides may conflict with the --moe-backend flydsl flag configured for the default INT4 model on MI355X, and recommends moving these environment variables to specific configurations or the mxfp4 variant's environment instead.
| VLLM_ROCM_USE_AITER_MOE: "1" | ||
| VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1" |
There was a problem hiding this comment.
Setting VLLM_ROCM_USE_AITER_MOE: "1" globally under hardware_overrides.amd applies it to all AMD platforms and variants. This includes the MI355X running the default INT4 model, which is documented to use --moe-backend flydsl (lines 230-242). In vLLM, having VLLM_ROCM_USE_AITER_MOE=1 in the environment can conflict with or override the --moe-backend flydsl flag, potentially disabling the optimized FlyDSL path or causing runtime errors. To prevent this conflict, these environment variables should be moved to the mxfp4 variant's extra_env or applied only to the specific configurations that require them.
Keep the AITER MoE and fused shared-experts flags on the MXFP4 variant so the default AMD FlyDSL path is not affected. Signed-off-by: haic0 <haichzha@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The tuned AITER MXFP4 MoE kernel-selection env vars were documented in the guide prose (vllm-project#661) but only two of them (VLLM_ROCM_USE_AITER_MOE, VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS) were promoted into the structured mxfp4 extra_env (vllm-project#680). The two behavior-changing vars that flip the kernel away from vLLM defaults were missing from the applied config: - AITER_MXFP4_INTERMEDIATE=1 selects the RadeonFlow MXFP4 intermediate GEMM path; default is "0", so without it the MoE oracle falls back to a slower kernel. - VLLM_ROCM_USE_SKINNY_GEMM=0 disables skinny GEMM; default is on. AITER_BYPASS_TUNE_CONFIG and AITER_MOE_SORT_BACKEND are left out because the values used in the guide (0 / auto) already match the AITER defaults. Signed-off-by: Fangzhou-Ai <Fangzhou-Ai@users.noreply.github.com>
3 participants
Summary
VLLM_ROCM_USE_AITER_MOE=1to the AMD Kimi-K2.5 hardware override.VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1to match the benchmark reproduction path documented in PR [ROCm] Document Kimi-K2.5 MXFP4 MI355X benchmark reproduction #673.Test plan
pythonYAML parse and top-level schema-order check formodels/moonshotai/Kimi-K2.5.yamlgit diff --check -- models/moonshotai/Kimi-K2.5.yamlReadLintsonmodels/moonshotai/Kimi-K2.5.yaml