[ROCm] Add MI300X and MI355X guidance for Moonshot Kimi - #571
Conversation
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: Hongxia Yang <hongxia.yang@amd.com> Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support and deployment guides for AMD GPUs (MI300X, MI325X, MI350X, and MI355X) across the Kimi-K2.5, Kimi-K2.6, and Kimi-K2.7-Code models. It updates hardware verification statuses, configures Docker images, and adjusts hardware overrides. The reviewer noted that the launch commands for AMD MI350X/MI355X in the documentation are missing several critical arguments (such as host, port, and tool-calling/reasoning parsers) that are present in the MI300X/MI325X commands, and suggested adding them for consistency and completeness.
| vllm serve moonshotai/Kimi-K2.5 \ | ||
| --tensor-parallel-size 8 \ | ||
| --trust-remote-code \ | ||
| --mm-encoder-tp-mode data \ | ||
| --moe-backend flydsl \ | ||
| --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}' |
There was a problem hiding this comment.
The launch command for AMD MI350X/MI355X is missing several critical arguments that are present in the MI300X/MI325X command, such as --host 0.0.0.0, --port 8000, and the tool-calling/reasoning parsers (--tool-call-parser kimi_k2, --enable-auto-tool-choice, --reasoning-parser kimi_k2). Adding these ensures consistency and enables essential model features.
vllm serve moonshotai/Kimi-K2.5 \\
--host 0.0.0.0 \\
--port 8000 \\
--trust-remote-code \\
--tensor-parallel-size 8 \\
--tool-call-parser kimi_k2 \\
--enable-auto-tool-choice \\
--reasoning-parser kimi_k2 \\
--mm-encoder-tp-mode data \\
--moe-backend flydsl \\
--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'| vllm serve moonshotai/Kimi-K2.6 \ | ||
| --tensor-parallel-size 8 \ | ||
| --trust-remote-code \ | ||
| --mm-encoder-tp-mode data \ | ||
| --moe-backend flydsl \ | ||
| --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}' |
There was a problem hiding this comment.
The launch command for AMD MI350X/MI355X is missing several critical arguments that are present in the MI300X/MI325X command, such as --host 0.0.0.0, --port 8000, and the tool-calling/reasoning parsers (--tool-call-parser kimi_k2, --enable-auto-tool-choice, --reasoning-parser kimi_k2). Adding these ensures consistency and enables essential model features.
vllm serve moonshotai/Kimi-K2.6 \\
--host 0.0.0.0 \\
--port 8000 \\
--trust-remote-code \\
--tensor-parallel-size 8 \\
--tool-call-parser kimi_k2 \\
--enable-auto-tool-choice \\
--reasoning-parser kimi_k2 \\
--mm-encoder-tp-mode data \\
--moe-backend flydsl \\
--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'| vllm serve moonshotai/Kimi-K2.7-Code \ | ||
| --tensor-parallel-size 8 \ | ||
| --trust-remote-code \ | ||
| --mm-encoder-tp-mode data \ | ||
| --moe-backend flydsl \ | ||
| --tool-call-parser kimi_k2 \ | ||
| --enable-auto-tool-choice \ | ||
| --reasoning-parser kimi_k2 \ | ||
| --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}' |
There was a problem hiding this comment.
The launch command for AMD MI350X/MI355X is missing the --host 0.0.0.0 and --port 8000 arguments, which are present in the MI300X/MI325X command. Adding these ensures consistency across the deployment guides.
vllm serve moonshotai/Kimi-K2.7-Code \\
--host 0.0.0.0 \\
--port 8000 \\
--tensor-parallel-size 8 \\
--trust-remote-code \\
--mm-encoder-tp-mode data \\
--moe-backend flydsl \\
--tool-call-parser kimi_k2 \\
--enable-auto-tool-choice \\
--reasoning-parser kimi_k2 \\
--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'
| min_vllm_version: "0.19.1" | ||
| docker_image: | ||
| nvidia: "vllm/vllm-openai:latest" | ||
| amd: "vllm/vllm-openai-rocm:nightly-4c626633159887b0f2c962058c17c78f1434556d" |
There was a problem hiding this comment.
nightly image like on CUDA, only have 14-days retention period, it will be removed after 14 days.
There was a problem hiding this comment.
changed back to nightly
| min_vllm_version: "0.19.1" | ||
| docker_image: | ||
| nvidia: "vllm/vllm-openai:latest" | ||
| amd: "vllm/vllm-openai-rocm:nightly-4c626633159887b0f2c962058c17c78f1434556d" |
There was a problem hiding this comment.
changed back to nightly
| min_vllm_version: "0.19.1" | ||
| docker_image: | ||
| nvidia: "vllm/vllm-openai:latest" | ||
| amd: "vllm/vllm-openai-rocm:nightly-4c626633159887b0f2c962058c17c78f1434556d" |
There was a problem hiding this comment.
changed back to nightly
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: Hongxia Yang <hongxia.yang@amd.com> Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com>
2 participants
Summary
Add AMD ROCm guidance for the Moonshot Kimi K2.5, K2.6, and K2.7-Code recipes.
This updates the recipes with:
VLLM_ROCM_USE_AITER=1VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4gfx942W4A16 MoE path.gfx950FlyDSL W4A16 MoE path.--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'Why
The MI350X / MI355X path depends on recent vLLM support from
vllm-project/vllm#44400, which enabled the W4A16 FlyDSL MoE path for
gfx950.Because this support is recent, the AMD recipe pins the ROCm Docker image to:
The AMD paths are split by GPU generation:
gfx942, so they use the existing W4A16 MoE implementation.gfx950, so they can use the new FlyDSL W4A16 MoE path with--moe-backend flydsl.Validation
Validated Kimi-K2.5, Kimi-K2.7-Code (Kimi-K2.6 should work)
Also benchmarked
--block-size 16vs--block-size 64with an 8k input / 1k output serving workload. Results were effectively tied, so the recipes do not force a block-size override and instead leave it as an optional tuning knob.We also tested
VLLM_ROCM_USE_AITER_RMSNORM=0; it was slower than leaving the RMSNorm setting at the image default, so the recipes remove that override and only keep the required AITER / INT4 QuickReduce environment variables.