Skip to content

[ROCm] Add MI300X and MI355X guidance for Moonshot Kimi #571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tjtanaa merged 2 commits into from
Jun 18, 2026
Merged

[ROCm] Add MI300X and MI355X guidance for Moonshot Kimi #571

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
acf8d39
Add AMD ROCm guidance for Moonshot Kimi recipes
tanpinsiang Jun 18, 2026
2d33173
change amd docker back to nightly
tanpinsiang Jun 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 49 additions & 3 deletions models/moonshotai/Kimi-K2.5.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ meta:
model:
model_id: "moonshotai/Kimi-K2.5"
min_vllm_version: "0.19.1"
docker_image:
nvidia: "vllm/vllm-openai:latest"
amd: "vllm/vllm-openai-rocm:nightly"
architecture: moe
parameter_count: "1T"
active_parameters: "32B"
Expand Down Expand Up @@ -96,12 +99,9 @@ hardware_overrides:
- "--attention-config.use_trtllm_ragged_deepseek_prefill=True"
amd:
# Verified on 8× MI300X / MI355X (MI325X listed as supported but not verified).
extra_args:
- "--block-size=1"
extra_env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
VLLM_ROCM_USE_AITER_RMSNORM: "0"

strategy_overrides:
single_node_dep:
Expand Down Expand Up @@ -174,6 +174,52 @@ guide: |
docker pull vllm/vllm-openai:latest
```

### AMD MI300X/MI325X

On 8x MI300X or MI325X (`gfx942`), use the standard W4A16 MoE path with AITER
and INT4 QuickReduce.

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.5 \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--tensor-parallel-size 8 \
--tool-call-parser kimi_k2 \
--enable-auto-tool-choice \
--reasoning-parser kimi_k2 \
--mm-encoder-tp-mode data
```

### AMD MI350X/MI355X

On 8x MI350X or MI355X (`gfx950`), add `--moe-backend flydsl` to use the
optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path.

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.5 \
--tensor-parallel-size 8 \
--trust-remote-code \
--mm-encoder-tp-mode data \
--moe-backend flydsl \
--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'
Comment on lines +206 to +211

@gemini-code-assist gemini-code-assist Bot Jun 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The launch command for AMD MI350X/MI355X is missing several critical arguments that are present in the MI300X/MI325X command, such as --host 0.0.0.0, --port 8000, and the tool-calling/reasoning parsers (--tool-call-parser kimi_k2, --enable-auto-tool-choice, --reasoning-parser kimi_k2). Adding these ensures consistency and enables essential model features.

  vllm serve moonshotai/Kimi-K2.5 \\
    --host 0.0.0.0 \\
    --port 8000 \\
    --trust-remote-code \\
    --tensor-parallel-size 8 \\
    --tool-call-parser kimi_k2 \\
    --enable-auto-tool-choice \\
    --reasoning-parser kimi_k2 \\
    --mm-encoder-tp-mode data \\
    --moe-backend flydsl \\
    --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'

```

Notes:
- The FlyDSL INT4 MoE path does not support expert parallelism; do not add
`--enable-expert-parallel`.
- Keep `--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'`;
it is required for this FlyDSL path on MI350X / MI355X.
- vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4.
- Keep vLLM's default block size unless you are tuning long-context
throughput; `--block-size 64` is safe to try.

## Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API:
Expand Down
54 changes: 51 additions & 3 deletions models/moonshotai/Kimi-K2.6.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,16 @@ meta:
hardware:
h200: verified
gb200: verified
mi300x: verified
mi325x: verified
mi355x: verified

model:
model_id: "moonshotai/Kimi-K2.6"
min_vllm_version: "0.19.1"
docker_image:
nvidia: "vllm/vllm-openai:latest"
amd: "vllm/vllm-openai-rocm:nightly"
architecture: moe
parameter_count: "1T"
active_parameters: "32B"
Expand Down Expand Up @@ -93,12 +99,9 @@ hardware_overrides:
- "--attention-config.use_trtllm_ragged_deepseek_prefill=True"
amd:
# Verified on 8× MI300X / MI355X (MI325X listed as supported but not verified).
extra_args:
- "--block-size=1"
extra_env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
VLLM_ROCM_USE_AITER_RMSNORM: "0"

strategy_overrides:
single_node_dep:
Expand Down Expand Up @@ -149,6 +152,51 @@ guide: |
- **Hardware (INT4):** 8x H200 GPUs (verified), or equivalent aggregate VRAM (~640 GB)
- **AMD support:** 8x MI300X / MI325X / MI355X with ROCm 7.2.1 and Python 3.12

### AMD MI300X/MI325X

On 8x MI300X or MI325X (`gfx942`), use the standard W4A16 MoE path with AITER
and INT4 QuickReduce.

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.6 \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--tensor-parallel-size 8 \
--tool-call-parser kimi_k2 \
--enable-auto-tool-choice \
--reasoning-parser kimi_k2 \
--mm-encoder-tp-mode data
```

### AMD MI350X/MI355X

On 8x MI350X or MI355X (`gfx950`), add `--moe-backend flydsl` to use the
optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path.

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.6 \
--tensor-parallel-size 8 \
--trust-remote-code \
--mm-encoder-tp-mode data \
--moe-backend flydsl \
--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'
Comment on lines +184 to +189

@gemini-code-assist gemini-code-assist Bot Jun 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The launch command for AMD MI350X/MI355X is missing several critical arguments that are present in the MI300X/MI325X command, such as --host 0.0.0.0, --port 8000, and the tool-calling/reasoning parsers (--tool-call-parser kimi_k2, --enable-auto-tool-choice, --reasoning-parser kimi_k2). Adding these ensures consistency and enables essential model features.

  vllm serve moonshotai/Kimi-K2.6 \\
    --host 0.0.0.0 \\
    --port 8000 \\
    --trust-remote-code \\
    --tensor-parallel-size 8 \\
    --tool-call-parser kimi_k2 \\
    --enable-auto-tool-choice \\
    --reasoning-parser kimi_k2 \\
    --mm-encoder-tp-mode data \\
    --moe-backend flydsl \\
    --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'

```

Notes:
- The FlyDSL INT4 MoE path does not support expert parallelism; do not add
`--enable-expert-parallel`.
- Keep `--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'`;
it is required for this FlyDSL path on MI350X / MI355X.
- vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4.
- Keep vLLM's default block size unless you are tuning long-context
throughput; `--block-size 64` is safe to try.

## Client Usage

Expand Down
60 changes: 60 additions & 0 deletions models/moonshotai/Kimi-K2.7-Code.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,16 @@ meta:
- "moonshotai/Kimi-K2-Thinking"
hardware:
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified

model:
model_id: "moonshotai/Kimi-K2.7-Code"
min_vllm_version: "0.19.1"
docker_image:
nvidia: "vllm/vllm-openai:latest"
amd: "vllm/vllm-openai-rocm:nightly"
architecture: moe
parameter_count: "1T"
active_parameters: "32B"
Expand Down Expand Up @@ -74,6 +80,11 @@ hardware_overrides:
blackwell:
extra_args:
- "--attention-config.use_trtllm_ragged_deepseek_prefill=True"
amd:
# Verified on 8× MI300X / MI355X (MI325X listed as supported but not verified).
extra_env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"

strategy_overrides: {}

Expand Down Expand Up @@ -125,6 +136,55 @@ guide: |
- `--mm-encoder-tp-mode data` — runs the small MoonViT encoder data-parallel to avoid TP
communication overhead.

### AMD MI300X/MI325X

On 8x MI300X or MI325X (`gfx942`), use the standard W4A16 MoE path with AITER
and INT4 QuickReduce.

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.7-Code \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--tensor-parallel-size 8 \
--tool-call-parser kimi_k2 \
--enable-auto-tool-choice \
--reasoning-parser kimi_k2 \
--mm-encoder-tp-mode data
```

### AMD MI350X/MI355X

On 8x MI350X or MI355X (`gfx950`), add `--moe-backend flydsl` to use the
optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path.

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.7-Code \
--tensor-parallel-size 8 \
--trust-remote-code \
--mm-encoder-tp-mode data \
--moe-backend flydsl \
--tool-call-parser kimi_k2 \
--enable-auto-tool-choice \
--reasoning-parser kimi_k2 \
--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'
Comment on lines +168 to +176

@gemini-code-assist gemini-code-assist Bot Jun 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The launch command for AMD MI350X/MI355X is missing the --host 0.0.0.0 and --port 8000 arguments, which are present in the MI300X/MI325X command. Adding these ensures consistency across the deployment guides.

  vllm serve moonshotai/Kimi-K2.7-Code \\
    --host 0.0.0.0 \\
    --port 8000 \\
    --tensor-parallel-size 8 \\
    --trust-remote-code \\
    --mm-encoder-tp-mode data \\
    --moe-backend flydsl \\
    --tool-call-parser kimi_k2 \\
    --enable-auto-tool-choice \\
    --reasoning-parser kimi_k2 \\
    --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'

```

Notes:
- The FlyDSL INT4 MoE path does not support expert parallelism; do not add
`--enable-expert-parallel`.
- Keep `--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'`;
it is required for this FlyDSL path on MI350X / MI355X.
- vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4.
- Keep vLLM's default block size unless you are tuning long-context
throughput; `--block-size 64` is safe to try.

## Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API. The recommended
Expand Down