Skip to content

[AMD] Use HIP Top-K for GLM-5.2 MI355X / 为 GLM-5.2 MI355X 使用 HIP Top-K #3117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jiejingzhangamd wants to merge 2 commits into
base: main
Choose a base branch
from
+12 −2
Open

[AMD] Use HIP Top-K for GLM-5.2 MI355X / 为 GLM-5.2 MI355X 使用 HIP Top-K #3117

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
34299f7
perf: use HIP Top-K for GLM-5.2 on MI355X
jiejingzhangamd Sep 15, 2026
aac8318
chore: link GLM-5.2 HIP Top-K changelog
jiejingzhangamd Sep 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions benchmarks/single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,9 @@ export AIPERF_HTTP_TCP_USER_TIMEOUT=900000
# inter-turn idle gaps can reuse a socket exactly as the server closes it.
# Outlast the client pool so the race cannot occur.
export SGLANG_TIMEOUT_KEEP_ALIVE=900
# PR #36684 & PR #36851 turned the v2 fused top-k on for GLM-5.x on ROCm.
export SGLANG_OPT_USE_TOPK_V2=true
# Use the precompiled HIP Top-K path; at c8 it improves P90 interactivity with
# essentially unchanged throughput per GPU compared with fused Top-K v2.
export SGLANG_OPT_USE_TOPK_V2=false

# HiCache L2 (host DRAM), optionally extended with Mooncake L3.
# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=hicache or mooncake.
Expand Down
9 changes: 9 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7753,3 +7753,12 @@
- "Set a 64-token CUDA graph capture floor for B300 DSpark at c1/c2/c4; preserve c8+ sizing, full AgentX traces, AL 3.51, and Engram UVA"
- "将 B300 DSpark 的 c1/c2/c4 CUDA graph 最小捕获范围设为 64 tokens；保持 c8+ 大小、完整 AgentX 轨迹、AL 3.51 和 Engram UVA"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3116

- config-keys:
- glm5.2-fp4-mi355x-sglang-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Disable SGLANG_OPT_USE_TOPK_V2 so GLM-5.2 uses the precompiled HIP Top-K path on MI355X; at concurrency 8, P90 interactivity improves from 96.97 to 106.69 tok/s/user (+10.0%) while throughput per GPU changes from 9,553.2 to 9,539.0 tok/s (-0.15%)."
- "关闭 SGLANG_OPT_USE_TOPK_V2，使 GLM-5.2 在 MI355X 上使用预编译 HIP Top-K 路径；并发 8 时，P90 interactivity 从 96.97 提升到 106.69 tok/s/user（+10.0%），单卡吞吐从 9,553.2 变为 9,539.0 tok/s（-0.15%）。"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3117
Loading