Skip to content

[CI][ROCm] Mirror large-model GSM8K evaluations on MI355 - #58683

Merged
AndreasKaratzas merged 3 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_amd_large_model_eval
Sep 29, 2026
Merged

AndreasKaratzas merged 3 commits into
vllm-project:mainfrom
AndreasKaratzas:akaratza_amd_large_model_eval

Conversation

@AndreasKaratzas

@AndreasKaratzas AndreasKaratzas commented Sep 25, 2026 •

Copy link
Copy Markdown
Member
  • Add four-GPU MI355 mirrors and matching native AMD jobs for all five large-model groups: DeepSeek R1 TP/DP, DeepSeek V3.2 TP/DP, and Nemotron Super BF16/FP8.
  • Use AITER MLA/MoE for DeepSeek and Triton Mamba/MoE for Nemotron; allow DeepSeek evaluations to execute on CDNA 4 and newer while retaining the older ROCm pool's infrastructure guard.
  • Preserve NVIDIA configurations and the full evaluation workload: identical checkpoints and precision, EP and MTP enabled with unchanged draft counts, 1,319 questions, five shots, 95% DeepSeek and 93% Nemotron targets, and the shared eight-percentage-point tolerance; AMD uses TP4/DP4 with EP4.
  • Validate all six configurations on both four and eight MI355s: 12 full GSM8K runs passed without skips or invalid answers, with accuracies of 93.03–95.75% and unchanged CI memory defaults.

The merged #57965 split H200 large-model evaluation into five groups without corresponding MI355 mirrors. This follow-up complements #34839 with executable MI355 DeepSeek and Nemotron coverage.

OpenAI Codex assisted with the implementation and local validation.

Configuration Four-GPU accuracy Four-GPU total Eight-GPU accuracy Eight-GPU total
DeepSeek R1 TP 95.30% 3:10 94.31% 3:23
DeepSeek R1 DP 95.75% 3:59 95.07% 3:41
DeepSeek V3.2 TP 95.53% 3:08 95.75% 3:22
DeepSeek V3.2 DP 95.60% 4:16 95.30% 4:16
Nemotron Super BF16 93.25% 2:04 93.33% 1:59
Nemotron Super FP8 93.03% 2:13 93.10% 2:18

All twelve cases passed the original assertions: DeepSeek's passing floor is 87% and Nemotron's is 85%, after the existing eight-percentage-point tolerance. Every case evaluated all 1,319 questions with five shots, maximum output length 256 and default GPU memory utilization 0.92. Four-GPU CI retains model, precision, attention, Mamba, EP and MTP coverage but does not replace continuous eight-rank coverage.

Add five four-GPU MI355 mirror groups and matching AMD jobs for the six DeepSeek and Nemotron evaluations. Preserve the upstream accuracy workload and enable supported AITER/Triton backends on gfx950.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
@mergify mergify Bot added ci/build rocm Related to AMD ROCm labels Sep 25, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Sep 25, 2026
Allow CDNA 4 and newer through the large-model evaluation guard and remove the preceding infrastructure comment.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
@AndreasKaratzas
AndreasKaratzas marked this pull request as ready for review September 28, 2026 22:11

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #91685 for commit e39b40d79624.

@AndreasKaratzas
AndreasKaratzas merged commit 0ae480f into vllm-project:main Sep 29, 2026
43 checks passed
@AndreasKaratzas
AndreasKaratzas deleted the akaratza_amd_large_model_eval branch September 29, 2026 07:07
michael-gpu pushed a commit to practical-inference/vllm-ampere-extended that referenced this pull request Oct 20, 2026
…t#58683)

Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants