Skip to content

ci(amd): consolidate stage-b-test-1-gpu-small-amd 14→6 shards (-14% GPU-min, -33% wall-clock) - #29594

Open
sunxxuns wants to merge 1 commit into
mainfrom
ci/amd-stage-b-shard-14to8
Open

sunxxuns wants to merge 1 commit into
mainfrom
ci/amd-stage-b-shard-14to8

Conversation

@sunxxuns

@sunxxuns sunxxuns commented Jun 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Consolidates stage-b-test-1-gpu-small-amd from 14 matrix shards to 6, saving ~14% GPU-minutes per PR run and ~33% wall-clock, with no test coverage change.

Why 6 (not 8 or 14)

The suite has 108 tests, ~256min total est_time. With ~6 concurrent MI325 1-GPU runners observed in practice:

Shards GPU-min Wall-clock Waves
6 292 49min 1 (zero idle)
8 304 76min 2 (6+2, 4 idle)
12 328 55min 2 (6+6)
14 340 73min 3 (6+6+2)

6 shards is optimal on both axes: 1 wave at 6 concurrent runners (zero idle), minimal bring-up overhead (6min × 6 = 36min vs 6min × 14 = 84min).

8 shards (a prior revision of this PR) was suboptimal: 2 waves (6+2) with 4 runners idle in wave 2, giving 76min wall-clock — slower than the 14-shard baseline (73min).

Timeout safety

Longest shard at 6 = ~53min (LPT-balanced), under the 60min job timeout and far under the 480min wait-for-stage-b gate. Per-file 2400s (40min) timeout is generous (max single-test est = 4.35min).

Changes

  • .github/workflows/pr-test-amd.yml: matrix part 14→6, --auto-partition-size 14→6, expected_count 14→6

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ✅ Run #29337465584
Latest PR Test (Extra): ✅ Run #29337465077
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions github-actions Bot added the amd label Jun 29, 2026
@sunxxuns sunxxuns added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) labels Jun 29, 2026
@sunxxuns
sunxxuns force-pushed the ci/amd-stage-b-shard-14to8 branch 2 times, most recently from 89172b0 to 1a9185b Compare July 7, 2026 23:35
@sunxxuns sunxxuns changed the title ci(amd): consolidate stage-b-test-1-gpu-small-amd 14→8 shards (-25% GPU-min) ci(amd): consolidate stage-b-test-1-gpu-small-amd 14→6 shards (-14% GPU-min, -33% wall-clock) Jul 7, 2026
@sunxxuns
sunxxuns force-pushed the ci/amd-stage-b-shard-14to8 branch from 1a9185b to c92fcd4 Compare July 8, 2026 01:17
@sunxxuns
sunxxuns force-pushed the ci/amd-stage-b-shard-14to8 branch 8 times, most recently from dc37c1e to ff872bd Compare July 8, 2026 10:20
@sunxxuns
sunxxuns force-pushed the ci/amd-stage-b-shard-14to8 branch 21 times, most recently from 36cb3d3 to cb8e306 Compare July 11, 2026 04:19
…PU-min, -33% wall-clock)

6 shards = 1 wave at 6 concurrent MI325 runners (zero idle), vs 14 shards
= 3 waves with 4 idle runners in the last wave. Saves 48 GPU-min/run of
bring-up overhead (6min × 8 fewer shards) and 24min wall-clock.

8 shards (prior revision) was suboptimal: 2 waves (6+2) with 4 idle in
wave 2, giving 76min wall-clock — slower than the 14-shard baseline
(73min). 6 is optimal on both GPU-min (292 vs 340) and wall-clock
(49min vs 73min).

Longest shard at 6 = ~53min (LPT-balanced), under the 60min job timeout
and far under the 480min wait-for-stage-b gate. Per-file 2400s (40min)
timeout is generous (max single-test est = 4.35min).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@sunxxuns

Copy link
Copy Markdown
Collaborator Author

Friendly nudge here — this one has been waiting for review for about three weeks. It's a CI-only change (consolidating stage-b shards 14→6) and all checks are green on the latest head. Is there anything else needed to move it forward? Happy to adjust if anything's unclear.

— sunxxuns

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants