Skip to content

CI: add an RDMA stage for p2p tests on sci H200 runners - #3988

Closed
yueming-yuan wants to merge 3 commits into
mainfrom
yueming/ci-rdma-stage
Closed

yueming-yuan wants to merge 3 commits into
mainfrom
yueming/ci-rdma-stage

Conversation

@yueming-yuan

@yueming-yuan yueming-yuan commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Adds stage-c-8-gpu-h200-rdma, a stage for tests that need RDMA between the GPUs of one host, such as p2p weight updates, to run on sci H200 runners labeled rdma. Tests that do not need it are unaffected.

Why

p2p weight updates write the rollout engines' GPU memory from the trainer's over RDMA inside one host. On aliyun-1-8gpu-0, today's only 8×H200 runner, the GLM-4.7-Flash p2p e2e fails at connect with [Handshake] Failed to modify QP to RTR between its NIC groups (engine on mlx5_bond_5..7, trainer on mlx5_bond_0..3). On an 8×H200 host of the sci cluster the same e2e passes.

What

piece change
tests/ci/hardware.py stage-c-8-gpu-h200-rdma = hopper, 8 GPUs, runs on ["h200", "8gpu", "rdma"]. A runner label after the GPU model and count is a stage capability: dispatch to another arch keeps the home stage's capabilities and prefers stages with fewer, so an RDMA test is not sent to a B200 stage without RDMA, and no other test is routed onto the RDMA runners
pr-test.yml, file_run.py, stage_selection.py the stage's job, /rerun-test mapping and PR stage selection
test_qwen3_30B_A3B_p2p.py stage-c-8-gpu-h100 → stage-c-8-gpu-h200-rdma
docs, .claude/rules/ci-test-registration.md stage roster row, dispatch rule, registration table

The RDMA runners also carry h200 and 8gpu, so stage-c-8-gpu-h200 runs on them too: they add 8-GPU capacity rather than fence it off.

Runner layout (needs a runner admin; not in this repo)

7 days of PR Test (2026-10-01 to 10-08, 429 GPU jobs):

stage GPU-hours share queue p50 / p90 GPUs today
stage-c-8-gpu-h200 840 50% 90 min / 6.4 h 8 (aliyun-1-8gpu-0)
stage-c-4-gpu-h200 668 40% 77 min / 6.0 h 20 (aliyun-2 ×2, scitix-71-29 ×2, scitix-71-36 ×1)
stage-*-2-gpu-h200 169 10% 23 min / 3.2 h 4 (scitix-71-36 ×2)

Proposed: register scitix-71-29 as one runner labeled h200, 8gpu, rdma (instead of its two 4-GPU runners) and keep the aliyun 8-GPU runner. That gives 16 / 12 / 4 GPUs for a 50 / 40 / 10 load split instead of today's 8 / 20 / 4, and stage-c-8-gpu-h200 no longer waits behind a single runner.

Alternative with no change in pool sizes: swap, so scitix-71-29 becomes the 8-GPU rdma runner and the aliyun 8-GPU host goes back to two 4-GPU runners (as it was earlier this week).

Before labeling, check RDMA across all GPU pairs of the sci CI host: a Mooncake loopback between GPU groups, then this stage's test on it.

Merge order: the rdma runner first. Until a runner carries the label, this job stays queued on any run that selects it.

Not covered

  • The GLM-4.7-Flash and Qwen3.5 MTP p2p e2e (stacked in p2p: update speculative drafts from trained MTP weights #3985) move to this stage in that stack once this lands.
  • B200: with no Blackwell rdma stage, run-on-blackwell no longer sends the Qwen3-30B-A3B p2p e2e to stage-c-8-gpu-b200. That runner cannot carry it today: on b200-oma-8gpu-0 only mlx5_10..13 have active ports, and every write between them fails with transport retry counter exceeded (CQ completion timeout ... submitted=1200, processed=0), job 113636275606. The p2p stack (p2p: replace full CPU replicas with two reusable transfer buffers #3959) drops blackwell from this test's hardware for that reason; this PR changes its suite on the same call, so whichever lands second resolves the overlap. A Blackwell runner with working RDMA would get its own stage.
  • The moved test's metric history is keyed by stage, so its gates start a new history (nightly seeds it).

Tests

  • tests/ci/test/ pass locally (1151). test_a_capability_of_the_home_stage_is_kept_on_another_arch fails with the capability rule removed. test_a_test_without_a_capability_never_takes_a_stage_that_offers_one guards the tie-break; today name order alone also satisfies it.
  • Inventory lock tests in test_run_suite.py count the new stage.

CI timing: tests/e2e/megatron/test_qwen3_30B_A3B_p2p.py moves from stage-c-8-gpu-h100 to stage-c-8-gpu-h200-rdma; est_time=800 kept and unmeasured on the new stage, which needs the rdma runner first.

p2p weight updates write the rollout engines' GPU memory from the
trainer's over RDMA inside one host. On the 8x H200 runner aliyun-1 that
fails at connect (Failed to modify QP to RTR between its NIC groups),
while miles' p2p e2e passes on an 8x H200 host of the sci cluster.
stage-c-8-gpu-h200-rdma runs on runners labeled h200, 8gpu and rdma;
they also take stage-c-8-gpu-h200, so tests that do not need RDMA never
wait on them, and other stages are unchanged.

A runner label after the GPU model and count is now a capability of the
stage: dispatch to another arch keeps the home stage's capabilities and
prefers stages with fewer, so an RDMA test is not sent to a B200 stage
without RDMA and no other test takes the RDMA runners.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant