Repository navigation
CI: add an RDMA stage for p2p tests on sci H200 runners - #3988
Closed
yueming-yuan wants to merge 3 commits into
Closed
yueming-yuan wants to merge 3 commits into
yueming-yuan wants to merge 3 commits into
Conversation
p2p weight updates write the rollout engines' GPU memory from the trainer's over RDMA inside one host. On the 8x H200 runner aliyun-1 that fails at connect (Failed to modify QP to RTR between its NIC groups), while miles' p2p e2e passes on an 8x H200 host of the sci cluster. stage-c-8-gpu-h200-rdma runs on runners labeled h200, 8gpu and rdma; they also take stage-c-8-gpu-h200, so tests that do not need RDMA never wait on them, and other stages are unchanged. A runner label after the GPU model and count is now a capability of the stage: dispatch to another arch keeps the home stage's capabilities and prefers stages with fewer, so an RDMA test is not sent to a B200 stage without RDMA and no other test takes the RDMA runners.
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
stage-c-8-gpu-h200-rdma, a stage for tests that need RDMA between the GPUs of one host, such as p2p weight updates, to run on sci H200 runners labeledrdma. Tests that do not need it are unaffected.Why
p2p weight updates write the rollout engines' GPU memory from the trainer's over RDMA inside one host. On
aliyun-1-8gpu-0, today's only 8×H200 runner, the GLM-4.7-Flash p2p e2e fails at connect with[Handshake] Failed to modify QP to RTRbetween its NIC groups (engine onmlx5_bond_5..7, trainer onmlx5_bond_0..3). On an 8×H200 host of the sci cluster the same e2e passes.What
tests/ci/hardware.pystage-c-8-gpu-h200-rdma= hopper, 8 GPUs, runs on["h200", "8gpu", "rdma"]. A runner label after the GPU model and count is a stage capability: dispatch to another arch keeps the home stage's capabilities and prefers stages with fewer, so an RDMA test is not sent to a B200 stage without RDMA, and no other test is routed onto the RDMA runnerspr-test.yml,file_run.py,stage_selection.py/rerun-testmapping and PR stage selectiontest_qwen3_30B_A3B_p2p.pystage-c-8-gpu-h100→stage-c-8-gpu-h200-rdma.claude/rules/ci-test-registration.mdThe RDMA runners also carry
h200and8gpu, sostage-c-8-gpu-h200runs on them too: they add 8-GPU capacity rather than fence it off.Runner layout (needs a runner admin; not in this repo)
7 days of
PR Test(2026-10-01 to 10-08, 429 GPU jobs):stage-c-8-gpu-h200aliyun-1-8gpu-0)stage-c-4-gpu-h200stage-*-2-gpu-h200Proposed: register
scitix-71-29as one runner labeledh200, 8gpu, rdma(instead of its two 4-GPU runners) and keep the aliyun 8-GPU runner. That gives 16 / 12 / 4 GPUs for a 50 / 40 / 10 load split instead of today's 8 / 20 / 4, andstage-c-8-gpu-h200no longer waits behind a single runner.Alternative with no change in pool sizes: swap, so
scitix-71-29becomes the 8-GPUrdmarunner and the aliyun 8-GPU host goes back to two 4-GPU runners (as it was earlier this week).Before labeling, check RDMA across all GPU pairs of the sci CI host: a Mooncake loopback between GPU groups, then this stage's test on it.
Merge order: the
rdmarunner first. Until a runner carries the label, this job stays queued on any run that selects it.Not covered
rdmastage,run-on-blackwellno longer sends the Qwen3-30B-A3B p2p e2e tostage-c-8-gpu-b200. That runner cannot carry it today: onb200-oma-8gpu-0onlymlx5_10..13have active ports, and every write between them fails withtransport retry counter exceeded(CQ completion timeout ... submitted=1200, processed=0), job 113636275606. The p2p stack (p2p: replace full CPU replicas with two reusable transfer buffers #3959) dropsblackwellfrom this test'shardwarefor that reason; this PR changes itssuiteon the same call, so whichever lands second resolves the overlap. A Blackwell runner with working RDMA would get its own stage.Tests
tests/ci/test/pass locally (1151).test_a_capability_of_the_home_stage_is_kept_on_another_archfails with the capability rule removed.test_a_test_without_a_capability_never_takes_a_stage_that_offers_oneguards the tie-break; today name order alone also satisfies it.test_run_suite.pycount the new stage.CI timing: tests/e2e/megatron/test_qwen3_30B_A3B_p2p.py moves from stage-c-8-gpu-h100 to stage-c-8-gpu-h200-rdma; est_time=800 kept and unmeasured on the new stage, which needs the
rdmarunner first.