Skip to content

ci(gb200): add 1-node mr-github functional test variants - #4334

Merged
ko3n1g merged 12 commits into
NVIDIA:mainfrom
ko3n1g:ko3n1g/feat/gb200-github-mr-tests
Apr 17, 2026
Merged

ci(gb200): add 1-node mr-github functional test variants#4334
ko3n1g merged 12 commits into
NVIDIA:mainfrom
ko3n1g:ko3n1g/feat/gb200-github-mr-tests

Conversation

@ko3n1g

@ko3n1g ko3n1g commented Apr 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Translates all remaining H100 GPT and MoE tests with `mr`/`mr-github` scope into GB200 1-node (4-GPU, `mr-github`) variants. Expands CI coverage with 49 new test cases:

  • 41 new GPT tests added to `gpt-1node.yaml`
  • 8 new MoE tests added to `moe-1node.yaml`

Parallelism adaptations (8 GPUs → 4 GPUs)

Category Original 1-node config Count
`tp2_pp2_cp2*` TP=2 PP=2 CP=2 PP 2→1 (preserves CP=2) 9
`tp4_pp2` TP=4 PP=2 PP 2→1 1
`ep8` MoE TP=1 PP=1 EP=8 EP 8→4 1
`tp4_ep2_etp2_pp2` MoE TP=4 PP=2 EP=2 ETP=2 PP 2→1, pipeline layout removed 1
Trivial copies TP×PP×CP ≤ 4 unchanged 37

All trivial cases have `TP × PP × CP ≤ 4` so reducing DP from 2 to 1 is sufficient. The `ep4` MoE test (`tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4`) is also trivial: `EP × ETP = 4 × 1 = 4 ≤ TP × DP = 2 × 2 = 4`.

All new tests use `scope: [mr-github]` only — no additions to `mr-github-slim` to keep the slim suite small.

Example

New `gpt-1node.yaml` products (excerpt):
```yaml

  • test_case: [gpt3_mcore_te_tp2_pp2_cp2_1node] # PP reduced 2→1
    products:
    • environment: [dev]
      scope: [mr-github]
      platforms: [dgx_gb200]
  • test_case: [gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline_1node] # trivial
    products:
    • environment: [dev]
      scope: [mr-github]
      platforms: [dgx_gb200]
      ```

Test plan

  • CI passes on GB200 `mr-github` pipeline
  • `mr-github-slim` suite duration is unchanged (no new slim entries)
  • Adapted configs (PP/EP reductions) use correct world size: `TP × PP × CP × DP = 4`

🤖 Generated with Claude Code

@ko3n1g @claude
ci(gb200): add 1-node mr-github functional test variants
1ee3b9f 
Onboard 8 new 1-node (4 GPU) GB200 functional tests for GitHub CI,
derived from existing 2-node mr-scoped tests in gpt.yaml and moe.yaml.

GPT tests (trivial copy, reduced DP): tp1pp1, tp2pp1, tp1pp2, tp1pp4, tp4pp1
MoE tests: ep8->4 for the two Muon tests, pp2->1 for the tp4ep2etp2pp2 test

Adds gpt-1node.yaml recipe (nodes:1 gpus:4, mr-github scope) and extends
moe-1node.yaml with the three new MoE entries.

Also splits the build-and-test skill into build-and-dependency (containers,
uv, linting) and testsystem (test layout, CI pipeline, failure investigation,
CI labels, trigger_internal_ci), and adds an onboard-gb200-1node-tests skill
documenting this workflow for future use.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Apr 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@svcnvidia-nemo-ci svcnvidia-nemo-ci added this to the Core 0.16 milestone Apr 16, 2026
@ko3n1g
docs(skills): clarify CI label selection for new test cases
5e7ac16 
New test cases require `Run functional tests` to generate golden values;
update the testsystem skill rule-of-thumb accordingly.

Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

ko3n1g and others added 2 commits April 16, 2026 10:52
@ko3n1g @claude
chore: auto-trigger testsystem skill for golden value downloads
c403fdf 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g @claude
ci(gb200): add remaining 1-node mr-github functional test variants
932adfd 
Translates all remaining H100 GPT and MoE tests with mr/mr-github scope
into GB200 1-node (4-GPU) variants, expanding CI coverage to 49 new test
cases split across gpt-1node.yaml (41 new) and moe-1node.yaml (8 new).

Parallelism adaptations for 4-GPU targets:
- tp2_pp2_cp2* (9 tests): PP 2→1, preserving CP=2 to keep context-parallel
  coverage
- tp4_pp2 (1 test): PP 2→1
- ep8 MoE (1 test): EP 8→4
- tp4_ep2_etp2_pp2 MoE (1 test): PP 2→1, pipeline layout removed (PP=1
  does not support multi-stage layout)

All remaining 37 GPT and 6 MoE tests are trivial copies where
TP×PP×CP ≤ 4 and EP satisfies the ETP constraint at DP=half.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@ko3n1g @claude
ci(gb200): remove copied golden values from new 1-node test dirs
04e8219 
Golden values copied from 2-node source configs are invalid for the
1-node GB200 variants. Remove them so CI generates fresh golden values
on first run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@ko3n1g @claude
ci(gb200): restore golden values for deepseek_proxy_fsdp_ep2_fsdp2_1node
68bcbed 
This file pre-existed in github/main and was incorrectly removed by the
bulk golden values cleanup in the previous commit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@ko3n1g ko3n1g closed this Apr 16, 2026
@ko3n1g ko3n1g reopened this Apr 16, 2026
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@ko3n1g @claude
ci(gb200): add golden values from pipeline 24507237865
ca37e8c 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

This was referenced Apr 16, 2026
Closed
Closed
Closed
ko3n1g and others added 2 commits April 16, 2026 14:50
@ko3n1g @claude
fix(ci): set EP=1 for MoE multi_dist_optimizer_instances 1-node test
cc897a5 
On a single DGX GB200 node (4 GPUs) with TP=2, PP=1, EP=2 the
expert_data_parallel_size collapses to 1 (4 / (2*1*2) = 1), which
is not divisible by num_distributed_optimizer_instances=2, causing the
assertion in parallel_state.py:1248 to fire.

Reduce --expert-model-parallel-size from 2 to 1 so that
expert_data_parallel_size = 4 / (2*1*1) = 2, which satisfies the
divisibility requirement and allows the test to exercise the
multi-distributed-optimizer-instance code path as intended.

Fixes: NVIDIA#4342

Signed-off-by: Oliver Koenig <okoenig@nvidia.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g @claude
ci(gb200): disable broken 1-node tests by marking scope as mr-github-…
eb5379b 
…broken

Two 1-node tests are currently failing and need to be excluded from
mr-github CI until they are fixed:

- gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap_1node (gpt-1node)
- gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances_1node (moe-1node)

Replace scope `mr-github` with `mr-github-broken` so they are skipped
in CI without removing the test definitions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@ko3n1g @claude
docs(skills): document how to disable a CI test without deleting it
d699840 
Add a "Disabling a Test Without Deleting It" section to the testsystem
skill: suffix the scope with `-broken` (e.g. `mr-github` →
`mr-github-broken`) to skip a test in CI while keeping the entry
intact for easy re-enablement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@ko3n1g
ko3n1g marked this pull request as ready for review April 16, 2026 18:17
@ko3n1g
ko3n1g requested a review from a team as a code owner April 16, 2026 18:17
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team April 16, 2026 18:17
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the Approved All necessary approvals have been made label Apr 16, 2026
@ko3n1g
ci(gb200): disable gpt3_mcore_te_tp1_pp4_vp1_1node
e309104 
Tracks NVIDIA#4344

Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e309104

@ko3n1g
ko3n1g enabled auto-merge April 16, 2026 20:50
@ko3n1g
ko3n1g added this pull request to the merge queue Apr 17, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24540904044

Merged via the queue into NVIDIA:main with commit 2aeaf56 Apr 17, 2026
184 checks passed
@ko3n1g
ko3n1g deleted the ko3n1g/feat/gb200-github-mr-tests branch April 17, 2026 01:15
Victarry pushed a commit to yanring/Megatron-LM that referenced this pull request Apr 20, 2026
Dennis Liu
Merge remote-tracking branch 'origin/main' into pr3431/fix-conflicts
9d4c1bc 
* origin/main: (286 commits)
  Rename MambaModel/MambaStack to HybridModel/HybridStack (NVIDIA#4099)
  Fix Megatron initialization with extra_args_provider (NVIDIA#4327)
  Fix RL to once again work with --skip-train (NVIDIA#4249)
  Add activation logging and tokens per expert logging (NVIDIA#3842)
  Make param_index_map always use unpacked (full numel) offsets (NVIDIA#4328)
  FA4 Inference (NVIDIA#4186)
  Fix RL reward due to stop token (NVIDIA#4096)
  cp: Fix UT timeout (NVIDIA#4310) (NVIDIA#4373)
  feat(ckpt): add --async-ckpt-use-cpu-shm argument (NVIDIA#4355)
  Update copy-pr-bot.yaml [skip ci]
  Docs: improve docstrings and comments in example training loop (NVIDIA#4041)
  Add QK layernorm support for dot-product attention in MambaModel (NVIDIA#4067)
  Fix bug with non-partial rollouts (NVIDIA#3964)
  [docs] ci: use parent-relative json_url for version picker (NVIDIA#4367)
  Add tables and histogram for RL staleness (NVIDIA#4097)
  Port DeepSeek Sparse Attention to `MambaModel` (NVIDIA#3553)
  docs: bump versions1.json to 0.17.0 (latest) (NVIDIA#4360)
  Fix potential coredump issue that occurs when saving a checkpoint (NVIDIA#1871)
  ci(gb200): add 1-node mr-github functional test variants (NVIDIA#4334)
  fix: wait for async P2P send before deallocating output tensor (NVIDIA#4047)
  ...

# Conflicts:
#	megatron/core/transformer/cuda_graphs.py
yangbofun pushed a commit to xlm-research/Megatron-LM that referenced this pull request May 22, 2026
@ko3n1g @claude
ci(gb200): add 1-node mr-github functional test variants (NVIDIA#4334)
dc4745d 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Oliver Koenig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
@ko3n1g ko3n1g mentioned this pull request May 26, 2026
Merged
yhgalaxy pushed a commit to yhgalaxy/Megatron-LM that referenced this pull request Jun 17, 2026
@ko3n1g @claude @yhgalaxy
ci(gb200): add 1-node mr-github functional test variants (NVIDIA#4334)
01e4a2b 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Oliver Koenig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>
jon-barker pushed a commit to jon-barker/Megatron-LM that referenced this pull request Jul 10, 2026
@ko3n1g @claude Jon Barker
ci(gb200): add 1-node mr-github functional test variants (NVIDIA#4334)
3ba1f2c 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Oliver Koenig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Jon Barker <jbarker@aws-cmh-slurm-1-vscode-02.cm.cluster>
terminator123 pushed a commit to 021ai/Megatron-LM that referenced this pull request Aug 3, 2026
@ko3n1g @claude @terminator123
ci(gb200): add 1-node mr-github functional test variants (NVIDIA#4334)
48a1126 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Oliver Koenig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chtruong814 chtruong814 chtruong814 approved these changes

Assignees

No one assigned

Labels

Approved All necessary approvals have been made complexity: high Run functional tests

Projects

None yet

Milestone

Core 0.16

Development

Successfully merging this pull request may close these issues.

3 participants

@ko3n1g @svcnvidia-nemo-ci @chtruong814