ci: Enable GB200 runners by chtruong814 · Pull Request #2017 · NVIDIA-NeMo/RL

chtruong814 · 2026-02-24T04:04:54Z

What does this PR do ?

Enable GB200 runners. Fallback to A100 runners if external contributor. Currently, we are unable to run CI for external contributors in the GB200 cluster. We have additional work to enable this.
fp8 vllm generation tests are failing on gb200. Skipping this for now. Will open an issue
The eval functional tests are failing on gb200 because the resulting score is different than expected. Updating the expected score to a wider range
The image name is updated by a Github Action variable. Currently it's "megatron-bridge" because we did not create the "rl" docker repo in the cluster environment yet. We will update that before we merge this.
Going forward, the CI needs to be kicked off by commenting with /ok to test <commit_sha>. The CI will still respect the labels applied. This aligns with how we are kicking off CI for other repos.

Issues

List issues that this PR closes (syntax):

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

...

Summary by CodeRabbit

Chores
- Updated CI/CD infrastructure with configurable registry and test data path parameters; simplified test setup by removing cloud-specific dependencies
- Modified PR bot auto-sync configuration
Bug Fixes
- Added hardware compatibility checks to skip FP8 tests on unsupported devices
Tests
- Disabled four GPU functional tests (eval and grpo variants)
Dependencies
- Replaced decord with decord2

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

copy-pr-bot · 2026-03-04T23:23:29Z

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

chtruong814 · 2026-03-04T23:23:51Z

/ok to test 73e70e8

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-03-04T23:48:36Z

/ok to test 836c8cb

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-03-06T02:02:29Z

/ok to test 7166bce

.github/workflows/cicd-main.yml

tests/functional/eval.sh

tests/functional/eval_async.sh

tests/unit/models/generation/test_vllm_generation.py

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-03-07T15:14:34Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-03-07T21:19:21Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-03-07T21:22:09Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-03-07T21:32:11Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-03-07T21:58:21Z

/ok to test

Test GB200 runner

c060437

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added the CI:docs Run doctest label Feb 24, 2026

github-actions bot added CI Relating to CI and removed CI:docs Run doctest labels Feb 24, 2026

chtruong814 had a problem deploying to nemo-ci February 24, 2026 04:05 — with GitHub Actions Failure

chtruong814 had a problem deploying to nemo-ci February 24, 2026 17:53 — with GitHub Actions Failure

Fix gb200 container build

02084e3

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added the CI:docs Run doctest label Feb 24, 2026

chtruong814 had a problem deploying to nemo-ci February 24, 2026 19:32 — with GitHub Actions Failure

chtruong814 had a problem deploying to nemo-ci February 25, 2026 16:11 — with GitHub Actions Failure

chtruong814 had a problem deploying to nemo-ci February 25, 2026 18:49 — with GitHub Actions Failure

chtruong814 had a problem deploying to nemo-ci February 25, 2026 20:16 — with GitHub Actions Failure

chtruong814 had a problem deploying to nemo-ci February 25, 2026 21:53 — with GitHub Actions Failure

Test updated registry

bcf8f81

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added CI:docs Run doctest and removed CI:docs Run doctest labels Feb 26, 2026

chtruong814 had a problem deploying to nemo-ci February 26, 2026 00:55 — with GitHub Actions Failure

chtruong814 added CI:docs Run doctest and removed CI:docs Run doctest labels Feb 26, 2026

chtruong814 temporarily deployed to nemo-ci February 26, 2026 05:14 — with GitHub Actions Inactive

Test gb200

e87f2e2

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added CI:L1 Run doctests, unit tests, and functional tests and removed CI:docs Run doctest labels Feb 28, 2026

Merge remote-tracking branch 'origin/main' into chtruong/gb200

2435ca5

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L1 Run doctests, unit tests, and functional tests labels Feb 28, 2026

Force gb200 build

f517e6a

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) labels Feb 28, 2026

chtruong814 temporarily deployed to nemo-ci February 28, 2026 03:09 — with GitHub Actions Inactive

chtruong814 mentioned this pull request Mar 4, 2026

feat: Allow custom registry in the build containers job NVIDIA-NeMo/FW-CI-templates#401

Merged

Ensure functional tests wait for unit tests

836c8cb

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Update preflight and build container to 0.78.0

7166bce

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

terrykong reviewed Mar 6, 2026

View reviewed changes

.github/workflows/cicd-main.yml Show resolved Hide resolved

terrykong reviewed Mar 6, 2026

View reviewed changes

tests/functional/eval.sh Show resolved Hide resolved

terrykong reviewed Mar 6, 2026

View reviewed changes

tests/functional/eval_async.sh Show resolved Hide resolved

terrykong reviewed Mar 6, 2026

View reviewed changes

tests/unit/models/generation/test_vllm_generation.py Outdated Show resolved Hide resolved

terrykong reviewed Mar 6, 2026

View reviewed changes

tests/unit/models/generation/test_vllm_generation.py Outdated Show resolved Hide resolved

terrykong reviewed Mar 6, 2026

View reviewed changes

tests/unit/models/generation/test_vllm_generation.py Outdated Show resolved Hide resolved

kajalj22 previously approved these changes Mar 6, 2026

View reviewed changes

chtruong814 added 3 commits March 6, 2026 17:09

Move skip of vllm fp8 generation to top of tests

2926902

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Add instructions to run CI

0c14ffd

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Fix markdown bullets in contribution guide

939e0da

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

terrykong previously approved these changes Mar 6, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into chtruong/gb200

98a7634

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added 2 commits March 7, 2026 10:13

Fix format in test_vllm_generation.py

f30ebd6

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Fix base and head refs

a36f5a9

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Fix pre-flight ref

99f82b9

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Fix checkout

f2c3fbb

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Fix test labels

1fbc780

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

thomasdhc approved these changes Mar 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Enable GB200 runners#2017

ci: Enable GB200 runners#2017
terrykong merged 49 commits intomainfrom
chtruong/gb200

chtruong814 commented Feb 24, 2026 •

edited

Loading

Uh oh!

copy-pr-bot bot commented Mar 4, 2026

Uh oh!

chtruong814 commented Mar 4, 2026

Uh oh!

chtruong814 commented Mar 4, 2026

Uh oh!

chtruong814 commented Mar 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

chtruong814 commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Issues

Usage

Before your PR is "Ready for review"

Additional Information

Summary by CodeRabbit

Uh oh!

copy-pr-bot bot commented Mar 4, 2026

Uh oh!

chtruong814 commented Mar 4, 2026

Uh oh!

chtruong814 commented Mar 4, 2026

Uh oh!

chtruong814 commented Mar 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

chtruong814 commented Mar 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chtruong814 commented Feb 24, 2026 •

edited

Loading