chore: Enable cuda-13 build by chtruong814 · Pull Request #2332 · NVIDIA-NeMo/RL

chtruong814 · 2026-04-24T13:05:21Z

What does this PR do ?

build: Enable cuda-13 build

Bump to nvcr.io/nvidia/cuda-dl-base:26.03-cuda13.2-devel-ubuntu24.04 base image
Set ENV CPLUS_INCLUDE_PATH=/usr/local/cuda/include/cccl to fix deep_gemm and deep_ep builds
- I assume we will bump deep_ep to hybrid_ep to use what MBridge uses. But just trying for minimal change needed.
Revert cuda 12 patches for MBridge and MCore
Ensure cuda 13 wheels were used for vllm and flash-attn
sglang-kernel had trouble building. Reduced the number of parallel jobs to allow build to complete. However, may try to rebuild a few times to see if I run into same issues again

Opening PR as draft to run CI tests here. After those pass, will check nightly tests.

Issues

#2111

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

...

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

copy-pr-bot · 2026-04-24T13:05:25Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

chtruong814 · 2026-04-24T13:05:37Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-04-24T13:33:19Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-04-24T14:20:57Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-04-24T16:28:35Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-04-24T20:47:30Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-04-24T23:11:56Z

/ok to test

chtruong814 · 2026-04-25T20:00:36Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-05-11T00:33:13Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-05-11T01:16:19Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 · 2026-05-11T02:07:06Z

/ok to test

Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: zswerth <zwertheimer@nvidia.com>

Upstream nemo-rl flipped its Dockerfile to `cuda-dl-base:26.03-cuda13.2` in NVIDIA-NeMo#2332, so the `nightly` base we now target is CUDA 13. nixl-cu12 wheels won't link against the cu13 runtime; install nixl-cu13 instead. Inline comment notes how to flip back to cu12 if pointing BASE_IMAGE at an older tag.

…base Consolidates the iterative overlay rework that happened while tracking the upstream nemo-rl image bump from 664d29c (CUDA-12) to :nightly (`cuda-dl-base:26.03-cuda13.2`, upstream NVIDIA-NeMo#2332). The Dockerfile.nemorl now: * Targets BASE_IMAGE=nvcr.io/nvidian/nemo-rl:nightly, MX_REF=8594fd6 (kavink/nemo_rl_moe sidecar fix + post-rebase fixups), NIXL_VERSION=0.10.1 (wire-compatible with the dynamo worker image). * Installs both `nixl` (Python bindings) and `nixl-cu13` (cu13 native libs sidecar) with --no-deps so the cu13 cuDNN/cuBLAS stack the base already pins isn't reinstalled from scratch. Same --no-deps treatment for modelexpress (whose pyproject pins nixl[cu12], wrong CUDA major for us) and tensordict (whose torch dep would re-touch the nvidia-* stack). * Layers back only the small deps the upstream base doesn't already supply: ai-dynamo-runtime, grpcio, pyvers (tensordict's only not-already-installed runtime dep). runai-model-streamer is dropped — modelexpress lists it but our v2 refit path doesn't exercise the runai-loader. * Bakes tensordict into both the main venv and per-actor Ray venvs so the :nightly base lag for upstream PR NVIDIA-NeMo#2439 doesn't bite at runtime. Validated end-to-end against the rebased dynamo-k8s-integration tree on the workplace-assistant dynamo-MX smoke (SUCCEEDED, driver_exit=0).

Add the TRT-LLM PyTorch backend integration needed to validate GRPO with draft-target speculative decoding, Mongo rollout logging, and weight-sync timing on AIHub. chore(trtllm): drop speculative decoding and MongoDB logging Remove the experimental speculative-decoding hooks (config, worker plumbing, spec_token_origins propagation, specdec exemplar config) and the MongoDB logger backend that were bundled with the TRT-LLM backend introduction. The TRT-LLM PyTorch backend itself is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> chore(trtllm): WIP — wire trtllm backend on top of CUDA 13 TMP commit (will amend or squash later). Layers TRT-LLM-specific wiring on top of the CUDA 13 squash from NVIDIA-NeMo#2332: - pyproject.toml: pin trtllm extra to tensorrt_llm==1.3.0rc13 - virtual_cluster.py: add PY_EXECUTABLES.TRTLLM - ray_actor_environment_registry.py: add TRTLLM_EXECUTABLE (with NEMO_RL_PY_EXECUTABLES_SYSTEM=1 fallback) - docker/Dockerfile: add SKIP_TRTLLM_BUILD arg + HPCX env exposure (cuda-compat removed — cu13.2 base image already provides it) - docs/docker.md: SKIP_TRTLLM_BUILD usage docs Known TODOs before this can be promoted: - `uv lock` likely fails: tensorrt-llm 1.3.0rc13 has no cp313 wheel. Either move trtllm to its own workspace (cp312) or relax pin. - Build + smoke test on a GB200 node not yet validated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> Downgrade to Python 3.12 Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> Successfully run nemorl+trtllm and verify convergence Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> Add async support Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> Support colocated case Cleanup Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> drop cp313 flash-attn wheel pin to unblock cp312 lock Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> drop branch-local nemo_gym changes Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> Refactor Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> Refactor code Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> Change flash attn wheel Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> Verify perf Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> Discard weights Signed-off-by: shuyixiong <219646547+shuyixiong@users.noreply.github.com> add py3.13 build infrastructure with torch 2.10 Signed-off-by: Shiki Wu <shikiw@ipp1-3010.ipp1a1.colossus.nvidia.com> Initial Async support Signed-off-by: Shiki Wu <shikiw@ipp1-3010.ipp1a1.colossus.nvidia.com> add async support for trtllm Signed-off-by: Shiki Wu <shikiw@nvidia.com>

chtruong814 added 3 commits April 24, 2026 04:31

build: Initial cuda 13 update

093c58f

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Fix deepep build for cuda 13

afec4ac

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Re-enable commented out docker build steps

73ad143

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 added the CI:L1 Run doctests, unit tests, and functional tests label Apr 24, 2026

copy-pr-bot Bot had a problem deploying to nemo-ci April 24, 2026 13:06 Failure

Do not use deepep fork

3e76b4e

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

copy-pr-bot Bot had a problem deploying to nemo-ci April 24, 2026 13:33 Error

Revert deep gemm change

23f0af2

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

copy-pr-bot Bot had a problem deploying to nemo-ci April 24, 2026 14:21 Failure

Limit parallel build for sglang kernel

2345fbf

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

copy-pr-bot Bot temporarily deployed to nemo-ci April 24, 2026 16:28 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci April 24, 2026 18:29 Inactive

chtruong814 added 3 commits April 24, 2026 19:13

Use vllm cuda 13 wheels

86a24f0

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Use flash-attn cu13 wheel

e651f8a

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Pin cublas

90cc00f

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

copy-pr-bot Bot had a problem deploying to nemo-ci April 24, 2026 20:47 Failure

Reduce sglang build to single thread

829a247

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

copy-pr-bot Bot temporarily deployed to nemo-ci April 24, 2026 23:12 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci April 25, 2026 01:17 Inactive

copy-pr-bot Bot had a problem deploying to nemo-ci April 25, 2026 02:52 Failure

Merge branch 'main' into chtruong/cuda-13

b0ed249

copy-pr-bot Bot temporarily deployed to nemo-ci April 25, 2026 20:00 Inactive

copy-pr-bot Bot had a problem deploying to nemo-ci May 10, 2026 18:57 Error

ci: Use gpu x 4 for sglang build

42d1397

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

chtruong814 requested a review from a team as a code owner May 11, 2026 00:33

github-actions Bot added the CI Relating to CI label May 11, 2026

copy-pr-bot Bot had a problem deploying to nemo-ci May 11, 2026 00:34 Failure

Set CMAKE_BUILD_PARALLEL_LEVEL = "12"

ce63ae4

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

copy-pr-bot Bot had a problem deploying to nemo-ci May 11, 2026 01:16 Failure

Set docker sglang build to CMAKE_BUILD_PARALLEL_LEVEL = "8",

136a6fa

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

copy-pr-bot Bot temporarily deployed to nemo-ci May 11, 2026 02:07 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci May 11, 2026 05:15 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci May 11, 2026 06:51 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci May 11, 2026 10:49 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci May 11, 2026 11:03 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci May 11, 2026 17:16 Inactive

kajalj22 approved these changes May 11, 2026

View reviewed changes

chtruong814 merged commit 477ab02 into main May 12, 2026
63 of 65 checks passed

chtruong814 deleted the chtruong/cuda-13 branch May 12, 2026 01:05

kajalj22 mentioned this pull request May 18, 2026

ci: build container on Azure for main builds #2521

Closed

4 tasks

kajalj22 mentioned this pull request May 26, 2026

Failing tests after CUDA 13 bump #2579

Closed

kajalj22 mentioned this pull request Jun 25, 2026

Perf test failing with NCCL error #2937

Closed

Navigation Menu

Uh oh!

chore: Enable cuda-13 build - #2332

chore: Enable cuda-13 build#2332
chtruong814 merged 31 commits into
mainfrom
chtruong/cuda-13

chtruong814 commented Apr 24, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 25, 2026

Uh oh!

chtruong814 commented May 11, 2026

Uh oh!

chtruong814 commented May 11, 2026

Uh oh!

chtruong814 commented May 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

chtruong814 commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Issues

Usage

Before your PR is "Ready for review"

Additional Information

Uh oh!

copy-pr-bot Bot commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 24, 2026

Uh oh!

chtruong814 commented Apr 25, 2026

Uh oh!

chtruong814 commented May 11, 2026

Uh oh!

chtruong814 commented May 11, 2026

Uh oh!

chtruong814 commented May 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chtruong814 commented Apr 24, 2026 •

edited

Loading