[Kernel/Docker/CMake] Add CCCL support for CUDA 13 in CMake and update Dockerfile

[NorthSecond]

Related: #14066 #17600 #18800

Motivation

CUDA 13 introduced CCCL 3.0, which moved several headers from:

• ${CTK_ROOT}/include/cuda/
• ${CTK_ROOT}/include/cub/
• ${CTK_ROOT}/include/thrust/

to:

• ${CTK_ROOT}/include/cccl/cuda/
• ${CTK_ROOT}/include/cccl/cub/
• ${CTK_ROOT}/include/cccl/thrust/

Per NVIDIA's migration guide, this can break builds for translation units compiled by the host compiler alone, because unlike nvcc, host compilers do not automatically pick up the new CCCL include layout. One common symptom is:

fatal error: cuda/atomic: No such file or directory

mscclpp is one of the affected dependencies. The mscclpp revision currently pinned in sgl-kernel predates the upstream CUDA 13 fix, so its CMake logic does not yet know that CCCL::CCCL must be linked for CUDA 13+ builds.

As a result, source builds of sgl-kernel on CUDA 13 currently rely on a Docker-specific include path workaround instead of handling the issue in the build system itself.

Problem with the current workaround

The current workaround injects CCCL-related include behavior from sgl-kernel/Dockerfile.(https://github.com/sgl-project/sglang/blob/25e38216b6af2472949f17b1ec0f4e476768d334/sgl-kernel/Dockerfile#L28-29)

This has a few drawbacks:

• It only helps Docker-based builds.
• It does not help local/source builds outside Docker, which is a common workflow for kernel development and experimentation.
• It places a compiler configuration issue in environment setup instead of in CMake.
• It applies outside the actual dependency boundary where the problem occurs.
• It can introduce unnecessary or hidden side effects for CUDA versions that do not need this workaround.

There is also an ongoing Docker-layer attempt in #18800, but that still addresses the symptom at the container/environment level rather than in the build configuration of the affected target.

Modifications

This PR moves the workaround to the correct layer by applying a minimal CMake-side fix:

• After adding the pinned mscclpp subdirectory, try to locate CCCL via:
  find_package(CCCL QUIET PATHS "${CUDAToolkit_LIBRARY_DIR}/cmake/cccl")
• If CCCL is found and the CUDA major version is greater than 12, link CCCL::CCCL to mscclpp_obj
• Remove the Dockerfile-only environment workaround

This keeps the fix scoped to the actual dependency that needs it, and makes both Docker and non-Docker source builds use the same build logic.

Why this approach

Upstream mscclpp has already addressed this in commit 51a86630ff8ffafd9b044581b383cf26658b4e6f ("Build fixes (#696)"), which adds CUDA 13 CCCL handling in CMake.

However, the pinned mscclpp revision in sgl-kernel is significantly older. Bumping the dependency directly would pull in a much larger set of upstream changes and would require broader validation.

Instead, this PR backports only the relevant CUDA 13 CCCL CMake fix in a minimal and readable form. This keeps the diff small, limits behavioral change, and aligns with the upstream direction without taking on the risk of a full dependency upgrade.

Scope and impact

• This change only affects build configuration.
• There is no kernel logic or runtime behavior change.
• CUDA 12 and earlier should remain unaffected because the additional linkage is only applied for CUDA 13+ when CCCL::CCCL is available.
• This is best understood as a mitigation for the currently pinned mscclpp revision.

Once sgl-kernel upgrades to an mscclpp version that already includes proper CUDA 13 support, this local backport may become unnecessary.

Scope and impact

• This change only affects build configuration.
• There is no kernel logic or runtime behavior change.
• CUDA 12 and earlier should remain unaffected because the additional linkage is only applied for CUDA 13+ when CCCL::CCCL is available.
• This is best understood as a mitigation for the currently pinned mscclpp revision.

Once sgl-kernel upgrades to an mscclpp version that already includes proper CUDA 13 support, this local backport may become unnecessary.

Accuracy Tests

N/A. This PR only changes build configuration and does not modify kernel math or runtime execution logic.

Benchmarking and Profiling

N/A. This PR does not change runtime code paths or performance-sensitive kernels.

References

• NVIDIA CCCL 2.x -> 3.0 migration guide:
  https://nvidia.github.io/cccl/unstable/cccl/3.0_migration_guide.html
• Upstream mscclpp CUDA 13 CMake fix:
  microsoft/mscclpp@51a8663

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the handling of CUDA 13's CCCL header changes by moving a previously Docker-specific workaround into the CMake build system. The change ensures that projects dependent on mscclpp can build correctly with CUDA 13 and newer versions, regardless of whether they are built within Docker or locally. This approach provides a more robust and consistent build experience without altering kernel logic or affecting older CUDA versions.

Highlights

• CUDA 13 CCCL Header Migration: Addressed the breaking change introduced by CUDA 13, which moved several CCCL headers, causing build failures for host compilers not automatically picking up the new include layout.
• Build System Fix for mscclpp: Implemented a CMake-side fix to correctly link CCCL::CCCL to the mscclpp_obj target when building with CUDA 13+, resolving a dependency issue where mscclpp's pinned revision predates CUDA 13 support.
• Removal of Docker-Specific Workaround: Eliminated the previous Dockerfile-only environment variable workaround for CCCL include paths, as the CMake fix now provides a more robust and universally applicable solution for both Docker and local builds.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• sgl-kernel/CMakeLists.txt
  • Added logic to find the CCCL package.
  • Conditionally linked CCCL::CCCL to mscclpp_obj if CCCL is found and the CUDA major version is greater than 12.
• sgl-kernel/Dockerfile
  • Removed CPLUS_INCLUDE_PATH and C_INCLUDE_PATH environment variables that previously injected CCCL-related include behavior.

Activity

• No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly addresses a build issue with CUDA 13 by moving the fix from a Docker-specific workaround to the CMake build system. This makes the build process more robust and works for non-Docker environments as well. The changes involve adding logic to CMakeLists.txt to link against CCCL for CUDA 13+ and removing the now-redundant include path workaround from the Dockerfile. The implementation is a clean backport of the upstream fix in mscclpp. I have one minor suggestion to improve consistency in the CMake version check.

[hnyls2002]

Thanks @NorthSecond! Two things changed under this PR: sgl-kernel/ was relocated to python/sglang/kernels/aot/ in #32648, and the mscclpp_obj target the CCCL link is attached to no longer exists anywhere in python/sglang/kernels/aot/CMakeLists.txt (grep for mscclpp there returns nothing). The Dockerfile CPLUS_INCLUDE_PATH/C_INCLUDE_PATH cccl workaround this removes is still present at python/sglang/kernels/aot/Dockerfile:33-34, so if you still hit the CUDA 13 CCCL header problem, a fresh PR against the new AOT tree would be welcome. Please reopen if I've missed something.