Unifying the CUDA C++ Core Libraries: Towards a More Delightful CUDA C++

[jrhemstad]

We're excited to announce that the CUDA C++ Core Libraries (CCCL) - Thrust, CUB, and libcudacxx - are now unified under the nvidia/cccl repository. This consolidation aims to offer a more cohesive experience, simplify development, and set the stage for future innovations. Dive into the details below to understand what this means for you and the broader community.

TL;DR:

• What’s happening?: Thrust, CUB, and libcudacxx, the "CUDA C++ Core Libraries" (CCCL), have moved into a unified nvidia/cccl repository.
• Why?: To streamline both development and user experience towards a more delightful CUDA C++.
• What's New?: Take a look at the refreshed README, contributor guide, new CI system, and Dev Containers.
• What to Expect?: For CUDA Toolkit users, there are no immediate changes. GitHub users should switch to the new repository and adapt to the new CMake infrastructure.
• Need Help?: Change can be a bit tricky, but help is available through examples, GitHub issues, and the discussion board.

Rationale

Bringing CCCL under one roof is the first critical step in building a more coherent and modern CUDA C++ platform. This consolidation will not only simplify development, but also ensure a smoother, more convenient user experience.

The big picture? A vision of a singular, modern CUDA C++ library that not only integrates but also enhances the features of its predecessors. Much like how the Standard C++ Library is the backbone to Standard C++, the ambition is for CCCL to be the go-to suite of general purpose, speed-of-light tools that free CUDA C++ developers to solve the problems that truly matter.

Changes

Active development has shifted and is now fully underway at the new repository. Soon, the current repositories will be archived, and all issues will be moved to the new repository. GitHub users will need to fetch from the new repository and make necessary updates to their build and CMake setups. An example project is available, illustrating how to fetch and utilize the new repository via CMake.

For those using CCCL from the CUDA Toolkit, no immediate changes are required. However, it might be beneficial to consider transitioning to the latest version of CCCL on GitHub. CCCL is now officially supported on both the latest and prior major series of the CUDA Toolkit. Therefore, moving to the latest version from GitHub grants access to the latest developments without any worries of incompatibility with older CTK versions. See the Platform Support documentation for more information.

Exciting New Features

The merger comes with improvements to infrastructure and documentation to streamline the experience for users, maintainers, and contributors.

• New CI System: A brand new CI system ushers in a new era of transparency, convenience, and reliability.
• Platform Support: A formal support policy for older CUDA versions and host compilers.
• Contributor Guide: A new contributor guide makes it easier than ever to get involved.
• Dev Containers: New Dev Containers offer a ready-to-use development environment for CCCL.
• GitHub Project Board: A new GitHub project board provides visibility into the ongoing work and future plans for CCCL.
• Godbolt Integration: The new repo is now available as a library on Godbolt, making it even easier to experiment with CCCL.

Let's Make CUDA C++ Better Together

CUDA C++ is evolving, and CCCL is at the forefront of making it more efficient and user-friendly.

Help from the community will be essential to make this vision a reality, and it is easier than ever to get involved.

Want to join the effort? Dive into open issues, try the new Dev Containers, check out the new project board, or open an issue. If CCCL is used in your project, consider adding to the list of users.

This merger is a testament to the power of collaboration that we want to foster. Thanks to the RAPIDS DevOps team (@ajschmidt8, @jjacobelli) for making our CI system happen, and @trxcllnt for creating the new Dev Container infrastructure. Kudos to @jarmak-nv for setting up the project board and processes, and to the CCCL team (@allisonvacanti, @ericniebler, @griwes, @miscco, @elstehle, @senior-zero, @wmaxey) for all their tireless work.

[eyalroz]

(Repeating and slightly expanding my comment from LinkedIn since I've now noticed this page has comments open...)

Unifying CUB and thrust might make sense, depending on how you approach it, and assuming you make the unified library more decomposable, so you don't drag everything in the moment you include anything, and are not required to follow a specific abstraction regime to use any functionality (which is an issue I've had with thrust in the past). Still, that's a surmountable challenge and possibly a good thing.

However, unifying these with libcudacxx is probably a bad idea, because libcudacxx is, IMHO, in itself a bad idea to begin with. You should not be trying to replace the standard language library, in the first place. For the host side - that's not NVIDIA's business; and for the device side - much of the standard library shouldn't exist at all, and much of the rest should not have the same design.

A minor point in this regard is amounts of code such a unification might drag in. thrust is already quite verbose; I wonder how it does when used in NVRTC. Tack on libcudacxx and I'm assuming there is even more of a burden there.

[harrism]

For the host side - that's not NVIDIA's business;

NVIDIA makes CPUs, and has for years now. This is only our latest.

Furthermore, NVIDIA needs to provide at least a device-side standard library. Doing so without providing a compatible host-side standard library leads to a lot of problems, even on systems without an NVIDIA CPU. The UX for the developer is much better with a unified standard library. I think system-wide std::atomic is a great example of this.

[eyalroz]

NVIDIA makes CPUs, and has for years now.

So do AMD, and Amazon (or ARM if you like) and Apple. They don't seem to think they need to offer their own C++ standard library.

(I'm not quite sure about Intel, because they do offer a compiler, so perhaps also a standard C++ library implementation; but those are not popular AFAICT.)

Furthermore, NVIDIA needs to provide at least a device-side standard library.

Well, in a sense, yes, but mostly no.

• It's a good idea to provide a standard(ized) library of device-side facilities; but
• It's a bad idea IMO to provide the C++ language standard library on the device.

The notion that there should be a unification of the code you use on the host an in GPU kernels is invalid, and I'm sure you know that, Mark. Yes, the "user experience" of using, say, std::vector in a CUDA kernel thread is great - it feels just like you're writing CPU code! Unfortunately, the experience will be soured somewhat by the dismal performance this will almost always get you.

I know the following is easier said than done, but - what I believe should be the course of action is collaborating with the development teams of relevant C++ standard library implementation (libstdc++, libc++, maybe EASTL, others?) to make sure your device-side libraries are ABI-compatible with their libraries, and that relevant parts of their libraries can be used transparently in device-side code (to give a counter-example to match that of the vector: span, tuples, arrays ).

Finally, a standard library of device-side code should quite possibly have also considered OpenCL C++ and maybe other vendor-specific execution environments like AMD's HIP. If it's supposed to be standard, it shouldn't be a single-vendor thing.

[miscco]

@eyalroz thanks for offering your insights. However, I believe that your comment misses the forest for the tree.

Yes we most likely will not need std::regex on device, but there is a whole breadth to the standard library that will improve user experience tremendously.

We just added std::optional and std::expected which greatly improve error handling and also backported it so that users can utilize it without switching to the latest standard. All of <type_traits> and utilities like std::tuple, std::pair and std::complex are incredibly useful for any programmer and being able to use those features seamlessly on host and device is helping a lot.

So I would greatly appreciate if we could focus on the value that libcu++ already brings on the table today, rather than some not-yet-implemented feature that might have downsides in certain circumstances.

Furthermore, I want to assure you that we do keep track of the development of most the major standard libraries and ensure that we interoperate seamless. However, it is also absolutely clear that we cannot force those libraries to adopt code changes that would increase their maintenance cost or go against their design philosophy.

[eyalroz]

@miscco : I think optional and expected are good examples to consider.

Let's start with the former. Indeed, it is convenient and useful to be able to write code with an algebraic "Unit + T" type. However:

• edit: (removed an invalid point about a pointer+boolean implementation relative to a T-and-boolean-on-the-stack-style impl)
• Using the CUDA-standard-library's optional type means either type conversions from and to host code which uses the host-compiler-associated std::optional; or no conversions, but insistence that the host-side code will also be using CUDA version of std::optional. This latter option means incompatibilities with any non-GPU (and non-NVIDIA) code and effective vendor lock-in.

As for std::expected - do we really want to encourage the user to let GPU kernels do complex error handling? I can't completely rule it out, but I wonder when this can be a good idea. After all, you can't return values from the kernel functions, nor are there exceptions you can propagate. There seems to be no alternative to using one of the kernel arguments to report back errors somehow - and even that is complicated if multiple threads are involved. If I had to advise someone about this, I'd tell them: Arrange it so that you can assume no errors when your computational kernel runs.

[griwes]

Optional never allocates dynamic memory. I'd appreciate it if you verified information like this before commenting in the future, thanks.

[eyalroz]

@griwes Well, it might, and that's a possible perf tradeoff, but - in practice it seems that's not the choice, so - point conceded.

[shkiss]

Will there be a unified documentation page?

[jrhemstad]

Yes! Overhauling and unifying the docs is high on our list of things to do. As you can imagine, we've got a ton of improvements planned now that we've merged the repositories, so please bear with us as we catch up on our long list!

If you're interested to follow, we have a tracking issue for docs improvements. This will give you an idea of what we're planning and how we're progressing. Let us know if there's anything you'd like to see that we don't have captured there already!

[Stonesjtu]

What's the difference between NVIDIA:cccl/cub and NVLABS/cub? Should the latter be archived too?

[pauleonix]

Looking at NVlabs/cub it is just a mirror of the archived NVIDIA/cub that wasn't synchronized with it for some time (it's 82 commits behind). I.e. yes, I think it should be archived (maybe after bringing it up to date with NVIDIA/cub).

[jrhemstad]

https://github.com/NVIDIA/cccl is the only actively maintained repository

All of the following are archived:
NVIDIA/cub
NVIDIA/thrust/
NVIDIA/libcudacxx
NVlabs/cub (just did this one as I'd forgotten to do it previously)