perf: prefer copy bandwidth over compute overlap for large batch memcpys - #4
perf: prefer copy bandwidth over compute overlap for large batch memcpys#4felipeblazing wants to merge 3 commits into
Conversation
cudf::detail::memcpy_batch_async unconditionally passed cudaMemcpyFlagPreferOverlapWithCompute. That flag caps device-to-device throughput at ~470 GB/s on GB300 regardless of copy size, while the default flag reaches ~3.2 TB/s -- a 6.8x gap on the multi-GB buffer copies behind cudf::concatenate's for_each_concatenate path. The flag is not useless: for batches of small buffers the driver fuses the copies far more effectively with it (213 GB/s vs 40 GB/s for 256 x 64 KiB), so dropping it outright regresses the small-copy IO paths by up to 5x. Measured crossover is ~1 MiB per copy and is nearly independent of the batch count, so select on the mean bytes per copy with a 2 MiB threshold. Host-device copies measured insensitive to the flag (within 1% pinned and pageable at every size), so this only changes device-to-device behavior. Also returns early on an empty batch, which the mean computation requires. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Have you benchmarked this crossover point on other hardware? |
|
@felipeblazing Let's sweep this on a few different types of hardware (suggesting L4, DGX Spark, H100, B200, RTX PRO 6000 Blackwell for decent coverage of various memory types) and file a PR upstream to cuDF. edit: I am working on that. |
|
I filed an issue here and will fix this in cuDF, rapidsmpf, and RMM. NVIDIA#23674 |
|
@felipeblazing @mike-wendt Since I see new commits here, I just wanted to let you know that I merged fixes for this in cudf, rapidsmpf, and RMM. It'll be in 26.10. |
|
Thanks @bdice for the heads up, I will defer to @felipeblazing for next steps. I was just getting our fork of |
4 participants
cudf::detail::memcpy_batch_async unconditionally passed
cudaMemcpyFlagPreferOverlapWithCompute. That flag caps device-to-device
throughput at ~470 GB/s on GB300 regardless of copy size, while the default
flag reaches ~3.2 TB/s -- a 6.8x gap on the multi-GB buffer copies behind
cudf::concatenate's for_each_concatenate path.
The flag is not useless: for batches of small buffers the driver fuses the
copies far more effectively with it (213 GB/s vs 40 GB/s for 256 x 64 KiB), so
dropping it outright regresses the small-copy IO paths by up to 5x. Measured
crossover is ~1 MiB per copy and is nearly independent of the batch count, so
select on the mean bytes per copy with a 2 MiB threshold.
Host-device copies measured insensitive to the flag (within 1% pinned and
pageable at every size), so this only changes device-to-device behavior.
Also returns early on an empty batch, which the mean computation requires.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com