Make nvCOMP linkage configurable - #23415
Closed
gerashegalov wants to merge 3 commits into
Closed
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughSummary by CodeRabbit
WalkthroughcuDF adds a CMake option for selecting nvCOMP linkage and updates target resolution to support validated static, shared, and automatic modes. ChangesnvCOMP linkage configuration
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>
Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>
Contributor
Author
|
Not needed at the moment per NVIDIA/cudf-spark-jni#4881 |
gerashegalov
added a commit
to NVIDIA/cudf-spark-jni
that referenced
this pull request
Jul 26, 2026
## Summary - rely on the static nvCOMP copy already linked into `libcudf.so` - stop packaging and loading the redundant nvCOMP shared libraries - localize symbols from statically linked CUDA compiler and nvCOMP implementation archives with `--exclude-libs` - retain the regular ELF symbol table for stack traces and debugging No GPU architectures are removed. No upstream cuDF change is required. cuDF already selects `nvcomp_static` when `CUDF_BUILD_STATIC_DEPS=FORCE`, as used by this build. This supersedes the previous dependency on NVIDIA/cudf#23415. Related to NVIDIA/cudf-spark#15145. ## Size results Focused CUDA 12.9.1 Release builds with `CMAKE_CUDA_ARCHITECTURES=75`: | Variant | Packaged native bytes | ZIP level 6 | | --- | ---: | ---: | | main: static nvCOMP in `libcudf.so` plus packaged `libnvcomp.so` | 754,765,376 B | 389,662,932 B | | this PR: static nvCOMP only in `libcudf.so`, private symbols localized | 693,387,896 B | 354,470,480 B | The focused build reduces packaged native data by 58.53 MiB uncompressed (8.13%) and 33.56 MiB after DEFLATE (9.03%). `libcudfjni.so` is unchanged and omitted from both rows. These measurements use one architecture to make repeated comparisons practical. The implementation does not change the production architecture list. ## Linking and symbol compatibility - the JAR contains `libcudf.so` and `libcudfjni.so`, with no nvCOMP shared-library entries - `libcudf.so` has no nvCOMP `DT_NEEDED` entry - all 12 nvCOMP JNI entry points remain dynamically exported - nvCOMP implementation symbols are absent from `.dynsym` - the regular ELF `.symtab` remains available for diagnostics ## Validation - native CUDA 12.9.1 Release build with `CMAKE_CUDA_ARCHITECTURES=75` - `mvn package -DskipTests` through the normal patch lifecycle - `NativeDepsLoaderTest`: 4 passed - GPU `NvcompTest` LZ4 and Zstd cases: 2 passed - `git diff --check` --------- Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CUDF_NVCOMP_LINKAGE=AUTO|STATIC|SHAREDAUTOmodeWhy
The cuDF Java distribution already packages the shared nvCOMP library. Downstream builds that otherwise force static dependencies currently embed a second copy of nvCOMP in
libcudf.sowhile also shippinglibnvcomp.so.This option lets those distributions choose the packaged shared library without changing the default linkage policy for other cuDF consumers.
Related to NVIDIA/cudf-spark#15145.
Validation
libcudf.sowith bothSTATICandSHAREDon CUDA 12.9.1 (sm_75)DT_NEEDED: libnvcomp.so.5libnvcomp.so,libcudf.so,libcudfjni.soMeasured on otherwise identical one-architecture builds:
libcudf.soThis removes 18.42 MiB uncompressed and 13.55 MiB after DEFLATE.