cudaPackages: fix regression, propagating too much

[SomeoneSerge]

Quite likely fixes #457391 #457406, but I'm waiting for the tests to make sure. Even if it does fix, I'm still analyzing how this could have lead to the extraneous reference in onnxrutime.

Regardless of the bearing on onnxruntime and firefox, this is the pre-refactoring logic that probably shouldn't have been changed (we've been propagating strictly less, and that seemed to have been sufficient).

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[Janrupf]

Even if it does fix, I'm still analyzing how this could have lead to the extraneous reference in onnxrutime.

Not sure if you have seen that I edited my comment, this is my guess as to how they ended up there

[SomeoneSerge]

Good news: with this build, I see no reference to nvcc in libonnxruntime_providers_cuda.so.
Bad news: even with this build, there's a requisite onnxruntime -> cudart -> nvcc, and I think the last edge didn't used to be there either

[Janrupf]

What exactly were you grepping for in libonnxruntime_providers_cuda.so? Keep in mind that not the full store path was in there, but only a prefix (grepping for the hash part of nvcc did yield it for me, so ygd3s9zm1pf77n3q3ac63v58www5scbc - not the full path and without the name). I'd be really confused if this change removed the reference magically from the binary, because it was embedded in the .nv_fatbin section in the ELF. Context was ! -L /nix/store/ygd3s9zm1pf77n3q3ac63v58www5scbc-9

[SomeoneSerge]

@Janrupf for the hash part:)

$ onnxruntime=$(nix build --no-link -f "<nixpkgs>" -I nixpkgs=flake:github:SomeoneSerge/nixpkgs/fix/cuda13/firefox --arg config '{ cudaSupport = true; cudaCapabilities = [ "8.9" ]; allowUnfree = true; }' onnxruntime --print-out-paths)
$ nix-store --query --references "$onnxruntime" | grep nvcc
$ nix-store --query --requisites "$onnxruntime" | grep nvcc
/nix/store/a9d5nqjvd81kq3rxpch647xxasvfvvpi-cuda12.8-cuda_nvcc-12.8.93

[Janrupf]

Ok now thats weird:

❯ nix why-depends --all --precise /nix/store/9bwh9xx4y08fw2384jkz3s3ai3fzg0np-onnxruntime-1.22.2 /nix/store/ygd3s9zm1pf77n3q3ac63v58www5scbc-cuda12.8-cuda_nvcc-12.8.93
/nix/store/9bwh9xx4y08fw2384jkz3s3ai3fzg0np-onnxruntime-1.22.2
├───lib/libonnxruntime_providers_cuda.so: …,rt,@.3dev......! -L /nix/store/ygd3s9zm1pf77n3q3ac63v58www5scbc-9.......c_nvcc-..../bin/..//lib…
│   → /nix/store/ygd3s9zm1pf77n3q3ac63v58www5scbc-cuda12.8-cuda_nvcc-12.8.93
├───lib/libonnxruntime_providers_cuda.so: …nn-9.13.0.50-lib/lib:/nix/store/80x699lyc99dahf85iqdv6z1f0vv6vz2-cuda12.8-cuda_cudart-12.8.90/li…
│   → /nix/store/80x699lyc99dahf85iqdv6z1f0vv6vz2-cuda12.8-cuda_cudart-12.8.90
│   └───nix-support/propagated-build-inputs: …fhjm-setup-cuda-hook /nix/store/ygd3s9zm1pf77n3q3ac63v58www5scbc-cuda12.8-cuda_nvcc-12.8.93 /nix…
│       → /nix/store/ygd3s9zm1pf77n3q3ac63v58www5scbc-cuda12.8-cuda_nvcc-12.8.93
└───lib/libonnxruntime.so.1.22.2: …y9-protobuf-32.1/lib:/nix/store/j55nsg143a22krndzhnr9f8ji78vlrdw-cuda12.8-nccl-2.27.6-1/lib:/nix…
    → /nix/store/j55nsg143a22krndzhnr9f8ji78vlrdw-cuda12.8-nccl-2.27.6-1
    └───lib/libnccl.so.2.27.6: …-m 64 -l:..&devrt -L /nix/store/ygd3s9zm1pf77n3q3ac63v58www5scbc-9..l....c_nvcc-..../bin/..//lib…
        → /nix/store/ygd3s9zm1pf77n3q3ac63v58www5scbc-cuda12.8-cuda_nvcc-12.8.93

Screenshot because of colors: .

Just figured out --precise exists. To be clear, this is pre your PR, but I don't understand the behavior of the compiler here

[SomeoneSerge]

requisite onnxruntime -> cudart -> nvcc, and I think the last edge didn't used to be there either

getOutput "include" v. getInclude 🙃

EDIT: THERE'S STILL MORE
EDIT2: OH! Nvcc's outputs are no longer split 🫠
EDIT3: Oh, but actually we only ever split out static. I think we failed to split out include because nvcc itself was assuming the merged layout, and we were running into circular references. So the real new behaviour here is cudart trying to propagate crt/host_config.h automatically

[Janrupf]

Might also be worth testing with multiple compute capabilities enabled - the problem with the hashes occurred in fatbin sections, I think those are only present when multiple compute capabilities are enabled?

EDIT: If I read the compiler documentation at https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/, section 5.6.2 this indeed sounds like fatbin is only emitted when multiple compute capabilities are enabled

[SomeoneSerge]

$ onnxruntime=$(nix build --refresh --no-link -f "<nixpkgs>" -I nixpkgs=flake:github:SomeoneSerge/nixpkgs/fix/cuda13/firefox-x --arg config '{ cudaSupport = true; cudaCapabilities = [ "8.6" "8.9" ]; allowUnfree = true; }' onnxruntime --print-out-paths)
$ nix-store --query --requisites "$onnxruntime" | grep nvcc

EDIT: Running a firefox build to make sure I'm not hallucinating

[SomeoneSerge]

I'd be really confused if this change removed the reference magically from the binary, because it was embedded in the .nv_fatbin section in the ELF

In Nixpkgs we sometimes have to open ourselves up to extreme possibilities

[Janrupf]

I'm happy as long as the reference is gone, lets hope this doesn't magically re-appear when building the full config as is done for cache lmao

[SomeoneSerge]

Good point, I'll start a full build as well

[SomeoneSerge]

Alright, I guess we're watching this one: https://hydra.nixos-cuda.org/build/2901 (@GaetanLepage, added a one-off jobset... damn it's nice to just immediately have access to hardware)

[GaetanLepage]

Still fails :'(

error: output '/nix/store/hms9xa16i406x9zxd9aci2i3cx5mg9zi-firefox-144.0.2' is not allowed to refer to the following paths:
         /nix/store/x8mydcgbry214s802nzvy7fdljx404ym-gcc-wrapper-14.3.0
error: 1 dependencies of derivation '/nix/store/h1jw7zmdwblqg5gih8bsyyhycm8xvqpg-review-shell.drv' failed to build

[SomeoneSerge]

$ firefox=$(nix build --refresh --no-link -f "<nixpkgs>" -I nixpkgs=flake:github:SomeoneSerge/nixpkgs/fix/cuda13/firefox --arg config '{ cudaSupport = true; allowUnfree = true; }' firefox-unwrapped --print-out-paths)
$ nvcc=$(nix-store --query --requisites "$firefox" | grep nvcc)
$ nix why-depends --precise "$firefox" "$nvcc"
/nix/store/7bhgsk9ck0vpbzyhy0zlvywnl0wzlisv-firefox-unwrapped-144.0.2
└───lib/firefox/libonnxruntime.so: …y9-protobuf-32.1/lib:/nix/store/d5y203f75h6ygm1cr3fbnwh5sn8fa668-cuda12.8-nccl-2.28.7-1/lib:/nix…
    → /nix/store/d5y203f75h6ygm1cr3fbnwh5sn8fa668-cuda12.8-nccl-2.28.7-1
    └───lib/libnccl.so.2.28.7: …-m 64 -l:..&devrt -L /nix/store/a9d5nqjvd81kq3rxpch647xxasvfvvpi-9..l....c_nvcc-..../bin/..//lib…
        → /nix/store/a9d5nqjvd81kq3rxpch647xxasvfvvpi-cuda12.8-cuda_nvcc-12.8.93

[GaetanLepage]

I bisected the presence of cuda_nvcc in nccl's runtime dependencies.
It happened in e4a66e9 (between 24.11 and 25.05).

[GaetanLepage]

On current master:

gaetan in 🌐 build02 in nixpkgs on  master
❮ nix-store --query --requisites "$(nix-build --arg config '{ allowUnfree = true; cudaSupport = true; }' -A cudaPackages_12_6.nccl)" | grep nvcc

gaetan in 🌐 build02 in nixpkgs on  master
❮ nix-store --query --requisites "$(nix-build --arg config '{ allowUnfree = true; cudaSupport = true; }' -A cudaPackages_12_8.nccl)" | grep nvcc
/nix/store/ygd3s9zm1pf77n3q3ac63v58www5scbc-cuda12.8-cuda_nvcc-12.8.93

[GaetanLepage]

Combining this PR and #457803 works!

❯ nix why-depends --precise $(nom-build --arg config '{ allowUnfree = true; cudaSupport = true; }' -A firefox-unwrapped) $(nom-build --arg config '{ allowUnfree = true; cudaSupport = true; }' -A cudaPackages.cuda_nvcc)
Finished at 20:17:12 after 1s
Finished at 20:17:13 after 0s
'/nix/store/vib4xmivlpznx50y6jr7smyd0d1l0yir-firefox-unwrapped-144.0.2' does not depend on '/nix/store/a9d5nqjvd81kq3rxpch647xxasvfvvpi-cuda12.8-cuda_nvcc-12.8.93'

[Janrupf]

I wonder if #457803 approach would be preferable? As in, is it actually a good idea to propagate stdenv.cc in nvcc and then strip out references after the fact? I'm not entirely sure what the reasoning behind the cuda hook propagation is, but at least in my mind it makes sense that a package which depends on nvcc also depends on gcc

[GaetanLepage]

I wonder if #457803 approach would be preferable? As in, is it actually a good idea to propagate stdenv.cc in nvcc and then strip out references after the fact? I'm not entirely sure what the reasoning behind the cuda hook propagation is, but at least in my mind it makes sense that a package which depends on nvcc also depends on gcc

Well, #457803 fixes the illegitimate cuda_nvcc path leaks (in nccl's libnccl.so and onnxruntime's libonnxruntime_providers_cuda.so), but does not prevent libonnxruntime from depending on cuda_nvcc at runtime through cuda_cudart.

Hence, we really need both PRs to fix the firefox build with cudaSupport enabled.

[Janrupf]

What I mean is whether this PR should actually also be changed to strip out the references. The problem is that I don't understand why the changes in this PR prevent the references from showing up in the first place. As far as I can tell they are meta/debug information placed there by the compiler, and I'm missing the connection between removing the propagated inputs and the references going missing.

As far as I can tell a CUDA runtime library should never reference NVCC, and thus the issue may be lurking in other packages as well where it was just not found yet.

[SomeoneSerge]

As far as I can tell they are meta/debug information placed there by the compiler, and I'm missing the connection between removing the propagated inputs and the references going missing.

My hypothesis was NVCC might have started recording flags it picks up from NVCC_{PREPEND,APPEND}_FLAGS, CUDAToolkit_INCLUDE_DIRS, etc c. I didn't have the time to debug this further, but the fact that the direct reference did disappear corroborates this version.

also be changed to strip out the references.

Yes, I think we merge this and #457803?

[UlyssesZh]

This PR breaks building ucx. Error log.

[SomeoneSerge]

@UlyssesZh probably the strictDeps comment from 1fceb96; try adding cuda_nvcc to buildInputs

[UlyssesZh]

Could you please open a PR for that?

[SomeoneSerge]

@UlyssesZh I'll try tomorrow

@Janrupf you were so much more on-point than I realized:

$ onnxruntime=$(nix build --refresh --no-link -f "<nixpkgs>" -I nixpkgs=flake:github:SomeoneSerge/nixpkgs/7aebac99c040df73d15198784a6da075e971e4e1 --arg config '{ cudaSupport = true; cudaCapabilities = [ "8.6" "8.9" ]; allowUnfree = true; }' onnxruntime --print-out-paths)
$ echo $onnxruntime
/nix/store/6id3nj2y9knjgsw4qiz3dlsw7gzwa2c7-onnxruntime-1.22.2
$ nix-store --query --requisites "$onnxruntime" | grep nvcc
$ onnxruntime=$(nix build --refresh --no-link -f "<nixpkgs>" -I nixpkgs=flake:github:SomeoneSerge/nixpkgs/7aebac99c040df73d15198784a6da075e971e4e1 --arg config '{ cudaSupport = true; allowUnfree = true; }' onnxruntime --print-out-paths)
$ nvcc=$(nix-store --query --requisites "$onnxruntime" | grep nvcc)
$ nix why-depends --precise "$onnxruntime" "$nvcc"
/nix/store/hmz27nmhys5pm2f86d0c5793234r8d4r-onnxruntime-1.22.2
└───lib/libonnxruntime_providers_cuda.so: …,rt,@.3dev......! -L /nix/store/a9d5nqjvd81kq3rxpch647xxasvfvvpi-9.......c_nvcc-..../bin/..//lib…
    → /nix/store/a9d5nqjvd81kq3rxpch647xxasvfvvpi-cuda12.8-cuda_nvcc-12.8.93

I.e. this PR disappears the direct reference otherwise present in onnxruntime, but only for cudaCapabilities = [ "8.6" "8.9" ], not for the default larger set

[Janrupf]

I honestly think the entire "propagating gcc" think is actually correct for nvcc. Best is probably to find some way to strip out the references since it seems to be just metadata - I'm not sure where there is some case where someone would want to have an nvcc string in the binary. Maybe there is a way to specifically strip the references from the fatbin sections?