cudaPackages.buildRedist: automatically remove runpath entries for stubs

[ConnorBaker]

Sometimes we have to link against stubs because the actual libraries are only available at runtime (like those provided by NVIDIA's drivers). Leaving the stubs in the RUNPATH is a mistake because the loader will search them and load the stubs rather than the real libraries provided at runtime if the stubs are discovered first.

This PR introduces a hook which replaces stub entries with a reference to /run/opengl-driver/lib. It is written such that at most one reference to the driver link lib directory is added if stub entries are found; the hook does not attempt to deduplicate existing entries; it may introduce a duplicate driver link lib directory reference if a stub entry occurs before an existing driver link lib directory reference.

CUDA sample which links against the stub before:

$ patchelf --print-rpath ./result/0_Introduction/matrixMul_nvrtc/matrixMul_nvrtc 
/nix/store/61cdmzcp5y82aapa1dg3d403rxpqi8h8-cuda12.8-cuda_nvrtc-12.8.93-lib/lib:/nix/store/daamdpmaz2vjvna55ccrc30qw3qb8h6d-glibc-2.40-66/lib:/nix/store/z7a34j3xnp66rpddayyxrxwsahxccbip-gcc-14.3.0-lib/lib:/nix/store/60bccal8rk5zm3nsxszvfvv6754imwcl-cuda12.8-cuda_cudart-12.8.90/lib/stubs

and after:

$ patchelf --print-rpath ./result/0_Introduction/matrixMul_nvrtc/matrixMul_nvrtc 
/nix/store/8d75zh50j3j0yv20pghnh6p67fvjf60j-cuda12.8-cuda_nvrtc-12.8.93-lib/lib:/nix/store/daamdpmaz2vjvna55ccrc30qw3qb8h6d-glibc-2.40-66/lib:/nix/store/z7a34j3xnp66rpddayyxrxwsahxccbip-gcc-14.3.0-lib/lib:/run/opengl-driver/lib

This hook is automatically enabled for all redistributables with a stubs output and can be manually enabled or disabled by providing buildRedist the includeRemoveStubsFromRunpathHook boolean.

With these changes, I'm also able to build UCX and UCC with

{
  # We cannot disallow cuda_cudart's stubs since they are in the same output as the libs.
  disallowedReferences = lib.optionals enableCuda [
    (lib.getOutput "stubs" cudaPackages.cuda_nvml_dev)
  ];
}

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[daniel-fahey]

Code looks nice running on my Mk1 eyeballs, I did notice your directory and file names don't follow Nixpkgs convention to use kebab-case? (No idea where there that is, if at all, documented. Seems to be the convention inside pkgs/build-support.)

[SomeoneSerge]

This will mean users don't have to set LD_LIBRARY_PATH before running e.g. vLLM like

Why? Does vllm suffer from the hook ordering issue, ordering stubs left of driverLink in DT_RUNPATH?

[daniel-fahey]

This will mean users don't have to set LD_LIBRARY_PATH before running e.g. vLLM like

Why? Does vllm suffer from the hook ordering issue, ordering stubs left of driverLink in DT_RUNPATH?

It doesn't. I was shooting from the hip. vLLM doesn't suffer from the hook ordering issue. The vLLM derivation uses autoAddDriverRunpath and all of its C/C++ extensions have /run/opengl-driver/lib as the first entry in their RUNPATHs 😳. Looks like the vLLM gotcha is due to it using ctypes. Thanks for pointing me in the right direction.

[ConnorBaker]

Mirroring from matrix: my preliminary assessment is that keeping stub paths is rather desirable than undesirable

Why? All I can think of is the error I (sometimes) get when loading the stub library that I’ve loaded a stub library. Other than that, having additional RUNPATH entries means more lookups while searching for other libraries, which I don’t enjoy.

... and that controlling the hook order (or commutativity & associativity & idempotence of hook execution) was the right approach

I absolutely agree that hooks should have those properties wherever possible (so nearly all of them).

But that’s a very heavy lift, which is why I looked into hook re-ordering. That’s the origin of my arrayUtilities PRs (#385960) -- I saw I needed this logic for array handling in a bunch of places and wanted to have it in a single place to reduce the likelihood I messed it up.

Unfortunately, not only was that PR not accepted as it was, requiring me to remove most of the functionality I needed from it, I also ran into an old bug (I tried to resolve it in #388767) which meant that I couldn’t use any of those utility functions (which are provided by setup hooks) until after phases started running. As such, I had to re-order hooks in later phases within a very early phase, which feels incredibly brittle and also depends on things like hook name not changing.

And as I was doing that, I encountered another problem: sometimes phases are just strings, not arrays! So I’d need to be very, very careful with how I parsed and interacted with them. I generally just don’t have the patience for that kind of surgery when we absolutely should be using structured attributes and arrays everywhere, full stop.

While I agree hooks should have these properties, there’s so much in the way of a clean, principled implementation.

Can we just remove the stub entries for now and make an issue to fix everything else later?

[SomeoneSerge]

To future readers: Connor notified us on #cuda:nixos.org matrix that he's unlikely to be available any time soon, due to dealing with the grimmer aspects of life.

All I can think of is the error I (sometimes) get when loading the stub library

Yeah we just need to rewrite the message 😂 Well, to ask for permission to rewrite the message

But that’s a very heavy lift, which is why I looked into hook re-ordering. That’s the origin of my arrayUtilities PRs [...] While I agree hooks should have these properties, there’s so much in the way of a clean, principled implementation.

I know, I've been there: https://github.com/NixOS/nixpkgs/pull/297590/files#diff-ec85191a2f968fa96c76a95ed2755151e530eebeb9852acb6a96e07e6eee9f81R10-R36

Can we just remove the stub entries for now and make an issue to fix everything else later?

If we do, I'd probably go with modifying mkDerivation with disallowReferences and removeReferencesTo. I guess let's ponder next time we chat @GaetanLepage ?

[SomeoneSerge]

Turns out the error messages aren't even defined in the stubs, but in e.g. cudart instead, so all my previous arguments are irrelevant

[SomeoneSerge]

Not a blocker: In future we probably want to replace this with a more general patchelf-structuredAttrs adapter, although patchelf is far from the only tool that suffers from this issue

[ConnorBaker]

Agreed... :(

[SomeoneSerge]

Aside from the out of date comment, ready?

[SomeoneSerge]

/me wonders if stdenv sets -o pipefail

[ConnorBaker]

setup.sh sets -euo pipefail when it begins and then at the end of being sourced restores e and u. (Interestingly, it does not unset pipefail!)

But no, unless someone sets it explicitly in a phase they're not set.

[GaetanLepage]

Good job guys!

[SuperSandro2000]

Do we really need to propagate this to runtime? Why no dev output?

[SomeoneSerge]

Damn github email integration broke again, I had replied to this days ago but the message doesn't appear. TLDR: stubs is a dev output

[SuperSandro2000]

I really want to get rid of patchelf from my final system as that is usually a sign that somewhere something gets propagated too much.

[SomeoneSerge]

It is a dev output (just not the "dev" output)

…

On November 30, 2025 2:55:36 PM UTC, Sandro ***@***.***> wrote: @SuperSandro2000 commented on this pull request. > @@ -332,6 +341,16 @@ extendMkDerivation { inherit doInstallCheck; inherit allowFHSReferences; + inherit includeRemoveStubsFromRunpathHook; + + postFixup = + postFixup + + optionalString finalAttrs.includeRemoveStubsFromRunpathHook '' + nixLog "installing stub removal runpath hook" + mkdir -p "''${!outputStubs:?}/nix-support" + printWords >>"''${!outputStubs:?}/nix-support/propagated-build-inputs" \ Do we really need to propagate this to runtime? Why no dev output?

[ConnorBaker]

@SuperSandro2000 there are a few ways to make that happen:

1. Remove patchelf as a dependency from arrayUtilities.getRunpathEntries, which will break its usage in any environment which does not provide patchelf.
2. Ensure all CUDA redists which have stubs have a stubs output and do not propagate them by default. This would ensure that the dependency on the stub removal hook is only pulled in when the stubs are explicitly requested. I may have to do that for an unrelated thing.

[SuperSandro2000]

The second option sounds like the best choice here

[SomeoneSerge]

Ensure all CUDA redists which have stubs have a stubs output and do not propagate them by default.

s/do not propagate by default/only propagate them in dev, not in out/