cudaPackages: add documentation on cross-compilation

[ConnorBaker]

Important

To-do:

• Revise language in pkgs/development/cuda-modules/ON-CROSS-COMPILATION.md, adding examples of prior work and possible routes for further investigation.

Description of changes

Split off from #301416 with the goal of discussing and revising docs on cross-compilation separately from otherwise unrelated fixes and improvements to cudaPackages.

CC @NixOS/cuda-maintainers @samuela @SomeoneSerge @kmittman

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[ConnorBaker]

Specific comments and points to address, from #301416.

A superficial analysis of the differences between redistributables and the CUDA Toolkit run file/Debian installer:

Superficially, I can say there's something going on with host-* and target-* subdirectories present in the CUDA Toolkit (of all kinds, I believe, cross or not).

In the non-cross CUDA Toolkit, on x86_64-linux (that's the NixOS system name) there will be a host-linux-x64 and target-linux-x64 subdirectory in the root of the CUDA Toolkit.

In the cross compilation CUDA Toolkit, the values might well be something like host-linux-x64 with target-linux-sbsa (ARM servers) or target-linux-aarch64 (Jetson devices).

I'd have to spin up an Ubuntu system to install the cross-compilation version of the CUDA Toolkit and trace through the flow of CMake to see how it's driving cross-compilation.

I fear that beyond the presence of these host and target directories, the binaries in the cross-compilation version of the CUDA Toolkit are different from those offered in the redistributables -- though I haven't done any sort of analysis on the individual binaries themselves, I worry they're compiled differently from NVIDIA's sources in such a way that we cannot mimic the changes necessary for cross-compilation in an ad-hoc manner.

At any rate, it would require more time than I'm able to allocate.

@ConnorBaker in #301416 (comment)

Cross-packages should be a subset of the native packages:

Hmmm, I'm not entirely sure why these issues are cropping up, seems to be related to the targets/ directory symlinks.

The binaries are the same, they are built once per platform/version and then packaged into each format; some formats (Conda and Python wheels) involve patchelf.

I would expect the "binary archive" tarballs, Debian packages, and RPM packages to be bit-identical.

The cross-packages are more-or-less a subset of the native packages; some libraries provide stub files, while others use the native runtime.

@kmittman in #301416 (comment)

On future approaches to a minimum working example:

We really should (have) start(ed) with an MWE: piecing together an nvcc derivation sufficient to run nvcc directly on a single file (saxpy but without cmake), manually copying stuff from archives for different systems, without hooks or much automation

@SomeoneSerge in #301416 (comment)