Patch the build system to better support conda-build

[leofang]

Patch taken from conda-forge/cupy-feedstock#199.

Context:

1. The conda compiler is a cross compiler by design (and it turns out nvcc is, too). It means that a conda package can be cross-compiled instead of native-compiled. In other words, the build platform is not necessary the same as the target platform. But this creates an issue in particular for CuPy, because CuPy has never promised to support build != target. In short, CuPy has an Autotool-like dependency checking system (backed by build_shlib() and build_and_run()) that essentially checks the dependencies on the build platform (say linux64), but the actual binaries on the target platform (say aarch64) cannot be linked by the cross compiler (on linux64). A linker failure is raised when attempting to do so.
2. The CUDA 12 bringup on conda-forge further reinforces the need to support this use case. As mentioned, we learned that nvcc is also a cross compiler (and people urge for this capability crazily in order to reduce the package build time), and when transitioning to the new package layout we see that it's best to assume a cross compiler is in use. So the patch that was originally written to only support cross compiling (+ CUDA 11), can also be tweaked to support CUDA 12 on conda-forge.

Approach:

• This patch is turned on only during conda-build. This is possible by detecting a few env vars set by conda-build; that is, when building CuPy on CF. It's a no-op for 99% of the time.
• This patch is written to support both the old layout (CUDA 11) and new layout (CUDA 12) on conda-forge.
• This patch works with both native and cross compiling.
• This patch has not been validated on Windows yet, because
  • It makes little sense to cross compile from linux64 to win64
  • CUDA 12 + win64 is not supported yet
  • Update: Windows support is included (Patch the build system to better support conda-build #7603 (comment))

Also close #8124.

[leofang]

/test mini

[leofang]

/test mini

[leofang]

FYI @kmaehashi this is ready 🙂

[leofang]

Update: I updated the PR with Windows support (from conda-forge/cupy-feedstock#228). If we plan to backport this PR, I can do it manually (the CUB path lookup needs to be patched).

[leofang]

/test mini

[leofang]

/test mini

[leofang]

@kmaehashi Shall we try to get this merged (and backported) before the next release, so that we can avoid maintaining the patch in cupy-feedstock?

[leofang]

Hi @kmaehashi any chance we can get this PR merged and backported to v13.2.0? It's getting more and more challenging for me to maintain a big patch for conda-forge... 😅

[leofang]

/test mini

[jakirkham]

/test mini

(to include now required CUDA 12.5 checks)