Fix warnings with -O3 in jax-0.6.2/CUDA

[Flamefire]

(created using eb --new-pr)

[jfgrimm]

Test report by @jfgrimm
FAILED
Build succeeded for 0 out of 1 (total: 33 mins 23 secs) (1 easyconfigs in total)
gpu23.viking2.yor.alces.network - Linux Rocky Linux 8.10, x86_64, AMD EPYC 7413 24-Core Processor, 1 x NVIDIA NVIDIA H100 PCIe, 580.105.08, Python 3.6.8
See https://gist.github.com/jfgrimm/92cd63089b5e2378ab3da7605ce5bf8e for a full test report.

[branfosj]

Test report by @branfosj
FAILED
Build succeeded for 4 out of 5 (total: 4 hours 40 mins 54 secs) (1 easyconfigs in total)
bear-pg0208u35a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), 1 x NVIDIA NVIDIA A100-SXM4-40GB, 580.95.05, Python 3.6.8
See https://gist.github.com/branfosj/e4a7e70447a25adb489700c9781f67bc for a full test report.

[Flamefire]

Test report by @Flamefire
SUCCESS
Build succeeded for 1 out of 1 (total: 7 hours 32 mins 26 secs) (1 easyconfigs in total)
i8021 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/e942c0d14d11b62102e3331cd615b10a for a full test report.

[Flamefire]

Both builds fail with

external/xla/xla/tsl/cuda/cupti_stub.cc:16:10: fatal error: 'third_party/gpus/cuda/extras/CUPTI/include/cupti.h' file not found
   16 | #include "third_party/gpus/cuda/extras/CUPTI/include/cupti.h"

I can't tell why that happens at all as I don't see that in my build.
And it can surely not be caused by this change to compiler warnings.
Any ideas?

[branfosj]

I am about to rebuild CUDA, as my install predates easybuilders/easybuild-easyblocks#3791 and try building jax again.

[jfgrimm]

Test report by @jfgrimm
FAILED
Build succeeded for 0 out of 1 (total: 1 hour 8 mins 24 secs) (1 easyconfigs in total)
node001.viking2.yor.alces.network - Linux Rocky Linux 8.10, x86_64, AMD EPYC 7643 48-Core Processor, Python 3.6.8
See https://gist.github.com/jfgrimm/81372dea0644985e5f3892b77e5ce4e3 for a full test report.

[Flamefire]

@jfgrimm Same issue.

In one environment where it did fail for me rebuilding CUDA-12.6.0.eb with easybuilders/easybuild-easyblocks#3791 made it pass afterwards

[jfgrimm]

I thought I did, but I accidentally rebuilt CUDA 11.6 not 12.6 🤦

[branfosj]

Test report by @branfosj
FAILED
Build succeeded for 0 out of 1 (total: 3 mins 46 secs) (1 easyconfigs in total)
bear-pg0210u03a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8480CL (sapphirerapids), Python 3.6.8
See https://gist.github.com/branfosj/90e7a15a56880334f087acbb30b1d36f for a full test report.

Clang / LLVM bug on SapphireRapids:

UNREACHABLE executed at /dev/shm/branfosj/build-up-EL8/Clang/18.1.8/GCCcore-13.3.0-CUDA-12.6.0/llvm-project-18.1.8.src/llvm/lib/Target/X86/X86InstrInfo.cpp:8220!

[branfosj]

Test report by @branfosj
SUCCESS
Build succeeded for 1 out of 1 (total: 6 hours 9 mins 44 secs) (1 easyconfigs in total)
bear-pg0208u35a - Linux RHEL 8.10, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), 1 x NVIDIA NVIDIA A100-SXM4-40GB, 580.95.05, Python 3.6.8
See https://gist.github.com/branfosj/6867b336f15732c011c21687462e30a2 for a full test report.

[jfgrimm]

Test report by @jfgrimm
SUCCESS
Build succeeded for 1 out of 1 (total: 3 hours 39 mins 52 secs) (1 easyconfigs in total)
gpu22.viking2.yor.alces.network - Linux Rocky Linux 8.10, x86_64, AMD EPYC 7413 24-Core Processor, 1 x NVIDIA NVIDIA H100 PCIe, 580.105.08, Python 3.6.8
See https://gist.github.com/jfgrimm/6c13860df0909435900eb99d643b3195 for a full test report.

[branfosj]

Going in, thanks @Flamefire!

[Flamefire]

Test report by @Flamefire
SUCCESS
Build succeeded for 1 out of 1 (total: 3 hours 8 mins 19 secs) (1 easyconfigs in total)
c112 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 9334 32-Core Processor (zen4), 4 x NVIDIA NVIDIA H100, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/296cd5024168a5d4a43db4a2408df200 for a full test report.