[CI/Build][BugFix] fix cuda/compat loading order issue in docker build

[wpc]

Purpose

In PR #30784 we persistents CUDA compat lib path with ldconfig. This caused a CUDA init issue when the docker image runs with new driver without requiring compatibility libs.

$ nvidia-smi
Mon Jan 26 13:09:50 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.105.08             Driver Version: 580.105.08     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+

$ podman run --entrypoint=/bin/python3  --rm --device nvidia.com/gpu=all docker.io/vllm/vllm-openai:cu130-nightly-aarch64  -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"

/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:182: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:119.)
 return torch._C._cuda_getDeviceCount() > 0
False 2

the issues is with prefix cuda compat path 00-*, the compat path now have higher priority even than normal CUDA lib

$ podman run --entrypoint=/bin/bash  --rm --device nvidia.com/gpu=all docker.io/vllm/vllm-openai:cu130-nightly-aarch64 -c "ls /etc/ld.so.conf.d/"
00-cuda-compat.conf
00-nvcr-4117280097.conf
000_cuda.conf
987_cuda-13.conf
aarch64-linux-gnu.conf
libc.conf
nvidia.conf
zz-nvcr-4281892442.conf

this patch remove the 00- prefix, lower down the compat lib loading priority to fix the problem

Test Plan

build new image with the fix and run follow, there should be no 830 error and result should be "True 2"

$ podman run --entrypoint=/bin/python3  --rm --device nvidia.com/gpu=all <new-image-tag>  -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"

Test Result

GB200 Driver Version: 580.105.08 (does not need compat libs):

$ podman run --entrypoint=/bin/python3  --rm --device nvidia.com/gpu=all localhost/vllm/vllm-openai:cu130-aarch64-785cd232b  -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"

True 2

H100 Driver Version: 535.183.06 (need compat libs to run cuda13)

$ podman run --entrypoint=/bin/python3  --rm --device nvidia.com/gpu=all localhost/vllm-openai:cu130-x86_64-673c04198  -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"

True 8

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request fixes a CUDA library loading order issue by renaming the ld.so.conf.d configuration file to lower its priority. The change is correct and well-justified. I've also identified a potential issue with how the CUDA_VERSION is parsed throughout the Dockerfile. The current method is fragile and can fail if the version number is an integer. I've provided suggestions to make the version parsing more robust. While I've commented on the lines changed in this PR, this issue is present in other parts of the file and should be addressed consistently.

[wangshangsam]

Thanks a lot for the fix, @wpc !

I've been very busy with MLPerf deadline recently and didn't have much time to get back to this problem. Since this PR is merged, I'll give the latest main a try on our internal dev GB200/GB300 clusters once the deadline is over, and report back if I encounter any further issues.

[ehfd]

@khluu

#33369

Maybe this PR should be backported to a patch release?

[ehfd]

#32373 (comment)

This PR did not really seem to fix things correctly...

[iori2333]

Unfortunately, this fix does not work for game cards with no cuda-compat support. Maybe reverting #30784 is neccessary for these cards...

[tahvane1]

I still get this error with CUDA 13.1 using nightly or cu130-nightly docker builds. I have also compatibility packages installed. We have A5000 cards.