Building Open3D on Windows with a recent MSVC and CUDA 12.6

[patrikhuber]

Hi,

Building Open3D on Windows with CUDA and a fairly recent MSVC version seems impossible, there are a variety of build issues. A few points:

• A more or less recent-ish CUDA version is needed, for example CUDA 12.3 and earlier don't work anymore with any MSVC >=14.40.
• Using the latest Open3D main and MSVC 14.40, 14.41 or 14.42, and any CUDA version (e.g. 12.5 or 12.6), fails with an is_proxy_reference error, see here Building failed when Windows 11 + CUDA 12.5 + MSVC 19.41 + CMake 3.29.4 + "STDGPU_BACKEND=STDGPU_BACKEND_CUDA" stotko/stdgpu#417. People in that issue say that this error happens on Ubuntu too.
• Using the latest stdgpu main compiles fine on MSVC >=14.40 and CUDA 12.6. But then using Open3D with USE_SYSTEM_STDGPU=ON and pointing it to that latest stdgpu fails with a variety of errors. Replacing thrust::pair with stdgpu::pair as suggested here stdgpu target fails the CMake configure step with CUDA 12.4 due to changes in Thrust version header #6743 (comment) results in more errors somewhere deep in Open3D's CUDA implementations.

In summary, it seems impossible to build Open3D with a somewhat recent MSVC and CUDA version, which is a very bad state of affairs, and it's been like that for almost a year now. And no one of the maintainers that might be familiar with those pieces seems to have any interest or time to fix things or even respond to issues.

Has anyone built Open3D with a somewhat recent MSVC and CUDA version on Windows? Are there no Windows users?

/CC @stotko, you are the only one that responds to issues and tries to help fix things, but I think you're the stdgpu author, and not open3d maintainer. Still, do you have any way to help perhaps?

[stotko]

Hi,

as you correctly pointed out, I'm the author and maintainer of stdgpu. I also worked and contributed to Open3D in the past but I'm not a maintainer of it. Nonetheless since Open3D is one of stdgpu's main users, I of course have some interest in fixing things on our side.

Since I mainly develop under Ubuntu, I typically rely on the CI to detect any issues on Windows which however worked quite well so far. Regarding your points:

• I observed this behavior as well in another project of mine. MSVC recently bumped the minimum required CUDA version to 12.4, see Toolset update: VS 2022 17.10 Preview 2, WinSDK 22621, CUDA 12.4.0 microsoft/STL#4475.
• The issue with is_proxy_reference has been fixed upstream in stdgpu. However, the fix required bumping the minimum thrust version to 1.13.1 (CUDA 11.5) as it built upon a thrust macro that has been introduced in this version. Unfortunately, Open3D still relies on an older version of stdgpu, presumably to maintain backward compatibility with very old CUDA versions/GPUs, see

  Open3D/3rdparty/stdgpu/stdgpu.cmake

  Line 10 in 9d0cfc8

  # Jun 20 2024. Later versions need CUDA 11.5 and an API update (stdgpu::pair)

• Glad to hear that the latest stdgpu version works. Why things fail again with USE_SYSTEM_STDGPU=ON or stdgpu::pair is strange though. I noticed that enabling the contract checks leads to weird CUDA errors deep within the reduce implementation of thrust/CUB. Do the same errors occur when the external/system stdgpu was built with STDGPU_ENABLE_CONTRACT_CHECKS=OFF?

[patrikhuber]

Hi @stotko,

Thanks a lot for the reply, and sorry for my late reply. I was now able to have another proper attempt at this. Indeed it seems pointless trying to build Open3D with its integrated old stdgpu version, as we'll hit the errors that are fixed already in stdgpu https://github.com/stotko/stdgpu/pull/428/files. So I tried a clean build again, compiling Open3D with USE_SYSTEM_STDGPU=ON and a self-compiled, latest version of stdgpu (from today). stdgpu itself compiled just fine and the tests ran through as well - so all the issues seem to be in Open3D. So:

Open3D with USE_SYSTEM_STDGPU=ON and a self-compiled, latest version of stdgpu (from today):

This results in a number of error messages when compiling Open3D. One of them revolves around thrust/stdgpu pair I think, the others are very long, but could also be pair-related. It looks like the same errors are repeated multiple times, so with any bit of luck, it might actually only be three errors / three places that need some fixing - but I have no idea how or what's going. Here's the gist of it:

Can you reproduce these errors on Linux with any recent Open3D main, USE_SYSTEM_STDGPU=ON, the latest stdgpu master, and CUDA 12.6? Or are they Windows-only?

Any ideas what I can potentially try to fix them?

Edit: Forgot to say one thing, STDGPU_ENABLE_CONTRACT_CHECKS should be OFF already because I did a Release build of stdgpu.

[stotko]

I performed some tests on Ubuntu 22.04 with the latest NVIDIA driver 565 and always did a clean build per run.

CUDA 12.4:

• default --> Error in stdgpu (ambiguous call), fixed upstream
• latest stdgpu --> Error in Open3D (thrust::pair vs. stdgpu::pair), same error as you observed
• latest stdgpu + stdgpu::pair fix --> Works
• latest stdgpu + stdgpu::pair fix + USE_SYSTEM_STDGPU=ON --> Works

CUDA 12.6:

• default --> Error (same as for CUDA 12.4)
• latest stdgpu --> Error (same as for CUDA 12.4)
• latest stdgpu + stdgpu::pair fix --> Works
• latest stdgpu + stdgpu::pair fix + USE_SYSTEM_STDGPU=ON --> Works

In addition to the above compile tests, when the compilation succeeded, I also ran the unit tests and the OnlineSLAMRGBD example (which seems to cause trouble on Windows in at least one other case #6743 (comment)). Both passed and worked smoothly on my setup.

In the initial post, you mentioned that you applied the pair fix suggested in #6743 (comment). Have you also applied the fix in the latest attempt? According to the above tests, it seems to resolve the errors in Open3D. Furthermore, it indicates that this is (fortunately) not Windows-specific. I also searched for further occurrences of thrust::pair in Open3D's code and only found one other unrelated place that does not require any changes.

[patrikhuber]

@stotko Thank you for doing your own tests on Ubuntu!

I can exactly reproduce what you did on CUDA 12.6 - and "latest stdgpu + stdgpu::pair fix + USE_SYSTEM_STDGPU=ON" "works" on Windows too. Finally! I think when I tried this earlier, I may have applied the pair-fix to the old stdgpu version, or did something else that didn't work. But applying the pair fix to the latest stdgpu version makes all CUDA/stdgpu issues go away in Open3D!

Above I put "works" in quotes: This is because it doesn't quite successfully compile to the end - there are a host of other issues ranging from 65536 symbols exceeded linker errors, issues with dynamic/static Windows CRT mixup in other Open3D dependencies, an --fPIC on Windows error, and a couple other issues - depending on whether you have BUILD_SHARED_LIBS On or Off and whether you use cmake-gui or Visual Studio's "Open->CMake" integrated CMake the build errors vary, and there are one or two combinations that more or less build (I think static on VS's cmake built). However none of that has anything to do with CUDA or stdgpu. So thank you very much for your help @stotko, it's much appreciated!

[patrikhuber]

I've renamed the title of this and I'll mark this as "Solved", as we did solve the CUDA issues here. I may or may not open an issue about the other build problems later.

[patrikhuber]

@stotko There is one more error when doing a Debug build of Open3D, any advice on this one?

I think I'm still using a Release build of stdgpu, if that could have anything to do with it.

[stotko]

Since I have the fix at hand anyways, I submitted at in #7083. Let's see if it gets accepted.

Regarding the other compile error you observed, I unfortuantely have no clue why this happens. The error description is pretty clear, but I cannot find any immediate issue in the surrounding code as only pure host functions seem to be called there. I only found this old issue (pytorch/pytorch#38024) which reported a similar issue in PyTorch. According to a NVIDIA dev contributing to the discussion, it could potentially be a CUDA regression/limitation. On Ubuntu, such an error would also appear, but I have not reproduced it in my previous testing above.