Cupy: cudaErrorIllegalAddress: an illegal memory access was encountered

[janbechtold]

Hi,
thanks for your great work. I run into the above mentioned cuda error when running the cupy script on the following normal map.
I was wondering whether you can reproduce the issue and have an idea where to start looking. Potentially somewhere where the masks are computed or the diagonal_data_term is updated.

It works with the numpy script, but I want to use the cupy script as it is faster for many of my normal_maps.

Looking forward to hearing from you.
Best,
Jan

K.txt

[xucao-42]

Hi, I have tried running the cupy script on your data, but I did not encounter any issue. I tested using cuda 11.3 with cupy-cuda11x.

I guess this cuda error are possibly due to the conflict with other threads using cuda (e.g., pytorch) or parallelization running.
Have you tried running the script along?

Hope it helps.

[janbechtold]

Hi,
thanks for your quick response and testing it on my data.
For me the error is reproducible for a number of shapes. I could not figure out yet what they have in common. Independent of the time or machine that I run on. Also there are no other GPU processes running when I start the script.

These are my cuda, cupy versions:

• cuda/10.2.89
• cudnn/10.2_v7.6
• cupy-cuda102 (name) 11.3.0 (version)

So I'll try with a newer cuda version. Thanks for the hint.

[xucao-42]

How about commenting out the two lines?
https://github.com/hoshino042/bilateral_normal_integration/blob/a224fb3897ac7c3a59f7e2c97d497962febd6604/bilateral_normal_integration_cupy.py#L16

The two lines are used to avoid out-of-memory for huge normal maps (like >2k). But in your case, the normal maps are several hundreds pixels wide. It is then safe to ignore such explicit memory management.

[janbechtold]

I tried it with the two lines commented out but the error remains.
I also tried with rather large images with 5000x3000 pixels but that worked. Even with the cuda memory allocator in place.

[janbechtold]

I updated my environment to use cuda/11.3 and respectively cupy-cuda11x (name) 11.4.0 (version)

Additional remark: I had to update pyvista to 0.37.0 to resolve another error
(ImportError: cannot import name 'vtkExtractEdges' from 'vtkmodules.vtkFiltersExtraction')

This resolves my issue. Thanks again for the hint.