pycuda._driver.LogicError: cuMemAlloc failed: context is destroyed (caused by scikit-cuda or even CUDA itself?)

[dimitsev]

Running the following in a clean python context throws pycuda._driver.LogicError: cuMemAlloc failed: context is destroyed on the last line:

import pycuda.autoinit

import numpy as np
import pycuda
import skcuda
import skcuda.fft as cufft

plan = cufft.Plan((2,2), np.complex64, np.complex64)

del plan # equivalent to `skcuda.cufft.cufftDestroy(plan.handle)`
#skcuda.cufft.cufftDestroy(plan.handle) # equivalent to `del plan`

pycuda.gpuarray.empty((2,2), np.float32)

Deleting the FFT plan in scikit-cuda destroys the pycuda context. This happens no matter if you use del plan or skcuda.cufft.cufftDestroy(plan.handle). So maybe CUDA itself is messing up here?

As long as skcuda.cufft.cufftDestroy() is not called, everything is ok. But I want to be able to destroy FFT plans...

(See also my stackoverflow question: https://stackoverflow.com/questions/72218532/deleting-an-fft-plan-in-scikit-cuda-destroys-the-pycuda-context.)

My system:

• Debian GNU/Linux 11 (bullseye) (stable)
• Python 3.9.2
• CUDA installed following https://wiki.debian.org/NvidiaGraphicsDrivers
• nvidia driver version 470.103.01
• CUDA version 11.4
• pyCUDA version: 2021.1
• scikit-cuda version: 0.5.3

[inducer]

By using pycuda.autoinit, you're putting pycuda in charge of context management. That's not typically a good recipe for interacting with libraries that use the CUDA runtime API (like cuFFT, to my understanding). You might be better off retaining the "primary context" made by/for the runtime API and using that instead:

https://documen.tician.de/pycuda/driver.html#pycuda.driver.Device.retain_primary_context

[inducer]

How come you feel the need to clean up? Just let the context go out of scope.

[dimitsev]

I found out that you have already written what I need: https://github.com/inducer/pycuda/blob/main/pycuda/autoprimaryctx.py

Your software is powerful. :)

I thought that I needed to clean up because CUDA wants people to always release the primary context.
So I tried to find some way to call this: https://github.com/inducer/pycuda/blob/main/src/cpp/cuda.hpp#L849 (but I failed miserably)

Thank you very much for your help, professor! I will answer & close the other questions / issues I opened in scikit-cuda and on stackoverflow.

[dimitsev]

Sorry, but I am curious: https://github.com/inducer/pycuda/blob/main/test/test_driver.py#L727 mentions a test_additional_primary_context(). Doesn't a second primary context on the same device conflict with the statement

The primary context is unique per device and shared with the CUDA runtime API.

from the CUDA API?

[inducer]

That test is about there being an existing context created by the driver API (PyCUDA's normal operating mode), after which the device-primary context is also retained. The name of the test is probably not the best.

[dimitsev]

I see now, thank you!