cudaErrorStreamCaptureUnsupported in cudagraph in multi-threads

[lix19937]

Description

When I use cudagraph in multi-threads(CPU threads), each thread has preprocess (default stream, with cudaDeviceSynchronize), and infer(create stream, not default stream), cudagraph only capture enqueueV2 .

Now it report error: cuda error code=900 cudaErrorStreamCaptureUnsupported. cudaDeviceSynchronize()

---

Question

In the case of 1 GPU and multiple CPU threads, if I call cudaDeviceSynchronize() from one CPU thread, will it wait for other CPU threads to complete their work on the GPU device ?

And A-thread(CPU) cudaDeviceSynchronize will break the B-thread(CPU) graph, so it report error code=900 ?

Thanks !

[nvpohanh]

Could you try capturing using cudaStreamCaptureModeThreadLocal mode?
See: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g24ae5ae00cc50614957ff8eba43e560e

[lix19937]

@nvpohanh yes I has set
GpuErrChk(cudaStreamBeginCapture(mStream, cudaStreamCaptureModeThreadLocal)); or
GpuErrChk(cudaStreamBeginCapture(mStream, cudaStreamCaptureModeGlobal)); both error

[nvpohanh]

I see... then it seems that cudaDeviceSynchronize() will break cuda stream capture regardless of the modes.

Is it possible to replace the cudaDeviceSynchronize() with something else?

[lix19937]

@nvpohanh Now I use cudaStreamSynchronize(0) replace of cudaDeviceSynchronize() , and no error report. Now closed, thanks

[lix19937]

@nvpohanh sometimes, error still exist, cuda error code=900 cudaErrorStreamCaptureUnsupported. cudaStreamSynchronize(0)

[lix19937]

Fixed, closed.