Make proxy tests with LocalCUDACluster asynchronous

[pentschev]

After dask/distributed#7429 was merged, some of those tests started hanging and I could confirm there were two threads concurrently attempting to take the UCX spinlock and the GIL, which led to such deadlock. UCX-Py is currently not thread-safe, and indeed can cause problems like this should two or more threads attempt to call communication routines that will required the UCX spinlock. My theory is that the synchronous cluster will indeed cause communication on the main thread (in this case, the pytest thread) upon attempting to shutdown the cluster, instead of only within the Distributed communication thread, likely being the reason behind the test hanging.

Asynchronous Distributed clusters seem not to cause any communication from the main thread, but only in the communication thread as expected, thus making the tests asynchronous suffice to resolve such issues. In practice, it's unlikely that people will use sync Distributed clusters from the same process (as pytest does), and thus it's improbable to happen in real use-cases.

[codecov-commenter]

Codecov Report

Base: 0.00% // Head: 87.04% // Increases project coverage by +87.04% 🎉

Coverage data is based on head (d71a5b9) compared to base (8c87288).
Patch coverage: 93.58% of modified lines in pull request are covered.

❗ Current head d71a5b9 differs from pull request most recent head 4e29b37. Consider uploading reports for the commit 4e29b37 to get more accurate results

Additional details and impacted files

@@                Coverage Diff                @@
##           branch-23.02    #1084       +/-   ##
=================================================
+ Coverage          0.00%   87.04%   +87.04%     
=================================================
  Files                26       18        -8     
  Lines              3439     2300     -1139     
=================================================
+ Hits                  0     2002     +2002     
+ Misses             3439      298     -3141     

Impacted Files	Coverage Δ
dask_cuda/utils.py	83.43% <82.35%> (+83.43%)	⬆️
dask_cuda/explicit_comms/dataframe/shuffle.py	98.65% <95.91%> (+98.65%)	⬆️
dask_cuda/cuda_worker.py	77.92% <100.00%> (+77.92%)	⬆️
dask_cuda/explicit_comms/comms.py	99.05% <100.00%> (+99.05%)	⬆️
dask_cuda/get_device_memory_objects.py	92.85% <100.00%> (+92.85%)	⬆️
dask_cuda/local_cuda_cluster.py	89.01% <100.00%> (+89.01%)	⬆️
dask_cuda/proxify_host_file.py	93.71% <100.00%> (+93.71%)	⬆️
dask_cuda/benchmarks/common.py
dask_cuda/benchmarks/local_cudf_groupby.py
dask_cuda/benchmarks/local_cudf_shuffle.py
... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[madsbk]

Awesome, thanks @pentschev !

[pentschev]

/merge

[pentschev]

Thanks @madsbk for the review/approval!