[ci][dask][gpu] Run Dask tests with LightGBM GPU version

[jgiannuzzi]

As it was suggested in #5282 (comment) (previous attempt: #5285).

Using findings from #5282 (comment):

My understanding is that the dask tests only fail with this error on Ubuntu 14.04 with Python 3.10. I ran them locally with Python 3.8 and 3.9 and they work fine. Do we maybe want to set PYTHON_VERSION again on the Linux gpu_source job?

On Ubuntu 20.04, the tests seem to fail because of a timeout. Enabling those tests on a fake GPU is expected to take more time. Maybe setting timeoutInMinutes like we do for the aarch64 job would help?

[jameslamb]

@jgiannuzzi I just updated this branch with the changes from latest master. I'm hoping that these changes that have accumulated since this PR was first opened will allow us to re-enable the Dask tests for gpu builds:

• [ci] migrate CI from macOS 10.15 to 11 (fixes #5391) #5396
• [tests][dask] fix workers without data test (fixes #5537) #5544
• [ci] prefer CPython in Linux and macOS test environment #5555
• [ci] switch to manylinux_2_28 for Linux artifacts (fixes #5514, fixes #5589) #5580

Thanks for experimenting with this and pushing it forward! The Dask tests are the most comprehensive tests on distributed training that this project has, so enabling them on more environment and installation types is really valuable to the project.

[jameslamb]

Happy to say that this is working!

I think that the timeout observed was actually not about the tests taking too long to run, but was related some combination of:

• the library-loading issues described in OSError: dlopen: cannot load any more object with static TLS #5584, and fixed by upgrading the Linux_* jobs to an environment with a newer GLIBC in [ci] switch to manylinux_2_28 for Linux artifacts (fixes #5514, fixes #5589) #5580
• the network-cleanup strategy in the Python package ([ci] prefer CPython in Windows test environment and use safer approach for cleaning up network (fixes #5509) #5510 (comment)), fixed by [ci] prefer CPython in Windows test environment and use safer approach for cleaning up network (fixes #5509) #5510

Given that, I pushed 06c2d3b removing the proposed timeouts for Azure DevOps from this PR.

The GPU jobs are taking less than 30 mins (build link).

Thanks very much for improving the test coverage @jgiannuzzi !

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.