[ci] improve and run CUDA jobs for every commit and PR

[StrikerRUS]

Due to

It seems currently both azure pipeline and github actions cannot support the power-off/turn-on machines.
Even with the VM scale set in azure pipeline, it needs at least one machine stand by.
I know there are some external solutions, but they seems to require my Azure (Microsoft) account permission, which seems to violate our policy.
#3424 (comment)

machine with NVIDIA GPU is always turned on and I don't see any reasons to not run CUDA jobs on it for every PR in this case. Running CUDA jobs on a regular basis will make us sure that no one new PR breaks CUDA source code.

CUDA jobs cannot be run in parallel but the duration of all 3 jobs I set in this PR are about 20 min. Is it OK?

[StrikerRUS]

@jameslamb
I'm very surprised that even Dask tests are passed with CUDA version!

[jameslamb]

@jameslamb
I'm very surprised that even Dask tests are passed with CUDA version!

that's great! I'd be surprised if it actually took advantage of the GPU, but at least the tests passing there is good. At some point (maybe when investigating #3776 ), we should look again at why the Dask tests fail on non-CUDA GPU builds.

CUDA jobs cannot be run in parallel but the duration of all 3 jobs I set in this PR are about 20 min. Is it OK?

I think this is totally fine!

I looked at the most recent one and it took about 24 minutes to run (https://github.com/microsoft/LightGBM/actions/runs/506269302).

That's quicker than other collections of jobs, so I don't think it would have too big of an impact on the peak level of activity in this repo. Because it gives us extra coverage of a part of the codebase that is very new (in terms of how many times it has been modified in its lifetime), I think it is very worth it.

[jameslamb]

I love this change! I think this will give us some new automatic test coverage that's really needed. This is going to help the pace of development a lot. Thank you for your efforts 🚀

[jameslamb]

I'm really glad to see building CMake from source replaced with an installation from a package manager 🙌

[StrikerRUS]

Huh, I decided that it is enough to have the most recent CMake for our other Linux jobs run inside Ubuntu 14.04 container. Thanks to default CMake from apt-get in Ubuntu 20.04 is quite fresh and passes our minimum version check!

LightGBM/CMakeLists.txt

Lines 21 to 22 in ac706e1

	elseif(USE_CUDA)
	cmake_minimum_required(VERSION 3.16)

As a bonus, we don't need to update it in one more place regularly 🙂

[StrikerRUS]

@guolinke Please take a look when have time.

[guolinke]

Thank you, LGTM

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.