Update VMs, CMake to 3.20.1, CUDA to 11.3, and pwsh to 7.1.3

[BillyONeal]

I tried updating to Ubuntu 20.04 as well but that's apparently not offered in a reasonable way on Azure right now.

[NancyLi1013]

The failures caused by cudnn depends on PR #17346.

For forest, related PR #16836.

[BillyONeal]

@NancyLi1013 Thanks for pointing those out!

[BillyONeal]

Looks like the many libvpx failures are #17287 ?

[BillyONeal]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[cenit]

you removed cudnn auto-download for recognized cuda versions. Wasn't it accepted that it was ok in other PRs?

This is removing a feature otherwise which might be the base on which someone has built something... (me)

[BillyONeal]

you removed cudnn auto-download for recognized cuda versions. Wasn't it accepted that it was ok in other PRs?

It was accepted in other PRs because from original contribution it looked like downloading any other closed source library, which would normally be OK. Unfortunately, it has been brought to our attention that this behavior is both (1) contrary to cudnn's license terms, and (2) contrary to anaconda.org's terms of service. As a result I feel forced to remove it. I did ask some nvidia folks to give us better options: https://twitter.com/MalwareMinigun/status/1384335333249097732

This is removing a feature otherwise which might be the base on which someone has built something... (me)

I'm sorry :(

[jacobkahn]

I've removed this functionality from the nccl port as well in #17431. It would be great to get a CUDA 11.3-compatible NCCL version on the CI machines as well.

[BillyONeal]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[PhoebeHui]

@BillyONeal, we noticed that many PRs failed with following failures on osx pipeline machine, could you please help take a look?

Unable to locate executable file: 'pwsh'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.

/bin/bash --noprofile --norc /Users/vagrant/Data/work/_temp/749d1b35-0c54-49fd-8da4-9c21db16a076.sh
Traceback (most recent call last):
  File "/Volumes/data/work/1/s/scripts/file_script.py", line 34, in main
    gen_all_file_strings(path, getFiles(path), headers, output)
  File "/Volumes/data/work/1/s/scripts/file_script.py", line 9, in getFiles
    files = os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/vagrant/Data/installed/vcpkg/info/'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Volumes/data/work/1/s/scripts/file_script.py", line 41, in <module>
    main(sys.argv[1])
  File "/Volumes/data/work/1/s/scripts/file_script.py", line 37, in main
    except e:
NameError: name 'e' is not defined
##[error]Bash exited with code '1'.

[BillyONeal]

@BillyONeal, we noticed that many PRs failed with following failures on osx pipeline machine, could you please help take a look?

Unfortunately I know very little about how the osx stuff works; Nicole was working on that

[BillyONeal]

/azp run

[BillyONeal]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[NancyLi1013]

All checks have passed now.

[strega-nil]

This should probably be a utilities.psm1, like in macOS, but not blocking

[BillyONeal]

We can't shove .psm1s through Invoke-AzVmRunCommand though; we're taking this script and find and replacing # REPLACE WITH UTILITY-PREFIX.ps1 with it.

[JackBoosY]

Great! Thanks Billy!