GPU queue experiments

[jaimergp]

• Add support in conda-smithy for rerenders
  • Add support for Cirun on self-hosted GHA runners conda-smithy#1703
• Log in with a conda-forge bot account to configure it with Quansight's OpenStack cloud provider
• Add support for automatic repository registration:
  • Enable Cirun app for given repository, on Github end
  • Enable Cirun OpenStack-GPU runner for given repository, on Cirun end
• Add support in admin-requests
• Documentation

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[jaimergp]

We need to rebuild the VM images with docker support.

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[jaimergp]

Updated the conda-smithy PR with support to recognize the need for extra docker flags (--gpus all). Next step: nvidia-docker runtime suppport in the VM!

[jaimergp]

Good news, the --gpus switch works now!

I think there is some issue with SSL downloads inside the Docker instance. With or without GPUs, we are failing to clone the repo that provides the source code to be built:

fatal: unable to access 'https://github.com/openmm/openmm.git/': SSL connection timeout

I have seen this in the two runs that managed to successfully start conda mambabuild (0486882 and 7901e95).

---

Another minor detail is that all actions checks are green despite the runner having obviously failed 🤔 See the green checks next to the commit hashes. Have you seen it elsewhere?

[jaimergp]

Might be a MTU mismatch. I recall we needed to route things through a VPN tunnel so I guess we are using a lower MTU than default, and Docker is insisting on using 1500. The daemon configuration would have to be amended.

[jaimergp]

Well, if you are reading this, please know you are witnessing the first ever conda-build run that has successfully used GPUs in conda-forge 🚀

Relevant block (ignore the warnings, that's OpenCL; omitted here for clarity):

+ python -m openmm.testInstallation

OpenMM Version: 7.7
Git Revision: 130124a3f9277b054ec40927360a6ad20c8f5fa6

There are 4 Platforms available:

1 Reference - Successfully computed forces
2 CPU - Successfully computed forces
3 CUDA - Successfully computed forces
4 OpenCL - Successfully computed forces

Median difference in forces between platforms:

Reference vs. CPU: 6.30618e-06
Reference vs. CUDA: 6.72268e-06
CPU vs. CUDA: 7.09326e-07
Reference vs. OpenCL: 6.74352e-06
CPU vs. OpenCL: 7.29696e-07
CUDA vs. OpenCL: 4.46884e-07

All differences are within tolerance.

Still a long way to go in terms of access control and so on, but this is a BIG milestone! Thanks @aktech for all the work in the backend!

[leofang]

Can we try it for CuPy? 🙂 Does it also support aarch64?

[jakirkham]

This is nice to see! Thanks Jaime 😄

For CuPy, we can run the tests during the build or on pre-built packages (using conda build --test <pkg file>). So it would make another nice test case.

[jaimergp]

Can we try it for CuPy? 🙂 Does it also support aarch64?

The server is native x64, but I guess we can cross-compile and/or emulate?

I'll try CuPy next week. I am out of hours for this one.

[leofang]

I am happy to help in any ways, just let me know. (And no rush, ofc!) Right now we use QEMU for aarch64/ppc64le, as it's not possible to cross-compile AFAIK.