Adding scripts to download and upload artifacts to the Github Artifact Store

[jayavenkatesh19]

Adding new scripts which use the Github Artifact Store to upload and download build artifacts towards https://github.com/rapidsai/build-infra/issues/237. No existing scripts or functionality is altered.

[jameslamb]

I'm going to join here as a reviewer alongside @gforsyth since you and I have been talking about this work and since I'd done an earlier version of it.

To start... how did you test this? Could we see some PRs showing how these would be used?

[jayavenkatesh19]

Hey @jameslamb, Here are a couple of PRs on RMM and cuDF which use the new gha-tools scripts which are a part of this PR

1. RMM: Streamlining wheel builds to use fixed location and uploading build artifacts to Github rmm#1810
2. cuDF: [DO NOT MERGE] Test PR for moving build artifacts to Github Artifact Store NVIDIA/cudf#18090

To test these changes, I used these scripts from my gha-tools fork which is overwriting the existing gha-tools folder present on the build Docker images. This customization is on the test-github-artifacts branch of the shared-workflows repository. You can see those changes here: rapidsai/shared-workflows@branch-25.04...test-github-artifacts.

Let me know if you need any additional details, I'll be happy to discuss them with you!

[gforsyth]

Thanks for putting this together @VenkateshJaya !

I have some bash simplifications (if such a thing can be said to exist) and a few questions about usage, but I think this tooling will be a big improvement!

[ajschmidt8]

Looks good far! Left a few comments, but we're getting closer.

[ajschmidt8]

nice! It's coming along.

I tested some of these scripts locally and found one issue. The rapids-download-*-from-github and rapids-get-pr-*-artifact-github scripts fails because rapids-package-name still returns the .tar.gz extension. To fix this, I updated the end of the rapids-package-name script to include the following logic:

diff --git a/tools/rapids-package-name b/tools/rapids-package-name
index eaadc5c..945f2d0 100755
--- a/tools/rapids-package-name
+++ b/tools/rapids-package-name
@@ -76,6 +76,11 @@ if (( append_arch == 1)); then
 fi
 
 # for cpp and python package types, it's a tarball, append .tar.gz and prepend project name
-pkg_name="${repo_name}_${pkg_name}.tar.gz"
+pkg_extension=".tar.gz"
+if [[ "${RAPIDS_NO_PKG_EXTENSION:-}" == "true" ]]; then
+  pkg_extension=""
+fi
+
+pkg_name="${repo_name}_${pkg_name}${pkg_extension}"
 
 echo -n "${pkg_name}"

then I was able to set RAPIDS_NO_PKG_EXTENSION=true in my local environment and everything started working correctly.

Can you make these changes to your scripts? This change will also require that you update the package extension handling in rapidsai/shared-workflows#297.

[ajschmidt8]

in hindsight, I probably should've suggested using basename to make that diff cleaner. but those conditionals and env vars will get removed after this migration is over, so it's a moot point.

[gforsyth]

One suggestion and one question, but nothing blocking. This looks really nice @VenkateshJaya !