Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tools/rapids-download-conda-from-github
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# A utility script that downloads a conda artifact archive from Github Artifacts, unzips it,
# Downloads a conda artifact archive from Github Artifacts, unzips it,
# and returns the path to the directory where it was unzipped.
# Positional Arguments:
# 1) a string of "cpp" or "python" which determines which conda artifact
Expand Down
3 changes: 2 additions & 1 deletion tools/rapids-download-from-github
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/bin/bash
# A utility script that downloads an artifact from Github Artifacts, unzips it to a temporary directory,
# Downloads an artifact from Github Artifacts, unzips it to a temporary directory,
# and prints the location where it was unzipped.
#
# Positional Arguments:
# 1) package name to download from Github Artifacts
set -euo pipefail
Expand Down
3 changes: 2 additions & 1 deletion tools/rapids-download-wheels-from-github
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/bin/bash
# A utility script that downloads a wheel artifact archive from Github Artifacts, unzips it,
# Downloads a wheel artifact archive from Github Artifacts, unzips it,
# and returns the path to the directory where it was unzipped.
#
# Positional Arguments:
# 1) a string of "cpp" or "python" which determines which wheel artifact
# should be downloaded
Expand Down
45 changes: 34 additions & 11 deletions tools/rapids-github-run-id
Original file line number Diff line number Diff line change
@@ -1,7 +1,22 @@
#!/bin/bash
# A utility script that examines environment variables provided
# by GitHub Actions to print out a Github Workflow Run ID where the expected artifact
# should be.
# Determines the GitHub Actions run ID to use in operations with the GitHub API.
#
# When environment variable GITHUB_RUN_ID is defined, just returns that value.
# GITHUB_RUN_ID is set automatically on GitHub Actions.
# ref: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables#default-environment-variables
#
# Otherwise, tries to determine a run ID based on other context in the environment.
# Most of that is provided by 'rapids-prompt-local-repo-config', see that
# script's docs for details.
#
# Where multiple run IDs match the supplied configuration, the most-recently-created one
# is chosen.
#
# Additional environment variables recognized by 'rapids-github-run-id':
#
# * RAPIDS_BUILD_WORKFLOW_NAME = Filename for the workflow that the run corresponds to.
# Defaults to 'pr.yaml' for RAPIDS_BUILD_TYPE="pull-request" and 'build.yaml' otherwise).
#
set -euo pipefail
export RAPIDS_SCRIPT_NAME="rapids-github-run-id"

Expand All @@ -13,8 +28,8 @@ case "${RAPIDS_BUILD_TYPE}" in
--branch "${RAPIDS_REF_NAME}" \
--commit "${RAPIDS_SHA}" \
--workflow "${RAPIDS_BUILD_WORKFLOW_NAME:-pr.yaml}" \
--json databaseId \
--jq '.[0] | .databaseId')}
--json 'createdAt,databaseId' \
--jq 'sort_by(.createdAt) | reverse | .[0] | .databaseId')}
Comment on lines +31 to +32

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See "How I tested this" in the PR description.

In short, gh run list does not make any strong guarantees about sorting, so just saying "choose the first element in the returned list" is not safe.

docs on gh run list: https://cli.github.com/manual/gh_run_list

;;
branch)
run_id=${GITHUB_RUN_ID:-$(rapids-retry --quiet gh run list \
Expand All @@ -23,22 +38,30 @@ case "${RAPIDS_BUILD_TYPE}" in
--commit "${RAPIDS_SHA}" \
--workflow "${RAPIDS_BUILD_WORKFLOW_NAME:-build.yaml}" \
--event "push" \
--json databaseId \
--jq '.[0] | .databaseId')}
--json 'createdAt,databaseId' \
--jq 'sort_by(.createdAt) | reverse | .[0] | .databaseId')}
;;
nightly)
# Notice that for nightly runs, this script intentionally does not return GITHUB_RUN_ID.
#
# In RAPIDS CI, projects often have 2 separate workflows for nightlies:
#
# * build.yaml = builds and uploads packages
# * test.yaml = installs nightly packages and runs tests
#
# From GitHub's perspective, those are different "runs", with different IDs.
# GitHub Actions organizes artifacts by run ID.
run_id=$(rapids-retry --quiet gh run list \
--repo "${RAPIDS_REPOSITORY}" \
--branch "${RAPIDS_REF_NAME}" \
--commit "${RAPIDS_SHA}" \
--workflow "${RAPIDS_BUILD_WORKFLOW_NAME:-build.yaml}" \
--event "workflow_dispatch" \
--created "${RAPIDS_NIGHTLY_DATE}" \

@jameslamb jameslamb May 8, 2025 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Proposing here that we remove --created from the call for nightly runs.

From the docs in gh run list --help:

FLAGS
  ...
      --created date      Filter runs by the date it was created
  ...

Using this flag creates an awkward association between the time when a workflow is scheduled and the nightly date it is "for".

Imagine this sequence:

  1. rmm nightly build job starts at 2025-05-10T05:00:00Z
  2. ... builds for all dependencies between rmm and cuml run...
  3. cuml nightly build job starts at 2025-05-10T23:59:59Z (almost 19 hours later)
  4. cuml nightly test job starts at 2025-05-11T01:00:00Z (the next day)

I think in that case, the test run would fail to download artifacts here:

https://github.com/rapidsai/cuml/blob/41e397d22e9985ec57f9a6ced44395efb86f2e0f/ci/test_python_common.sh#L9-L10

Because RAPIDS_NIGHTLY_DATE on the test run would be 2025-05-11, but the latest nightly artifacts were generated by a workflow run that started on 2025-05-10.

Is this likely? I don't know.

On the most recent run, it took about 3 hours from when rmm-build began to when cuml-tests began

Situations where we could encounter this problem:

  • builds take much longer than that run (e.g. because of a high rate of sccache misses)
  • we change the nightly pipeline to be triggered later than 5am UTC (code link)
  • build parallelism is worsened by some change in the dependency graph (e.g., cugraph adding a build-time dependency on cuml)

I cannot think of any situation where "get the artifacts from the latest nightly run for this specific branch + commit" would produce the wrong behavior, so even if we think the scenario I described above is unlikely, I still think we should remove --created here because it's possible.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good writeup -- I agree.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reading through it!

--json databaseId \
--jq '.[0] | .databaseId')
--json 'createdAt,databaseId' \
--jq 'sort_by(.createdAt) | reverse | .[0] | .databaseId')
;;
*)
rapids-echo-stderr "please pass a valid RAPIDS_BUILD_TYPE"
rapids-echo-stderr "RAPIDS_BUILD_TYPE must be one of [branch, nightly, pull-request]"
exit 1
;;
esac
Expand Down
1 change: 1 addition & 0 deletions tools/rapids-prompt-local-github-auth
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/bin/bash
#
# A utility script that prompts user to authenticate with GitHub in
# local environments

Expand Down
20 changes: 18 additions & 2 deletions tools/rapids-prompt-local-repo-config
Original file line number Diff line number Diff line change
@@ -1,6 +1,22 @@
#!/bin/bash
# A utility script that prompts user to setup repository information in
# local environments
# Sets environment variables reused by other scripts here to interact with
# git, the GitHub API, and external artifact storage.
#
# If those variables are already defined when this script is called, it will not overwrite them.
# If not, it prompts interactively for them.
#
# Variables:
#
# * RAPIDS_BUILD_TYPE = One of "branch", "nightly", or "pull-request".
# * RAPIDS_NIGHTLY_DATE = Date in YYYY-MM-DD format, used to organize nightly uploads.
# If not provided, the current system time is used.
# Only used when `RAPIDS_BUILD_TYPE` is "nightly".
# * RAPIDS_REF_NAME = Branch or tag (e.g. "branch-25.08" or "pull-request/1234")
# * RAPIDS_REPOSITORY = Repository the run happened in, in {org}/{repo} format (e.g. "rapidsai/rmm")
# * RAPIDS_SHA = Commit SHA on the repository indicated by `RAPIDS_REPOSITORY`.
# If not provided, this script assumes it's being run from within that repo
# and checks the commit pointed to by HEAD.
#

if [ "${CI:-false}" = "false" ]; then
rapids-echo-stderr "Local run detected."
Expand Down