ensure nightly builds always produce new packages - #508
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
This comment was marked as resolved.
This comment was marked as resolved.
…or non-release builds (#268) After this PR, if environment variable `RAPIDS_VERSION_SUFFIX` is non-empty, its value is appended to the output of `rapids-generate-version` on non-release builds. ## Notes for Reviewers ### Motivation Helps resolve a long-standing issue where newly-built nightly packages aren't published because their version numbers match existing published packages (rapidsai/build-planning#218). ### How I tested this ```console $ export PATH="$(pwd)/tools:${PATH}" $ pushd ~/repos/cudf $ git fetch upstream --tags $ python -m venv .venv $ source .venv/bin/activate $ pip install dunamai $ env -u RAPIDS_VERSION_SUFFIX rapids-generate-version 26.10.00a93 $ RAPIDS_VERSION_SUFFIX="+abcd" rapids-generate-version 26.10.00a93+abcd $ RAPIDS_VERSION_SUFFIX=".post1785251654" rapids-generate-version 26.10.00a93.post1785251654 ``` See more testing on these PRs: * rapidsai/cugraph-gnn#508 * rapidsai/shared-workflows#603 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #268
…uild time (#603) Contributes to rapidsai/build-planning#218 Proposes adding a new workflow, `compute-build-details`, which computes information to be passed into builds. Right now this only calculates a version suffix like `.post{current-time-in-epoch-seconds}` to help with distinguishing nightlies, but proposing giving it a generic workflow name in case we want to stick other data in here in the future. ## Notes for Reviewers ### How does this solve the problem with nightlies? With this time-based version component, every new non-release run of `build.yaml` in a RAPIDS repo should produce packages with a never-before-seen version, so uploads to anaconda.org should succeed. e.g. we might see this version sequence for nightlies: ```text 26.08.00a35.post1785176675 # 35 commits past '26.08.00a' tag, CI run started at epoch time 1785176675 26.08.00a35.post1785180275 # 35 commits past '26.08.00a' tag, CI run started 1 hour later 26.08.00a36.post1785180475 # 36 commits past '26.08.00a' tag, CI run started a little later ``` These versions will only be used in pre-releases built on `main`, not releases or packages built in PR CI (rapidsai/gha-tools#268). ### What about timing problems? This workflow will run once at the very beginning of the branch/nightly workflow run, then its output is passed as input to all other build jobs. That makes the value static across an entire workflow run, so packages will have the same version even for: * builds starting at different times due to scheduling delays * builds starting at different times from clicking "re-run failed jobs" (e.g. for network failures) ### How I tested this Confirmed that versions like this are recognized as valid PEP 440 versions and sort the way we expect in `packaging.Version`. ```python from packaging.version import Version Version("2026.08.00") > Version("2026.08.00a35.post1785176675") # True Version("2026.08.00rc0") > Version("2026.08.00a35.post1785176675") # True # same commit, newer build wins Version("2026.08.00a35.post1885176675") > Version("2026.08.00a35.post1785176675") # True # later commit wins Version("2026.08.00a36.post1885176675") > Version("2026.08.00a35.post1785176675") # True ``` Also tested live with on `cugraph-gnn`, see "How I tested this" on rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #603
| node_type: cpu8 | ||
| script: ci/build_cpp.sh | ||
| sha: ${{ inputs.sha }} | ||
| version-suffix: ${{ needs.build-details.outputs.package-version-suffix }} |
There was a problem hiding this comment.
I should've realized this earlier, but this suffix only makes sense for wheels (it contains .post) and yet its name doesn't indicate anything about wheels.
We aren't going to suffix conda versions, we're just going to use build strings, so I don't want to imply that this is generic to all build types.
We should name this something like RAPIDS_DATETIME_STRING (to align with conda's usage of RAPIDS_DATE_STRING) and then use .post${RAPIDS_DATETIME_STRING} for wheels and insert ${RAPIDS_DATETIME_STRING} into the conda recipes where we currently use RAPIDS_DATE_STRING.
There was a problem hiding this comment.
We aren't going to suffix conda versions, we're just going to use build strings
As written, I AM proposing using this suffix conda packages too and that's how it's working right now:
...
cugraph-pyg 26.10.00a21.post260728210952 260728_9fc66978 file:///tmp/tmp.vYeKSGpHdu
...
(example from the test runs linked in the description)
Because:
- I wasn't sure how the build string participates in sorting (would the latest nightly always be preferred?)
- I didn't want to mess with the existing YYMMDD component in the build strings and break the
test-nightly-conda-envstuff inintegration(code link) and anything else that might be relying on that formatting.
Fine with me to change the naming of these things and try what you're suggesting, and I'll happily patch integration and anything else that needs to change as a result of changing the build string.
There was a problem hiding this comment.
I think there may be issues with conda solvers with both pre- and post- suffixes. I am not 100% sure about that. I do think we should stick to the status quo for conda, aside from adopting the same build string date format proposed for wheels.
The only place we explicitly rely on that date format string is in the integration repo's parser, to my knowledge.
…les' lists (#2502) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #2502
…les' lists (#275) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #275
…les' lists (#5620) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #5620
Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #1127
…les' lists (#23533) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #23533
Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #1673
…les' lists (#1018) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #1018
…les' lists (#2414) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #2414
…les' lists (#188) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #188
…les' lists (#3103) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #3103
…les' lists (#1148) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #1148
…les' lists (#722) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #722
Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #162
Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` For wheels: **unchanged**. Unlike most other projects in this series, `rapids-cli` is using `hatchling version` for its nightly wheel versions and doesn't have scheduled nightly builds (only on new commits / tags), so doesn't face the same problems as most other RAPIDS projects with needing to upload new packages from an already-built commit: <img width="405" height="194" alt="image" src="https://github.com/user-attachments/assets/19eb35e8-2a3e-40de-a1de-275f1fa202ac" /> https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/rapids-cli/ ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Naty Clementi (https://github.com/ncclementi) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #156
…les' lists (#8448) Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` Other changes: * updates `changed-files` lists to avoid triggering test jobs in PR CI when only `.github/workflows/{build,test}.yaml` are changed ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #8448
Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #303
Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #863
Contributes to rapidsai/build-planning#218 * uses the new patterns for versioning nightly packages (see rapidsai/shared-workflows#603), to ensure they're always uploaded on new builds. For wheels: ```text # before 26.10.0a55 # after 26.10.0a55.post260803200854 ``` And for conda: ```text # before 26.10.0a55[build=cuda12_260803_9da146ef] # after 26.10.0a55[build=cuda12_260803200854_9da146ef] ``` ## Notes for Reviewers ### How I tested this See rapidsai/cugraph-gnn#508 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Ramakrishna Prabhu (https://github.com/ramakrishnap-nv) URL: #1661
Contributes to rapidsai/build-planning#218
For wheels:
And for conda:
Other changes:
changed-fileslists to avoid triggering test jobs in PR CI when only.github/workflows/{build,test}.yamlare changedNotes for Reviewers
How I tested this
round 1: epoch seconds versions (click me)
Triggered a build from this branch.
gh workflow run \ build.yaml \ --ref nightly-versions \ --repo rapidsai/cugraph-gnn \ -f branch=nightly-versions \ -f build_type=branch \ -f date=2026-07-27 \ -f sha="$(git rev-parse HEAD)"That started this: https://github.com/rapidsai/cugraph-gnn/actions/runs/30321225539
Then did a second run, to show that new packages would be published from the same commit: https://github.com/rapidsai/cugraph-gnn/actions/runs/30322425531
Then triggered a testing run.
gh workflow run \ test.yaml \ --ref nightly-versions \ --repo rapidsai/cugraph-gnn \ -f branch=nightly-versions \ -f build_type=branch \ -f date=2026-07-27 \ -f sha="$(git rev-parse HEAD)"Build link: https://github.com/rapidsai/cugraph-gnn/actions/runs/30323823418
That all looks good!
The second build uploaded new packages with the same beginning version (
26.10.0a21) but different suffixes (.post1785203206vs..post1785204732).All packages from all jobs within a run had the same version (
26.10.00a21.post1785204732).The newer packages were successfully pulled by all the test jobs. e.g. output like this in a conda job:
And this in a wheels job:
(build link)
round 2: YYMMDDhhmmss versions (click me)
During review, we switched to
YYMMDDhhmmss-formatted versions strings (rapidsai/shared-workflows#603 (comment)), so I repeated the testing above with that new version format.build.yaml1: https://github.com/rapidsai/cugraph-gnn/actions/runs/30396789538build.yaml2: https://github.com/rapidsai/cugraph-gnn/actions/runs/30399476266test.yaml: https://github.com/rapidsai/cugraph-gnn/actions/runs/30402598280/job/90422013211After more review, we switched away from updating versions for conda packages, towards just replacing the YYMMDD component of build strings with a more precise date-time (#508 (comment)), so repeated the testing again.
build.yaml1: https://github.com/rapidsai/cugraph-gnn/actions/runs/30648937831build.yaml2: https://github.com/rapidsai/cugraph-gnn/actions/runs/30652238487test.yaml: https://github.com/rapidsai/cugraph-gnn/actions/runs/30653529575