Add and use new artifact/workflow functionality for CI scripts.#4861
Merged
alliepiper merged 9 commits intoNVIDIA:mainfrom Jul 21, 2025
Merged
Add and use new artifact/workflow functionality for CI scripts.#4861alliepiper merged 9 commits intoNVIDIA:mainfrom
alliepiper merged 9 commits intoNVIDIA:mainfrom
Conversation
Contributor
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
/ok to test |
Contributor
Author
|
/ok to test |
Contributor
Author
|
/ok to test |
Contributor
🟥 CI finished in 0m 06s: Pass: 0%/2 | Total: 0m 06s | Avg: 0m 03s | Max: 0m 06s
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 2)
| # | Runner |
|---|---|
| 2 | linux-amd64-cpu16 |
Contributor
Author
|
/ok to test |
Contributor
🟥 CI finished in 0m 06s: Pass: 0%/2 | Total: 0m 06s | Avg: 0m 03s | Max: 0m 06s
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 2)
| # | Runner |
|---|---|
| 2 | linux-amd64-cpu16 |
Contributor
Author
|
/ok to test |
Contributor
🟥 CI finished in 0m 06s: Pass: 0%/2 | Total: 0m 06s | Avg: 0m 03s | Max: 0m 06s
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 2)
| # | Runner |
|---|---|
| 2 | linux-amd64-cpu16 |
Contributor
Author
|
/ok to test |
Contributor
🟥 CI finished in 2m 13s: Pass: 0%/2 | Total: 2m 13s | Avg: 1m 06s | Max: 2m 13s
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 2)
| # | Runner |
|---|---|
| 2 | linux-amd64-cpu16 |
Contributor
Author
|
/ok to test |
Contributor
🟥 CI finished in 2m 14s: Pass: 0%/2 | Total: 2m 14s | Avg: 1m 07s | Max: 2m 14s
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 2)
| # | Runner |
|---|---|
| 2 | linux-amd64-cpu16 |
Contributor
Author
|
/ok to test |
Contributor
🟥 CI finished in 2m 17s: Pass: 0%/2 | Total: 2m 17s | Avg: 1m 08s | Max: 2m 17s
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 2)
| # | Runner |
|---|---|
| 2 | linux-amd64-cpu16 |
Contributor
Author
|
/ok to test |
Contributor
Author
|
/ok to test |
2 similar comments
Contributor
Author
|
/ok to test |
Contributor
Author
|
/ok to test |
Contributor
Author
|
pre-commit.ci autofix |
Contributor
Author
|
/ok to test |
1 similar comment
Contributor
Author
|
/ok to test |
Contributor
🟨 CI finished in 1h 10m: Pass: 62%/189 | Total: 1d 04h | Avg: 9m 06s | Max: 40m 40s | Hits: 95%/211372
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| +/- | CUB |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 189)
| # | Runner |
|---|---|
| 125 | linux-amd64-cpu16 |
| 15 | windows-amd64-cpu16 |
| 12 | linux-arm64-cpu16 |
| 12 | linux-amd64-gpu-l4-latest-1 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 5 | linux-amd64-gpu-h100-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
Contributor
Author
|
/ok to test |
Contributor
🟩 CI finished in 1h 27m: Pass: 100%/189 | Total: 1d 05h | Avg: 9m 23s | Max: 36m 33s | Hits: 97%/272659
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| +/- | CUB |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 189)
| # | Runner |
|---|---|
| 125 | linux-amd64-cpu16 |
| 15 | windows-amd64-cpu16 |
| 12 | linux-arm64-cpu16 |
| 12 | linux-amd64-gpu-l4-latest-1 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 5 | linux-amd64-gpu-h100-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
alliepiper
commented
Jun 17, 2025
|
|
||
| def generate_dispatch_job_name(matrix_job, job_type): | ||
| job_info = get_job_type_info(job_type) | ||
| ctk = matrix_job["ctk"] |
Contributor
Author
There was a problem hiding this comment.
This change just moves CTK info from the CI group name -> CI job name to support producers/consumers on different CTK versions. This is to allow Python to build wheels on 12.9 and test on older 12.x.
Contributor
Author
|
/ok to test |
Contributor
🟩 CI finished in 1h 51m: Pass: 100%/189 | Total: 3d 13h | Avg: 26m 59s | Max: 1h 24m | Hits: 82%/273365
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 189)
| # | Runner |
|---|---|
| 125 | linux-amd64-cpu16 |
| 15 | windows-amd64-cpu16 |
| 12 | linux-arm64-cpu16 |
| 12 | linux-amd64-gpu-l4-latest-1 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 5 | linux-amd64-gpu-h100-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
trxcllnt
reviewed
Jun 17, 2025
Contributor
🟩 CI finished in 3h 02m: Pass: 100%/204 | Total: 1d 11h | Avg: 10m 28s | Max: 32m 58s | Hits: 94%/296367
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 204)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 12 | linux-arm64-cpu16 |
| 12 | linux-amd64-gpu-l4-latest-1 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 9 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
trxcllnt
approved these changes
Jul 16, 2025
Contributor
🟩 CI finished in 46m 09s: Pass: 100%/44 | Total: 7h 41m | Avg: 10m 29s | Max: 19m 31s
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 44)
| # | Runner |
|---|---|
| 40 | linux-amd64-gpu-l4-latest-1 |
| 4 | linux-amd64-cpu16 |
Contributor
🟩 CI finished in 38m 00s: Pass: 100%/60 | Total: 8h 53m | Avg: 8m 53s | Max: 18m 46s
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 60)
| # | Runner |
|---|---|
| 56 | linux-amd64-gpu-l4-latest-1 |
| 4 | linux-amd64-cpu16 |
Contributor
🟩 CI finished in 1h 10m: Pass: 100%/209 | Total: 1d 11h | Avg: 10m 04s | Max: 36m 53s | Hits: 97%/319216
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
🏃 Runner counts (total jobs: 209)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 16 | linux-amd64-gpu-l4-latest-1 |
| 12 | linux-arm64-cpu16 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 10 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a rather large PR that consists of several interrelated changes that would be difficult to implement separately. The bulk of the added lines are boilerplate usage/arg parsing logic that is repeated across several new bash scripts for artifact upload and workflow querying.
The details and motivation behind the changes are summarized below and may be helpful before jumping into the diff.
Overview
This is a significant update to the artifact handling options available to CCCL. It reduces CUB test artifact preparation time from 15-20 minutes to ~90 seconds (over 90% reduction), resulting in CUB test job times dropping from 30-40 minutes to 13-25 minutes. Thrust CI times are similarly improved.
Dynamic job information, including details about related producers/consumers, is exposed through a new set of scripts. This information is used to limit the size and number of uploaded testing artifacts. This is also used to embed packaging-related info in python wheel artifact names.
Artifact stage / upload / download script
upload-artifactsgithub action enables arbitrary artifact uploads (github artifact upload options are very limited...)ci/util/artifact/*.shscripts provide convenience interfaces around dynamically staging / uploading / downloading artifacts from CI scripts, on-demand and as-needed.CI Workflow scripting
ci/util/workflow/*.shscripts interact with theworkflowartifact that contains data about allmatrix.yamljobs in the current run.get_python_wheel_name.sh).Updated existing artifact packaging
z_/zz_prefixes so things like wheels are more visible in UIs.workflow/scripts.artifact/scripts.Add test artifact uploads
sccachetest artifact caching with compressed archives containing the minimal set of files required to run tests.ci/util/create_mock_job_env.shmay be used to locally test artifact handling for a specific CI job.build_<proj>.shinstead of fetching artifacts.pbzip2.Added
force_producer_ctk:job property for major-version compatibility testingConsolidate python test coverage spec to
ci/matrix.yamlbuild-and-test-python-wheelsworkflow runs.Misc:
ci-overview.md.workflow-run-job-linuxaction has been updated to support the new artifact passing mechanism.