Introduce new reusable workflow jobs and cleanup manual trigger (#1954)

There are 8 reusable workflow "components" that we can use to build different scenarios: reusable_checks.yml - These are all the checks that run to ensure the code is formatted, reusable_bench.yml - This job runs the benchmarks to check for performance regressions. reusable_deploy_docs- This job deploys the python and rust documentation to https://ref.rerun.io reusable_build_and_test_wheels.yml - This job builds the wheels, runs the end-to-end test, and produces a sample RRD. The artifacts are accessible via GitHub artifacts, but not otherwise uploaded anywhere. reusable_upload_wheels.yml- This job uploads the wheels to google cloud reusable_build_web.yml - This job builds the wasm artifacts for the web. reusable_upload_web.yml - This job uploads the web assets to google cloud. By default this only uploads to: app.rerun.io/commit/<commit>/ reusable_pr_summary.yml - This job updates the PR summary with the results of the CI run. Example summary can be found at: https://storage.googleapis.com/rerun-builds/pull_request/1954/index.html This also introduces a manual_dispatch.yml helper as a convenience for testing these workflows and their different parameterizations.
rerun-io · Apr 25, 2023 · 7eaf158 · 7eaf158
1 parent 840a127
commit 7eaf158
Show file tree

Hide file tree

Showing 18 changed files with 1,734 additions and 103 deletions.
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
@@ -0,0 +1,66 @@
+# Overview
+
+Our CI workflows make heavy usage of [Reusable Workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows). These reusable workflows can then be tested manually via the `manual_dispatch.yml` workflow.
+Or integrated into CI jobs such has `on_pull_request.yml` or `on_main.yml`.
+
+By convention:
+- All reusable workflows start with the `reusable_` prefix.
+- All workflows that are triggered via `workflow_dispatch` start with the `manual_` prefix.
+- All workflows that are triggered via an event start with the `on_` prefix.
+  - `on_pull_request` is triggered on pull requests.
+  - `on_main` is triggered on pushes to the main branch.
+
+If you are going to be doing any editing of workflows, the
+[VS Code extension](https://marketplace.visualstudio.com/items?itemName=cschleiden.vscode-github-actions)
+for GitHub Actions is highly recommended.
+
+## Reusable Workflows
+- [reusable_checks.yml](reusable_checks.yml) - These are all the checks that run to ensure the code is formatted,
+  linted, and tested. This job produces no artifacts other than a pass/fail criteria for the build.
+  - `SAVE_CACHE` - If true, the rust cache will be saved. Generally we only do this for builds on `main`
+- [reusable_bench.yml](reusable_bench.yml) - This job runs the benchmarks to check for performance regressions.
+  - `SAVE_BENCH` - If true, then the benchmark results are saved to update https://ref.rerun.io/dev/bench/
+- [reusable_deploy_docs](reusable_deploy_docs.yml) - This job deploys the python and rust documentation to https://ref.rerun.io
+  - `PY_DOCS_VERSION_NAME` - The name to use for versioning the python docs. This should generally match the version in
+    `Cargo.toml`.
+  - `UPDATE_LATEST` - If true, then the docs will be deployed to `latest/` as well as the versioned directory.
+- [reusable_build_and_test_wheels.yml](reusable_build_and_test_wheels.yml) - This job builds the wheels, runs the
+end-to-end test, and produces a sample RRD. The artifacts are accessible via GitHub artifacts, but not otherwise
+uploaded anywhere.
+  - `MATURIN_FEATURE_FLAGS` - The feature flags to pass to maturin.
+  - `PLATFORM` - Which platform to build for: `linux`, `macos-arm`, `macos-intel`, or `windows`.
+  - `RELEASE_VERSION` - If producing a release, the version number. This must match the version in `Cargo.toml`.
+  - `RRD_ARTIFACT_NAME` - Intermediate name of the GitHub rrd artifact for passing to `reusable_upload_wheels.yml`
+  - `SAVE_CACHE` - If true, the rust cache will be saved. Generally we only do this for builds on `main`
+  - `WHEEL_ARTIFACT_NAME` - Intermediate name of the GitHub wheel artifact for passing to `reusable_upload_wheels.yml`
+- [reusable_upload_wheels.yml](reusable_upload_wheels.yml) - This job uploads the wheels to google cloud
+  - `RRD_ARTIFACT_NAME` - Intermediate name of the GitHub rrd artifact. This should match the name passed to
+    `reusable_build_and_test_wheels.yml`
+  - `WHEEL_ARTIFACT_NAME` - Intermediate name of the GitHub wheel artifact. This should match the name passed to
+    `reusable_build_and_test_wheels.yml`
+- [reusable_build_web.yml](reusable_build_web.yml) - This job builds the wasm artifacts for the web.
+  - `RELEASE_VERSION` - If producing a release, the version number. This must match the version in `Cargo.toml`.
+- [reusable_upload_web.yml](reusable_upload_web.yml) - This job uploads the web assets to google cloud. By default this
+  only uploads to: `app.rerun.io/commit/<commit>/`
+  - `MARK_PRERELEASE_FOR_MAINLINE` - If true, then the web assets will go to `app.rerun.io/preleease/
+  - `MARK_TAGGED_VERSION` - If true, then the web assets will go to `app.rerun.io/version/<RELEASE_VERSION>`
+  - `RELEASE_VERSION` - If producing a release, the version number.
+  - `RRD_ARTIFACT_NAME` - Intermediate name of the GitHub rrd artifact. This should match the name passed to
+    `reusable_build_and_test_wheels.yml`
+  - `UPLOAD_COMMIT_OVERRIDE` - If set, will replace the value of `<commit>`. This is necessary because we want pull
+  request builds associated with their originating commit, even if the web-build happens on an ephemeral merge-commit.
+- [reusable_pr_summary.yml](reusable_pr_summary.yml) - This job updates the PR summary with the results of the CI run.
+  - This summary can be found at:
+  `https://storage.googleapis.com/rerun-builds/pull_request/<PR_NUMBER>/index.html`
+  - `PR_NUMBER` - The PR number to update. This will generally be set by the `on_pull_request.yml` workflow using:
+  `${{github.event.pull_request.number}}`
+
+## Manual Workflows
+- [manual_dispatch](manual_dispatch.yml) - This workflow is used to manually trigger the assorted reusable workflows for
+  testing.
+  - See the workflow file for the list of parameters.
+- [manual_build_wheels_for_pr.yml](manual_build_wheels_for_pr.yml) - This workflow can be dispatched on a branch and
+  will build all of the wheels for the associated pull-request. Uses:
+  - [reusable_build_and_test_wheels.yml](reusable_build_and_test_wheels.yml)
+  - [reusable_upload_wheels.yml](reusable_upload_wheels.yml)
+  - [reusable_pr_summary.yml](reusable_pr_summary.yml)
diff --git a/.github/workflows/manual_build_wheels_for_pr.yml b/.github/workflows/manual_build_wheels_for_pr.yml
@@ -5,7 +5,7 @@ on:
 
 jobs:
 
-  check_for_pr:
+  check-for-pr:
     runs-on: ubuntu-latest
     outputs:
       PR_NUMBER: ${{ steps.get_pr.outputs.PR_NUMBER }}
@@ -24,5 +24,89 @@ jobs:
             exit 1
           else
             echo "Commit is associated with PR: $pr_number"
-            echo "PR_NUMBER=$pr_number" >> "$GITHUB_OUTPUTS"
+            echo "PR_NUMBER=$pr_number" >> "$GITHUB_OUTPUT"
           fi
+
+  build-linux:
+    needs: [check-for-pr]
+    name: 'Linux: Build/Test Wheels'
+    uses: ./.github/workflows/reusable_build_and_test_wheels.yml
+    with:
+      PLATFORM: linux
+      WHEEL_ARTIFACT_NAME: linux-wheel
+      RRD_ARTIFACT_NAME: linux-rrd
+    secrets: inherit
+
+  build-windows:
+    needs: [check-for-pr]
+    name: 'Windows: Build/Test Wheels'
+    uses: ./.github/workflows/reusable_build_and_test_wheels.yml
+    with:
+      PLATFORM: windows
+      WHEEL_ARTIFACT_NAME: windows-wheel
+      RRD_ARTIFACT_NAME: ''
+    secrets: inherit
+
+  build-macos-arm:
+    needs: [check-for-pr]
+    name: 'Macos-Arm: Build/Test Wheels'
+    uses: ./.github/workflows/reusable_build_and_test_wheels.yml
+    with:
+      PLATFORM: macos-arm
+      WHEEL_ARTIFACT_NAME: macos-arm-wheel
+      RRD_ARTIFACT_NAME: ''
+    secrets: inherit
+
+  build-macos-intel:
+    needs: [check-for-pr]
+    name: 'Macos-Intel: Build/Test Wheels'
+    uses: ./.github/workflows/reusable_build_and_test_wheels.yml
+    with:
+      PLATFORM: macos-intel
+      WHEEL_ARTIFACT_NAME: 'macos-intel-wheel'
+      RRD_ARTIFACT_NAME: ''
+    secrets: inherit
+
+  upload-wheels-linux:
+    name: 'Linux: Upload Wheels'
+    needs: [build-linux]
+    uses: ./.github/workflows/reusable_upload_wheels.yml
+    with:
+      WHEEL_ARTIFACT_NAME: linux-wheel
+      RRD_ARTIFACT_NAME: linux-rrd
+    secrets: inherit
+
+  upload-wheels-windows:
+    name: 'Windows: Upload Wheels'
+    needs: [build-linux, build-windows]
+    uses: ./.github/workflows/reusable_upload_wheels.yml
+    with:
+      WHEEL_ARTIFACT_NAME: windows-wheel
+      RRD_ARTIFACT_NAME: linux-rrd
+    secrets: inherit
+
+  upload-wheels-macos-arm:
+    name: 'Macos-Arm: Upload Wheels'
+    needs: [build-linux, build-macos-arm]
+    uses: ./.github/workflows/reusable_upload_wheels.yml
+    with:
+      WHEEL_ARTIFACT_NAME: macos-arm-wheel
+      RRD_ARTIFACT_NAME: linux-rrd
+    secrets: inherit
+
+  upload-wheels-macos-intel:
+    name: 'Macos-Intel: Upload Wheels'
+    needs: [build-linux, build-macos-intel]
+    uses: ./.github/workflows/reusable_upload_wheels.yml
+    with:
+      WHEEL_ARTIFACT_NAME: macos-intel-wheel
+      RRD_ARTIFACT_NAME: linux-rrd
+    secrets: inherit
+
+  update-pr-summary:
+    name: 'Update PR Summary'
+    needs: [check-for-pr, upload-wheels-linux, upload-wheels-windows, upload-wheels-macos-arm, upload-wheels-macos-intel]
+    uses: ./.github/workflows/reusable_pr_summary.yml
+    with:
+      PR_NUMBER: ${{ needs.check-for-pr.outputs.PR_NUMBER}}
+    secrets: inherit