drone: Switch Mac (darwin) pipelines to GitHub Actions#24102
Merged
drone: Switch Mac (darwin) pipelines to GitHub Actions#24102
Conversation
camscale
requested review from
fheinecke,
r0mant,
tcsc and
wadells
and removed request for
hugoShaka and
timothyb89
Contributor
Author
|
I've run push builds on teleport and teleport-private and a tag build on teleport. All working as expected, although I'm not sure how to check the release server for You can get the artifacts from the workflow run page: https://github.com/gravitational/teleport.e/actions/runs/4618294083 - they're only there for 1 day though before they expire. |
camscale
force-pushed
the
camh/drone-mac-to-gha
branch
from
436b3ce to
57926ed
Compare
r0mant
approved these changes
Apr 5, 2023
Collaborator
r0mant
left a comment
r0mant
left a comment
There was a problem hiding this comment.
This may be my favorite PR this year yet 🚀
camscale
force-pushed
the
camh/drone-mac-to-gha
branch
from
57926ed to
65ff2bc
Compare
tcsc
approved these changes
Apr 6, 2023
Sort the GitHub Actions inputs when generating the `gh-trigger-workflow` command line so that it does not randomly change order, as happens when iterating a map directly.
camscale
force-pushed
the
camh/drone-mac-to-gha
branch
from
187023f to
a6c545c
Compare
Update the darwin pipelines to run workflows on GitHub Actions instead of locally on drone builders. This replaces four pipelines with a single GitHub actions workflow as the one workflow builds the tarballs, Mac packages and Mac disk images. We continue to drive the push build from drone until we work out how secrets are safely managed in the Teleport OSS repo.
To regenerate the `.drone.yml` file, first three pipelines were manually
removed:
- build-darwin-amd64-pkg
- build-darwin-amd64-pkg-tsh
- build-darwin-amd64-connect
Then
make dronegen
was run to update the pipelines:
- push-build-darwin-amd64
- build-darwin-amd64
camscale
force-pushed
the
camh/drone-mac-to-gha
branch
from
a6c545c to
7bf1188
Compare
Contributor
|
🙌 |
camscale
added a commit
that referenced
this pull request
Apr 12, 2023
* dronegen: Sort workflow inputs for stable output
Sort the GitHub Actions inputs when generating the `gh-trigger-workflow`
command line so that it does not randomly change order, as happens when
iterating a map directly.
* dronegen: Have darwin pipelines call out to GitHub Actions
Update the darwin pipelines to run workflows on GitHub Actions instead
of locally on drone builders. This replaces four pipelines with a single
GitHub actions workflow as the one workflow builds the tarballs, Mac
packages and Mac disk images.
We continue to drive the push build from drone until we work out how
secrets are safely managed in the Teleport OSS repo.
* drone: Regenerate .drone.yml for Mac pipeline changes
To regenerate the `.drone.yml` file, first three pipelines were manually
removed:
- build-darwin-amd64-pkg
- build-darwin-amd64-pkg-tsh
- build-darwin-amd64-connect
Then
make dronegen
was run to update the pipelines:
- push-build-darwin-amd64
- build-darwin-amd64
Backport: #24102
camscale
added a commit
that referenced
this pull request
Apr 19, 2023
* dronegen: Sort workflow inputs for stable output
Sort the GitHub Actions inputs when generating the `gh-trigger-workflow`
command line so that it does not randomly change order, as happens when
iterating a map directly.
* dronegen: Have darwin pipelines call out to GitHub Actions
Update the darwin pipelines to run workflows on GitHub Actions instead
of locally on drone builders. This replaces four pipelines with a single
GitHub actions workflow as the one workflow builds the tarballs, Mac
packages and Mac disk images.
We continue to drive the push build from drone until we work out how
secrets are safely managed in the Teleport OSS repo.
* drone: Regenerate .drone.yml for Mac pipeline changes
To regenerate the `.drone.yml` file, first three pipelines were manually
removed:
- build-darwin-amd64-pkg
- build-darwin-amd64-pkg-tsh
- build-darwin-amd64-connect
Then
make dronegen
was run to update the pipelines:
- push-build-darwin-amd64
- build-darwin-amd64
Backport: #24102
camscale
added a commit
that referenced
this pull request
Apr 19, 2023
* notarize: Accept parameters for dev and bundle ID Add a couple of parameters for the developer key ID and bundle ID for signing/notarizing binaries. Keep the hard-coded values as defaults for now, but we will remove these soon when all the call sites of the tool have been updated to pass these values. We want to parameterize these values so we can use different signing keys in GitHub Actions and to make the tool agnostic to which binaries it is signing. Backport: #23092 * release: Prepare for MacOS builds on GitHub Actions * release: Move Mac signing vars from script to Makefile Move the variables for Mac signing from the `build-common.sh` shell script to the `Makefile`. These vars will need to be passed to other build processes to parameterize the signing for different GitHub Actions build environments. The switch on `ENVIRONMENT_NAME` allows different secrets to be available in GitHub Actions for production (promote) vs developer (build) builds. The default environment name is `promote` so as to be compatible with the existing Drone setup, which does not define `ENVIRONMENT_NAME`. * release: Determine Mac signing key IDs automatically Remove the hard-coded MacOS signing key IDs from the Makefile and find them dynamically based on the name of the key. This allows GitHub Actions to be set up with new keys different to the ones on the Drone builders. As long as we keep the same name on the keys, we can rotate the keys without needing to update the IDs in the Makefile. This requires us to be more judicious about exporting the variables as exporting them causes them to be evaluated. We do not want to evaluate them on non-darwin targets, and on darwin, we should only evaluate it if needed for a recipe. So use a dynamic `eval` in the recipes that need the environment variables. * release: Pass key & team ID to notarize tool Override the hard-coded values in `notarize-apple-binaries` and pass the values we get based on the GitHub Actions environment. This allows us to sign and notarize software in a development branch more easily when working on the signing and notarizing process. This will not happen automatically, but it is expected that a developer can manually trigger a workflow to perform building, signing and notarizing from a dev branch where the workflow has temporarily changed the environment to `build`. A similar change to the `Makefile` in the teleport.e repository goes with this change. This adds a new bundle ID of `com.goteleport.dev` for the dev build of Teleport. This follows the same pattern as used for the dev build of the `tsh` binary and the current production bundle ID for Teleport. Previously there was no dev signing/notarizing process for the set of Teleport binaries. * release: Add script to setup the MacOS keychain for signing Add a script for setting up the MacOS keychain for signing applications and packages. It encapsulates the `security` commands to add either or both application keys and installer keys. The keys can be either base64-encoded in environment variables, or `.p12` files on disk, making it useful for local development. * release: Split MacOS signing vars into separate mk file Put the MacOS signing variables into a separate `.mk` file and include it from the main `Makefile`. Add more comments to document the purpose of the vars and where some of the values come from. * release: Add some more comments to keychain-setup.sh Explain that the purpose of the script is to be run on CI, but can also be run manually. Add the default values used to the usage message for the keychain and password. * Address PR comments on keychain-setup.sh script * Change shebang to /bin/bash * Use heredoc instead of multiple printfs for usage message * Move `local` declaration next to setting of kpath var * release: Export DEVELOPER_ID_APPLICATION in release-darwin The sub-make for enterprise needs this to be set or it cannot sign the enterprise binaries. Export it if we are doing signing/notarizing. Backport: #23407 * release: Dont look for signing keys if not needed Use a `make` conditional instead of a shell conditional when deciding whether to run notarization or not, based on APPLE_{USERNAME,PASSWORD} being set. When done as a shell conditional, $(DEVELOPER_ID_APPLICATION) was still being evaluated, and that causes a key error if the key does not exist in the keychain. If the `APPLE_{USERNAME,PASSWORD}` env vars are not set, it should not matter if the key is present or not as it will not be used. By switching to `$(if ...)` in make, the recipe will not be evaluated at all if the condition is false. To ensure the logs say what is going on, add a message when we do not notarize the binaries. Move the logic for notarization and checking the username/password into `darwin-signing.mk` with the other signing/notarization variables. This will make it reusable by the enterprise Makefile later. Backport: #23636 * release: Add build-connect target to Makefile * release: Add build-connect target to Makefile Add a `build-connect` target to the `Makefile` to build Teleport Connect via yarn on MacOS. Linux uses the `build.assets/Makefile` `teleterm` target, and Windows build pipelines do not use `make`. Add the `CSC_NAME` make variable containing the Developer Key ID to tell electron-builder which key to use to sign the package it produces. This gives us a little more control over how the Connect built and packaged and will simplify the CI scripts to have them just call make. It is also required in order to set the `CSC_NAME` environment name correctly as the developer ID is determined by the Makefile. * darwin: Fix echo command when not notarizing Remove the leading `@` from the echo command that is run when we are not notarizing binaries. It works fine from the OSS repo, but the enterprise repo needs to do `cd .. && $(NOTARIZE_COMMAND)` and the leading `@` causes the build error: `@echo: command not found`. We'll just have to live with a little extra noise amongst all the other noise. Backport: #23750 * release: Update build for product signing Update the build scripts to properly set up the key for signing packages using `productsign`, and parameterise the bundle ID for packages in the packaging scripts. Backport: #23820 * dronegen: Add workflow-tag arg to gh-trigger-workflow This was deliberately not backported in "[v12] Backport Distroless OCI builds (#22814)" (f1bfd5a) , but now we want it. * Increase GHA build timeout Our ARM64 build fails quite frequently due to 30 minutes timeout. This PR increases the build timeout to one hour. Backport: #22926 * drone: Switch Mac (darwin) pipelines to GitHub Actions * dronegen: Sort workflow inputs for stable output Sort the GitHub Actions inputs when generating the `gh-trigger-workflow` command line so that it does not randomly change order, as happens when iterating a map directly. * dronegen: Have darwin pipelines call out to GitHub Actions Update the darwin pipelines to run workflows on GitHub Actions instead of locally on drone builders. This replaces four pipelines with a single GitHub actions workflow as the one workflow builds the tarballs, Mac packages and Mac disk images. We continue to drive the push build from drone until we work out how secrets are safely managed in the Teleport OSS repo. * drone: Regenerate .drone.yml for Mac pipeline changes To regenerate the `.drone.yml` file, first three pipelines were manually removed: - build-darwin-amd64-pkg - build-darwin-amd64-pkg-tsh - build-darwin-amd64-connect Then make dronegen was run to update the pipelines: - push-build-darwin-amd64 - build-darwin-amd64 Backport: #24102 * Update `e` submodule for mac gha backport This commit will be updated after #1119 in teleport.e has been merged to point to the head of branch/v12. --------- Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
camscale
added a commit
that referenced
this pull request
Apr 19, 2023
Sort the GitHub Actions inputs when generating the `gh-trigger-workflow` command line so that it does not randomly change order, as happens when iterating a map directly. This is a backport of a single commit of #24102 (drone: Switch Mac (darwin) pipelines to GitHub Actions), as it changes dronegen, but the pipelines themselves are not being backported to `branch/v10`.
This was referenced Apr 19, 2023
camscale
added a commit
that referenced
this pull request
Apr 20, 2023
* [v10] Make GHA pipeline structure more generic Make the GHA pipeline structure generic instead of assuming it only calls the linux-arm64 workflows. This backports part of dff5cd4 from "Integrates distroless OCI publishing into drone (#22707)", but does not backport the distroless OCI pipelines themselves. * [v10] Removes unnecessary workflow arguments for GHA pipelines Removes unnecessary workflow arguments from the Drone workflow invocation. Refactors some of the `dronegen` generators to make this easier to express insife `dronegen` This backports "Fix OCI promotion (#22867)" without the changes to the OCI pipelines as they have not been backported to v11. * [v10] Increase GHA build timeout Our ARM64 build fails quite frequently due to 30 minutes timeout. This PR increases the build timeout to one hour. * [v10] dronegen: Sort workflow inputs for stable output Sort the GitHub Actions inputs when generating the `gh-trigger-workflow` command line so that it does not randomly change order, as happens when iterating a map directly. This is a backport of a single commit of #24102 (drone: Switch Mac (darwin) pipelines to GitHub Actions), as it changes dronegen, but the pipelines themselves are not being backported to `branch/v10`. --------- Co-authored-by: Trent Clarke <trent@goteleport.com> Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Switch the pipelines for the darwin push and tag builds to call GitHub
Actions workflows, similar to what has been done for the Linux
pipelines.
The pipeline
build-darwin-amd64now incorporates what was done inthe
build-darwin-amd64-pkg,build-darwin-amd64-pkg-tshandbuild-darwin-amd64-connectpipelines, so these pipelines have beenremoved.
The pipeline
push-build-darwin-amd64also now calls GHA to perform thebuild.
dronegen has been changed so that GHA pipelines emit the workflow inputs
in sorted order rather than random map iteration order, to avoid spurious
changes to
.drone.ymlwhen a pipeline has multiple inputs..drone.ymlwas manually edited to remove the extra pipelines no longerneeded before running
dronegen.Issue: #20319