release: Prepare for MacOS builds on GitHub Actions#23407
Merged
Conversation
Move the variables for Mac signing from the `build-common.sh` shell script to the `Makefile`. These vars will need to be passed to other build processes to parameterize the signing for different GitHub Actions build environments. The switch on `ENVIRONMENT_NAME` allows different secrets to be available in GitHub Actions for production (promote) vs developer (build) builds. The default environment name is `promote` so as to be compatible with the existing Drone setup, which does not define `ENVIRONMENT_NAME`.
Remove the hard-coded MacOS signing key IDs from the Makefile and find them dynamically based on the name of the key. This allows GitHub Actions to be set up with new keys different to the ones on the Drone builders. As long as we keep the same name on the keys, we can rotate the keys without needing to update the IDs in the Makefile. This requires us to be more judicious about exporting the variables as exporting them causes them to be evaluated. We do not want to evaluate them on non-darwin targets, and on darwin, we should only evaluate it if needed for a recipe. So use a dynamic `eval` in the recipes that need the environment variables.
Override the hard-coded values in `notarize-apple-binaries` and pass the values we get based on the GitHub Actions environment. This allows us to sign and notarize software in a development branch more easily when working on the signing and notarizing process. This will not happen automatically, but it is expected that a developer can manually trigger a workflow to perform building, signing and notarizing from a dev branch where the workflow has temporarily changed the environment to `build`. A similar change to the `Makefile` in the teleport.e repository goes with this change. This adds a new bundle ID of `com.goteleport.dev` for the dev build of Teleport. This follows the same pattern as used for the dev build of the `tsh` binary and the current production bundle ID for Teleport. Previously there was no dev signing/notarizing process for the set of Teleport binaries.
Add a script for setting up the MacOS keychain for signing applications and packages. It encapsulates the `security` commands to add either or both application keys and installer keys. The keys can be either base64-encoded in environment variables, or `.p12` files on disk, making it useful for local development.
camscale
requested review from
fheinecke,
r0mant,
tcsc and
wadells
and removed request for
greedy52 and
probakowski
Contributor
Author
|
@codingllama Maybe RTYI, as it reworks the key and other parameter handling used in the mac build scripts you wrote. Would be happy to have your feedback too. |
wadells
reviewed
Mar 22, 2023
Contributor
wadells
left a comment
wadells
left a comment
There was a problem hiding this comment.
Needs the correct cert info for promote/release. Otherwise, it looks reasonable to me.
Put the MacOS signing variables into a separate `.mk` file and include it from the main `Makefile`. Add more comments to document the purpose of the vars and where some of the values come from.
camscale
force-pushed
the
camh/prepare-for-gha-mac
branch
from
f964764 to
a7778d8
Compare
Explain that the purpose of the script is to be run on CI, but can also be run manually. Add the default values used to the usage message for the keychain and password.
codingllama
reviewed
Mar 23, 2023
Contributor
codingllama
left a comment
codingllama
left a comment
There was a problem hiding this comment.
Thanks for the heads up, Cam.
Did a quick pass and let a few comments, not a full review by any means. (Leaving the full review for assigned folks.)
Developers can manually export ENVIRONMENT_NAME=build to use the development keys locally
Nice. Is that somewhere easy to find? Would the scripts tell me that on error?
Contributor
Author
The script does not rely or use ENVIRONMENT_NAME itself. I have put some docs in the |
* Change shebang to /bin/bash * Use heredoc instead of multiple printfs for usage message * Move `local` declaration next to setting of kpath var
Contributor
Author
r0mant
approved these changes
Mar 24, 2023
The sub-make for enterprise needs this to be set or it cannot sign the enterprise binaries. Export it if we are doing signing/notarizing.
wadells
approved these changes
Mar 24, 2023
Comment on lines
+61
to
+62
| DEVELOPER_KEY_NAME_build = Developer ID Application: Ada Lin | ||
| INSTALLER_KEY_NAME_build = Developer ID Installer: Ada Lin |
Contributor
There was a problem hiding this comment.
cc @adaadb6 This developer account is getting codified in our release pipeline. It is only used for signing binaries that are pending release. When they're ready to be promoted the signatures are stripped, and it is re-signed with the Gravitational Inc certificates.
Contributor
Author
There was a problem hiding this comment.
@wadells I'm not sure what the release process is meant to be - I was not planning on stripping any signatures and re-signing. The builds from master and branch/* would be built from source and signed with the promote keys. This is how the pipelines currently work. Are we changing the release process around this?
camscale
added a commit
that referenced
this pull request
Apr 12, 2023
* release: Move Mac signing vars from script to Makefile Move the variables for Mac signing from the `build-common.sh` shell script to the `Makefile`. These vars will need to be passed to other build processes to parameterize the signing for different GitHub Actions build environments. The switch on `ENVIRONMENT_NAME` allows different secrets to be available in GitHub Actions for production (promote) vs developer (build) builds. The default environment name is `promote` so as to be compatible with the existing Drone setup, which does not define `ENVIRONMENT_NAME`. * release: Determine Mac signing key IDs automatically Remove the hard-coded MacOS signing key IDs from the Makefile and find them dynamically based on the name of the key. This allows GitHub Actions to be set up with new keys different to the ones on the Drone builders. As long as we keep the same name on the keys, we can rotate the keys without needing to update the IDs in the Makefile. This requires us to be more judicious about exporting the variables as exporting them causes them to be evaluated. We do not want to evaluate them on non-darwin targets, and on darwin, we should only evaluate it if needed for a recipe. So use a dynamic `eval` in the recipes that need the environment variables. * release: Pass key & team ID to notarize tool Override the hard-coded values in `notarize-apple-binaries` and pass the values we get based on the GitHub Actions environment. This allows us to sign and notarize software in a development branch more easily when working on the signing and notarizing process. This will not happen automatically, but it is expected that a developer can manually trigger a workflow to perform building, signing and notarizing from a dev branch where the workflow has temporarily changed the environment to `build`. A similar change to the `Makefile` in the teleport.e repository goes with this change. This adds a new bundle ID of `com.goteleport.dev` for the dev build of Teleport. This follows the same pattern as used for the dev build of the `tsh` binary and the current production bundle ID for Teleport. Previously there was no dev signing/notarizing process for the set of Teleport binaries. * release: Add script to setup the MacOS keychain for signing Add a script for setting up the MacOS keychain for signing applications and packages. It encapsulates the `security` commands to add either or both application keys and installer keys. The keys can be either base64-encoded in environment variables, or `.p12` files on disk, making it useful for local development. * release: Split MacOS signing vars into separate mk file Put the MacOS signing variables into a separate `.mk` file and include it from the main `Makefile`. Add more comments to document the purpose of the vars and where some of the values come from. * release: Add some more comments to keychain-setup.sh Explain that the purpose of the script is to be run on CI, but can also be run manually. Add the default values used to the usage message for the keychain and password. * Address PR comments on keychain-setup.sh script * Change shebang to /bin/bash * Use heredoc instead of multiple printfs for usage message * Move `local` declaration next to setting of kpath var * release: Export DEVELOPER_ID_APPLICATION in release-darwin The sub-make for enterprise needs this to be set or it cannot sign the enterprise binaries. Export it if we are doing signing/notarizing. Backport: #23407
camscale
added a commit
that referenced
this pull request
Apr 19, 2023
* release: Move Mac signing vars from script to Makefile Move the variables for Mac signing from the `build-common.sh` shell script to the `Makefile`. These vars will need to be passed to other build processes to parameterize the signing for different GitHub Actions build environments. The switch on `ENVIRONMENT_NAME` allows different secrets to be available in GitHub Actions for production (promote) vs developer (build) builds. The default environment name is `promote` so as to be compatible with the existing Drone setup, which does not define `ENVIRONMENT_NAME`. * release: Determine Mac signing key IDs automatically Remove the hard-coded MacOS signing key IDs from the Makefile and find them dynamically based on the name of the key. This allows GitHub Actions to be set up with new keys different to the ones on the Drone builders. As long as we keep the same name on the keys, we can rotate the keys without needing to update the IDs in the Makefile. This requires us to be more judicious about exporting the variables as exporting them causes them to be evaluated. We do not want to evaluate them on non-darwin targets, and on darwin, we should only evaluate it if needed for a recipe. So use a dynamic `eval` in the recipes that need the environment variables. * release: Pass key & team ID to notarize tool Override the hard-coded values in `notarize-apple-binaries` and pass the values we get based on the GitHub Actions environment. This allows us to sign and notarize software in a development branch more easily when working on the signing and notarizing process. This will not happen automatically, but it is expected that a developer can manually trigger a workflow to perform building, signing and notarizing from a dev branch where the workflow has temporarily changed the environment to `build`. A similar change to the `Makefile` in the teleport.e repository goes with this change. This adds a new bundle ID of `com.goteleport.dev` for the dev build of Teleport. This follows the same pattern as used for the dev build of the `tsh` binary and the current production bundle ID for Teleport. Previously there was no dev signing/notarizing process for the set of Teleport binaries. * release: Add script to setup the MacOS keychain for signing Add a script for setting up the MacOS keychain for signing applications and packages. It encapsulates the `security` commands to add either or both application keys and installer keys. The keys can be either base64-encoded in environment variables, or `.p12` files on disk, making it useful for local development. * release: Split MacOS signing vars into separate mk file Put the MacOS signing variables into a separate `.mk` file and include it from the main `Makefile`. Add more comments to document the purpose of the vars and where some of the values come from. * release: Add some more comments to keychain-setup.sh Explain that the purpose of the script is to be run on CI, but can also be run manually. Add the default values used to the usage message for the keychain and password. * Address PR comments on keychain-setup.sh script * Change shebang to /bin/bash * Use heredoc instead of multiple printfs for usage message * Move `local` declaration next to setting of kpath var * release: Export DEVELOPER_ID_APPLICATION in release-darwin The sub-make for enterprise needs this to be set or it cannot sign the enterprise binaries. Export it if we are doing signing/notarizing. Backport: #23407
camscale
added a commit
that referenced
this pull request
Apr 19, 2023
* notarize: Accept parameters for dev and bundle ID Add a couple of parameters for the developer key ID and bundle ID for signing/notarizing binaries. Keep the hard-coded values as defaults for now, but we will remove these soon when all the call sites of the tool have been updated to pass these values. We want to parameterize these values so we can use different signing keys in GitHub Actions and to make the tool agnostic to which binaries it is signing. Backport: #23092 * release: Prepare for MacOS builds on GitHub Actions * release: Move Mac signing vars from script to Makefile Move the variables for Mac signing from the `build-common.sh` shell script to the `Makefile`. These vars will need to be passed to other build processes to parameterize the signing for different GitHub Actions build environments. The switch on `ENVIRONMENT_NAME` allows different secrets to be available in GitHub Actions for production (promote) vs developer (build) builds. The default environment name is `promote` so as to be compatible with the existing Drone setup, which does not define `ENVIRONMENT_NAME`. * release: Determine Mac signing key IDs automatically Remove the hard-coded MacOS signing key IDs from the Makefile and find them dynamically based on the name of the key. This allows GitHub Actions to be set up with new keys different to the ones on the Drone builders. As long as we keep the same name on the keys, we can rotate the keys without needing to update the IDs in the Makefile. This requires us to be more judicious about exporting the variables as exporting them causes them to be evaluated. We do not want to evaluate them on non-darwin targets, and on darwin, we should only evaluate it if needed for a recipe. So use a dynamic `eval` in the recipes that need the environment variables. * release: Pass key & team ID to notarize tool Override the hard-coded values in `notarize-apple-binaries` and pass the values we get based on the GitHub Actions environment. This allows us to sign and notarize software in a development branch more easily when working on the signing and notarizing process. This will not happen automatically, but it is expected that a developer can manually trigger a workflow to perform building, signing and notarizing from a dev branch where the workflow has temporarily changed the environment to `build`. A similar change to the `Makefile` in the teleport.e repository goes with this change. This adds a new bundle ID of `com.goteleport.dev` for the dev build of Teleport. This follows the same pattern as used for the dev build of the `tsh` binary and the current production bundle ID for Teleport. Previously there was no dev signing/notarizing process for the set of Teleport binaries. * release: Add script to setup the MacOS keychain for signing Add a script for setting up the MacOS keychain for signing applications and packages. It encapsulates the `security` commands to add either or both application keys and installer keys. The keys can be either base64-encoded in environment variables, or `.p12` files on disk, making it useful for local development. * release: Split MacOS signing vars into separate mk file Put the MacOS signing variables into a separate `.mk` file and include it from the main `Makefile`. Add more comments to document the purpose of the vars and where some of the values come from. * release: Add some more comments to keychain-setup.sh Explain that the purpose of the script is to be run on CI, but can also be run manually. Add the default values used to the usage message for the keychain and password. * Address PR comments on keychain-setup.sh script * Change shebang to /bin/bash * Use heredoc instead of multiple printfs for usage message * Move `local` declaration next to setting of kpath var * release: Export DEVELOPER_ID_APPLICATION in release-darwin The sub-make for enterprise needs this to be set or it cannot sign the enterprise binaries. Export it if we are doing signing/notarizing. Backport: #23407 * release: Dont look for signing keys if not needed Use a `make` conditional instead of a shell conditional when deciding whether to run notarization or not, based on APPLE_{USERNAME,PASSWORD} being set. When done as a shell conditional, $(DEVELOPER_ID_APPLICATION) was still being evaluated, and that causes a key error if the key does not exist in the keychain. If the `APPLE_{USERNAME,PASSWORD}` env vars are not set, it should not matter if the key is present or not as it will not be used. By switching to `$(if ...)` in make, the recipe will not be evaluated at all if the condition is false. To ensure the logs say what is going on, add a message when we do not notarize the binaries. Move the logic for notarization and checking the username/password into `darwin-signing.mk` with the other signing/notarization variables. This will make it reusable by the enterprise Makefile later. Backport: #23636 * release: Add build-connect target to Makefile * release: Add build-connect target to Makefile Add a `build-connect` target to the `Makefile` to build Teleport Connect via yarn on MacOS. Linux uses the `build.assets/Makefile` `teleterm` target, and Windows build pipelines do not use `make`. Add the `CSC_NAME` make variable containing the Developer Key ID to tell electron-builder which key to use to sign the package it produces. This gives us a little more control over how the Connect built and packaged and will simplify the CI scripts to have them just call make. It is also required in order to set the `CSC_NAME` environment name correctly as the developer ID is determined by the Makefile. * darwin: Fix echo command when not notarizing Remove the leading `@` from the echo command that is run when we are not notarizing binaries. It works fine from the OSS repo, but the enterprise repo needs to do `cd .. && $(NOTARIZE_COMMAND)` and the leading `@` causes the build error: `@echo: command not found`. We'll just have to live with a little extra noise amongst all the other noise. Backport: #23750 * release: Update build for product signing Update the build scripts to properly set up the key for signing packages using `productsign`, and parameterise the bundle ID for packages in the packaging scripts. Backport: #23820 * dronegen: Add workflow-tag arg to gh-trigger-workflow This was deliberately not backported in "[v12] Backport Distroless OCI builds (#22814)" (f1bfd5a) , but now we want it. * Increase GHA build timeout Our ARM64 build fails quite frequently due to 30 minutes timeout. This PR increases the build timeout to one hour. Backport: #22926 * drone: Switch Mac (darwin) pipelines to GitHub Actions * dronegen: Sort workflow inputs for stable output Sort the GitHub Actions inputs when generating the `gh-trigger-workflow` command line so that it does not randomly change order, as happens when iterating a map directly. * dronegen: Have darwin pipelines call out to GitHub Actions Update the darwin pipelines to run workflows on GitHub Actions instead of locally on drone builders. This replaces four pipelines with a single GitHub actions workflow as the one workflow builds the tarballs, Mac packages and Mac disk images. We continue to drive the push build from drone until we work out how secrets are safely managed in the Teleport OSS repo. * drone: Regenerate .drone.yml for Mac pipeline changes To regenerate the `.drone.yml` file, first three pipelines were manually removed: - build-darwin-amd64-pkg - build-darwin-amd64-pkg-tsh - build-darwin-amd64-connect Then make dronegen was run to update the pipelines: - push-build-darwin-amd64 - build-darwin-amd64 Backport: #24102 * Update `e` submodule for mac gha backport This commit will be updated after #1119 in teleport.e has been merged to point to the head of branch/v12. --------- Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update the
Makefileandbuild.assets/build-common.shto allowsigning keys to be parameterized for the signing and notarization
process, used for making releases for MacOS.
This moves some variables from the
build-common.shshell script to theMakefile, which then exports those variables in the recipies that run
the scripts.
The hard-coded key IDs have been replaced with querying the keyring
based on the name of the key. This allows us to use different keys in
Drone and GitHub Actions as requested by the security team. It also
makes rotating the keys easier as no repo changes are needed to use the
new keys.
The default IDs and other signing parameters are for the production
environment (call
promotein GitHub) as Drone has no concept of thisenvironment config. Developers can manually export
ENVIRONMENT_NAME=buildto use the development keys locally. Whendeveloping the GitHub Actions workflows in the enterprise repository,
changing the
environment: promotetoenvironment: buildwillautomatically pick up the correct environment name and use the dev keys.
Issue: #20319