Skip to content

Setup binary release CI pipeline#1085

Merged
mergify[bot] merged 11 commits intomasterfrom
bowd/release-pipeline
Jul 2, 2020
Merged

Setup binary release CI pipeline#1085
mergify[bot] merged 11 commits intomasterfrom
bowd/release-pipeline

Conversation

@bowd
Copy link
Copy Markdown
Contributor

@bowd bowd commented Jun 29, 2020

Description

This PR includes the necessary bits to setup a CI pipeline that cross-compiles all geth tools into release binaries for the platforms we want/support. Currently these are:

Elements of the PR:

  • Dockerfile.binaries for the container where cross-compilation occurs. It's based on a fork of xgo, and includes some back-ported mingw packages for the windows build.

  • Changes to the geth build scripts:

    • Introduce a new CI environment in internal/build/env.go which which can be used for Google Cloud Build.
    • Create a new ci task in build/ci.go that bundles the binaries into well named release archive
  • cloudbuild-binaries.yaml which defines the CI pipeline. It should be used in conjunction with a trigger on release/.* and master branches and the trigger should have to variables:

    • _BUCKET with the name of the google cloud storage bucket where the artefacts will be stored
    • _BUILD_TARGETS comma-separated string of build targets. e.g. linux/amd64,linux/386

Shortcomings

  • 🔴 The geth build.Environment struct requires the commit timestamp, which is then passed to main.gitDate in the binaries that it builds. Usually this is extracted from the git but in Cloud Build in the CI steps the git data is stripped and all we have is what's passed through env vars. I have substituted the commit timestamp with the build timestamp instead.
  • 🟡 The xgo image is really large (3gb) and adds about ~4minutes to the build time

Release artefacts

The pipeline outputs release artefacts into <bucket>/<branch>/<file>. For example here's the output of a test build:

Screenshot 2020-06-29 at 11 41 00

A few things to notice here:

  • The folder is <bucket>/release/1.1 yet the version is 1.0.0-unstable this is because there isn't any enforced relationship between the branch name and the version sourced from params/version.go which is the authoritative source. So in this case I just created the dummy release/1.1 branch on my fork in order to play around. Because we're using the branch name in the path all "nightly" version will be stored in the <bucket>/master folder, if we setup the trigger on master.

  • There are 2 archives per platform, one containing only geth and the other containing all binary tools located inside cmd, just like geth releases. ⚠️ The archives also contain the "COPYING" file, we need to decide if we want to make changes to that ⚠️

Post-merge TODOs:

  • Create a bucket and triggers in celo-testnet to start running the pipeline
  • Enable more build targets as they are unblocked

Tested

I have currently tested the CI pipeline in a personal google cloud project and tested the linux binaries in docker.

Related issues

Backwards compatibility

Changes are only related to tooling so no problem with compatibility.

@bowd bowd requested review from asaj and timmoreton as code owners June 29, 2020 09:45
@bowd bowd requested a review from nategraf June 29, 2020 09:47
@bowd bowd force-pushed the bowd/release-pipeline branch from 4e266b2 to bd731c0 Compare June 30, 2020 06:47
@bowd bowd mentioned this pull request Jun 30, 2020
bowd added a commit that referenced this pull request Jul 1, 2020
Required by #1085 to enable compiling celo-blockchain for windows
Copy link
Copy Markdown

@nategraf nategraf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just a few out-of-date comments

Comment thread Dockerfile.binaries
Comment thread Dockerfile.binaries
Comment thread Dockerfile.binaries
@bowd bowd force-pushed the bowd/release-pipeline branch from c930f60 to 69b1e0b Compare July 2, 2020 09:42
@bowd bowd force-pushed the bowd/release-pipeline branch from 69b1e0b to a4142e6 Compare July 2, 2020 13:31
@bowd bowd added the automerge label Jul 2, 2020
@mergify mergify Bot merged commit 3f9380c into master Jul 2, 2020
@mergify mergify Bot deleted the bowd/release-pipeline branch July 2, 2020 14:02
@mcortesi mcortesi added this to the 1.0.2 milestone Sep 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Setup pipeline that builds release artefacts

3 participants