Skip to content

build: Parallelize the CI image builds#25564

Closed
sebastianliebscher wants to merge 6 commits intoapache:masterfrom
sebastianliebscher:parallelize_image_builds
Closed

build: Parallelize the CI image builds#25564
sebastianliebscher wants to merge 6 commits intoapache:masterfrom
sebastianliebscher:parallelize_image_builds

Conversation

@sebastianliebscher
Copy link
Copy Markdown
Contributor

@sebastianliebscher sebastianliebscher commented Oct 8, 2023

SUMMARY

This PR replaces the custom script docker_build_push.sh that runs the docker builds sequentially with GitHub action jobs running in parallel.

  • set tags with docker/metadata-action GitHub Action
  • build images with docker/build-push-action GitHub Action
  • on release tag docker image with docker/metadata-action tags: type=pep440,pattern={{version}}

No functionality changed, except:

  • removes docker buildx local cache usage as every build runs on its own job hence on different machines where local cache does not apply
  • docker buildx cache will be a follow-up PR to use GitHub's container registry to speed up builds

TESTING INSTRUCTIONS

Use case: PR from fork (can be verified by the successful jobs from this PR)

  • detect that no secrets are present
  • images build without failures
  • the subsequent steps to push the images are skipped as intended (PR from fork)
  • tags: (no tags)

Use case: PR from Superset repo

Use case: commit to master

Use case: release


ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

- replaces custom script to set tags with docker/metadata-action GitHub Action
- replaces custom script to sequentially build images with docker/build-push-action GitHub Action
- moves docker-release.yml logic into docker.yml by utilizing docker/metadata-action 'tags: type=pep440,pattern={{version}}'
- removes docker buildx local cache usage as every build runs on its own job hence on different machines (docker buildx registry cache will be a follow-up PR)
else
echo "has-secrets=0" >> "$GITHUB_OUTPUT"
echo "no secrets!"
fi
Copy link
Copy Markdown
Contributor Author

@sebastianliebscher sebastianliebscher Oct 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not work as expected. The check if: needs.config.outputs.has-secrets returns true in both cases.

Either check for if: needs.config.outputs.has-secrets == 1 or don't set has-secrets when there are no secrets. I went with the latter.

@sebastianliebscher sebastianliebscher changed the title [WIP] build: Parallelize the CI image builds build: Parallelize the CI image builds Oct 8, 2023
@sebastianliebscher sebastianliebscher marked this pull request as ready for review October 8, 2023 13:01
@craig-rueda
Copy link
Copy Markdown
Member

Any updates here? Looks like there's conflicts :)

@mistercrunch
Copy link
Copy Markdown
Member

I pull this, rebased and reopened here #26697

@rusackas
Copy link
Copy Markdown
Member

Closing this since #26698 merged

@rusackas rusackas closed this Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants