Build multi-arch images if PLATFORMS env var is set #143
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -49,9 +49,55 @@ jobs: | |
| run: | | ||
| if [ "$DO_RELEASE" != "true" ]; then | ||
| echo "Skipping the release step because not starting from a snapshot version" | ||
| exit 0 | ||
| fi | ||
| single_arch=( | ||
|
Member
There was a problem hiding this comment. Now is there some way to make sure that all images get released. Looks like now anything no in either list gets skipped silently. Is it not reasonable to assume anything not in multi_arch list is single-arch?
Member
Author
There was a problem hiding this comment. I'll try to figure out how to get a list of all available images and add some validation here to throw an error if there are any not mentioned in any of these two lists. I wouldn't make any assumptions about them being single or multi-arch. Also see this old comment: #143 (comment)
Member
There was a problem hiding this comment. One idea I had was to place a marker file in each directory for the multi-arch supported ones and both ci.yml and release.yml can use that to decide what needs to be done. (Very hacky and I feel dirty for even suggesting it but I don't see better ways yet to be honest).
Member
Author
There was a problem hiding this comment. Done! While testing this, I noticed we were missing the recently added |
||
| testing/accumulo | ||
| testing/cdh5.12-hive | ||
| testing/cdh5.12-hive-kerberized | ||
| testing/cdh5.15-hive | ||
| testing/cdh5.15-hive-kerberized | ||
| testing/cdh5.15-hive-kerberized-kms | ||
| testing/gpdb-6 | ||
| testing/hdp2.6-hive | ||
| testing/hdp2.6-hive-kerberized | ||
| testing/hdp2.6-hive-kerberized-2 | ||
| testing/hdp3.1-hive | ||
| testing/hdp3.1-hive-kerberized | ||
| ) | ||
| multi_arch=( | ||
| testing/centos7-oj11 | ||
| testing/centos7-oj17 | ||
| testing/centos7-oj17-openldap | ||
| testing/centos7-oj17-openldap-referrals | ||
| testing/dns | ||
| testing/hive3.1-hive | ||
| testing/kerberos | ||
| testing/phoenix5 | ||
| testing/spark3-delta | ||
| testing/spark3-iceberg | ||
| testing/spark3-hudi | ||
| ) | ||
| to_release=("${single_arch[@]}" "${multi_arch[@]}") | ||
| make meta | ||
| read -a images < <(make list) | ||
| export LC_ALL=C | ||
| mapfile -t ignored < <( \ | ||
| comm -23 \ | ||
| <(printf '%s\n' "${images[@]}" | sort) \ | ||
| <(printf '%s\n' "${to_release[@]}" | sort) \ | ||
| ) | ||
| if [ "${#ignored[@]}" -ne 0 ]; then | ||
| echo "Images that would not get released: ${ignored[*]}" | ||
| echo "Must be explicitly added to either single_arch or multi_arch list" | ||
| exit 2 | ||
| fi | ||
| make prepare-release | ||
|
Member
There was a problem hiding this comment. " Extract prepare-release target from release " - should this come before others (or maybe even together with the change which allows releasing multi-arch images)? Also some context is useful.
Member
Author
There was a problem hiding this comment. If I'd put them into a single commit, you'd ask me to extract it :-) I moved it so it's before the commit that adds multi-arch releases, but now it adds a target that's unused until the next commit.
Member
Author
There was a problem hiding this comment. Can you suggest something for more context?
Member
There was a problem hiding this comment.
Sorry I has misordered commits while reviewing. It's all good here.
Member
There was a problem hiding this comment. For context I'd probably say: With multi-arch releases images for all platforms must be built before they can be published - which requires decoupling prepare-release from release goal.
Member
Author
There was a problem hiding this comment.
But I moved it, so are you sure? |
||
| make "${single_arch[@]/%/@$VERSION}" | ||
| make "${multi_arch[@]/%/@$VERSION}" PLATFORM="linux/amd64,linux/arm64" | ||
| remote=("${to_release[@]/#/ghcr.io/trinodb/}") | ||
| remote=("${remote[@]/%/:$VERSION}") | ||
| ./bin/push.sh "${remote[@]}" | ||
|
|
||
| - name: Set next development version | ||
| run: | | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -xeuo pipefail | ||
|
|
||
| usage() { | ||
| echo "$0 {image} [args]" >&2 | ||
| } | ||
|
|
||
| if [ $# -lt 1 ]; then | ||
| usage | ||
| exit 1 | ||
| fi | ||
|
|
||
| image=$1 | ||
| shift | ||
|
|
||
| if [ -z "${PLATFORMS:-}" ]; then | ||
| docker buildx build \ | ||
| --compress \ | ||
| --progress=plain \ | ||
| --add-host hadoop-master:127.0.0.2 \ | ||
| -t "$image" \ | ||
| --load \ | ||
| "$@" \ | ||
| . | ||
| exit 0 | ||
| fi | ||
|
|
||
| IFS=, read -ra platforms <<<"$PLATFORMS" | ||
| export ARCH | ||
| for platform in "${platforms[@]}"; do | ||
| IFS=: read -r name tag <<<"$image" | ||
| ARCH="-${platform//\//-}" | ||
| docker buildx build \ | ||
| --platform "$platform" \ | ||
| --compress \ | ||
| --progress=plain \ | ||
| --add-host hadoop-master:127.0.0.2 \ | ||
| -t "${name}:${tag}${ARCH}" \ | ||
| --load \ | ||
| "$@" \ | ||
| . | ||
| done |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -xeuo pipefail | ||
|
|
||
| function expand_multiarch_tags() { | ||
| local platforms | ||
| local name | ||
| local tag=$1 | ||
| shift | ||
|
|
||
| if [ -z "${PLATFORMS:-}" ]; then | ||
| echo "$tag" | ||
| return | ||
| fi | ||
|
|
||
| IFS=, read -ra platforms <<<"$PLATFORMS" | ||
| IFS=: read -r name tag <<<"$tag" | ||
|
|
||
| for platform in "${platforms[@]}"; do | ||
| echo "${name}:${tag}-${platform//\//-}" | ||
| done | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -xeuo pipefail | ||
|
|
||
| usage() { | ||
| echo "$0 {src} {dst}" >&2 | ||
| } | ||
|
|
||
| if [ $# -lt 2 ]; then | ||
| usage | ||
| exit 1 | ||
| fi | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)" | ||
| # shellcheck source=bin/lib.sh | ||
| source "$SCRIPT_DIR/lib.sh" | ||
|
|
||
| mapfile -t src_tags < <(expand_multiarch_tags "$1") | ||
| mapfile -t dst_tags < <(expand_multiarch_tags "$2") | ||
|
|
||
| for i in "${!src_tags[@]}"; do | ||
| src=${src_tags[$i]} | ||
| dst=${dst_tags[$i]} | ||
| docker tag "$src" "$dst" | ||
| done |
Uh oh!
There was an error while loading. Please reload this page.