Skip to content

[CI] Fixed container images packaging #6972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pazone merged 24 commits into from
Mar 5, 2025
Merged

[CI] Fixed container images packaging #6972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
4aa8682
[CI] Fixed container images packaging
Feb 21, 2025
2e78431
Packaging Ubuntu matrix
Feb 21, 2025
fa0aa63
Improve matrix representation
Feb 24, 2025
cb26440
Removed matrix packaging if favour to unique step keys
Feb 24, 2025
97ca59d
Cleanup
Feb 24, 2025
4f97115
Added binfmt
Feb 24, 2025
08efd9f
Merge branch 'main' into ci_fix_packaging_containers
Feb 24, 2025
dcfd931
Pinned binfmt to v8.1.5
Feb 25, 2025
57591b2
Merge branch 'ci_fix_packaging_containers' of github.com:elastic/elas…
Feb 25, 2025
841ab70
Correct step names
Feb 26, 2025
e8e4546
Changed packaging ARM machine type
Feb 26, 2025
0a73c85
pinned qemu-v9.2.0-51
Feb 26, 2025
cd3e102
Resolved conflicts
Feb 26, 2025
ea79860
Remove qemu
Feb 26, 2025
9cdbb8b
Resolved conflicts
Mar 3, 2025
1f76c24
Removed deb and rpm integration packaging
Mar 3, 2025
ccce73c
Reordered packaging steps
Mar 3, 2025
8e20566
Merge branch 'main' into ci_fix_packaging_containers
Mar 4, 2025
b4c15c4
[CI] Build FIPS ARM Linux on ARM machine. Added missing retries
Mar 4, 2025
4c6d45e
Resolved conflicts
Mar 4, 2025
a404b1d
Fixed FIPS
Mar 4, 2025
c84271d
[CI] Corrected integration tests packaging dependencies
Mar 4, 2025
cbc3489
[CI] single buildkite download invocation
Mar 4, 2025
7677e85
Added missing retries
Mar 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions .buildkite/bk.integration.pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ steps:
- integration-ess
steps:
- label: "Win2022:sudo:{{matrix}}"
depends_on:
- packaging-windows
Comment thread
dliappis marked this conversation as resolved.
command: |
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
.buildkite/scripts/integration-tests.ps1 {{matrix}} true
Expand All @@ -51,6 +53,8 @@ steps:
- install-uninstall

- label: "Win2022:non-sudo:{{matrix}}"
depends_on:
- packaging-windows
command: |
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
.buildkite/scripts/integration-tests.ps1 {{matrix}} false
Expand All @@ -68,6 +72,8 @@ steps:
- default

- label: "Win2025:sudo:{{matrix}}"
depends_on:
- packaging-windows
command: |
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
.buildkite/scripts/integration-tests.ps1 {{matrix}} true
Expand All @@ -78,6 +84,9 @@ steps:
provider: "gcp"
machineType: "n1-standard-8"
Comment thread
dliappis marked this conversation as resolved.
image: "family/platform-ingest-elastic-agent-windows-2025"
retry:
automatic:
limit: 1
matrix:
- default
- fleet
Expand All @@ -89,12 +98,17 @@ steps:
- install-uninstall

- label: "Win2025:non-sudo:{{matrix}}"
depends_on:
- packaging-windows
command: |
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
.buildkite/scripts/integration-tests.ps1 {{matrix}} false
artifact_paths:
- build/**
- build/diagnostics/**
retry:
automatic:
limit: 1
agents:
provider: "gcp"
machineType: "n1-standard-8"
Expand All @@ -108,7 +122,7 @@ steps:
- integration-ess
steps:
- label: "x86_64:non-sudo: {{matrix}}"
# only packaging-ubuntu-x86-64 artifact dependency is required
depends_on: packaging-ubuntu-x86-64
Comment thread
dliappis marked this conversation as resolved.
command: |
buildkite-agent artifact download build/distributions/** . --step 'packaging-ubuntu-x86-64'
.buildkite/scripts/steps/integration_tests_tf.sh {{matrix}} false
Expand All @@ -126,9 +140,11 @@ steps:
- default

- label: "x86_64:sudo: {{matrix}}"
depends_on:
- packaging-ubuntu-x86-64
# due to deb group present in matrix tar.gz and deb packages artifacts are required
command: |
buildkite-agent artifact download build/distributions/** .
buildkite-agent artifact download build/distributions/** . --step packaging-ubuntu-x86-64
.buildkite/scripts/steps/integration_tests_tf.sh {{matrix}} true
artifact_paths:
- build/**
Expand Down Expand Up @@ -157,6 +173,8 @@ steps:
- container

- label: "arm:sudo: {{matrix}}"
depends_on:
- packaging-ubuntu-arm64
command: |
buildkite-agent artifact download build/distributions/** . --step 'packaging-ubuntu-arm64'
.buildkite/scripts/steps/integration_tests_tf.sh {{matrix}} true
Expand Down Expand Up @@ -189,6 +207,8 @@ steps:

- label: "arm:non-sudo: {{matrix}}"
skip: true
depends_on:
- packaging-ubuntu-arm64
command: |
buildkite-agent artifact download build/distributions/** . --step 'packaging-ubuntu-arm64'
.buildkite/scripts/steps/integration_tests_tf.sh {{matrix}} false
Expand All @@ -211,8 +231,10 @@ steps:
- integration-ess
steps:
- label: "x86_64:sudo:rpm"
depends_on:
- packaging-ubuntu-x86-64
command: |
buildkite-agent artifact download build/distributions/** .
buildkite-agent artifact download build/distributions/** . --step packaging-ubuntu-x86-64
.buildkite/scripts/steps/integration_tests_tf.sh rpm true
artifact_paths:
- build/**
Expand Down
75 changes: 44 additions & 31 deletions .buildkite/integration.pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,16 @@ steps:
- group: "Integration tests: packaging"
key: "int-packaging"
steps:
- label: "Packaging: Ubuntu x86_64"
key: "packaging-ubuntu-x86-64"
# Build matrix is not used for packaging in favor to unique step keys
# Packaging linux/amd64
- label: "Packaging: linux/amd64 rpm"
Comment thread
dliappis marked this conversation as resolved.
key: packaging-ubuntu-x86-64
env:
PACKAGES: "tar.gz"
PLATFORMS: "linux/amd64"
PACKAGES: "tar.gz,rpm,deb"
command: ".buildkite/scripts/steps/integration-package.sh"
artifact_paths:
- build/distributions/**
- build/distributions/**
retry:
automatic:
limit: 1
Expand All @@ -31,25 +33,30 @@ steps:
FIPS: "true"
command: ".buildkite/scripts/steps/integration-package.sh"
artifact_paths:
- build/distributions/**
- build/distributions/**
retry:
automatic:
limit: 1
agents:
provider: "gcp"
machineType: "n2-standard-4"

- label: "Packaging: Ubuntu arm64"
key: "packaging-ubuntu-arm64"
# Packaging linux/arm64
- label: "Packaging: linux/arm64 tar.gz"
key: packaging-ubuntu-arm64
env:
PACKAGES: "tar.gz"
PLATFORMS: "linux/arm64"
PACKAGES: "tar.gz"
command: ".buildkite/scripts/steps/integration-package.sh"
artifact_paths:
- build/distributions/**
retry:
automatic:
limit: 1
agents:
provider: "gcp"
machineType: "n2-standard-8"
provider: "aws"
instanceType: "c6g.4xlarge"
imagePrefix: "core-ubuntu-2204-aarch64"

- label: "Packaging: Ubuntu arm64 FIPS"
key: "packaging-ubuntu-arm64-fips"
Expand All @@ -60,12 +67,16 @@ steps:
command: ".buildkite/scripts/steps/integration-package.sh"
artifact_paths:
- build/distributions/**
retry:
automatic:
limit: 1
agents:
provider: "gcp"
machineType: "n2-standard-4"
provider: "aws"
instanceType: "c6g.4xlarge"
imagePrefix: "core-ubuntu-2204-aarch64"

- label: "Packaging: Windows"
key: "packaging-windows"
- label: "Packaging: windows/amd64 zip"
key: packaging-windows
env:
PACKAGES: "zip"
PLATFORMS: "windows/amd64"
Expand All @@ -79,28 +90,31 @@ steps:
provider: "gcp"
machineType: "n2-standard-8"

- label: "Packaging: Containers {{matrix.ext}} {{matrix.arch}}"
key: "packaging-containers"
- label: "Packaging: Containers linux/amd64"
key: packaging-containers-x86-64
env:
PACKAGES: "{{matrix.ext}}"
PLATFORMS: "{{matrix.arch}}"
PACKAGES: "docker"
PLATFORMS: "linux/amd64"
command: ".buildkite/scripts/steps/integration-package.sh"
artifact_paths:
- build/distributions/**
retry:
automatic:
limit: 1
agents:
provider: "gcp"
machineType: "n2-standard-8"
matrix:
setup:
arch:
- linux/amd64
- linux/arm64
ext:
- rpm
- deb

- label: "Packaging: Containers linux/arm64"
key: packaging-containers-arm64
env:
PACKAGES: "docker"
PLATFORMS: "linux/arm64"
command: |
.buildkite/scripts/steps/integration-package.sh
artifact_paths:
- build/distributions/**
agents:
provider: "aws"
instanceType: "c6g.4xlarge"
imagePrefix: "core-ubuntu-2204-aarch64"

- label: "Serverless integration test"
key: "serverless-integration-tests"
Expand Down Expand Up @@ -157,9 +171,8 @@ steps:
command: "buildkite-agent pipeline upload .buildkite/bk.integration.pipeline.yml"

- label: "Serverless Beats Tests"
# To speedup the build process only packaging-ubuntu-x86-64 artifact dependency is required
depends_on:
- packaging-ubuntu-x86-64
- int-packaging
key: "serverless-beats-integration-tests"
concurrency_group: elastic-agent-extended-testing/beats-integration
concurrency: 8
Expand Down
2 changes: 1 addition & 1 deletion dev-tools/mage/crossbuild.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ func CrossBuildImage(platform string) (string, error) {
case platform == "darwin/arm64" || platform == "darwin/universal":
tagSuffix = "darwin-arm64-debian11"
case platform == "linux/arm64":
tagSuffix = "arm"
tagSuffix = "base-arm-debian9"
case platform == "linux/armv5" || platform == "linux/armv6":
tagSuffix = "armel"
case platform == "linux/armv7":
Expand Down