[8.x](backport #7690) Fips packaging by mergify[bot] · Pull Request #7790 · elastic/elastic-agent

mergify · 2025-04-09T14:06:53Z

What does this PR do?

This PR backports FIPS packaging support for elastic-agent.
This change is the combination of 5 PRs:
#7602
#7690
#7795
#7805
#7818

There are 2 extra commits added on top of the 5 backported PRs that deal with cloud-defend packaging.

What follows is the description of PR #7690:

When specifying "FIPS=true" the agent will pull FIPS-enable components, compile/download a FIPS-enabled agent executable, package only specs that are marked with fips: true in dev-tools/packaging/packages.yml.

It is also possible to package FIPS-compliant elastic-agent-core artifact specifying FIPS=true

In order to download and use a different set of dependencies, components are now defined in dev-tools/packaging/packages.yml and referenced directly in the packaging specs.

Elastic Agent packaging code will collect all the enabled specs, compute the full set of components needed and pass that list to the download/unpack/flatten/compress steps.

The high-level packaging flow looks like this:

detect the packaging specs that are selected from dev-tools/packaging/packages.yml according to PLATFORMS, PACKAGES, FIPS filters
go through the components listed under each spec and deduplicate them
download the dependencies from CI, DROP_PATH (if specified) using the BinarySpec.GetPackageName() method to locate the correct file
flatten the dependencies in per-platform directories, extracting the dependencies' archive contents under BinarySpec.GetRootDir()
Copy the dependencies for each spec in a dedicated package directory
compile the agent binaries or download them from the elastic-agent-core project with a similar mechanism to the one used for the dependencies
create the package according to the specification, creating a package manifest that depends on the spec fips value.
run package tests (running additional tests for FIPS packages, checking that the elastic-agent and all components binaries have been built with the correct tags and libraries)

This PR introduces the new packages below:

elastic-agent-fips-9.1.0-SNAPSHOT-linux-*.tar.gz archive
elastic-agent-fips-9.1.0-SNAPSHOT-linux-*.docker.tar.gz / docker.elastic.co/elastic-agent/elastic-agent-fips:9.1.0-SNAPSHOT docker image
elastic-agent-fips-cloud-9.1.0-SNAPSHOT-linux-*.docker.tar.gz/docker.elastic.co/beats-ci/elastic-agent-fips-cloud:9.1.0-SNAPSHOT docker image
elastic-agent-core-fips-9.1.0-linux-*.tar.gz package

Why is it important?

To have FIPS-compliant Elastic Agent packages that contain only FIPS-compliant binaries.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
~~[ ] I have added tests that prove my fix is effective or that my feature works~~
~~[ ] I have added an entry in ./changelog/fragments using the changelog tool~~
~~[ ] I have added an integration test or an E2E test~~

Disruptive User Impact

No disruptive user impact as existing packages (and the command line to package elastic-agent) should be unchanged.

How to test this PR locally

Package a linux/amd64 elastic-agent tar.gz package using local elastic-agent code

Run:

FIPS=true SNAPSHOT=true EXTERNAL=true PLATFORMS="linux/amd64" PACKAGES="tar.gz"  mage clean package

Sample output:

➜  elastic-agent git:(fips-packaging) ✗ FIPS=true SNAPSHOT=true EXTERNAL=true PLATFORMS="linux/amd64" PACKAGES="tar.gz"  mage clean package
--- Package Elastic-Agent
Downloading from https://snapshots.elastic.co/9.1.0-de2e77fc/downloads/beats/agentbeat/agentbeat-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz
Downloading from https://snapshots.elastic.co/9.1.0-de2e77fc/downloads/fleet-server/fleet-server-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz
Done downloading agentbeat-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz into /home/paolo/dev/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-x86_64.tar.gz
Done downloading fleet-server-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz into /home/paolo/dev/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-x86_64.tar.gz
>> File /home/paolo/dev/elastic-agent/build/distributions/elastic-agent-drop/linux-x86_64.tar.gz/fleet-server.spec.yml does not exist, reverting to local specfile
>> Building internal/pkg/otel/README.md
>> BuildPGP from GPG-KEY-elasticsearch to internal/pkg/release/pgp.go
>> BuildFleetCfg _meta/elastic-agent.fleet.yml to internal/pkg/agent/application/configuration_embed.go
>> Building elastic-agent.yml for linux/amd64
>> Building elastic-agent.reference.yml for linux/amd64
>> Building elastic-agent.docker.yml for linux/amd64
>> check: Checking for invalid links in "internal/pkg/otel/README.md"
--- CrossBuildGoDaemon Elastic-Agent
--- CrossBuild Elastic-Agent
--- CrossBuild Elastic-Agent
>> golangCrossBuild: Building for linux/amd64
>> buildGoDaemon: Building for linux/amd64
>> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, GOTOOLCHAIN=local, PLATFORM_ID=linux-amd64]
>> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, GOTOOLCHAIN=local, PLATFORM_ID=linux-amd64]
/usr/bin/ld: /tmp/ccy8Jm4S.o: in function `main':
god.c:(.text+0x2cd): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: god.c:(.text+0x263): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
--- Package artifact
>> package: Building linux-amd64
>> package: Building elastic-agent-fips type=tar.gz for platform=linux/amd64 fips=true
--- TestPackages, the generated packages (i.e. file modes, owners, groups).
--- TestPackages
>> Testing package contents
package ran for 3m11.456684868s
➜

Package a linux/amd64 elastic-agent-core tar.gz package using local elastic-agent code

Run:

FIPS=true SNAPSHOT=true PLATFORMS="linux/amd64" mage clean packageAgentCore

Sample output:

➜  elastic-agent git:(fips-packaging) ✗ FIPS=true PLATFORMS="linux/amd64" mage clean packageAgentCore
--- CrossBuildGoDaemon Elastic-Agent
--- CrossBuild Elastic-Agent
--- CrossBuild Elastic-Agent
>> golangCrossBuild: Building for linux/amd64
>> buildGoDaemon: Building for linux/amd64
>> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, GOTOOLCHAIN=local, PLATFORM_ID=linux-amd64]
>> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, GOTOOLCHAIN=local, PLATFORM_ID=linux-amd64]
/usr/bin/ld: /tmp/ccn8qRRH.o: in function `main':
god.c:(.text+0x2cd): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: god.c:(.text+0x263): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
--- Package artifact
>> package: Building linux-amd64
>> package: Building elastic-agent-core-fips type=tar.gz for platform=linux/amd64 fips=true
packageAgentCore ran for 2m6.82510297s
➜  elastic-agent git:(fips-packaging) ✗ tree -L 1 build/distributions
build/distributions
├── elastic-agent-core-fips-9.1.0-linux-x86_64.tar.gz
└── elastic-agent-core-fips-9.1.0-linux-x86_64.tar.gz.sha512

0 directories, 2 files

Package a linux/amd64 elastic-agent-core tar.gz and docker images using local elastic-agent code

Run:

FIPS=true SNAPSHOT=true EXTERNAL=true PLATFORMS="linux/amd64" PACKAGES="tar.gz,docker" mage clean package

Sample output:

➜  elastic-agent git:(fips-packaging) ✗ FIPS=true SNAPSHOT=true EXTERNAL=true PLATFORMS="linux/amd64" PACKAGES="tar.gz,docker" mage clean package
--- Package Elastic-Agent
Downloading from https://snapshots.elastic.co/9.1.0-0a06c833/downloads/beats/agentbeat/agentbeat-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz
Downloading from https://snapshots.elastic.co/9.1.0-0a06c833/downloads/beats/agentbeat/agentbeat-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz
Downloading from https://snapshots.elastic.co/9.1.0-0a06c833/downloads/fleet-server/fleet-server-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz
Done downloading agentbeat-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz into /home/paolo/dev/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-x86_64.tar.gz
Done downloading fleet-server-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz into /home/paolo/dev/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-x86_64.tar.gz
Downloading from https://snapshots.elastic.co/9.1.0-0a06c833/downloads/fleet-server/fleet-server-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz
Done downloading fleet-server-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz into /home/paolo/dev/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-x86_64.tar.gz
Done downloading agentbeat-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz into /home/paolo/dev/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-x86_64.tar.gz
>> File /home/paolo/dev/elastic-agent/build/distributions/elastic-agent-drop/linux-x86_64.tar.gz/fleet-server.spec.yml does not exist, reverting to local specfile
>> Building internal/pkg/otel/README.md
>> BuildPGP from GPG-KEY-elasticsearch to internal/pkg/release/pgp.go
>> BuildFleetCfg _meta/elastic-agent.fleet.yml to internal/pkg/agent/application/configuration_embed.go
>> Building elastic-agent.yml for linux/amd64
>> Building elastic-agent.reference.yml for linux/amd64
>> Building elastic-agent.docker.yml for linux/amd64
>> check: Checking for invalid links in "internal/pkg/otel/README.md"
--- CrossBuildGoDaemon Elastic-Agent
--- CrossBuild Elastic-Agent
--- CrossBuild Elastic-Agent
>> buildGoDaemon: Building for linux/amd64
>> golangCrossBuild: Building for linux/amd64
>> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, GOTOOLCHAIN=local, PLATFORM_ID=linux-amd64]
>> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, GOTOOLCHAIN=local, PLATFORM_ID=linux-amd64]
/usr/bin/ld: /tmp/ccxMs6ZP.o: in function `main':
god.c:(.text+0x2cd): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: god.c:(.text+0x263): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
--- Package artifact
>> package: Building linux-amd64
>> package: Building elastic-agent-fips type=docker for platform=linux/amd64 fips=true
>> package: Building elastic-agent-fips type=docker for platform=linux/amd64 fips=true
>> package: Building elastic-agent-fips type=tar.gz for platform=linux/amd64 fips=true
[+] Building 23.7s (10/25)                                                                                                                                                                            docker:default
[+] Building 23.8s (16/22)                                                                                                                                                                            docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                            0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                            0.0s
 => => transferring dockerfile: 7.44kB                                                                                                                                                                          0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 5)                                                                                            0.0s
 => => transferring dockerfile: 6.59kB                                                                                                                                                                          0.0s
 => [internal] load metadata for docker.elastic.co/wolfi/chainguard-base-fips:latest                                                                                                                            1.6s
 => [internal] load metadata for cgr.dev/chainguard/wolfi-base:latest                                                                                                                                           0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 5)                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                 0.0s
 => [internal] load metadata for docker.elastic.co/wolfi/chainguard-base-fips:latest                                                                                                                            1.7s
 => [stage-1  1/14] FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:e436fc2621091a21f223c9cb6bc0156b04ed93284debaeaab2ac27d1ed53c387                                                            0.0s
 => [home 1/5] FROM cgr.dev/chainguard/wolfi-base:latest                                                                                                                                                        0.0s
 => [internal] load metadata for cgr.dev/chainguard/wolfi-base:latest                                                                                                                                           0.0s
[+] Building 24.0s (17/22)                                                                                                                                                                            docker:default
[+] Building 27.0s (22/23)                                                                                                                                                                            docker:default
[+] Building 27.2s (22/23)                                                                                                                                                                            docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                            0.0s
 => => transferring dockerfile: 6.59kB                                                                                                                                                                          0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 5)                                                                                            0.0s
[+] Building 27.5s (22/23)                                                                                                                                                                            docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                            0.0s
 => => transferring dockerfile: 6.59kB                                                                                                                                                                          0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 5)                                                                                            0.0s
 => [internal] load metadata for docker.elastic.co/wolfi/chainguard-base-fips:latest                                                                                                                            1.7s
 => [internal] load metadata for cgr.dev/chainguard/wolfi-base:latest                                                                                                                                           0.0s
 => [auth] wolfi/chainguard-base-fips:pull token for docker.elastic.co                                                                                                                                          0.0s
[+] Building 28.5s (28/30)                                                                                                                                                                            docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                            0.0s
[+] Building 30.5s (31/31) FINISHED                                                                                                                                                                   docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                            0.0s
 => => transferring dockerfile: 7.44kB                                                                                                                                                                          0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 5)                                                                                            0.0s
 => [internal] load metadata for docker.elastic.co/wolfi/chainguard-base-fips:latest                                                                                                                            1.6s
 => [internal] load metadata for cgr.dev/chainguard/wolfi-base:latest                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                 0.0s
 => [stage-1  1/14] FROM docker.elastic.co/wolfi/chainguard-base-fips:latest@sha256:e436fc2621091a21f223c9cb6bc0156b04ed93284debaeaab2ac27d1ed53c387                                                            0.0s
 => [home 1/5] FROM cgr.dev/chainguard/wolfi-base:latest                                                                                                                                                        0.0s
 => [internal] load build context                                                                                                                                                                              11.8s
 => => transferring context: 1.24GB                                                                                                                                                                            11.7s
 => CACHED [home 2/5] RUN for iter in {1..10}; do         apk fix &&         apk add --no-cache shadow libcap-utils &&         exit_code=0 && break || exit_code=$? && echo "apk error: retry $iter in 10s" &&  0.0s
 => [home 3/5] COPY beat /usr/share/elastic-agent                                                                                                                                                               1.8s
 => [home 4/5] RUN true &&     chmod 0777 /usr/share/elastic-agent &&     mkdir -p /usr/share/elastic-agent/data /usr/share/elastic-agent/data/elastic-agent-0c1aa5/logs &&     find /usr/share/elastic-agent   8.6s
 => CACHED [stage-1  5/11] RUN groupadd --gid 1000 elastic-agent &&     useradd -M --uid 1000 --gid 1000 elastic-agent &&     chmod 755 /usr/local/bin/docker-entrypoint &&     true                            0.0s
 => [home 5/5] RUN setcap =p /usr/share/elastic-agent/data/elastic-agent-0c1aa5/elastic-agent && true                                                                                                           1.1s
 => [stage-1  6/11] COPY --chown=elastic-agent:elastic-agent --from=home /usr/share/elastic-agent /usr/share/elastic-agent                                                                                      1.2s
 => [stage-1  7/11] RUN chmod 0777 /usr/share/elastic-agent &&     usermod -d /usr/share/elastic-agent elastic-agent &&     find /usr/share/elastic-agent/data/elastic-agent-0c1aa5/components -name "*.yml*"   1.7s
 => [stage-1  8/11] RUN mkdir /licenses                                                                                                                                                                         0.5s
 => [stage-1  9/11] COPY --from=home /usr/share/elastic-agent/LICENSE.txt /licenses                                                                                                                             0.1s
 => [stage-1 10/11] COPY --from=home /usr/share/elastic-agent/NOTICE.txt /licenses                                                                                                                              0.2s
 => CACHED [stage-1  2/14] RUN for iter in {1..10}; do         apk fix &&         apk add --no-cache ca-certificates curl gawk shadow bash &&         exit_code=0 && break || exit_code=$? && echo "apk error:  0.0s
 => CACHED [stage-1  3/14] RUN set -e ;   TINI_BIN="";   TINI_SHA256="";   TINI_VERSION="v0.19.0";   case "$(arch)" in     x86_64)         TINI_BIN="tini-amd64";         TINI_SHA256="93dcc18adc78c65a028a847  0.0s
 => CACHED [stage-1  4/14] COPY docker-entrypoint /usr/local/bin/docker-entrypoint                                                                                                                              0.0s
 => CACHED [stage-1  5/14] RUN groupadd --gid 1000 elastic-agent &&     useradd -M --uid 1000 --gid 1000 elastic-agent &&     chmod 755 /usr/local/bin/docker-entrypoint &&     true                            0.0s
 => CACHED [stage-1  6/14] COPY --chown=elastic-agent:elastic-agent --from=home /usr/share/elastic-agent /usr/share/elastic-agent                                                                               0.0s
 => CACHED [stage-1  7/14] RUN chmod 0777 /usr/share/elastic-agent &&     usermod -d /usr/share/elastic-agent elastic-agent &&     find /usr/share/elastic-agent/data/elastic-agent-0c1aa5/components -name "*  0.0s
 => CACHED [stage-1  8/14] RUN mkdir /licenses                                                                                                                                                                  0.0s
 => CACHED [stage-1  9/14] COPY --from=home /usr/share/elastic-agent/LICENSE.txt /licenses                                                                                                                      0.0s
 => CACHED [stage-1 10/14] COPY --from=home /usr/share/elastic-agent/NOTICE.txt /licenses                                                                                                                       0.0s
 => [stage-1 11/14] COPY --from=home /opt /opt                                                                                                                                                                  0.7s
 => [stage-1 12/14] RUN mkdir /app &&     chown elastic-agent:elastic-agent /app                                                                                                                                0.4s
 => [stage-1 13/14] WORKDIR /usr/share/elastic-agent                                                                                                                                                            0.3s
 => [stage-1 14/14] RUN echo -e '#!/bin/sh\nexec /usr/local/bin/docker-entrypoint' > /app/apm.sh &&     chmod 0555 /app/apm.sh                                                                                  0.4s
 => exporting to image                                                                                                                                                                                          1.0s
 => => exporting layers                                                                                                                                                                                         0.9s
 => => writing image sha256:38f90038e97054750c5e0e7c123297cd33daa44a9feeaa779e9c3d1e82958ffe                                                                                                                    0.0s
 => => naming to docker.elastic.co/beats-ci/elastic-agent-fips-cloud:9.1.0-SNAPSHOT                                                                                                                             0.0s
--- TestPackages, the generated packages (i.e. file modes, owners, groups).
--- TestPackages
>> Testing package contents
package ran for 4m43.228406644s
➜  elastic-agent git:(fips-packaging) ✗ tree build/distributions
build/distributions
├── elastic-agent-fips-9.1.0-SNAPSHOT-linux-amd64.docker.tar.gz
├── elastic-agent-fips-9.1.0-SNAPSHOT-linux-amd64.docker.tar.gz.sha512
├── elastic-agent-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz
├── elastic-agent-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz.sha512
├── elastic-agent-fips-cloud-9.1.0-SNAPSHOT-linux-amd64.docker.tar.gz
└── elastic-agent-fips-cloud-9.1.0-SNAPSHOT-linux-amd64.docker.tar.gz.sha512

0 directories, 6 files

Package a linux/amd64 elastic-agent tar.gz package using a DRA manifest, downloading the EA executable from it (no local compilation)

We need a small hack as the elastic-agent-core-fips-* artifacts will only be available after this PR is merged: flip fips flag for components &comp-elastic-agent-core here and &comp-elastic-agent-core-fips here. This will force the packaging code to select the non-FIPS compliant elastic-agent-core artifact and complete the packaging (without this the packaging will fail because the elastic-agent-core-fips-* cannot be downloaded)

After modifying the components definitions, run:

FIPS=true MANIFEST_URL="https://snapshots.elastic.co/9.1.0-94eefa61/manifest-9.1.0-SNAPSHOT.json" AGENT_DROP_PATH=build/elastic-agent-drop PLATFORMS="linux/amd64" PACKAGES="tar.gz" mage clean downloadManifest packageUsingDRA

Sample output:

➜  elastic-agent git:(fips-packaging) ✗ FIPS=true MANIFEST_URL="https://snapshots.elastic.co/9.1.0-94eefa61/manifest-9.1.0-SNAPSHOT.json" AGENT_DROP_PATH=build/elastic-agent-drop PLATFORMS="linux/amd64" PACKAGES="tar.gz" mage clean downloadManifest packageUsingDRA
--- Downloading manifest
Downloading manifest took 14.439813263s
--- Package Elastic-Agent
>> File build/elastic-agent-drop/linux-x86_64.tar.gz/fleet-server.spec.yml does not exist, reverting to local specfile
>> Building internal/pkg/otel/README.md
>> BuildPGP from GPG-KEY-elasticsearch to internal/pkg/release/pgp.go
>> BuildFleetCfg _meta/elastic-agent.fleet.yml to internal/pkg/agent/application/configuration_embed.go
>> Building elastic-agent.yml for linux/amd64
>> Building elastic-agent.reference.yml for linux/amd64
>> Building elastic-agent.docker.yml for linux/amd64
>> check: Checking for invalid links in "internal/pkg/otel/README.md"
--- CrossBuildGoDaemon Elastic-Agent
--- CrossBuild Elastic-Agent
>> buildGoDaemon: Building for linux/amd64
>> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, GOTOOLCHAIN=local, PLATFORM_ID=linux-amd64]
/usr/bin/ld: /tmp/ccqUWk62.o: in function `main':
god.c:(.text+0x2cd): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: god.c:(.text+0x263): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
--- Package artifact
>> package: Building linux-amd64
>> package: Building elastic-agent-fips type=tar.gz for platform=linux/amd64 fips=true
--- TestPackages, the generated packages (i.e. file modes, owners, groups).
--- TestPackages
>> Testing package contents
package ran for 51.409934986s
➜  elastic-agent git:(fips-packaging) ✗ tree -L 1 build/distributions
build/distributions
├── elastic-agent-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz
└── elastic-agent-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz.sha512

0 directories, 2 files

Related issues

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

This is an automatic backport of pull request #7690 done by [Mergify](https://mergify.com).

mergify · 2025-04-09T14:06:57Z

Cherry-pick of 6abd585 has failed:

On branch mergify/bp/8.x/pr-7690
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit 6abd585e5.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/integration.pipeline.yml
	modified:   .buildkite/pipeline.elastic-agent-binary-dra.yml
	modified:   dev-tools/mage/build.go
	modified:   dev-tools/mage/crossbuild.go
	modified:   dev-tools/mage/pkg.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   dev-tools/mage/checksums.go
	both modified:   dev-tools/mage/dockerbuilder.go
	both modified:   dev-tools/mage/manifest/manifest.go
	both modified:   dev-tools/mage/manifest/manifest_test.go
	both modified:   dev-tools/mage/pkgtypes.go
	both modified:   dev-tools/mage/settings.go
	both modified:   dev-tools/packaging/package_test.go
	both modified:   dev-tools/packaging/packages.yml
	deleted by us:   dev-tools/packaging/settings.go
	both modified:   magefile.go
	both modified:   testing/integration/upgrade_standalone_same_commit_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

elasticmachine · 2025-04-09T14:07:15Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

elasticmachine · 2025-04-09T14:07:15Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

pchila · 2025-04-10T07:43:29Z

Cherry-picked commits:
894ef93 (PR #7602)
6abd585 (PR #7690)
b93e749 (PR #7795)
0f1f85f (PR #7805)

* Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package

* Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec

Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name.

mergify · 2025-04-14T04:10:12Z

This pull request has not been merged yet. Could you please review and merge it @pchila? 🙏

pchila · 2025-04-14T09:05:32Z

buildkite test this

elastic-sonarqube · 2025-04-14T11:35:06Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-04-14T12:16:04Z

💚 Build Succeeded

Buildkite Build
Commit: 886a65c

History

💔 Build #19961 failed 953a7ff
💔 Build #19925 failed 2c12bea
💔 Build #19882 failed dd4798e
💔 Build #19872 failed 1de471b
💔 Build #19871 failed 38d1c96

cc @pchila

pchila · 2025-04-16T06:33:07Z

Test packaging build https://buildkite.com/elastic/elastic-agent-package/builds/5384

* Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <paolo.chila@elastic.co> # Conflicts: # .buildkite/integration.pipeline.yml # dev-tools/mage/checksums.go # dev-tools/mage/dockerbuilder.go # dev-tools/mage/manifest/manifest.go # dev-tools/mage/pkgtypes.go # dev-tools/mage/settings.go # dev-tools/packaging/packages.yml # dev-tools/packaging/testing/package_test.go # testing/integration/ess/upgrade_standalone_same_commit_test.go

* Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <paolo.chila@elastic.co>

* ci: build agent from snapshot DRA (#9048) * feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA * feat: incorporate USE_PACKAGE_VERSION in mage * experiment: bump version.go * Revert "experiment: bump version.go" This reverts commit a57ee10. * chore: bump .package-version * feat: allow AGENT_VERSION to be overridden by env var * fix: use named args for all args in integration_tests_tf.ps1 * feat: panic on err of initPackageVersion * fix: don't panic when .package-version file doesn't exist, log it instead * feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE * feat: use os.WriteFile in writePackageVersion * chore: bump to latest snapshot DRA * fix: always DownloadManifest if PackagingFromManifest is set in mage package * fix: check err of filepath.Abs(dropPath) (cherry picked from commit a155660) # Conflicts: # .buildkite/integration.pipeline.yml # .buildkite/scripts/steps/ess.ps1 # .package-version # dev-tools/mage/manifest/manifest.go * [8.x](backport #7690) Fips packaging (#7790) * Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <paolo.chila@elastic.co> * fix: 8.18.5 snapshot DRA * fix: resolve QF1004: could use strings.ReplaceAll linter errors --------- Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Paolo Chilà <paolo.chila@elastic.co>

* ci: build agent from snapshot DRA (#9048) * feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA * feat: incorporate USE_PACKAGE_VERSION in mage * experiment: bump version.go * Revert "experiment: bump version.go" This reverts commit a57ee10. * chore: bump .package-version * feat: allow AGENT_VERSION to be overridden by env var * fix: use named args for all args in integration_tests_tf.ps1 * feat: panic on err of initPackageVersion * fix: don't panic when .package-version file doesn't exist, log it instead * feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE * feat: use os.WriteFile in writePackageVersion * chore: bump to latest snapshot DRA * fix: always DownloadManifest if PackagingFromManifest is set in mage package * fix: check err of filepath.Abs(dropPath) (cherry picked from commit a155660) # Conflicts: # .buildkite/integration.pipeline.yml # .buildkite/scripts/steps/ess.ps1 # .package-version # dev-tools/mage/manifest/manifest.go * [8.x](backport #7690) Fips packaging (#7790) * Make components in packages configurable (#7602) * Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package * Fips packaging (#7690) * Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec * Correctly inject dependency list when packaging using DROP_PATH (#7795) * Restore qualifier=core for elastic-agent-core packaging specs (#7805) Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name. * Modify fips core spec qualifier and name (#7818) * Reintroduce cloud-defend component * Filter components by package-type --------- Co-authored-by: Paolo Chilà <paolo.chila@elastic.co> * fix: point .package-version to latest 8.17 SNAPSHOT DRA * fix: resolve QF1004: could use strings.ReplaceAll linter errors * fix: remove unused var hintsInputsDFilePattern --------- Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Paolo Chilà <paolo.chila@elastic.co>

mergify bot requested review from a team as code owners April 9, 2025 14:06

mergify bot added backport conflicts There is a conflict in the backported pull request labels Apr 9, 2025

mergify bot requested review from blakerouse and pkoutsovasilis and removed request for a team April 9, 2025 14:06

mergify bot assigned pchila Apr 9, 2025

github-actions bot added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Team:Elastic-Agent Label for the Agent team skip-changelog labels Apr 9, 2025

pchila marked this pull request as draft April 9, 2025 18:34

pchila force-pushed the mergify/bp/8.x/pr-7690 branch from a246e8d to 38d1c96 Compare April 10, 2025 07:39

pchila force-pushed the mergify/bp/8.x/pr-7690 branch 2 times, most recently from 263d54e to dd4798e Compare April 10, 2025 09:45

pchila added 6 commits April 11, 2025 11:25

Correctly inject dependency list when packaging using DROP_PATH (#7795)

aa826db

Modify fips core spec qualifier and name (#7818)

47db082

Reintroduce cloud-defend component

953a7ff

pchila force-pushed the mergify/bp/8.x/pr-7690 branch from dd4798e to 2c12bea Compare April 11, 2025 09:25

Filter components by package-type

886a65c

pchila force-pushed the mergify/bp/8.x/pr-7690 branch from 2c12bea to 953a7ff Compare April 14, 2025 09:16

pchila marked this pull request as ready for review April 14, 2025 13:03

pchila removed the conflicts There is a conflict in the backported pull request label Apr 15, 2025

pchila requested a review from ycombinator April 15, 2025 14:08

pkoutsovasilis approved these changes Apr 16, 2025

View reviewed changes

pchila merged commit 9737b5f into 8.x Apr 16, 2025
13 checks passed

pchila deleted the mergify/bp/8.x/pr-7690 branch April 16, 2025 08:07

pkoutsovasilis mentioned this pull request Aug 7, 2025

[9.0] (backport #9048) ci: build agent from snapshot DRA #9281

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.x](backport #7690) Fips packaging#7790

[8.x](backport #7690) Fips packaging#7790
pchila merged 7 commits into8.xfrom
mergify/bp/8.x/pr-7690

mergify bot commented Apr 9, 2025 •

edited by pchila

Loading

Uh oh!

mergify bot commented Apr 9, 2025

Uh oh!

elasticmachine commented Apr 9, 2025

Uh oh!

elasticmachine commented Apr 9, 2025

Uh oh!

pchila commented Apr 10, 2025 •

edited

Loading

Uh oh!

mergify bot commented Apr 14, 2025

Uh oh!

pchila commented Apr 14, 2025

Uh oh!

elastic-sonarqube bot commented Apr 14, 2025

Uh oh!

elasticmachine commented Apr 14, 2025

Uh oh!

pchila commented Apr 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mergify bot commented Apr 9, 2025 • edited by pchila Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Package a linux/amd64 elastic-agent tar.gz package using local elastic-agent code

Package a linux/amd64 elastic-agent-core tar.gz package using local elastic-agent code

Package a linux/amd64 elastic-agent-core tar.gz and docker images using local elastic-agent code

Package a linux/amd64 elastic-agent tar.gz package using a DRA manifest, downloading the EA executable from it (no local compilation)

Related issues

Questions to ask yourself

Uh oh!

mergify bot commented Apr 9, 2025

Uh oh!

elasticmachine commented Apr 9, 2025

Uh oh!

elasticmachine commented Apr 9, 2025

Uh oh!

pchila commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Apr 14, 2025

Uh oh!

pchila commented Apr 14, 2025

Uh oh!

elastic-sonarqube bot commented Apr 14, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Apr 14, 2025

💚 Build Succeeded

History

Uh oh!

pchila commented Apr 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mergify bot commented Apr 9, 2025 •

edited by pchila

Loading

pchila commented Apr 10, 2025 •

edited

Loading