[9.0] (backport #9048) ci: build agent from snapshot DRA #9281

mergify · 2025-08-07T08:45:17Z

What does this PR do?

This PR reworks the .package-version file format and integrates it into our CI and mage workflows to enable deterministic, snapshot-based builds of Elastic Agent based on published DRA (Downloadable Release Artifacts).

Key changes:

.package-version is now a structured JSON file containing version metadata (version, build_id, stack_build_id, manifest_url, etc.).
CI scripts and mage targets now conditionally derive the agent version and build metadata from .package-version when USE_PACKAGE_VERSION=true is set.
The mage target UpdatePackageVersion replaces previous logic with a structured update mechanism for .package-version.
CI integration steps now consistently use the snapshot version and manifest provided in .package-version.

Why is it important?

We previously had a fundamental dependency issue in the release flow: to produce a new Elastic Agent DRA, we needed to bump the agent version — but our CI integration tests required that same DRA to already exist in order to pass. This created a timing problem and made the release workflow fragile.

With this PR:

The CI system and mage targets now honour .package-version when USE_PACKAGE_VERSION=true is set, allowing packaging and testing to rely on a known-good, previously published DRA.
This ensures that builds and tests only proceed against DRAs that have been published by the unified release process, making test results and releases more predictable and reliable.
PRs such as #9163, which update .package-version, now become the authoritative source of truth for the Elastic Agent version used in the repo and must be handled with utmost care.
Note: Even if a component publishes its own DRA, Elastic Agent will only package it once the unified release has run and the respective PR to update the .package-version has been merged.

This enables a reproducible and version-pinned CI system while allowing us to automate snapshot bumps in a controlled and testable way.

An experimental version bump under commit a57ee10 verified this flow successfully. The CI run is available here.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

None expected. Existing workflows are unaffected unless USE_PACKAGE_VERSION=true is explicitly set. In that case, .package-version is required and must be up to date.

How to test this PR locally

specify USE_PACKAGE_VERION=true and call any mage target e.g.

USE_PACKAGE_VERSION="true" EXTERNAL="true" PLATFORMS="linux/arm64" PACKAGES="docker" DOCKER_VARIANTS="basic" mage package

Related issues

N/A

This is an automatic backport of pull request #9048 done by [Mergify](https://mergify.com).

mergify · 2025-08-07T08:45:20Z

Cherry-pick of a155660 has failed:

On branch mergify/bp/9.0/pr-9048
Your branch is up to date with 'origin/9.0'.

You are currently cherry-picking commit a155660c4.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/scripts/buildkite-integration-tests.ps1
	modified:   .buildkite/scripts/buildkite-integration-tests.sh
	modified:   .buildkite/scripts/buildkite-k8s-integration-tests.sh
	modified:   .buildkite/scripts/steps/ess.sh
	modified:   .buildkite/scripts/steps/ess_start.sh
	new file:   .buildkite/scripts/steps/integration-cloud-image-push.sh
	modified:   .buildkite/scripts/steps/integration-package.sh
	modified:   .buildkite/scripts/steps/integration_tests.sh
	modified:   .buildkite/scripts/steps/integration_tests_tf.ps1
	modified:   .buildkite/scripts/steps/integration_tests_tf.sh
	modified:   .github/workflows/bump-agent-versions.sh
	new file:   dev-tools/mage/packageversion.go
	modified:   dev-tools/mage/settings.go
	modified:   magefile.go
	modified:   test_infra/ess/deployment.tf
	modified:   testing/integration/ess/upgrade_broken_package_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   .buildkite/integration.pipeline.yml
	both modified:   .buildkite/scripts/steps/ess.ps1
	both modified:   .package-version
	both modified:   dev-tools/mage/manifest/manifest.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

elasticmachine · 2025-08-07T08:45:33Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

pkoutsovasilis · 2025-08-07T08:56:51Z

first I wanna see how this PR behaves on main and if everything checks out, I am gonna deal with the backports

pkoutsovasilis · 2025-08-07T10:52:32Z

ok CI run on main is green and the automation PR looks correct #9283 proceeding with addressing the conflicts

pkoutsovasilis · 2025-08-07T11:47:34Z

This branch is lacking the changes introduced by #7790 which apparently are needed for packaging to work. @pchila I am gonna try to cherry-pick the changes here. cc @ebeahan for awareness

* Redefine ExpectedBinaries as YAML config * Move ExpectedBinaries closer to package spec file * Fix error formatting in downloadDRAArtifacts * add packageName template to ExpectedPackages * use a relaxed dependencies version for IAR releases * Remove FIPS hack introduced in PR #7486 * Allow for a looser match on relaxing dependencies versions * Add debug logging when packaging with EXTERNAL=true * move package tests to dedicated package

* Add component list to specs * extract component dependencies from the packages to be built * Refactor component extraction from package specs * Fix package tests error handling * Inject dependencies and remove references to ExpectedBinaries * Remove ExpectedBinaries global * Add rootdir to components * Extract actual version matched on the package file and use it to render RootDir * Package elastic-agent FIPS specs when FIPS=true is specified * refactor ResolveManifestPackage * Move FIPS compile settings in packages.yml * Add more FIPS components * Properly handle dependenciesVersion when calling mage package * Refactor ChecksumsWithoutManifest to use list of dependencies instead of globbing files * Rework useDRAAgentBinaryForPackage for repackaging agent Define elastic-agent-core components (both FIPS and non-FIPS variants) and define package name and root dir templates. Implement some filtering on component list to extract the correct component definition according to the FIPSBuild flag. Refactor code that downloads pre-compiled elastic-agent binaries and places them in the golangCrossBuild directory to make use of the new component definition. * Write spec FIPS flag into manifest.yaml when packaging * Add FIPS elastic agent basic and cloud docker images * Build FIPS docker images in CI packaging * Fix FIPS .tar.gz package tests * Restructure package tests * Extend FIPS check to all binaries in components directory * Create FIPS elastic-agent-core artifacts in elastic-agent-binary-dra pipeline * Cleanup ChecksumsWithManifest and improve godoc * Improve godoc for BinarySpec

Restore qualifier for elastic-agent-core packaging specs to avoid changing the rootDir name of the archives. The qualifier had been removed in PR #7690 trying to use the spec name: this worked to get the desired file name but changed the root Dir name which uses '{{.BeatName}}{{if .Qualifier}}-{{.Qualifier}}{{end}}' in the template definition instead of '{{.Name}}' which would render the spec name.

* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA * feat: incorporate USE_PACKAGE_VERSION in mage * experiment: bump version.go * Revert "experiment: bump version.go" This reverts commit a57ee10. * chore: bump .package-version * feat: allow AGENT_VERSION to be overridden by env var * fix: use named args for all args in integration_tests_tf.ps1 * feat: panic on err of initPackageVersion * fix: don't panic when .package-version file doesn't exist, log it instead * feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE * feat: use os.WriteFile in writePackageVersion * chore: bump to latest snapshot DRA * fix: always DownloadManifest if PackagingFromManifest is set in mage package * fix: check err of filepath.Abs(dropPath)

elastic-sonarqube · 2025-08-07T23:30:23Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-08-08T00:03:28Z

💚 Build Succeeded

Buildkite Build
Commit: e98e93f

History

💔 Build #24932 failed 2133791
💔 Build #24909 failed 496a48b
💔 Build #24904 failed a4cc8ab
💔 Build #24899 failed c67a0e5
💔 Build #24893 failed d930d16

cc @pkoutsovasilis

pkoutsovasilis · 2025-08-08T07:47:00Z

did a dry-run of elastic-agent-package to test that all my backporting didn't break anything here

So @pchila would you mind skimming through this one, since I needed to backport all you packaging changes?

pchila

Minor nitpick on FIPS settings struct, not blocking and can be fixed in a follow-up PR.
Package DRA build looks good.
Thank you for the backport!

dev-tools/packaging/settings.go

mergify bot requested review from a team as code owners August 7, 2025 08:45

mergify bot added backport conflicts There is a conflict in the backported pull request labels Aug 7, 2025

mergify bot requested review from blakerouse and michalpristas and removed request for a team August 7, 2025 08:45

mergify bot assigned pkoutsovasilis Aug 7, 2025

mergify bot mentioned this pull request Aug 7, 2025

ci: build agent from snapshot DRA #9048

Merged

8 tasks

github-actions bot added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog Testing:run:TestUpgradeIntegrationsServer labels Aug 7, 2025

pkoutsovasilis removed the conflicts There is a conflict in the backported pull request label Aug 7, 2025

pkoutsovasilis force-pushed the mergify/bp/9.0/pr-9048 branch from 0fc66a5 to 6818837 Compare August 7, 2025 12:51

pchila and others added 6 commits August 7, 2025 22:19

Correctly inject dependency list when packaging using DROP_PATH (#7795)

b1905a7

Modify fips core spec qualifier and name (#7818)

be256c1

pkoutsovasilis force-pushed the mergify/bp/9.0/pr-9048 branch 2 times, most recently from 6ff5311 to b1d731f Compare August 7, 2025 21:23

pkoutsovasilis added 3 commits August 8, 2025 01:29

fix: remove fips-related changes

d53c5ed

fix: resolve QF1004: could use strings.ReplaceAll linter errors

51e3e51

fix: remove unused imports

e98e93f

pkoutsovasilis force-pushed the mergify/bp/9.0/pr-9048 branch from 2133791 to e98e93f Compare August 7, 2025 22:29

pkoutsovasilis requested a review from pchila August 8, 2025 08:14

pchila approved these changes Aug 8, 2025

View reviewed changes

dev-tools/packaging/settings.go Show resolved Hide resolved

pkoutsovasilis approved these changes Aug 8, 2025

View reviewed changes

pkoutsovasilis merged commit b29094d into 9.0 Aug 8, 2025
19 checks passed

pkoutsovasilis deleted the mergify/bp/9.0/pr-9048 branch August 8, 2025 09:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[9.0] (backport #9048) ci: build agent from snapshot DRA #9281

[9.0] (backport #9048) ci: build agent from snapshot DRA #9281

Uh oh!

mergify bot commented Aug 7, 2025

Uh oh!

mergify bot commented Aug 7, 2025

Uh oh!

elasticmachine commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

elastic-sonarqube bot commented Aug 7, 2025

Uh oh!

elasticmachine commented Aug 8, 2025

Uh oh!

pkoutsovasilis commented Aug 8, 2025

Uh oh!

pchila left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[9.0] (backport #9048) ci: build agent from snapshot DRA #9281

[9.0] (backport #9048) ci: build agent from snapshot DRA #9281

Uh oh!

Conversation

mergify bot commented Aug 7, 2025

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Uh oh!

mergify bot commented Aug 7, 2025

Uh oh!

elasticmachine commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

elastic-sonarqube bot commented Aug 7, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Aug 8, 2025

💚 Build Succeeded

History

Uh oh!

pkoutsovasilis commented Aug 8, 2025

Uh oh!

pchila left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants