[8.17] (backport #9048) ci: build agent from snapshot DRA

[mergify]

What does this PR do?

This PR reworks the .package-version file format and integrates it into our CI and mage workflows to enable deterministic, snapshot-based builds of Elastic Agent based on published DRA (Downloadable Release Artifacts).

Key changes:

• .package-version is now a structured JSON file containing version metadata (version, build_id, stack_build_id, manifest_url, etc.).
• CI scripts and mage targets now conditionally derive the agent version and build metadata from .package-version when USE_PACKAGE_VERSION=true is set.
• The mage target UpdatePackageVersion replaces previous logic with a structured update mechanism for .package-version.
• CI integration steps now consistently use the snapshot version and manifest provided in .package-version.

Why is it important?

We previously had a fundamental dependency issue in the release flow: to produce a new Elastic Agent DRA, we needed to bump the agent version — but our CI integration tests required that same DRA to already exist in order to pass. This created a timing problem and made the release workflow fragile.

With this PR:

• The CI system and mage targets now honour .package-version when USE_PACKAGE_VERSION=true is set, allowing packaging and testing to rely on a known-good, previously published DRA.
• This ensures that builds and tests only proceed against DRAs that have been published by the unified release process, making test results and releases more predictable and reliable.
• PRs such as #9163, which update .package-version, now become the authoritative source of truth for the Elastic Agent version used in the repo and must be handled with utmost care.
• Note: Even if a component publishes its own DRA, Elastic Agent will only package it once the unified release has run and the respective PR to update the .package-version has been merged.

This enables a reproducible and version-pinned CI system while allowing us to automate snapshot bumps in a controlled and testable way.

An experimental version bump under commit a57ee10 verified this flow successfully. The CI run is available here.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None expected. Existing workflows are unaffected unless USE_PACKAGE_VERSION=true is explicitly set. In that case, .package-version is required and must be up to date.

How to test this PR locally

specify USE_PACKAGE_VERION=true and call any mage target e.g.

USE_PACKAGE_VERSION="true" EXTERNAL="true" PLATFORMS="linux/arm64" PACKAGES="docker" DOCKER_VARIANTS="basic" mage package

Related issues

• N/A

---

This is an automatic backport of pull request #9048 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of a155660 has failed:

On branch mergify/bp/8.17/pr-9048
Your branch is up to date with 'origin/8.17'.

You are currently cherry-picking commit a155660c4.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/scripts/buildkite-integration-tests.ps1
	modified:   .buildkite/scripts/buildkite-integration-tests.sh
	modified:   .buildkite/scripts/buildkite-k8s-integration-tests.sh
	modified:   .buildkite/scripts/steps/ess.sh
	modified:   .buildkite/scripts/steps/ess_start.sh
	new file:   .buildkite/scripts/steps/integration-cloud-image-push.sh
	modified:   .buildkite/scripts/steps/integration-package.sh
	modified:   .buildkite/scripts/steps/integration_tests.sh
	modified:   .buildkite/scripts/steps/integration_tests_tf.ps1
	modified:   .buildkite/scripts/steps/integration_tests_tf.sh
	modified:   .github/workflows/bump-agent-versions.sh
	new file:   dev-tools/mage/packageversion.go
	modified:   dev-tools/mage/settings.go
	modified:   magefile.go
	modified:   test_infra/ess/deployment.tf
	modified:   testing/integration/ess/upgrade_broken_package_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   .buildkite/integration.pipeline.yml
	both modified:   .buildkite/scripts/steps/ess.ps1
	both modified:   .package-version
	both modified:   dev-tools/mage/manifest/manifest.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[pkoutsovasilis]

first I wanna see how this PR behaves on main and if everything checks out, I am gonna deal with the backports

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: da314cd

History

• 💔 Build #24931 failed 2217142
• 💔 Build #24915 failed 50463b7
• 💔 Build #24880 failed f12fc8a

cc @pkoutsovasilis

[pkoutsovasilis]

did a dry-run of elastic-agent-package to test that all my backporting didn't break anything here

So @pchila would you mind skimming through this one, since I needed to backport all you packaging changes?