Add a mage target that packages elastic-agent using elastic-agent-core DRA

[pchila]

What does this PR do?

This PR adds 2 new mage targets:

• FetchLatestAgentCoreStagingDRA : fetches the latest staging artifact for agent core for a given branch passed as argument
• PackageUsingDRA: start the packaging process but instead of compiling the agent binaries using Crossbuild, downloads the elastic-agent-core DRAs specified in the manifest passed using ManifestURL environment variable.
  This target assumes that downloadManifest has been called with the same manifest

The algorithm is pretty straightforward: packageAgent now receives an mg.F parameter to run as dependency to obtain the agent binaries:

• in the existing Package target we pass GolangCrossbuild as the dependency to run
• in the new PackageUsingDRA we pass mg.F(useDRAAgentBinaryForPackage, manifestUrl) to retrieve the binaries from the elastic-agent-core DRA packages specified in the manifest

There is also some refactoring to break up the packageAgent function in more manageable pieces: collectPackageDependencies and flattenDependencies have been extracted from the original function body.

Why is it important?

When repackaging agent for early releases we want to specify what agent binaries to use via the manifest instead of rebuilding from the HEAD of a give release branch.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Author's Checklist

• [ ]

How to test this PR locally

To test the packaging functionality we can run mage packageUsingDRA specifiying a manifest and a version, for example:

mage clean && AGENT_DROP_PATH="build/elastic-agent-drop" DRA_VERSION="8.13.0+build202403190000" AGENT_PACKAGE_VERSION="8.13.0+build202403190000" DRA_WORKFLOW="staging" ManifestURL="https://staging.elastic.co/independent-agent/8.13.0+build202403190101/agent-artifacts-8.13.0+build202403190101.json"  mage -v  downloadManifest packageUsingDRA

The dependencies and agent binaries will be downloaded according to what is specified in the manifest and packages will be created with the correct package version in build/distributions

To just fetch the agent-core binaries (useful for debugging scenarios) we call FetchLatestAgentCoreStagingDRA with a release branch as argument (we can have a look at what is available in the DRA report):

mage FetchLatestAgentCoreStagingDRA 8.13

Related issues

• Closes Add a mage target that packages the elastic-agent-core package instead of recompiling the agent #4367

Use cases

Screenshots

Logs

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @pchila? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[dwhyrock]

LGTM.

Love all the added verbose logging, btw.

[pchila]

buildkite package this

[pchila]

buildkite package this

[cmacknz]

Verified the MacOS artifact works now. Thanks!

When we merge this, it is going to make it so that snapshots and build candidates depend on the output of the elastic-agent-binary-dra buildkite pipeline that produces the elastic-agent-core artifact.

Looking at the pipeline https://buildkite.com/elastic/elastic-agent-binary-dra I don't think it runs often enough. Main seems to be on a daily schedule, and I'm not sure what triggers 8.13 as it is now several days behind the head of the 8.13 branch.

It looks like this job only takes 5m most of the time, as part of this PR can we change this agent core DRA step to execute for every commit? It is cheap and that will avoid the situation where someone forgets to kick it manually for a build candidate.

Looking at the buildkite pipeline definition it isn't obvious to me why it isn't already working this way for these branches. https://github.com/elastic/elastic-agent/blob/8.13/.buildkite/pipeline.elastic-agent-binary-dra.yml

[pchila]

Verified the MacOS artifact works now. Thanks!

When we merge this, it is going to make it so that snapshots and build candidates depend on the output of the elastic-agent-binary-dra buildkite pipeline that produces the elastic-agent-core artifact.

Looking at the pipeline https://buildkite.com/elastic/elastic-agent-binary-dra I don't think it runs often enough. Main seems to be on a daily schedule, and I'm not sure what triggers 8.13 as it is now several days behind the head of the 8.13 branch.

It looks like this job only takes 5m most of the time, as part of this PR can we change this agent core DRA step to execute for every commit? It is cheap and that will avoid the situation where someone forgets to kick it manually for a build candidate.

Looking at the buildkite pipeline definition it isn't obvious to me why it isn't already working this way for these branches. https://github.com/elastic/elastic-agent/blob/8.13/.buildkite/pipeline.elastic-agent-binary-dra.yml

There's nothing in the pipeline definition that would limit the elastic-agent-binary-dra from running on any release branch or main (we just have a regex identifiying \d+.\d+ branches or exact match with main).
You are right about the schedule as we have a daily on main in the buildkite settings directly.

I think that the reason why we don't get a build out of every commit is that the option has not been selected on the pipeline Github configuration Build branches option in the pic below

If we want to build on every commit on main and release branches I think we need to enable that option (not sure if we need to keep the schedule ?)

I would like a second opinion from our eng-prod team before making any changes though...

Edit: had an OOB chat with @alexsapran, in the end the most sensible thing seems to be having a scheduled buildkite job for each of the release branches. I will raise that modification in a separate PR and crossreference with this one.

Edit2: switched to building branches, PR: #4550

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube