[8.x](backport #7602) Make components in packages configurable

[mergify]

What does this PR do?

This PR allows declarative definitions of elastic-agent components in dev-tools/packaging/packages.yml.
In the declaration of such components it's now possible to specify a packageName template that will be rendered using platform and version during packaging.

Why is it important?

This is a prerequisite for selecting different component packages for specific packaging specs.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Disruptive User Impact

No disruptive user impact as neither the mage build targets nor the final packages changed

How to test this PR locally

Package using a manifest

MANIFEST_URL="https://snapshots.elastic.co/9.1.0-15b6399f/agent-package/agent-artifacts-9.1.0-SNAPSHOT.json" AGENT_DROP_PATH=build/elastic-agent-drop  PLATFORMS="linux/amd64 windows/amd64 darwin/amd64" mage clean downloadManifest package

Package without using a manifest

EXTERNAL=true PLATFORMS="linux/amd64" mage clean package

Related issues

• Closes Refactor required packages definition for packaging to correctly select dependencies #7610

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

---

This is an automatic backport of pull request #7602 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 894ef93 has failed:

On branch mergify/bp/8.x/pr-7602
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit 894ef9385.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   dev-tools/mage/manifest/manifest_test.go
	modified:   dev-tools/mage/pkg.go
	modified:   dev-tools/packaging/packages.yml
	new file:   dev-tools/packaging/settings.go
	renamed:    dev-tools/packaging/package_test.go -> dev-tools/packaging/testing/package_test.go
	modified:   magefile.go
	modified:   pkg/version/version_parser.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   dev-tools/mage/checksums.go
	both modified:   dev-tools/mage/manifest/manifest.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: dfedb66

Failed CI Steps

• check-ci
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Windows 2022
• Unit tests - Windows 2022
• Unit tests - Windows 2016
• Unit tests - Windows 2016
• Unit tests - macOS 15 ARM
• Unit tests - macOS 15 ARM
• Unit tests - Windows 10
• Unit tests - Windows 10
• Unit tests - Windows 11
• Unit tests - Windows 11
• :junit: Junit annotate
• Merge coverage reports
• :sonarqube: Continuous Code Inspection
• Packaging: linux/amd64 rpm
• Packaging: linux/amd64 rpm
• Packaging: Ubuntu x86_64 FIPS
• Packaging: Ubuntu x86_64 FIPS
• Packaging: linux/arm64 tar.gz
• Packaging: linux/arm64 tar.gz
• Packaging: Ubuntu arm64 FIPS
• Packaging: Ubuntu arm64 FIPS
• Packaging: windows/amd64 zip
• Packaging: windows/amd64 zip
• Packaging: Containers linux/amd64
• Packaging: Containers linux/arm64

History

cc @pchila

[pchila]

Closing this as for backport on 8.x we need also the changes introduced with #7690

Will backport both commit at once in #7690 backport