[8.x](backport #5349) Add elastic-agent-service container to packaging

[mergify]

What does this PR do?

Adds a new docker variant of elastic-agent-service that includes the python connectors.

Currently this PR includes the following PR until that PR is merged - #5338

Why is it important?

Used by the Elastic Agent service.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool (not really user facing at this level)
• [ ] I have added an integration test or an E2E test

Disruptive User Impact

None.

How to test this PR locally

$ PLATFORMS="linux/arm64" PACKAGES="docker" SNAPSHOT="true" EXTERNAL="true" DOCKER_VARIANTS="service" mage package
$ docker run -it --rm docker.elastic.co/beats-ci/elastic-agent-service:8.16.0-SNAPSHOT

---

This is an automatic backport of pull request #5349 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 701f8b9 has failed:

On branch mergify/bp/8.x/pr-5349
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit 701f8b9491.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/integration.pipeline.yml
	modified:   dev-tools/mage/checksums.go
	modified:   dev-tools/mage/dockervariants.go
	modified:   dev-tools/mage/manifest/manifest_test.go
	new file:   dev-tools/packaging/files/linux/connectors.sh
	modified:   dev-tools/packaging/packages.yml
	modified:   dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl
	modified:   magefile.go
	modified:   pkg/testing/kubernetes/supported.go
	new file:   specs/connectors.spec.yml
	new file:   testing/integration/kubernetes_agent_service_test.go
	modified:   testing/integration/kubernetes_agent_standalone_test.go
	new file:   testing/integration/testdata/connectors.agent.yml

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   dev-tools/mage/manifest/manifest.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[jlind23]

This change broke the unified release on main, I would hold off before this is fixed on main.

[ycombinator]

This change broke the unified release on main, I would hold off before this is fixed on main.

Probably should merge this PR and the 8.x backport for #5647 at the same time.

[seanstory]

Wait I don't think we want this in 8.x, right?
Connectors on agentless is targeting 9.0

[seanstory]

Disabling auto-merge out of an abundance of caution - I don't want anything to sneak into 8.16.0-SNAPSHOT that the agentless team isn't ready for.

[blakerouse]

@seanstory I assumed this was going into 8.16. If it's 9.0 only then we don't need this.

[seanstory]

Relevant thread here: https://github.com/elastic/ingest-dev/issues/3767#issuecomment-2388796205

My team doesn't need this for 8.16.x, but perhaps the agentless team would rather use the new image sooner than later.

[jlind23]

My team doesn't need this for 8.16.x, but perhaps the agentless team would rather use the new image sooner than later.

@tehilashn @olegsu any answer for Sean's point? If you don't need the new variant then we'll keep this change in 9.0 instead of backporting it to 8.x.

[tehilashn]

@oren-zohar @orestisfl - do we need this for 8.16?

[oren-zohar]

@oren-zohar @orestisfl - do we need this for 8.16?

No, as far as I'm aware, the search connectors will not be supported for 8.16.x. We might want it for the next 8.x, even if the connectors are only planned for 9.0, just for testing, but for 8.16 we can skip it

[mergify]

This pull request has not been merged yet. Could you please review and merge it @blakerouse? 🙏

[jlind23]

Thanks @oren-zohar, I'll close this PR for now and will reopen later if we need it in 8.17+.

[blakerouse]

We are re-opening this PR to be merged into 8.x, but only after the 8.16 release is cut. This PR has two follow-up PR fixes already merged in so that when it goes into 8.x it doesn't cause any issues.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[mergify]

This pull request has not been merged yet. Could you please review and merge it @blakerouse? 🙏