[8.18](backport #6697) Add ability to run components in the Otel manager

[mergify]

What does this PR do?

Allows to run components in the Otel manager instead of the runtime manager. Currently, only filestream is supported. This implements option 2 from https://github.com/elastic/ingest-dev/issues/4931.

Some notes about the implementation:

• We use the same otel collector as hybrid mode. We add a prefix to each component and pipeline name in an effort to avoid conflicts.
• The Otel configuration is generated from the component model agent already uses for beats processes. It is easier to manipulate and lets us emit the same status information in the future.
• This behaviour is controlled by a single attribute of the Coordinator, the only way to enable it is by changing the code. This is a temporary measure until we decide what feature flags we want for it.
• Status reporting uses the otel collector status, and doesn't try to be compatible with beats processes status.
• Which input types are supported is hardcoded.

Why is it important?

This is a preprequisite for migrating to Otel within agent, which is our long-term plan. In the short-term, it will make it easier for us to run tests.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works
• I have added an integration test or an E2E test

How to test this PR locally

1. Change the boolean value in

   elastic-agent/internal/pkg/agent/application/coordinator/coordinator.go

   Line 402 in 0c79718

   runComponentsInOtelManager: false, // change this to run supported components in the Otel manager

2. Build an agent package
3. Run agent with a configuration containing a filestream input

Related issues

• Closes https://github.com/elastic/ingest-dev/issues/3530
• Relates https://github.com/elastic/ingest-dev/issues/3517

---

This is an automatic backport of pull request #6697 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of eeffc0e has failed:

On branch mergify/bp/8.18/pr-6697
Your branch is up to date with 'origin/8.18'.

You are currently cherry-picking commit eeffc0e5a.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/agent/application/coordinator/coordinator.go
	modified:   internal/pkg/agent/application/coordinator/coordinator_unit_test.go
	modified:   internal/pkg/agent/application/upgrade/upgrade.go
	new file:   internal/pkg/otel/configtranslate/otelconfig.go
	new file:   internal/pkg/otel/configtranslate/otelconfig_test.go
	modified:   pkg/component/runtime/command.go
	modified:   pkg/component/runtime/runtime_comm.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   NOTICE.txt
	both modified:   go.mod

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
4 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
82.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: adf5f38

Failed CI Steps

• Win2022:non-sudo:default
• Win2022:non-sudo:default
• Win2025:non-sudo:default
• Win2025:non-sudo:default
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• x86_64:non-sudo: default

History

• 💔 Build #20480 failed 3bed429
• 💔 Build #20454 failed 68877c6
• 💔 Build #20439 failed bb09e16

cc @swiatekm

[mergify]

This pull request has not been merged yet. Could you please review and merge it @swiatekm? 🙏

[swiatekm]

Closing, we don't need to backport this change to 8.18. It's good enough that it's in 8.19, which is our long-term support branch.