Skip to content

[8.x](backport #6697) Add ability to run components in the Otel manager #6995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
swiatekm merged 1 commit into from
Feb 25, 2025
Merged

[8.x](backport #6697) Add ability to run components in the Otel manager #6995

swiatekm merged 1 commit into from
Feb 25, 2025

Conversation

@mergify
Copy link
Contributor

@mergify mergify bot commented Feb 24, 2025

What does this PR do?

Allows to run components in the Otel manager instead of the runtime manager. Currently, only filestream is supported. This implements option 2 from https://github.com/elastic/ingest-dev/issues/4931.

Some notes about the implementation:

  • We use the same otel collector as hybrid mode. We add a prefix to each component and pipeline name in an effort to avoid conflicts.
  • The Otel configuration is generated from the component model agent already uses for beats processes. It is easier to manipulate and lets us emit the same status information in the future.
  • This behaviour is controlled by a single attribute of the Coordinator, the only way to enable it is by changing the code. This is a temporary measure until we decide what feature flags we want for it.
  • Status reporting uses the otel collector status, and doesn't try to be compatible with beats processes status.
  • Which input types are supported is hardcoded.

Why is it important?

This is a preprequisite for migrating to Otel within agent, which is our long-term plan. In the short-term, it will make it easier for us to run tests.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an integration test or an E2E test

How to test this PR locally

  1. Change the boolean value in

    elastic-agent/internal/pkg/agent/application/coordinator/coordinator.go

    Line 402 in 0c79718

    runComponentsInOtelManager: false, // change this to run supported components in the Otel manager
  2. Build an agent package
  3. Run agent with a configuration containing a filestream input

Related issues

This is an automatic backport of pull request #6697 done by [Mergify](https://mergify.com).

@mergify mergify bot added the backport label Feb 24, 2025
@mergify mergify bot requested a review from a team as a code owner February 24, 2025 15:51
@mergify mergify bot removed the request for review from a team February 24, 2025 15:51
@github-actions github-actions bot added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog labels Feb 24, 2025
@elasticmachine
Copy link
Contributor

elasticmachine commented Feb 24, 2025

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@swiatekm swiatekm mentioned this pull request Feb 25, 2025
Merged
8 tasks
@swiatekm
Add ability to run components in the Otel manager (#6697)
b8f1463 
* Add ability to run components in the Otel manager

# Conflicts:
#	NOTICE.txt
#	go.mod
#	go.sum

* Add coordinator test

* Set metricbeat receiver signal type to logs

* Drop unnecessary transform processor

The conversion now happens in the otel consumer in beats.

* Determine default datastream type from beat name

* Fix diagnostics tests

* Promote output queue settings to receivers

* Move otel config translation to the otel package

* Emit the otel component diagnostic conditionally

* Add more otel config translation tests

* Code review fixes

* Fix diagnostics tests

* Code Review fixes

* Correctly set input types if not present

* More code review fixes

(cherry picked from commit eeffc0e)
@swiatekm swiatekm force-pushed the mergify/bp/8.x/pr-6697 branch from fc56291 to b8f1463 Compare February 25, 2025 14:16
@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Feb 25, 2025

Quality Gate passed Quality Gate passed

Issues
4 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
82.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link
Contributor

elasticmachine commented Feb 25, 2025 •
edited
Loading

💚 Build Succeeded

History

cc @swiatekm

@swiatekm swiatekm merged commit f9eb06f into 8.x Feb 25, 2025
14 checks passed
@swiatekm swiatekm deleted the mergify/bp/8.x/pr-6697 branch February 25, 2025 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@swiatekm swiatekm swiatekm approved these changes

@blakerouse blakerouse Awaiting requested review from blakerouse blakerouse was automatically assigned from elastic/elastic-agent-control-plane

@andrzej-stencel andrzej-stencel Awaiting requested review from andrzej-stencel andrzej-stencel was automatically assigned from elastic/elastic-agent-control-plane

Assignees

@swiatekm swiatekm

Labels

backport skip-changelog Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@elasticmachine @swiatekm