otel: allow filebeat otel to load otel config files by mauri870 · Pull Request #45093 · elastic/beats

mauri870 · 2025-06-27T17:10:14Z

Proposed commit message

Currently the filebeat otel command only loads beats config files, which it then translates to beats receivers in order to run a OTel collector.

This PR augments this functionality to allow the command to load otel configuration files, this aids in testing and running beats receivers in a standalone collector without depending on EDOT.

How to test

cd x-pack/filebeat/
go test -c -tags otelbeat
mage docker:composeUp
go test   -tags "integration" -run 'TestFilebeatOTel.*' ./tests/integration -v -count=1

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

Currently the filebeat otel command only loads beats config files, which it then translates to beats receivers in order to run a OTel collector. This PR augments this functionality to allow the command to load otel configuration files, this aids in testing and running beats receivers in a standalone collector.

mergify · 2025-06-30T15:23:29Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filebeat-otel-config upstream/filebeat-otel-config
git merge upstream/main
git push upstream filebeat-otel-config

elasticmachine · 2025-06-30T15:42:37Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

mauri870 · 2025-06-30T18:16:04Z

/test

khushijain21

LGTM

khushijain21 · 2025-07-01T12:41:48Z

This should be backported to 8.19 as well, I believe

… files (#45133) * otel: allow filebeat otel to load otel config files (#45093) * otel: allow filebeat otel to load otel config files Currently the filebeat otel command only loads beats config files, which it then translates to beats receivers in order to run a OTel collector. This PR augments this functionality to allow the command to load otel configuration files, this aids in testing and running beats receivers in a standalone collector. * update notice * add test for multiple receivers * fix monitoring port * test refactor for template * restore condition removed by mistake * allow monitoring http request with variable url (cherry picked from commit 46e855e) * fix linter error --------- Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

This test is flaking for a while with a high failure rate. We have an integration test that covers the same scenario added by elastic#45093. Skip this test for now.

khushijain21 · 2025-07-03T07:54:52Z

+          id: filestream-fbreceiver
+          enabled: true
+          paths:
+            - {{$receiver.InputFile}}


Hello @mauri870, I just realized that both the receivers are using the same directory. The registry is going to be updated by both receivers at the same time. And since they are ingesting the same log file - there is likely going to be data loss here.

Were you able to test this? I am afarid we will turn out to see this as another flaky test.

It is also best to use unqiue namespaces for both receivers

Thanks, you are absolutely right. I ran these tests with stress for half an hour but did not find any issues, but they are definitely bound to happen at some point.

khushijain21

Sorry for bringing up few comments after this was merged

This test is flaking for a while with a high failure rate. We have an integration test that covers the same scenario added by #45093. Skip this test for now.

This test is flaking for a while with a high failure rate. We have an integration test that covers the same scenario added by #45093. Skip this test for now. (cherry picked from commit 5069a00) # Conflicts: # x-pack/filebeat/fbreceiver/receiver_test.go

This test is flaking for a while with a high failure rate. We have an integration test that covers the same scenario added by #45093. Skip this test for now. (cherry picked from commit 5069a00)

This test is flaking for a while with a high failure rate. We have an integration test that covers the same scenario added by #45093. Skip this test for now. (cherry picked from commit 5069a00) # Conflicts: # x-pack/filebeat/fbreceiver/receiver_test.go

This test is flaking for a while with a high failure rate. We have an integration test that covers the same scenario added by #45093. Skip this test for now. (cherry picked from commit 5069a00) Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

…vers (#45166) * [oteltest][fbreceiver]: skip TestMultipleReceivers (#45147) This test is flaking for a while with a high failure rate. We have an integration test that covers the same scenario added by #45093. Skip this test for now. (cherry picked from commit 5069a00) # Conflicts: # x-pack/filebeat/fbreceiver/receiver_test.go * fix conflicts --------- Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

…ivers (#45168) * [oteltest][fbreceiver]: skip TestMultipleReceivers (#45147) This test is flaking for a while with a high failure rate. We have an integration test that covers the same scenario added by #45093. Skip this test for now. (cherry picked from commit 5069a00) # Conflicts: # x-pack/filebeat/fbreceiver/receiver_test.go * fix conflicts --------- Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

As a leftover from elastic#45093, the tests added reuse the index names which may cause issues when running the tests in sequence without cleaning up first. Adjusted the existing tests to use unique indexes.

As a leftover from #45093, the tests added reuse the index names which may cause issues when running the tests in sequence without cleaning up first. Adjusted the existing tests to use unique indexes.

As a leftover from #45093, the tests added reuse the index names which may cause issues when running the tests in sequence without cleaning up first. Adjusted the existing tests to use unique indexes. (cherry picked from commit ea98564) # Conflicts: # x-pack/filebeat/tests/integration/otel_test.go

As a leftover from #45093, the tests added reuse the index names which may cause issues when running the tests in sequence without cleaning up first. Adjusted the existing tests to use unique indexes. (cherry picked from commit ea98564)

…45319) As a leftover from #45093, the tests added reuse the index names which may cause issues when running the tests in sequence without cleaning up first. Adjusted the existing tests to use unique indexes. (cherry picked from commit ea98564) Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

* otel: allow filebeat otel to load otel config files Currently the filebeat otel command only loads beats config files, which it then translates to beats receivers in order to run a OTel collector. This PR augments this functionality to allow the command to load otel configuration files, this aids in testing and running beats receivers in a standalone collector. * update notice * add test for multiple receivers * fix monitoring port * test refactor for template * restore condition removed by mistake * allow monitoring http request with variable url (cherry picked from commit 46e855e) # Conflicts: # NOTICE.txt # go.mod # x-pack/libbeat/common/otelbeat/otel.go

mauri870 · 2025-07-11T13:38:25Z

Huh, for some reason there was no backport for 9.1 even with the backport-active-9 label.

botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 27, 2025

mergify Bot assigned mauri870 Jun 27, 2025

mauri870 added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 27, 2025

mauri870 force-pushed the filebeat-otel-config branch from 7571341 to 2d668c9 Compare June 27, 2025 17:42

mauri870 added 4 commits June 30, 2025 10:57

update notice

a7eb7fe

add test for multiple receivers

b3cdc6a

fix monitoring port

74657ba

test refactor for template

2df0132

mauri870 added 2 commits June 30, 2025 12:32

Merge branch 'main' into filebeat-otel-config

3c255ca

restore condition removed by mistake

56030dd

mauri870 marked this pull request as ready for review June 30, 2025 15:42

mauri870 requested review from a team as code owners June 30, 2025 15:42

mauri870 requested review from VihasMakwana and khushijain21 June 30, 2025 15:42

allow monitoring http request with variable url

818c4c8

mauri870 force-pushed the filebeat-otel-config branch from a47c49f to 818c4c8 Compare June 30, 2025 16:18

Merge branch 'main' into filebeat-otel-config

67a8fd5

mauri870 mentioned this pull request Jun 30, 2025

[Flaky Test] x-pack/filebeat/fbreceiver TestMultipleReceivers and TestFilebeatOTelMultipleReceiversE2E #43832

Closed

khushijain21 reviewed Jul 1, 2025

View reviewed changes

Comment thread x-pack/filebeat/tests/integration/otel_test.go

khushijain21 approved these changes Jul 1, 2025

View reviewed changes

mauri870 mentioned this pull request Jul 2, 2025

[oteltest][fbreceiver]: skip TestMultipleReceivers #45147

Merged

6 tasks

khushijain21 reviewed Jul 3, 2025

View reviewed changes

Comment thread x-pack/filebeat/tests/integration/otel_test.go

khushijain21 reviewed Jul 3, 2025

View reviewed changes

mergify Bot mentioned this pull request Jul 3, 2025

[9.0](backport #45147) [oteltest][fbreceiver]: skip TestMultipleReceivers #45166

Merged

6 tasks

mergify Bot mentioned this pull request Jul 3, 2025

[9.1](backport #45147) [oteltest][fbreceiver]: skip TestMultipleReceivers #45167

Merged

6 tasks

mergify Bot mentioned this pull request Jul 3, 2025

[8.19](backport #45147) [oteltest][fbreceiver]: skip TestMultipleReceivers #45168

Merged

6 tasks

mauri870 mentioned this pull request Jul 10, 2025

otel: update filebeat integration tests to run in isolation #45314

Merged

6 tasks

mauri870 added the backport-9.1 Automated backport to the 9.1 branch label Jul 11, 2025

mergify Bot mentioned this pull request Jul 11, 2025

[9.1](backport #45093) otel: allow filebeat otel to load otel config files #45324

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

otel: allow filebeat otel to load otel config files#45093

otel: allow filebeat otel to load otel config files#45093
mauri870 merged 9 commits intoelastic:mainfrom
mauri870:filebeat-otel-config

mauri870 commented Jun 27, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 27, 2025

Uh oh!

mergify Bot commented Jun 27, 2025

Uh oh!

mergify Bot commented Jun 30, 2025

Uh oh!

elasticmachine commented Jun 30, 2025

Uh oh!

mauri870 commented Jun 30, 2025

Uh oh!

Uh oh!

khushijain21 left a comment

Uh oh!

khushijain21 commented Jul 1, 2025

Uh oh!

khushijain21 Jul 3, 2025 •

edited

Loading

Uh oh!

mauri870 Jul 3, 2025

Uh oh!

Uh oh!

khushijain21 left a comment

Uh oh!

mauri870 commented Jul 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mauri870 commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

How to test

Checklist

Related

Uh oh!

github-actions Bot commented Jun 27, 2025

🤖 GitHub comments

Uh oh!

mergify Bot commented Jun 27, 2025

Uh oh!

mergify Bot commented Jun 30, 2025

Uh oh!

elasticmachine commented Jun 30, 2025

Uh oh!

mauri870 commented Jun 30, 2025

Uh oh!

Uh oh!

khushijain21 left a comment

Choose a reason for hiding this comment

Uh oh!

khushijain21 commented Jul 1, 2025

Uh oh!

khushijain21 Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mauri870 Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

khushijain21 left a comment

Choose a reason for hiding this comment

Uh oh!

mauri870 commented Jul 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mauri870 commented Jun 27, 2025 •

edited

Loading

khushijain21 Jul 3, 2025 •

edited

Loading