wip: run filestream input in Otel

[swiatekm]

What does this PR do?

Run filestream inputs within the otel manager instead of the component model. This is a proof of concept of how this could work. We do a straightforward configuration translation from agent inputs and outputs to Otel receivers and exporters, and refactor configuration handling in the Coordinator to allow the configuration to either be handled by the Otel Manager or the Runtime Manager.

To Do:

• Translate input and output configuration to Otel configuration
• Run the Otel configuration in the Otel manager used for hybrid mode
• Ensure inputs handled by Otel are not handled by the component model
• Ensure statuses are as similar as possible between the otel manager and the component model
• Inject monitoring config at a level where it can be run in Otel mode

Why is it important?

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elastic-sonarqube]

Quality Gate failed

Failed conditions
1.8% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube

[khushijain21]

FYI, the package path has been updated https://github.com/elastic/beats/blob/main/libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feat/filebeat-otel-receiver upstream/feat/filebeat-otel-receiver
git merge upstream/main
git push upstream feat/filebeat-otel-receiver

[khushijain21]

Also, to note that queue settings defined under outputs/[output-type] is promoted one level up by beats. The queue settings from output will have to be updated under beatsreceiver to stay consistent. For example:

filebeatreceiver:
   filebeat:
     inputs:
   output:
    otelconsumer: nil
   queue:  // queue settings coming from ES should be updated here
    mem: ....

FYI, for how it is done for beats in otel mode

[khushijain21]

we may also want to inject the monitoring config before we generate the OTel components

[swiatekm]

Closing in favor of #6697