[9.0](backport #7328) Allow switching inputs to the otel runtime manager

[mergify]

What does this PR do?

Allows switching individual inputs to the Otel runtime manager. This is done by introducing a new configuration field: _runtime_experimental with possible values of process (the default) and otel. It is no longer necessary to modify agent code and build a local binary to run inputs as beats receivers in an otel collector.

This change is only intended for testing, and the configuration is not stable. See https://github.com/elastic/ingest-dev/issues/5117 for more details. Its only effect on users not opting in to using the otel runtime is that diagnostics now unconditionally emit an additional file containing the final otel collector configuration.

In terms of implementation, we add a new field to Components in the component model, and ensure inputs with a different runtime manager setting end up in different Components.

Why is it important?

It will make testing beats receivers in agent a lot easier. The component model machinery is also easy to repurpose even if we ultimately decide this needs to be configured differently.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works
• I have added an integration test or an E2E test

How to test this PR locally

Build the agent binary and use the following configuration:

outputs:
  default:
    type: elasticsearch
    hosts: [localhost:9200]
    username: elastic
    password: ""
    preset: "balanced"

inputs:
  # Collecting log files
  - type: filestream
    # Input ID allowing Elastic Agent to track the state of this input. Must be unique.
    id: your-input-id
    use_output: default
    _runtime_experimental: otel
    streams:
      - id: input2
        data_stream:
          dataset: myfile
        paths:
          - /tmp/input.txt
        prospector.scanner.fingerprint.enabled: false
        file_identity.native: ~

Related issues

Implements the agent part of https://github.com/elastic/ingest-dev/issues/5117. It's possible the actual config key and values will change, but the changes to the component model should be universal.

---

This is an automatic backport of pull request #7328 done by [Mergify](https://mergify.com).

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: d2bbd68

Failed CI Steps

• check-ci
• Serverless integration test
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• x86_64:non-sudo: default

History

cc @swiatekm

[khushijain21]

The CI fails with this error

max_size` must be greater than or equal to mix_size

I think this is because beats version is not updated in 9.0 branch and changes made to https://github.com/elastic/beats/blob/9.0/libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go are not reflected. This must have happened due to multiple 9.0 backports incoming. I'll try to raise a separate PR to unblock us

[swiatekm]

The CI fails with this error

max_size` must be greater than or equal to mix_size

I think this is because beats version is not updated in 9.0 branch and changes made to https://github.com/elastic/beats/blob/9.0/libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go are not reflected. This must have happened due to multiple 9.0 backports incoming. I'll try to raise a separate PR to unblock us

It'll be unblocked by #7953, this branch points to the beats main branch right now.

[khushijain21]

QQ. Shouldn't "9.0-elastic-agent" branch point to the head of beats 9.0? So that we do not bring in changes not targeted to 9.0?

I had a similar comment from @cmacknz here #7676 (comment)

[swiatekm]

QQ. Shouldn't "9.0-elastic-agent" branch point to the head of beats 9.0? So that we do not bring in changes not targeted to 9.0?

I had a similar comment from @cmacknz here #7676 (comment)

It should, yes. But I wanted to wait until the workstream meeting on Monday to hash out which changes need to go into which branches, exactly. In the meantime, we can merge both this PR and the ones depending on it.

[elastic-sonarqube]

Quality Gate passed

Issues
2 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
93.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube