Allow switching inputs to the otel runtime manager

[swiatekm]

What does this PR do?

Allows switching individual inputs to the Otel runtime manager. This is done by introducing a new configuration field: _runtime_experimental with possible values of process (the default) and otel. It is no longer necessary to modify agent code and build a local binary to run inputs as beats receivers in an otel collector.

This change is only intended for testing, and the configuration is not stable. See https://github.com/elastic/ingest-dev/issues/5117 for more details. Its only effect on users not opting in to using the otel runtime is that diagnostics now unconditionally emit an additional file containing the final otel collector configuration.

In terms of implementation, we add a new field to Components in the component model, and ensure inputs with a different runtime manager setting end up in different Components.

Why is it important?

It will make testing beats receivers in agent a lot easier. The component model machinery is also easy to repurpose even if we ultimately decide this needs to be configured differently.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works
• I have added an integration test or an E2E test

How to test this PR locally

Build the agent binary and use the following configuration:

outputs:
  default:
    type: elasticsearch
    hosts: [localhost:9200]
    username: elastic
    password: ""
    preset: "balanced"

inputs:
  # Collecting log files
  - type: filestream
    # Input ID allowing Elastic Agent to track the state of this input. Must be unique.
    id: your-input-id
    use_output: default
    _runtime_experimental: otel
    streams:
      - id: input2
        data_stream:
          dataset: myfile
        paths:
          - /tmp/input.txt
        prospector.scanner.fingerprint.enabled: false
        file_identity.native: ~

Related issues

Implements the agent part of https://github.com/elastic/ingest-dev/issues/5117. It's possible the actual config key and values will change, but the changes to the component model should be universal.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pkoutsovasilis]

since the following is the case

This change is only intended for testing, and the configuration is not stable. See https://github.com/elastic/ingest-dev/issues/5117 for more details. Its only effect on users not opting in to using the otel runtime is that diagnostics now unconditionally emit an additional file containing the final otel collector configuration.

any reason we aren't guarding this behind a Feature Flag? cc @cmacknz

[swiatekm]

since the following is the case

This change is only intended for testing, and the configuration is not stable. See elastic/ingest-dev#5117 for more details. Its only effect on users not opting in to using the otel runtime is that diagnostics now unconditionally emit an additional file containing the final otel collector configuration.

any reason we aren't guarding this behind a Feature Flag? cc @cmacknz

I'm not against adding an undocumented feature flag for it. I was trying to minimize the set of changes I'd potentially need to redo in the future, but a flag isn't that much work.

[swiatekm]

since the following is the case

This change is only intended for testing, and the configuration is not stable. See elastic/ingest-dev#5117 for more details. Its only effect on users not opting in to using the otel runtime is that diagnostics now unconditionally emit an additional file containing the final otel collector configuration.

any reason we aren't guarding this behind a Feature Flag? cc @cmacknz

We discussed this during a meeting today, and decided on the following:

• We don't need a flag specifically to enable the new setting at all, but we decided to make it less accessible. It's now called _runtime_experimental to discourage accidental use.
• We do want to have a feature flag controlling the default runtime, but that can come in a separate PR.

Are you ok with this approach @pkoutsovasilis ?

[pkoutsovasilis]

We discussed this during a meeting today, and decided on the following:

• We don't need a flag specifically to enable the new setting at all, but we decided to make it less accessible. It's now called _runtime_experimental to discourage accidental use.
• We do want to have a feature flag controlling the default runtime, but that can come in a separate PR.

Are you ok with this approach @pkoutsovasilis ?

ok @swiatekm since this was discussed and agreed to be in this way, no argument on my end. Waiting for the CI to be green 🙂

[elastic-sonarqube]

Quality Gate passed

Issues
2 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
93.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: fa909c6

History

• 💛 Build #18921 was flaky 6259aec
• 💔 Build #18908 failed 173efa6
• 💔 Build #18892 failed ca54854
• 💔 Build #18850 failed 88fa83b
• 💛 Build #18705 was flaky a9650a9
• 💔 Build #18590 failed d238651

cc @swiatekm