[hybrid agent] add state storage extension

[VihasMakwana]

What does this PR do?

This PR adds the elasticsearch_storage extension to EDOT.

Why is it important?

This is important because, unlike the process runtime, the otel runtime does not initialize the ES state store , which causes the pipeline to block in waitReady() and results in no events being emitted.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

Testing

I had a sync with @nkvoll and he tested my changes out on the agentless-serverless buildkite pipeline: https://buildkite.com/elastic/agentless-serverless-release/builds/48/canvas.

Previously:

• The build wasn't succeeding and was stuck in the first phase.

After my changes:

• Build succeeds
• Test cases are passing.
• Data is verified.

Related issues

• Relates [otel] otel runtime does not work with remote state storage beats#48824

[mergify]

This pull request does not have a backport label. Could you fix it @VihasMakwana? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b state-store-extension upstream/state-store-extension
git merge upstream/main
git push upstream state-store-extension

[VihasMakwana]

Blocked on #13106

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b state-store-extension upstream/state-store-extension
git merge upstream/main
git push upstream state-store-extension

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: ac1f1b2

Failed CI Steps

• x86_64:non-sudo: default
• x86_64:non-sudo: default - platform-ingest-elastic-agent-debian-11-1772525581

History

• 💛 Build #35785 was flaky 0ef4e4c
• 💔 Build #35727 failed dfb69d0
• 💔 Build #35717 failed 018e539
• 💔 Build #35547 failed 7718a19
• 💔 Build #35546 failed 2b94429
• 💔 Build #35427 failed 458e2d4

cc @VihasMakwana

[swiatekm]

LGTM. I don't like the way we check for the feature, and I'd prefer if we made it more explicit in the future, but I'm not going to block the PR over that.

[VihasMakwana]

I'll make this explicit in a follow-up. For now, I'll merge this to unblock the agentless team.