[8.19](backport #46413) otel: add test for otelconsumer + elasticsearchexporter document-level retries

[mergify]

Proposed commit message

This commit adds a test to verify that elasticsearchexporter applies
document-level retries correctly, and that the combination of
otelconsumer and the exporter guarantees at-least-once delivery when
a batch is acknowledged.

The test simulates retryable and permanent errors, checks that
max_retries is respected, and ensures that documents retried after
temporary failures are ingested while permanent failures are dropped.

Currenty, otelconsumer output metrics are not tracked correctly, and it
will always ACK the whole batch even if there were retries or dropped
events by the exporter.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

$ cd x-pack/filebeat
$ mage buildSystemTestBinary
$ go test -run ^TestFilebeatOTelDocumentLevelRetries$ -v -count=1 -tags=integration ./tests/integration

$ ./script/stresstest.sh --tags integration ./x-pack/filebeat/tests/integration ^TestFilebeatOTelDocumentLevelRetries$ -p 1
10m20s: 30 runs so far, 0 failures, 1 active
10m25s: 30 runs so far, 0 failures, 1 active

Related issues

• Depends on otel: handle compression_level 0 correctly in config translation #46391
• Depends on sending_queue from [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter #46111
• For [beats receivers] The interactions between otelconsumer and the ES exporter lose data on partial failure of _bulk #45707

  ---

  This is an automatic backport of pull request otel: add test for otelconsumer + elasticsearchexporter document-level retries #46413 done by Mergify.

[mergify]

Cherry-pick of 2948886 has failed:

On branch mergify/bp/8.19/pr-46413
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 29488865a.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   NOTICE.txt
	both modified:   go.mod
	both modified:   go.sum
	both modified:   x-pack/filebeat/tests/integration/otel_test.go

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

This needs some adjustment from a breaking change in mock-es api.NewApiHandler to use otel sdk metrics.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @mauri870? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @mauri870? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @mauri870? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @mauri870? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @mauri870? 🙏

[pierrehilbert]

@mauri870 could you please resolve the conflicts?

[mauri870]

Thanks for the reminder. This is on my to-do list, but since it requires some missing backports due to breaking changes in mock-es, I’ve been postponing it because of other priorities. I should have all backports merged this week.

[mauri870]

Depends on the following backport #47089.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @mauri870? 🙏

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.19/pr-46413 upstream/mergify/bp/8.19/pr-46413
git merge upstream/8.19
git push upstream mergify/bp/8.19/pr-46413