otel: add test for otelconsumer + elasticsearchexporter document-level retries by mauri870 · Pull Request #46413 · elastic/beats

mauri870 · 2025-09-05T14:19:08Z

Proposed commit message

This commit adds a test to verify that elasticsearchexporter applies
document-level retries correctly, and that the combination of
otelconsumer and the exporter guarantees at-least-once delivery when
a batch is acknowledged.

The test simulates retryable and permanent errors, checks that
max_retries is respected, and ensures that documents retried after
temporary failures are ingested while permanent failures are dropped.

Currenty, otelconsumer output metrics are not tracked correctly, and it
will always ACK the whole batch even if there were retries or dropped
events by the exporter.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

$ cd x-pack/filebeat
$ mage buildSystemTestBinary
$ go test -run ^TestFilebeatOTelDocumentLevelRetries$ -v -count=1 -tags=integration ./tests/integration

$ ./script/stresstest.sh --tags integration ./x-pack/filebeat/tests/integration ^TestFilebeatOTelDocumentLevelRetries$ -p 1
10m20s: 30 runs so far, 0 failures, 1 active
10m25s: 30 runs so far, 0 failures, 1 active

Related issues

Depends on otel: handle compression_level 0 correctly in config translation #46391
Depends on sending_queue from [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter #46111
For [beats receivers] The interactions between otelconsumer and the ES exporter lose data on partial failure of _bulk #45707

github-actions · 2025-09-05T14:19:19Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2025-09-08T17:12:36Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

mauri870 · 2025-09-10T22:32:47Z

/test

mergify · 2025-09-11T05:13:23Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fbreceiver-mockes-test upstream/fbreceiver-mockes-test
git merge upstream/main
git push upstream fbreceiver-mockes-test

mauri870 · 2025-09-11T13:11:09Z

/test

mauri870 · 2025-09-11T19:57:24Z

This is ready for review now, pending CI.

mergify · 2025-09-12T02:33:13Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fbreceiver-mockes-test upstream/fbreceiver-mockes-test
git merge upstream/main
git push upstream fbreceiver-mockes-test

VihasMakwana

LGTM, just a nit.

mauri870 · 2025-09-12T13:10:40Z

TestFilestreamTruncateBlockedOutput is flaking https://github.com/elastic/ingest-dev/issues/6087.

Edit: Fix here #46576.

mauri870 · 2025-09-12T17:20:52Z

@cmacknz When you have a chance, could you please review this and approve if it looks good? Thanks!

github-actions · 2025-09-12T19:56:02Z

@Mergifyio backport 9.0 9.1

mergify · 2025-09-12T19:56:16Z

backport 9.0 9.1

✅ Backports have been created

Details

#46617 [9.0](backport #46413) otel: add test for otelconsumer + elasticsearchexporter document-level retries has been created for branch 9.0 but encountered conflicts
#46619 [9.1](backport #46413) otel: add test for otelconsumer + elasticsearchexporter document-level retries has been created for branch 9.1 but encountered conflicts

…l retries (#46413) * otel: add test for document-level retries * fix linter errors * add dummy assert for dropped events * use deterministic handler from mock-es * assert mock-es metrics * sync go.mod and go.sum * simplify handler logic * update comments * use slices.Contains * go get mock-es latest (cherry picked from commit 2948886) # Conflicts: # NOTICE.txt # go.mod # go.sum # x-pack/filebeat/tests/integration/otel_test.go

…hexporter document-level retries (#46619) * otel: add test for otelconsumer + elasticsearchexporter document-level retries (#46413) * otel: add test for document-level retries * fix linter errors * add dummy assert for dropped events * use deterministic handler from mock-es * assert mock-es metrics * sync go.mod and go.sum * simplify handler logic * update comments * use slices.Contains * go get mock-es latest (cherry picked from commit 2948886) # Conflicts: # NOTICE.txt # go.mod # go.sum # x-pack/filebeat/tests/integration/otel_test.go * fix conflicts * go mod tidy --------- Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

…chexporter document-level retries (#46616) * otel: add test for otelconsumer + elasticsearchexporter document-level retries (#46413) * otel: add test for document-level retries * fix linter errors * add dummy assert for dropped events * use deterministic handler from mock-es * assert mock-es metrics * sync go.mod and go.sum * simplify handler logic * update comments * use slices.Contains * go get mock-es latest (cherry picked from commit 2948886) # Conflicts: # NOTICE.txt # go.mod # go.sum # x-pack/filebeat/tests/integration/otel_test.go * fix conflicts * remove test not in 8.19 --------- Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>

otel: add test for document-level retries

9c9ce6c

mauri870 added this to the otel milestone Sep 5, 2025

mauri870 self-assigned this Sep 5, 2025

mauri870 added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches backport-8.19 Automated backport to the 8.19 branch labels Sep 5, 2025

botelastic Bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 5, 2025

mauri870 added 4 commits September 5, 2025 13:27

fix linter errors

ee93a6f

Merge branch 'main' into fbreceiver-mockes-test

e255135

Merge branch 'main' into fbreceiver-mockes-test

a5ceaf2

add dummy assert for dropped events

75c538d

mauri870 marked this pull request as ready for review September 8, 2025 17:12

mauri870 requested a review from a team as a code owner September 8, 2025 17:12

mauri870 requested review from faec and leehinman September 8, 2025 17:12

cmacknz reviewed Sep 8, 2025

View reviewed changes

Comment thread x-pack/filebeat/tests/integration/otel_test.go Outdated

cmacknz reviewed Sep 8, 2025

View reviewed changes

Comment thread x-pack/filebeat/tests/integration/otel_test.go Outdated

mauri870 marked this pull request as draft September 9, 2025 13:23

mauri870 added 3 commits September 10, 2025 17:12

use deterministic handler from mock-es

9290a37

assert mock-es metrics

c94188a

Merge branch 'main' into fbreceiver-mockes-test

59942e6

mauri870 force-pushed the fbreceiver-mockes-test branch from 7a0f602 to 59942e6 Compare September 10, 2025 20:43

sync go.mod and go.sum

56a5e5c

mauri870 force-pushed the fbreceiver-mockes-test branch from 05af572 to 56a5e5c Compare September 10, 2025 20:46

simplify handler logic

83e6acb

mauri870 marked this pull request as ready for review September 11, 2025 19:56

mauri870 requested a review from a team as a code owner September 11, 2025 19:56

mauri870 requested a review from cmacknz September 12, 2025 01:39

VihasMakwana reviewed Sep 12, 2025

View reviewed changes

Comment thread x-pack/filebeat/tests/integration/otel_test.go Outdated

VihasMakwana approved these changes Sep 12, 2025

View reviewed changes

mauri870 added 2 commits September 12, 2025 06:47

use slices.Contains

c874279

Merge branch 'main' into fbreceiver-mockes-test

05a4cbf

mauri870 force-pushed the fbreceiver-mockes-test branch from 19ac3fe to 05a4cbf Compare September 12, 2025 09:51

mauri870 added 2 commits September 12, 2025 06:55

go get mock-es latest

3ba284c

Merge branch 'main' into fbreceiver-mockes-test

965ed64

Merge branch 'main' into fbreceiver-mockes-test

3185f74

cmacknz approved these changes Sep 12, 2025

View reviewed changes

mauri870 enabled auto-merge (squash) September 12, 2025 19:55

mauri870 merged commit 2948886 into elastic:main Sep 12, 2025
207 checks passed

mergify Bot mentioned this pull request Sep 12, 2025

[8.19](backport #46413) otel: add test for otelconsumer + elasticsearchexporter document-level retries #46616

Merged

6 tasks

mergify Bot mentioned this pull request Sep 12, 2025

[9.0](backport #46413) otel: add test for otelconsumer + elasticsearchexporter document-level retries #46617

Merged

6 tasks

mergify Bot mentioned this pull request Sep 12, 2025

[9.1](backport #46413) otel: add test for otelconsumer + elasticsearchexporter document-level retries #46619

Merged

6 tasks

mauri870 mentioned this pull request Sep 12, 2025

[beats receivers] The interactions between otelconsumer and the ES exporter lose data on partial failure of _bulk #45707

Closed

Conversation

mauri870 commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

github-actions Bot commented Sep 5, 2025

🤖 GitHub comments

Uh oh!

elasticmachine commented Sep 8, 2025

Uh oh!

Uh oh!

Uh oh!

mauri870 commented Sep 10, 2025

Uh oh!

mergify Bot commented Sep 11, 2025

Uh oh!

mauri870 commented Sep 11, 2025

Uh oh!

mauri870 commented Sep 11, 2025

Uh oh!

mergify Bot commented Sep 12, 2025

Uh oh!

Uh oh!

VihasMakwana left a comment

Choose a reason for hiding this comment

Uh oh!

mauri870 commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mauri870 commented Sep 12, 2025

Uh oh!

Uh oh!

github-actions Bot commented Sep 12, 2025

Uh oh!

mergify Bot commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mauri870 commented Sep 5, 2025 •

edited

Loading

mauri870 commented Sep 12, 2025 •

edited

Loading

mergify Bot commented Sep 12, 2025 •

edited

Loading