otel: add test for document equivalence between agent filestream input and filebeat receiver #6681

mauri870 · 2025-01-31T13:52:25Z

What does this PR do?

This PR adds a test that compares a log line ingested by a filestream agent input and a filebeat OTel receiver with equivalent configuration.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Related issues

elasticmachine · 2025-01-31T14:03:31Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

testing/integration/otel_test.go

leehinman

I think if you add the default processors listed at https://github.com/elastic/beats/blob/76f40863c9918ee1a691a2bb29cbab98a5d83878/x-pack/filebeat/cmd/root.go#L48 and add the processors listed in the generate.go below you should get a much shorter list of fields to ignore.

testing/integration/otel_test.go

tests are failing

pierrehilbert · 2025-02-07T06:49:26Z

I don't think the failing tests are related to this PR:

=== RUN   TestDownloadFile
    utils_test.go:21: 
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1738851441003538166/elastic/elastic-agent/dev-tools/mage/downloads/utils_test.go:21
        	Error:      	Expected nil, but got: &fmt.wrapError{msg:"downloading file https://www.elastic.co/robots.txt: GET request failed with 504", err:(*errors.errorString)(0xc00085a970)}
        	Test:       	TestDownloadFile
--- FAIL: TestDownloadFile (20.57s)

Will update the branch to have them running again and hope the gateway failure was temporary

mergify · 2025-02-10T15:50:16Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hybrid-agent-test upstream/hybrid-agent-test
git merge upstream/main
git push upstream hybrid-agent-test

leehinman

LGTM

elastic-sonarqube · 2025-02-11T15:11:54Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

…t and filebeat receiver (#6681) * otel: add test for agent hybrid * add missing fingerprint config options to index small files * ensure new lines are written to the input file * disable compression level so we inspect the requests * add queue timeout to normal beats and separate path.home for fbreceiver * add path.home for normal filebeat * fix agent not starting * print elastic-agent output if test fails * add note for metadata field * fail if ignored field is not present in both maps * add host metadata processor, remove 'host.*' from ignored fields * Use global processor * more strict rules for ignored fields to avoid false positives * use add_fields to add missing fields, use a single index * add add_cloud_metadata processor * avoid checking cmd.Wait error since it is bogus (cherry picked from commit 41882ac)

…t and filebeat receiver (#6681) (#6868) * otel: add test for agent hybrid * add missing fingerprint config options to index small files * ensure new lines are written to the input file * disable compression level so we inspect the requests * add queue timeout to normal beats and separate path.home for fbreceiver * add path.home for normal filebeat * fix agent not starting * print elastic-agent output if test fails * add note for metadata field * fail if ignored field is not present in both maps * add host metadata processor, remove 'host.*' from ignored fields * Use global processor * more strict rules for ignored fields to avoid false positives * use add_fields to add missing fields, use a single index * add add_cloud_metadata processor * avoid checking cmd.Wait error since it is bogus (cherry picked from commit 41882ac) Co-authored-by: Mauri de Souza Meneguzzo <[email protected]>

…t and filebeat receiver (#6681) (#6867) * otel: add test for agent hybrid * add missing fingerprint config options to index small files * ensure new lines are written to the input file * disable compression level so we inspect the requests * add queue timeout to normal beats and separate path.home for fbreceiver * add path.home for normal filebeat * fix agent not starting * print elastic-agent output if test fails * add note for metadata field * fail if ignored field is not present in both maps * add host metadata processor, remove 'host.*' from ignored fields * Use global processor * more strict rules for ignored fields to avoid false positives * use add_fields to add missing fields, use a single index * add add_cloud_metadata processor * avoid checking cmd.Wait error since it is bogus (cherry picked from commit 41882ac) Co-authored-by: Mauri de Souza Meneguzzo <[email protected]>

mauri870 added 10 commits January 3, 2025 11:06

otel: add test for agent hybrid

219af16

Merge branch 'main' into hybrid-agent-test

f2915df

add missing fingerprint config options to index small files

74e9b51

ensure new lines are written to the input file

309640b

disable compression level so we inspect the requests

83e9896

add queue timeout to normal beats and separate path.home for fbreceiver

7051ed8

add path.home for normal filebeat

f86787c

fix agent not starting

f130377

print elastic-agent output if test fails

6e7e882

add note for metadata field

db4caef

mauri870 added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Cleanup skip-changelog backport-8.x Automated backport to the 8.x branch with mergify labels Jan 31, 2025

mauri870 self-assigned this Jan 31, 2025

mauri870 changed the title ~~Hybrid agent test~~ otel: add test for document equivalence between filebeat and filebeat receiver Jan 31, 2025

mauri870 changed the title ~~otel: add test for document equivalence between filebeat and filebeat receiver~~ otel: add test for document equivalence between agent filestream input and filebeat receiver Jan 31, 2025

mauri870 marked this pull request as ready for review January 31, 2025 14:03

mauri870 requested a review from a team as a code owner January 31, 2025 14:03

mauri870 requested review from blakerouse and kaanyalti January 31, 2025 14:03

mauri870 requested review from leehinman and removed request for kaanyalti January 31, 2025 14:03

mauri870 commented Jan 31, 2025

View reviewed changes

testing/integration/otel_test.go Outdated Show resolved Hide resolved

leehinman reviewed Jan 31, 2025

View reviewed changes

testing/integration/otel_test.go Outdated Show resolved Hide resolved

mauri870 marked this pull request as draft February 3, 2025 12:33

mauri870 added 2 commits February 5, 2025 09:45

fail if ignored field is not present in both maps

3a9c9ac

Merge branch 'main' into hybrid-agent-test

4f8c12d

mauri870 force-pushed the hybrid-agent-test branch from 0d77c63 to 4f8c12d Compare February 5, 2025 12:46

mauri870 added 2 commits February 5, 2025 17:16

Use global processor

59253fa

more strict rules for ignored fields to avoid false positives

c0f6415

mauri870 force-pushed the hybrid-agent-test branch from f751674 to c0f6415 Compare February 5, 2025 20:25

use add_fields to add missing fields, use a single index

9359f0c

mauri870 force-pushed the hybrid-agent-test branch from d022ccb to 9359f0c Compare February 6, 2025 12:13

Merge branch 'main' into hybrid-agent-test

2d94e09

mauri870 marked this pull request as ready for review February 6, 2025 14:16

mauri870 requested review from khushijain21 and leehinman February 6, 2025 14:16

khushijain21 previously approved these changes Feb 7, 2025

View reviewed changes

mauri870 added 2 commits February 7, 2025 12:27

add add_cloud_metadata processor

1238a43

Merge branch 'main' into hybrid-agent-test

541729d

mauri870 force-pushed the hybrid-agent-test branch from 5b5e4cd to 541729d Compare February 7, 2025 15:28

avoid checking cmd.Wait error since it is bogus

763318b

khushijain21 approved these changes Feb 10, 2025

View reviewed changes

Merge branch 'main' into hybrid-agent-test

7de70ca

mauri870 force-pushed the hybrid-agent-test branch from 351981b to 7de70ca Compare February 10, 2025 17:44

leehinman approved these changes Feb 10, 2025

View reviewed changes

Merge branch 'main' into hybrid-agent-test

ff2f810

mauri870 merged commit 41882ac into elastic:main Feb 14, 2025
14 checks passed

This was referenced Feb 14, 2025

[8.x](backport #6681) otel: add test for document equivalence between agent filestream input and filebeat receiver #6867

Merged

[9.0](backport #6681) otel: add test for document equivalence between agent filestream input and filebeat receiver #6868

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

otel: add test for document equivalence between agent filestream input and filebeat receiver #6681

otel: add test for document equivalence between agent filestream input and filebeat receiver #6681

Uh oh!

mauri870 commented Jan 31, 2025 •

edited

Loading

Uh oh!

elasticmachine commented Jan 31, 2025

Uh oh!

Uh oh!

leehinman left a comment

Uh oh!

Uh oh!

pierrehilbert commented Feb 7, 2025

Uh oh!

mergify bot commented Feb 10, 2025

Uh oh!

leehinman left a comment

Uh oh!

elastic-sonarqube bot commented Feb 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

otel: add test for document equivalence between agent filestream input and filebeat receiver #6681

otel: add test for document equivalence between agent filestream input and filebeat receiver #6681

Uh oh!

Conversation

mauri870 commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Related issues

Uh oh!

elasticmachine commented Jan 31, 2025

Uh oh!

Uh oh!

leehinman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pierrehilbert commented Feb 7, 2025

Uh oh!

mergify bot commented Feb 10, 2025

Uh oh!

leehinman left a comment

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Feb 11, 2025

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mauri870 commented Jan 31, 2025 •

edited

Loading