Skip to content

otelconsumer: set document id attribute for elasticsearchexporter#42412

Merged
mauri870 merged 6 commits intoelastic:mainfrom
mauri870:otelconsumer-document-id
Jan 31, 2025
Merged

otelconsumer: set document id attribute for elasticsearchexporter#42412
mauri870 merged 6 commits intoelastic:mainfrom
mauri870:otelconsumer-document-id

Conversation

@mauri870
Copy link
Member

Proposed commit message

In order for beats receivers to support deduplication of data like the normal beats we need to have special handling for the @metadata._id field in the beats event. In the elasticsearch beats output this metadata field is used to set the document id when ingesting the event.

The elasticsearchexporter will support dynamic id for log records in the upcoming v0.119 release via a log record attribute.

This PR adds special handling for the metadata _id field and translates it to the elasticsearch.document_id attribute.

Related issues

@mauri870 mauri870 added cleanup Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Jan 23, 2025
@mauri870 mauri870 self-assigned this Jan 23, 2025
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 23, 2025
@mergify
Copy link
Contributor

mergify bot commented Jan 23, 2025

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @mauri870? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

@mergify
Copy link
Contributor

mergify bot commented Jan 23, 2025

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Jan 23, 2025
@mauri870 mauri870 marked this pull request as ready for review January 23, 2025 19:17
@mauri870 mauri870 requested a review from a team as a code owner January 23, 2025 19:17
@mauri870 mauri870 requested review from belimawr and rdner January 23, 2025 19:17
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

@mauri870 mauri870 force-pushed the otelconsumer-document-id branch from 6ee84d1 to 48e1852 Compare January 27, 2025 13:38
@mauri870 mauri870 force-pushed the otelconsumer-document-id branch from 48e1852 to f48d7c6 Compare January 27, 2025 13:41
@mauri870 mauri870 force-pushed the otelconsumer-document-id branch from f981713 to bc8828a Compare January 30, 2025 12:48
@mauri870 mauri870 enabled auto-merge (squash) January 30, 2025 14:58
@mauri870 mauri870 merged commit 04eac62 into elastic:main Jan 31, 2025
142 checks passed
mergify bot pushed a commit that referenced this pull request Jan 31, 2025
…2412)

* otelconsumer: set document id attribute for elasticsearchexporter

* add comment about field being removed from elasticsearch

(cherry picked from commit 04eac62)
mauri870 added a commit that referenced this pull request Jan 31, 2025
…2412) (#42535)

* otelconsumer: set document id attribute for elasticsearchexporter

* add comment about field being removed from elasticsearch

(cherry picked from commit 04eac62)

Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
@mauri870 mauri870 added the backport-9.0 Automated backport to the 9.0 branch label Feb 10, 2025
mergify bot pushed a commit that referenced this pull request Feb 10, 2025
…2412)

* otelconsumer: set document id attribute for elasticsearchexporter

* add comment about field being removed from elasticsearch

(cherry picked from commit 04eac62)
mauri870 added a commit that referenced this pull request Feb 11, 2025
…2412) (#42651)

* otelconsumer: set document id attribute for elasticsearchexporter

* add comment about field being removed from elasticsearch

(cherry picked from commit 04eac62)

Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-8.x Automated backport to the 8.x branch with mergify backport-9.0 Automated backport to the 9.0 branch cleanup Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants