Skip to content

[8.x](backport #41636) [libbeat] fix: aws & openstack metadata conflict in add_cloud_metadata processor#41815

Merged
Kavindu-Dodan merged 1 commit into8.xfrom
mergify/bp/8.x/pr-41636
Nov 27, 2024
Merged

[8.x](backport #41636) [libbeat] fix: aws & openstack metadata conflict in add_cloud_metadata processor#41815
Kavindu-Dodan merged 1 commit into8.xfrom
mergify/bp/8.x/pr-41636

Conversation

@mergify
Copy link
Contributor

@mergify mergify bot commented Nov 27, 2024

Proposed commit message

This PR fixes incorrect recognition of EC2/AWS cloud provider as Openstack. The root cause was the common metadata endpoints used by both AWS SDK & Openstack logic. And this happened when IMDSv2 is disabled in AWS.

I attempted to migrate Openstack logic to another metadata implementation. However, I did not manage to create a fully functioning setup to validate the implementation. Hence, this PR focuses on a priority-based solution where priority is given for SDK-backed metadata fetching over HTTP endpoints.

Current priory providers are - aws/ec2 & azure

Note - I have done a minor refactoring to rename Local struct property to DefaultEnabled to make intention clearer

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

You need a local copy and an EC2 instance to validate the fix.

  • Enable metadata service in EC2 instance and make IMDSv2 optional
  • Build a beats (ex:- metricbeat) module based on this libbeat change
  • Copy the beats module to EC2 instance and start the module with add_cloud_metadata processor enabled & logs set to debug for more in-depth logs & no provider enforced
  • Observe logs and see data (ex:- system monitoring) through Kibana to validate cloud provider detection

Related issues

Screenshots

-IMDSv2 disabled

Screenshot 2024-11-13 at 11 45 38 AM

  • Processor enabled but no provider enforced

Screenshot 2024-11-13 at 11 50 24 AM

  • Debug logs on multi-result and priority based selection

Screenshot 2024-11-13 at 11 44 47 AM

  • Cloud provider detected correctly,

Screenshot 2024-11-13 at 11 45 13 AM


This is an automatic backport of pull request #41636 done by [Mergify](https://mergify.com).

…a processor (#41636)

* rename misleading variable

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* introduce provider priority

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

# Conflicts:
#	libbeat/processors/add_cloud_metadata/providers.go

* isolate priority logic and add testing

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* documentation

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* review changes

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

---------

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
(cherry picked from commit 6d4e641)
@mergify mergify bot added the backport label Nov 27, 2024
@mergify mergify bot requested a review from a team as a code owner November 27, 2024 15:31
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 27, 2024
@Kavindu-Dodan Kavindu-Dodan added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Nov 27, 2024
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 27, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

@Kavindu-Dodan Kavindu-Dodan enabled auto-merge (squash) November 27, 2024 16:03
@Kavindu-Dodan Kavindu-Dodan merged commit d40087a into 8.x Nov 27, 2024
@Kavindu-Dodan Kavindu-Dodan deleted the mergify/bp/8.x/pr-41636 branch November 27, 2024 17:11
@khushijain21 khushijain21 mentioned this pull request Jun 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:obs-ds-hosted-services Label for the Observability Hosted Services team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants