Skip to content

[Metricbeat] Add storage metricset to Google Cloud Platform module#15598

Merged
sayden merged 2 commits intoelastic:masterfrom
sayden:feature/xp/mb/gcp/storage
Mar 24, 2020
Merged

[Metricbeat] Add storage metricset to Google Cloud Platform module#15598
sayden merged 2 commits intoelastic:masterfrom
sayden:feature/xp/mb/gcp/storage

Conversation

@sayden
Copy link
Copy Markdown
Contributor

@sayden sayden commented Jan 15, 2020 •
edited by andresrc
Loading

Add the storage metricset and the field region to the input config

cf. #15810

@sayden sayden requested a review from a team as a code owner January 15, 2020 20:26
@sayden sayden self-assigned this Jan 15, 2020
ChrsMark
ChrsMark reviewed Jan 16, 2020
View reviewed changes
Comment thread x-pack/metricbeat/module/googlecloud/storage/_meta/data.json Outdated
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark Jan 16, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a suggestion, since this is being included in the docs it would be nice if we could have more metrics-fields as a more complete sample for our users. I would try to include as many as possible of them https://github.com/elastic/beats/pull/15598/files#diff-71e43a13b3f822a4956f547a646aced4R10.

wdyt?

ChrsMark
ChrsMark reviewed Jan 16, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall lgtm @sayden already! I just left some minor concerns.

Also do you plan to add a dashboard too in this or in followup PR? It would be nice to update the PR's description with this info too.

Comment thread x-pack/metricbeat/module/googlecloud/stackdriver/metadata_services.go Outdated
Copy link
Copy Markdown
Member

@ChrsMark ChrsMark Jan 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that ServicePubsub and ServiceLoadBalancing are not strictly related to this patch and related to different upcoming metricsets, right?

Also ServiceStorage returns nil metadata too?

If we plan to keep it like this for now could you please provide some content here for reference so as to be more clear?

Copy link
Copy Markdown
Contributor Author

@sayden sayden Jan 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, main thing here is that I'm trying to avoid too many potential conflicts because there are 3 different follow-up PR's touching the same function and I have to wait until only one is left to see overall final look.

About the question of storage: This file, metadata_services.go "controls / knows" which metricsets / gcp-services have an specific implementation to retrieve extra metadata (call it labels or properties) using their specific API's, instead of taking just the labels that are returned in the Stackdriver API. Only one metricset has such implementation: compute.

This is caused by a direct limitation on lightweight modules. Being so generic is a con in this case.

Copy link
Copy Markdown
Member

@ChrsMark ChrsMark Jan 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough! Thanks for elaborating!

@andresrc andresrc requested a review from a team January 27, 2020 10:34
@kaiyan-sheng
Copy link
Copy Markdown
Contributor

kaiyan-sheng commented Feb 14, 2020

Need a rebase and a changelog entry please 😬

@kaiyan-sheng kaiyan-sheng mentioned this pull request Feb 14, 2020
Merged
5 tasks
kaiyan-sheng
kaiyan-sheng reviewed Feb 19, 2020
View reviewed changes
Comment thread x-pack/metricbeat/modules.d/googlecloud.yml.disabled Outdated
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng Feb 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should compute and storage be in separate config section? Because one is taking zone and the other is taking region as config.

Copy link
Copy Markdown
Member

@ChrsMark ChrsMark Feb 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say that yes 🤔

Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng Mar 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sayden ^^ WDYT? 😄

Copy link
Copy Markdown
Contributor Author

@sayden sayden Mar 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I overlooked at this the first time but yeap, I think it's better to separate them too. Thanks! 😬

@ChrsMark
Copy link
Copy Markdown
Member

ChrsMark commented Feb 20, 2020

@sayden @kaiyan-sheng could you please update the status of this after the changes/rebase?
It is floating around for some time now and I'm not sure if it is in_review status or there are still things to be done. So an update would be useful , if there is something blocking this or whatever 🙂 .

@sayden
Copy link
Copy Markdown
Contributor Author

sayden commented Feb 20, 2020 •
edited
Loading

It was blocked by the loadbalancing metricset and now it needs rebase. I'll do it asap It's still blocked after #16422

@sayden sayden force-pushed the feature/xp/mb/gcp/storage branch from 9541366 to a635414 Compare February 20, 2020 22:05
@sayden sayden force-pushed the feature/xp/mb/gcp/storage branch from a635414 to 9e87030 Compare March 4, 2020 18:56
@sayden sayden force-pushed the feature/xp/mb/gcp/storage branch from e9b87d1 to cdcca53 Compare March 16, 2020 21:28
@sayden
Copy link
Copy Markdown
Contributor Author

sayden commented Mar 16, 2020

@kaiyan-sheng whenever you can take a look at this, I'm not sure if the final "merge" is correct, for example the function constructFilter in metrics_requester.go is only being used in metric_requester_test.go. I'm not fully sure if this was intentional or I mess it up with conflicts so a second pair of eyes is more than welcome 😬

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

kaiyan-sheng commented Mar 18, 2020

@sayden Sorry for the late response. I believe constructFilter is covered in getFilterForMetric function now. You can remove constructFilter function and the unit test 😄

@sayden sayden force-pushed the feature/xp/mb/gcp/storage branch from 0d22a64 to 0c3ba1c Compare March 24, 2020 16:10
kaiyan-sheng
kaiyan-sheng approved these changes Mar 24, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can create a separate PR to address the leftover comments 👍 Let's get this in first! Thanks!

@sayden sayden force-pushed the feature/xp/mb/gcp/storage branch from 67894b4 to 4406e09 Compare March 24, 2020 16:50
@sayden sayden merged commit 52b9330 into elastic:master Mar 24, 2020
sayden added a commit to sayden/beats that referenced this pull request Mar 24, 2020
@sayden
[Metricbeat] Add storage metricset to Google Cloud Platform module (e…
3eedd4f 
…lastic#15598)

(cherry picked from commit 52b9330)

# Conflicts:
#	metricbeat/docs/modules/googlecloud.asciidoc
#	metricbeat/docs/modules_list.asciidoc
#	x-pack/metricbeat/metricbeat.reference.yml
#	x-pack/metricbeat/module/googlecloud/_meta/config.yml
#	x-pack/metricbeat/module/googlecloud/fields.go
#	x-pack/metricbeat/modules.d/googlecloud.yml.disabled
@sayden sayden mentioned this pull request Mar 24, 2020
Merged
@sayden sayden added the v7.7.0 label Mar 24, 2020
sayden added a commit that referenced this pull request Mar 24, 2020
@sayden
Cherry-pick #15598 to 7.x: [Metricbeat] Add storage metricset to Goog…
ebc46ce 
…le Cloud Platform module (#17219)
@sayden sayden added the test-plan Add this PR to be manual test plan label Mar 31, 2020
@andresrc andresrc added the test-plan-added This PR has been added to the test plan label Apr 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark left review comments

+1 more reviewer

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@sayden sayden

Labels

enhancement Metricbeat Metricbeat Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.7.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sayden @kaiyan-sheng @ChrsMark @andresrc