[Metricbeat] gcp: respect Beats naming conventions by endorama · Pull Request #27974 · elastic/beats

endorama · 2021-09-16T16:05:24Z

What does this PR do?

Changes fiels exported by all GCP metricset to adhere to ECS and Beats naming conventions.

This PR flattens all fields definition, to help correlating the mapping between GCP Monitoring and ECS fields.

The remapping operation is performed as a map[string]string lookup on a manually built map mapping the GCP Monitoring metric name to the appropriate ECS field.

Why is it important?

We want our metricset to adhere to ECS and Beats naming convention, so the data exported is easily consumed in Elasticsearch.

As 7.16 is the last minor before 8.0 and must be compatible with 8.0 is the last chance to introduce these breaking changes.

Checklist

My code follows the style guidelines of this project
~~I have commented my code, particularly in hard-to-understand areas~~
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [Metricbeat] gcp: fields do not respect naming best practices #27231
Requires [Metricbeat] gcp: fix integration tests #28364
Requires [Metricbeat] gcp.storage: uniform metricset's fields #28372
Requires [Metricbeat] gcp.pubsub: uniform metricset's fields #28371
Requires [Metricbeat] gcp.loadbalancing: uniform metric's fields #28370
Requires [Metricbeat] gcp.gke: uniform metric's fields #28368
Requires [Metricbeat] gcp.compute: uniform metricset's fields #28367
Requires [Metricbeat] gcp.billing: uniform metricset's fields #28366

Use cases

Screenshots

Logs

elasticmachine · 2021-09-16T16:19:41Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-18T09:46:39.839+0000
Duration: 105 min 13 sec
Commit: e9cf813

Test stats 🧪

Test	Results
Failed	0
Passed	9538
Skipped	2537
Total	12075

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

endorama · 2021-09-23T10:13:32Z

@kaiyan-sheng I was expecting tests to fail due to the rename. As the are all green I'm confused.

elasticmachine · 2021-09-23T10:17:49Z

Pinging @elastic/integrations (Team:Integrations)

kaiyan-sheng · 2021-09-23T15:46:28Z

@endorama I think it's because only billing metricset has the check TestMetricsetFieldsDocumented in integration test enabled: https://github.com/elastic/beats/blob/master/x-pack/metricbeat/module/gcp/billing/billing_integration_test.go#L31. This should also be added to all the other metricsets.

mergify · 2021-10-05T11:30:30Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b master upstream/master
git merge upstream/master
git push upstream master

x-pack/metricbeat/module/gcp/compute/_meta/fields.yml

kaiyan-sheng · 2021-10-06T22:45:11Z

@endorama Are you planning to add TestMetricsetFieldsDocumented for GCP integration tests as well in this PR?

endorama · 2021-10-07T13:30:41Z

@kaiyan-sheng I was planning to at least try to do that, so I was able to test these changes. But it's taking a lot of time and I'm still facing some strange errors.

endorama · 2021-10-18T09:48:10Z

@kaiyan-sheng dashboards updated

(cherry picked from commit 5375f0a) # Conflicts: # metricbeat/docs/fields.asciidoc

(cherry picked from commit 5375f0a) # Conflicts: # metricbeat/docs/fields.asciidoc Co-authored-by: endorama <526307+endorama@users.noreply.github.com>

After we aligned metrics with Beats naming convention (elastic#27974) the docs has not been adequately updated.

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

Co-authored-by: DeDe Morton <dede.morton@elastic.co> (cherry picked from commit 923e11f) # Conflicts: # x-pack/metricbeat/module/gcp/dataproc/_meta/docs.asciidoc # x-pack/metricbeat/module/gcp/firestore/_meta/docs.asciidoc

Co-authored-by: DeDe Morton <dede.morton@elastic.co> (cherry picked from commit 923e11f)

…es after #27974 (#33080) Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Edoardo Tenani <526307+endorama@users.noreply.github.com>

…ies after #27974 (#33079) Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Edoardo Tenani <526307+endorama@users.noreply.github.com> Co-authored-by: Edoardo Tenani <edoardo.tenani@elastic.co>

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

endorama added Team: Ingest Team:Integrations Label for the Integrations team labels Sep 16, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 16, 2021

endorama removed the Team: Ingest label Sep 16, 2021

endorama force-pushed the master branch from 62c296e to fcafd7d Compare September 20, 2021 16:06

endorama added the backport-v7.16.0 Automated backport with mergify label Sep 20, 2021

endorama force-pushed the master branch from 4062198 to 502b212 Compare September 22, 2021 10:35

endorama changed the title ~~[Metricbeat] gcp: respect ECS and Beats naming conventions~~ [Metricbeat] gcp: respect Beats naming conventions Sep 22, 2021

endorama force-pushed the master branch from 502b212 to 3000c72 Compare September 22, 2021 13:05

endorama requested a review from kaiyan-sheng September 23, 2021 10:12

endorama marked this pull request as ready for review September 23, 2021 10:17

endorama added the breaking change label Sep 23, 2021

endorama requested review from a team as code owners October 5, 2021 08:16

botelastic bot added the Team:Automation Label for the Observability productivity team label Oct 5, 2021

endorama self-assigned this Oct 6, 2021

endorama closed this Oct 6, 2021

endorama force-pushed the master branch from 2fb569d to 3c1731d Compare October 6, 2021 14:30

endorama reopened this Oct 6, 2021

kaiyan-sheng reviewed Oct 6, 2021

View reviewed changes

x-pack/metricbeat/module/gcp/compute/_meta/fields.yml Outdated Show resolved Hide resolved

v1v removed the request for review from a team October 7, 2021 11:05

endorama force-pushed the master branch from 620a44b to 63db98f Compare October 7, 2021 13:29

endorama added 13 commits October 18, 2021 11:44

gcp dashboard: replace l3.internal.ingress_packets_count.value

c9deff6

gcp dashboard: replace tcp_ssl_proxy.egress_bytes_count.value

052eb6e

gcp dashboard: replace tcp_ssl_proxy.ingress_bytes_count.value

5dbe5e1

gcp dashboard: replace snapshot.backlog_bytes.value

183373a

gcp dashboard: replace snapshot.oldest_message_age.value

c75074e

gcp dashboard: replace subscription.backlog_bytes.value

25f3678

gcp dashboard: replace subscription.oldest_unacked_message_age.value

8cfea88

gcp dashboard: replace topic.send_request_count.value

e72ee8a

gcp dashboard: replace api.request_count.value

eaa825f

gcp dashboard: replace network.received_bytes_count.value

874327f

gcp dashboard: replace network.sent_bytes_count.value

fd042c7

gcp dashboard: replace storage.object_count.value

04f567a

gcp dashboard: replace storage.total_bytes.value

e9cf813

kaiyan-sheng approved these changes Oct 18, 2021

View reviewed changes

endorama merged commit 5375f0a into elastic:master Oct 18, 2021

mergify bot pushed a commit that referenced this pull request Oct 18, 2021

[Metricbeat] gcp: respect Beats naming conventions (#27974)

4b7490f

(cherry picked from commit 5375f0a) # Conflicts: # metricbeat/docs/fields.asciidoc

mergify bot mentioned this pull request Oct 18, 2021

[7.x](backport #27974) [Metricbeat] gcp: respect Beats naming conventions #28510

Merged

endorama added a commit that referenced this pull request Oct 19, 2021

[Metricbeat] gcp: respect Beats naming conventions (#27974)

65989ea

(cherry picked from commit 5375f0a) # Conflicts: # metricbeat/docs/fields.asciidoc

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021

[Metricbeat] gcp: respect Beats naming conventions (elastic#27974)

4c9dc3c

endorama mentioned this pull request Dec 29, 2021

[Metricbeat] gcp: update sample datas #29637

Closed

6 tasks

endorama added a commit to endorama/beats that referenced this pull request Sep 7, 2022

change wording for metricset metrics names

0d1959f

After we aligned metrics with Beats naming convention (elastic#27974) the docs has not been adequately updated.

endorama mentioned this pull request Sep 7, 2022

[GCP] Update fields docs and fix inconsistencies after #27974 #33006

Merged

6 tasks

endorama added a commit that referenced this pull request Sep 14, 2022

[GCP] Update fields docs and fix inconsistencies after #27974 (#33006)

923e11f

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

mergify bot pushed a commit that referenced this pull request Sep 14, 2022

[GCP] Update fields docs and fix inconsistencies after #27974 (#33006)

0a30bd9

Co-authored-by: DeDe Morton <dede.morton@elastic.co> (cherry picked from commit 923e11f)

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

[GCP] Update fields docs and fix inconsistencies after #27974 (#33006)

1275704

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] gcp: respect Beats naming conventions#27974

[Metricbeat] gcp: respect Beats naming conventions#27974
endorama merged 55 commits intoelastic:masterfrom
endorama:master

endorama commented Sep 16, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Sep 16, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Uh oh!

endorama commented Sep 23, 2021

Uh oh!

elasticmachine commented Sep 23, 2021

Uh oh!

kaiyan-sheng commented Sep 23, 2021

Uh oh!

mergify bot commented Oct 5, 2021

Uh oh!

Uh oh!

kaiyan-sheng commented Oct 6, 2021

Uh oh!

endorama commented Oct 7, 2021

Uh oh!

endorama commented Oct 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

endorama commented Sep 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

elasticmachine commented Sep 16, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

endorama commented Sep 23, 2021

Uh oh!

elasticmachine commented Sep 23, 2021

Uh oh!

kaiyan-sheng commented Sep 23, 2021

Uh oh!

mergify bot commented Oct 5, 2021

Uh oh!

Uh oh!

kaiyan-sheng commented Oct 6, 2021

Uh oh!

endorama commented Oct 7, 2021

Uh oh!

endorama commented Oct 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

endorama commented Sep 16, 2021 •

edited

Loading

elasticmachine commented Sep 16, 2021 •

edited by jenkins-beats-ci bot

Loading