Add azure storage metricset by narph · Pull Request #15342 · elastic/beats

narph · 2020-01-06T18:43:28Z

Should handle:
#14548
#14549
#14551
#14552

Configuration:

- module: azure
  metricsets:
  - storage
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s
  resources:
  - resource_group: [""]
    service_type: ["blob", "file"]
  - resource_id: [""]

Output:

 {
        "_index" : "metricbeat-8.0.0-2020.01.06-000001",
        "_type" : "_doc",
        "_id" : "JRT4e28BQe78yNOEWjXE",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2020-01-06T17:42:00.000Z",
          "agent" : {
            "type" : "metricbeat",
            "ephemeral_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "hostname" : "DESKTOP-RFOOE09",
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "version" : "8.0.0"
          },
          "ecs" : {
            "version" : "1.2.0"
          },
          "host" : {
            "os" : {
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.535 (WinBuild.160101.0800)",
              "build" : "18362.535",
              "platform" : "windows",
              "version" : "10.0"
            },
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "hostname" : "DESKTOP-RFOOE09",
            "architecture" : "x86_64",
            "name" : "DESKTOP-RFOOE09"
          },
          "service" : {
            "type" : "azure"
          },
          "cloud" : {
            "provider" : "azure",
            "region" : "westeurope"
          },
          "event" : {
            "dataset" : "azure.storage",
            "module" : "azure",
            "duration" : 9280652400
          },
          "metricset" : {
            "name" : "storage",
            "period" : 300000
          },
          "azure" : {
            "subscription_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "namespace" : "Microsoft.Storage/storageAccounts",
            "storage" : {
              "successe2elatency" : {
                "avg" : 26.516666666666666
              },
              "availability" : {
                "avg" : 100
              },
              "transactions" : {
                "total" : 310
              },
              "ingress" : {
                "total" : 7723808.0
              },
              "egress" : {
                "total" : 1473483.0
              },
              "successserverlatency" : {
                "avg" : 16.883333333333333
              }
            },
            "resource" : {
              "type" : "Microsoft.Storage/storageAccounts",
              "group" : "obs-infrastructure",
              "name" : "obsdiaglinux"
            }
          }
        },
        "sort" : [
          1578332520000
        ]
      },
      {
        "_index" : "metricbeat-8.0.0-2020.01.06-000001",
        "_type" : "_doc",
        "_id" : "KBT4e28BQe78yNOEWjXE",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2020-01-06T17:42:00.000Z",
          "agent" : {
            "version" : "8.0.0",
            "type" : "metricbeat",
            "ephemeral_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "hostname" : "DESKTOP-RFOOE09",
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b"
          },
          "service" : {
            "type" : "azure"
          },
          "cloud" : {
            "provider" : "azure",
            "region" : "westeurope"
          },
          "event" : {
            "dataset" : "azure.storage",
            "module" : "azure",
            "duration" : 9281658700
          },
          "metricset" : {
            "name" : "storage",
            "period" : 300000
          },
          "azure" : {
            "subscription_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "namespace" : "Microsoft.Storage/storageAccounts/tableServices",
            "storage" : {
                "successserverlatency" : {
                  "avg" : 115.76666666666667
                },
                "successe2elatency" : {
                  "avg" : 118.63333333333334
                },
                "availability" : {
                  "avg" : 100
                },
                "transactions" : {
                  "total" : 30
                },
                "ingress" : {
                  "total" : 2248122.0
                },
                "egress" : {
                  "total" : 1209276.0
              }
            },
            "resource" : {
              "name" : "obsdiagwin",
              "type" : "Microsoft.Storage/storageAccounts",
              "group" : "obs-infrastructure"
            }
          },
          "ecs" : {
            "version" : "1.2.0"
          },
          "host" : {
            "name" : "DESKTOP-RFOOE09",
            "hostname" : "DESKTOP-RFOOE09",
            "architecture" : "x86_64",
            "os" : {
              "platform" : "windows",
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.535 (WinBuild.160101.0800)",
              "build" : "18362.535"
            },
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b"
          }
        },
        "sort" : [
          1578332520000
        ]
      },

PR also contains:

fix on generated fields and modified dashboard
change on the way metric values are retrieved and events are sent. The events are sent now after all metric values are retrieved for each resource. In the past, events were sent only after all metric value calls are returned for all resources in the config (this took some time if the number of resources is larger)

ChrsMark · 2020-01-13T10:27:38Z

-	for i, metric := range client.Resources.Metrics {
+	for _, metric := range metrics {
 		// select period to collect metrics, will double the interval value in order to retrieve any missing values
+		//if timegrain is larger than intervalx2 then interval will be assigned the timegrain value


Just a suggestion here: wouldn't it be better to be calculated in a different helping method? :)

kaiyan-sheng

Overall, data.json and the code looks good to me. But I haven't tested this PR locally.

narph · 2020-01-13T17:32:03Z

Overall, data.json and the code looks good to me. But I haven't tested this PR locally.

thanks for the review, the PR will be in the test plan.

* create metricset * work in prog * Work on metricset * Add serviceType config option * Fix tests * Add func to retrieve on all dimensions * work on different intervals * revert custom event format * work on reducing api calls * fix fields and dashboards, work on reducing amount of api calls * Add json example * refactor * refactor * fix tests * fix tests * feedback (cherry picked from commit ad8c503)

* Add azure storage metricset (#15342) * create metricset * work in prog * Work on metricset * Add serviceType config option * Fix tests * Add func to retrieve on all dimensions * work on different intervals * revert custom event format * work on reducing api calls * fix fields and dashboards, work on reducing amount of api calls * Add json example * refactor * refactor * fix tests * fix tests * feedback (cherry picked from commit ad8c503) * Fix changelog

narph added 3 commits December 19, 2019 12:14

create metricset

b9ac9b8

work in prog

e25da30

Work on metricset

5f1bb1f

narph requested a review from a team as a code owner January 6, 2020 18:43

houndci-bot reviewed Jan 6, 2020

View reviewed changes

Comment thread x-pack/metricbeat/module/azure/storage/client_helper.go Outdated

narph self-assigned this Jan 6, 2020

houndci-bot reviewed Jan 6, 2020

View reviewed changes

Comment thread x-pack/metricbeat/module/azure/storage/client_helper.go Outdated

narph added Metricbeat Metricbeat Team:Integrations Label for the Integrations team [zube]: In Progress labels Jan 6, 2020

narph added 4 commits January 7, 2020 11:24

Merge branch 'master' into azure-storage

74c556f

Add serviceType config option

e31ded5

Fix tests

f111fbc

Add func to retrieve on all dimensions

8f5704f

houndci-bot reviewed Jan 7, 2020

View reviewed changes

Comment thread x-pack/metricbeat/module/azure/data.go

houndci-bot reviewed Jan 7, 2020

View reviewed changes

Comment thread x-pack/metricbeat/module/azure/data.go

narph added 5 commits January 8, 2020 16:03

work on different intervals

1101521

revert custom event format

23c05f8

work on reducing api calls

be2ceac

fix fields and dashboards, work on reducing amount of api calls

69a3c55

Add json example

5e83336

narph added [zube]: In Review and removed [zube]: In Progress labels Jan 9, 2020

ChrsMark reviewed Jan 13, 2020

View reviewed changes

Comment thread x-pack/metricbeat/module/azure/data.go Outdated

ChrsMark reviewed Jan 13, 2020

View reviewed changes

Comment thread x-pack/metricbeat/module/azure/monitor/client_helper.go

narph added 3 commits January 13, 2020 11:57

Merge branch 'master' into azure-storage

0a4b3f5

refactor

639fab1

refactor

861847c

narph requested a review from ChrsMark January 13, 2020 11:50

fix tests

947d8bd

narph added needs_backport PR is waiting to be backported to other branches. v7.6.0 labels Jan 13, 2020

fix tests

e792c4b

kaiyan-sheng reviewed Jan 13, 2020

View reviewed changes

Comment thread x-pack/metricbeat/module/azure/azure.go Outdated

kaiyan-sheng reviewed Jan 13, 2020

View reviewed changes

Comment thread x-pack/metricbeat/module/azure/client.go

kaiyan-sheng approved these changes Jan 13, 2020

View reviewed changes

feedback

c721565

narph merged commit ad8c503 into elastic:master Jan 13, 2020

narph deleted the azure-storage branch January 13, 2020 20:22

zube Bot added [zube]: Done and removed [zube]: In Review labels Jan 13, 2020

narph mentioned this pull request Jan 13, 2020

Cherry-pick #15342 to 7.x: Add azure storage metricset #15514

Merged

narph added test-plan Add this PR to be manual test plan and removed needs_backport PR is waiting to be backported to other branches. labels Jan 14, 2020

andresrc unassigned narph Jan 15, 2020

kaiyan-sheng self-assigned this Jan 15, 2020

andresrc removed the [zube]: Done label Jan 17, 2020

This was referenced Jan 21, 2020

[Metricbeat] Azure storage file metricset #14549

Closed

[Metricbeat] Azure storage blob metricset #14548

Closed

[Metricbeat] Azure storage queue metricset #14551

Closed

[Metricbeat] Azure storage table metricset #14552

Closed

kaiyan-sheng added the test-plan-ok This PR passed manual testing label Jan 24, 2020

urso added the release-highlight label Feb 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add azure storage metricset#15342

Add azure storage metricset#15342
narph merged 18 commits intoelastic:masterfrom
narph:azure-storage

narph commented Jan 6, 2020 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChrsMark Jan 13, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaiyan-sheng left a comment

Uh oh!

narph commented Jan 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

narph commented Jan 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChrsMark Jan 13, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

narph commented Jan 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

narph commented Jan 6, 2020 •

edited

Loading