Skip to content

Cherry-pick #15342 to 7.x: Add azure storage metricset#15514

Merged
narph merged 2 commits intoelastic:7.xfrom
narph:backport_15342_7.x
Jan 14, 2020
Merged

Cherry-pick #15342 to 7.x: Add azure storage metricset#15514
narph merged 2 commits intoelastic:7.xfrom
narph:backport_15342_7.x

Conversation

@narph
Copy link
Copy Markdown
Contributor

@narph narph commented Jan 13, 2020

Cherry-pick of PR #15342 to 7.x branch. Original message:

Should handle:
#14548
#14549
#14551
#14552

Configuration:

- module: azure
  metricsets:
  - storage
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s
  resources:
  - resource_group: [""]
    service_type: ["blob", "file"]
  - resource_id: [""]  

Output:

 {
        "_index" : "metricbeat-8.0.0-2020.01.06-000001",
        "_type" : "_doc",
        "_id" : "JRT4e28BQe78yNOEWjXE",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2020-01-06T17:42:00.000Z",
          "agent" : {
            "type" : "metricbeat",
            "ephemeral_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "hostname" : "DESKTOP-RFOOE09",
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "version" : "8.0.0"
          },
          "ecs" : {
            "version" : "1.2.0"
          },
          "host" : {
            "os" : {
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.535 (WinBuild.160101.0800)",
              "build" : "18362.535",
              "platform" : "windows",
              "version" : "10.0"
            },
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "hostname" : "DESKTOP-RFOOE09",
            "architecture" : "x86_64",
            "name" : "DESKTOP-RFOOE09"
          },
          "service" : {
            "type" : "azure"
          },
          "cloud" : {
            "provider" : "azure",
            "region" : "westeurope"
          },
          "event" : {
            "dataset" : "azure.storage",
            "module" : "azure",
            "duration" : 9280652400
          },
          "metricset" : {
            "name" : "storage",
            "period" : 300000
          },
          "azure" : {
            "subscription_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "namespace" : "Microsoft.Storage/storageAccounts",
            "storage" : {
              "successe2elatency" : {
                "avg" : 26.516666666666666
              },
              "availability" : {
                "avg" : 100
              },
              "transactions" : {
                "total" : 310
              },
              "ingress" : {
                "total" : 7723808.0
              },
              "egress" : {
                "total" : 1473483.0
              },
              "successserverlatency" : {
                "avg" : 16.883333333333333
              }
            },
            "resource" : {
              "type" : "Microsoft.Storage/storageAccounts",
              "group" : "obs-infrastructure",
              "name" : "obsdiaglinux"
            }
          }
        },
        "sort" : [
          1578332520000
        ]
      },
      {
        "_index" : "metricbeat-8.0.0-2020.01.06-000001",
        "_type" : "_doc",
        "_id" : "KBT4e28BQe78yNOEWjXE",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2020-01-06T17:42:00.000Z",
          "agent" : {
            "version" : "8.0.0",
            "type" : "metricbeat",
            "ephemeral_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "hostname" : "DESKTOP-RFOOE09",
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b"
          },
          "service" : {
            "type" : "azure"
          },
          "cloud" : {
            "provider" : "azure",
            "region" : "westeurope"
          },
          "event" : {
            "dataset" : "azure.storage",
            "module" : "azure",
            "duration" : 9281658700
          },
          "metricset" : {
            "name" : "storage",
            "period" : 300000
          },
          "azure" : {
            "subscription_id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b",
            "namespace" : "Microsoft.Storage/storageAccounts/tableServices",
            "storage" : {
                "successserverlatency" : {
                  "avg" : 115.76666666666667
                },
                "successe2elatency" : {
                  "avg" : 118.63333333333334
                },
                "availability" : {
                  "avg" : 100
                },
                "transactions" : {
                  "total" : 30
                },
                "ingress" : {
                  "total" : 2248122.0
                },
                "egress" : {
                  "total" : 1209276.0
              }
            },
            "resource" : {
              "name" : "obsdiagwin",
              "type" : "Microsoft.Storage/storageAccounts",
              "group" : "obs-infrastructure"
            }
          },
          "ecs" : {
            "version" : "1.2.0"
          },
          "host" : {
            "name" : "DESKTOP-RFOOE09",
            "hostname" : "DESKTOP-RFOOE09",
            "architecture" : "x86_64",
            "os" : {
              "platform" : "windows",
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Pro",
              "kernel" : "10.0.18362.535 (WinBuild.160101.0800)",
              "build" : "18362.535"
            },
            "id" : "297967de-3c3a-40eb-a7c6-e7b06fbe774b"
          }
        },
        "sort" : [
          1578332520000
        ]
      },

PR also contains:

  • fix on generated fields and modified dashboard
  • change on the way metric values are retrieved and events are sent. The events are sent now after all metric values are retrieved for each resource. In the past, events were sent only after all metric value calls are returned for all resources in the config (this took some time if the number of resources is larger)

* create metricset

* work in prog

* Work on metricset

* Add serviceType config option

* Fix tests

* Add func to retrieve on all dimensions

* work on different intervals

* revert custom event format

* work on reducing api calls

* fix fields and dashboards, work on reducing amount of api calls

* Add json example

* refactor

* refactor

* fix tests

* fix tests

* feedback

(cherry picked from commit ad8c503)
@narph narph requested a review from a team as a code owner January 13, 2020 20:32
@narph narph self-assigned this Jan 13, 2020
Comment thread CHANGELOG.next.asciidoc Outdated

*Metricbeat*

- Add AWS SQS metricset. {pull}10684[10684] {issue}10053[10053]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These extra changelog entries need to be removed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that was fast, just updated changelog

@narph narph merged commit 4d0bacd into elastic:7.x Jan 14, 2020
@narph narph deleted the backport_15342_7.x branch January 14, 2020 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Integrations Label for the Integrations team v7.6.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants