From 349c90707c9b77299db862c84f59ea33366f1399 Mon Sep 17 00:00:00 2001 From: Scott Addie Date: Wed, 28 Jul 2021 12:05:31 -0500 Subject: [PATCH 1/2] Edit pass on Monitor Query README file --- sdk/monitor/azure-monitor-query/README.md | 199 +++++++++++----------- 1 file changed, 95 insertions(+), 104 deletions(-) diff --git a/sdk/monitor/azure-monitor-query/README.md b/sdk/monitor/azure-monitor-query/README.md index e7acaf4e1305..6776139db74b 100644 --- a/sdk/monitor/azure-monitor-query/README.md +++ b/sdk/monitor/azure-monitor-query/README.md @@ -1,126 +1,133 @@ # Azure Monitor Query client library for Python -Azure Monitor helps you maximize the availability and performance of your applications and services. It delivers a comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments. +Azure Monitor helps you maximize the availability and performance of your apps. It delivers a comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments. -All data collected by Azure Monitor fits into one of two fundamental types, metrics and logs. Metrics are numerical values that describe some aspect of a system at a particular point in time. They are lightweight and capable of supporting near real-time scenarios. Logs contain different kinds of data organized into records with different sets of properties for each type. Telemetry such as events and traces are stored as logs in addition to performance data so that it can all be combined for analysis. +All data collected by Azure Monitor fits into one of two fundamental types: + +- **Metrics** - Numerical values that describe some aspect of a system at a particular time. They're lightweight and can support near real-time scenarios. +- **Logs** - Disparate types of data organized into records with different sets of properties for each type. Performance data and telemetry such as events, exceptions, and traces are stored as logs. + +To programmatically analyze these data sources, the Azure Monitor Query client library can be used. [Source code][python-query-src] | [Package (PyPI)][python-query-pypi] | [API reference documentation][python-query-ref-docs] | [Product documentation][python-query-product-docs] | [Samples][python-query-samples] | [Changelog][python-query-changelog] ## Getting started ### Prerequisites -* Python 2.7, or 3.6 or later is required to use this package. -* You must have an [Azure subscription][azure_subscription]. +- Python 2.7, or 3.6 or later. +- An [Azure subscription][azure_subscription]. ### Install the package + Install the Azure Monitor Query client library for Python with [pip][pip]: ```bash pip install azure-monitor-query --pre ``` -### Create and Authenticate the client -A **token credential** is necessary to instantiate both the LogsQueryClient and the MetricsQueryClient object. +### Create the client -```Python -from azure.monitor.query import LogsQueryClient -from azure.identity import DefaultAzureCredential +To interact with the Azure Monitor service, create an instance of a token credential. Use that instance when creating a `LogsQueryClient` or `MetricsQueryClient`. +#### Synchronous clients -credential = DefaultAzureCredential() +Consider the following example, which creates synchronous clients for both logs and metrics querying: -client = LogsQueryClient(credential) +```python +from azure.identity import DefaultAzureCredential +from azure.monitor.query import LogsQueryClient, MetricsQueryClient + +credential = DefaultAzureCredential() +logs_client = LogsQueryClient(credential) +metrics_client = MetricsQueryClient(credential) ``` -```Python -from azure.monitor.query import MetricsQueryClient -from azure.identity import DefaultAzureCredential +#### Asynchronous clients +The asynchronous forms of the query client APIs are found in the `.aio`-suffixed namespace. For example: -credential = DefaultAzureCredential() +```python +from azure.identity.aio import DefaultAzureCredential +from azure.monitor.query.aio import LogsQueryClient, MetricsQueryClient -client = MetricsQueryClient(credential) +credential = DefaultAzureCredential() +async_logs_client = LogsQueryClient(credential) +async_metrics_client = MetricsQueryClient(credential) ``` ## Key concepts ### Logs -Azure Monitor Logs is a feature of Azure Monitor that collects and organizes log and performance data from monitored -resources. Data from different sources such as platform logs from Azure services, log and performance data from virtual -machines agents, and usage and performance data from applications can be consolidated into a single workspace so they -can be analyzed together using a sophisticated query language that's capable of quickly analyzing millions of records. -You may perform a simple query that just retrieves a specific set of records or perform sophisticated data analysis to -identify critical patterns in your monitoring data. +Azure Monitor Logs collects and organizes log and performance data from monitored resources. Data from different sources can be consolidated into a single workspace. Examples of data sources include: + +- Platform logs from Azure services. +- Log and performance data from virtual machine agents. +- Usage and performance data from apps. -#### Log Analytics workspaces +#### Azure Log Analytics workspaces -Data collected by Azure Monitor Logs is stored in one or more Log Analytics workspaces. The workspace defines the -geographic location of the data, access rights defining which users can access data, and configuration settings such as -the pricing tier and data retention. +Data collected by Azure Monitor Logs is stored in one or more [Log Analytics workspaces](https://docs.microsoft.com/azure/azure-monitor/logs/data-platform-logs#log-analytics-workspaces). The workspace defines the: -You must create at least one workspace to use Azure Monitor Logs. A single workspace may be sufficient for all of your -monitoring data, or may choose to create multiple workspaces depending on your requirements. For example, you might have -one workspace for your production data and another for testing. +- Geographic location of the data. +- Access rights defining which users can access data. +- Configuration settings, such as the pricing tier and data retention. #### Log queries -Data is retrieved from a Log Analytics workspace using a log query which is a read-only request to process data and -return results. Log queries are written -in [Kusto Query Language (KQL)](https://docs.microsoft.com/azure/data-explorer/kusto/query/), which is the same query -language used by Azure Data Explorer. You can write log queries in Log Analytics to interactively analyze their results, -use them in alert rules to be proactively notified of issues, or include their results in workbooks or dashboards. -Insights include prebuilt queries to support their views and workbooks. +Data from the disparate sources can be analyzed together using [Kusto Query Language (KQL)](https://docs.microsoft.com/azure/data-explorer/kusto/query/)—the same query language used by [Azure Data Explorer](https://docs.microsoft.com/azure/data-explorer/data-explorer-overview). Data is retrieved from a Log Analytics workspace using a KQL query—a read-only request to process data and return results. For more information, see [Log queries in Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/logs/log-query-overview). ### Metrics -Azure Monitor Metrics is a feature of Azure Monitor that collects numeric data from monitored resources into a time -series database. Metrics are numerical values that are collected at regular intervals and describe some aspect of a -system at a particular time. Metrics in Azure Monitor are lightweight and capable of supporting near real-time scenarios -making them particularly useful for alerting and fast detection of issues. You can analyze them interactively with -metrics explorer, be proactively notified with an alert when a value crosses a threshold, or visualize them in a -workbook or dashboard. +Azure Monitor Metrics collects numeric data from monitored resources into a time series database. Metrics are collected at regular intervals and describe some aspect of a system at a particular time. Metrics in Azure Monitor are lightweight and can support near real-time scenarios. They're useful for alerting and fast detection of issues. Metrics can be: + +- Analyzed interactively with [Metrics Explorer](https://docs.microsoft.com/azure/azure-monitor/essentials/metrics-getting-started). +- Used to receive notifications with an alert when a value crosses a threshold. +- Visualized in a workbook or dashboard. #### Metrics data structure -Data collected by Azure Monitor Metrics is stored in a time-series database which is optimized for analyzing -time-stamped data. Each set of metric values is a time series with the following properties: +Each set of metric values is a time series with the following characteristics: - The time the value was collected -- The resource the value is associated with +- The resource associated with the value - A namespace that acts like a category for the metric - A metric name - The value itself -- Some metrics may have multiple dimensions as described in Multi-dimensional metrics. Custom metrics can have up to 10 - dimensions. - +- Some metrics may have multiple dimensions as described in multi-dimensional metrics. Custom metrics can have up to 10 dimensions. ## Examples -### Get logs for a query +- [Single logs query](#single-logs-query) + - [Specify duration](#specify-duration) + - [Query multiple workspaces](#query-multiple-workspaces) +- [Batch logs query](#batch-logs-query) +- [Set logs query timeout](#set-logs-query-timeout) +- [Query metrics](#query-metrics) +- [Handle metrics response](#handle-metrics-response) + - [Example of handling response](#example-of-handling-response) + +### Single logs query -This sample shows getting a log query. to handle the response and view it in a tabular form, the [pandas](https://pypi.org/project/pandas/) library is used. Please look at the samples if you don't want to use the pandas library. +This example shows getting a log query. To handle the response and view it in a tabular form, the [pandas](https://pypi.org/project/pandas/) library is used. See the [samples][python-query-samples] if you choose not to use pandas. -#### Specifying a duration +#### Specify duration -The `duration` parameter can be used to specify the time duration for which to query the data. This can also be accompanied with either `start_time` or `end_time`. If either `start_time` or `end_time` are not provided, the current time is taken as the end time. -Alternatively, the `start_time` and `end_time` keyword arguments can be provided together instead of `duration` parameter like shown in the example below. +The `duration` parameter specifies the time duration for which to query the data. This argument can also be accompanied with either `start_time` or `end_time`. If either `start_time` or `end_time` aren't provided, the current time is used as the end time. As an alternative, the `start_time` and `end_time` arguments can be provided together instead of the `duration` argument. For example: -```Python +```python import os import pandas as pd from datetime import datetime from azure.monitor.query import LogsQueryClient from azure.identity import DefaultAzureCredential - -credential = DefaultAzureCredential() - +credential = DefaultAzureCredential() client = LogsQueryClient(credential) # Response time trend -# request duration over the last 12 hours. +# request duration over the last 12 hours query = """AppRequests | summarize avgRequestDuration=avg(DurationMs) by bin(TimeGenerated, 10m), _ResourceId""" @@ -140,12 +147,11 @@ for table in response.tables: print(df) ``` -#### Single query on multiple workspaces. +#### Query multiple workspaces -`additional_workspaces` param can be used to pass a list of workspaces that are included in the query when querying a single query over multiple workspaces. These can be qualified workspace names, workspace IDs or Azure resource IDs. -Note that a primary `workspace_id` must still be provided when querying additional workspaces like in the snippet below. +The `additional_workspaces` parameter can be used to pass a list of workspaces that are included in the query when querying a single query over multiple workspaces. The parameter's list items can be qualified workspace names, workspace IDs, or Azure resource IDs. A primary workspace ID must be provided when querying multiple workspaces, as in the following example: -```Python +```python client.query( , query, @@ -153,20 +159,18 @@ Note that a primary `workspace_id` must still be provided when querying addition ) ``` -### Get Logs for multiple queries +### Batch logs query -This sample shows sending multiple queries at the same time using batch query API. For each query, a `LogQueryRequest` object can be used. Alternatively, a dictionary can be used as well. +The following example demonstrates sending multiple queries at the same time using batch query API. The queries can either be represented as a list of `LogQueryRequest` objects or a dictionary. This example uses the former approach. -```Python +```python import os from datetime import timedelta import pandas as pd from azure.monitor.query import LogsQueryClient, LogsQueryRequest from azure.identity import DefaultAzureCredential - -credential = DefaultAzureCredential() - +credential = DefaultAzureCredential() client = LogsQueryClient(credential) requests = [ @@ -199,19 +203,17 @@ for rsp in response: print(df) ``` -### Get logs with server timeout +### Set logs query timeout -This sample shows setting a server timeout in seconds. A GateWay timeout is raised if the query takes more time than the mentioned timeout. The default is 180 seconds and can be set uptio 10 minutes (600 seconds). +The following example shows setting a server timeout in seconds. A gateway timeout is raised if the query takes more time than the mentioned timeout. The default is 180 seconds and can be set up to 10 minutes (600 seconds). -```Python +```python import os import pandas as pd from azure.monitor.query import LogsQueryClient from azure.identity import DefaultAzureCredential - -credential = DefaultAzureCredential() - +credential = DefaultAzureCredential() client = LogsQueryClient(credential) response = client.query( @@ -221,11 +223,11 @@ response = client.query( ) ``` -### Get Metrics +### Query metrics -This example shows getting the metrics for an EventGrid subscription. The resource URI is that of an eventgrid topic. +The following example gets metrics for an Event Grid subscription. The resource URI is that of an event grid topic. -**Note** The resource URI must be that of the resource for which metrics are being queried. It's normally of the format, `/subscriptions//resourceGroups//providers//topics/`. +The resource URI must be that of the resource for which metrics are being queried. It's normally of the format `/subscriptions//resourceGroups//providers//topics/`. To find the resource URI: @@ -233,15 +235,13 @@ To find the resource URI: 2. From the **Overview** blade, select the **JSON View** link. 3. In the resulting JSON, copy the value of the `id` property. -```Python +```python import os from datetime import timedelta from azure.monitor.query import MetricsQueryClient from azure.identity import DefaultAzureCredential - -credential = DefaultAzureCredential() - +credential = DefaultAzureCredential() client = MetricsQueryClient(credential) metrics_uri = os.environ['METRICS_RESOURCE_URI'] @@ -259,16 +259,9 @@ for metric in response.metrics: print(metric_value.time_stamp) ``` -### Handling the response for metrics +### Handle metrics response -The metrics query API returns a `MetricsResult` object. This `MetricsResult` object has multiple properties, including: a list of `Metric` type objects, `interval`, `namespace`, and `timespan`. - -This can be fetched using the `metrics` param. - -Each of the items in this list is a `Metric` object that contains a list of `TimeSeriesElement` objects. Each `TimeSeriesElement` contains `data` and `metadata_values` properties. - - -#### Object heirarchy of the response +The metrics query API returns a `MetricsResult` object. The `MetricsResult` object contains properties such as a list of `Metric`-typed objects, `interval`, `namespace`, and `timespan`. The `Metric` objects list can be accessed using the `metrics` param. Each `Metric` object in this list contains a list of `TimeSeriesElement` objects. Each `TimeSeriesElement` contains `data` and `metadata_values` properties. In visual form, the object hierarchy of the response resembles the following structure: ``` MetricsResult @@ -287,15 +280,15 @@ MetricsResult |---data (list of data points represented by `MetricValue` objects) ``` -#### Sample usage of the API to handle response. +#### Example of handling response -```Python +```python import os from datetime import datetime, timedelta from azure.monitor.query import MetricsQueryClient, AggregationType from azure.identity import DefaultAzureCredential -credential = DefaultAzureCredential() +credential = DefaultAzureCredential() client = MetricsQueryClient(credential) metrics_uri = os.environ['METRICS_RESOURCE_URI'] @@ -322,33 +315,31 @@ for metric in response.metrics: ## Troubleshooting -- Enable `azure.monitor.query` logger to collect traces from the library. +Enable the `azure.monitor.query` logger to collect traces from the library. ### General + Monitor Query client library will raise exceptions defined in [Azure Core][azure_core_exceptions]. ### Logging -This library uses the standard -[logging][python_logging] library for logging. -Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO -level. -### Optional Configuration +This library uses the standard [logging][python_logging] library for logging. Basic information about HTTP sessions, such as URLs and headers, is logged at the `INFO` level. + +### Optional configuration -Optional keyword arguments can be passed in at the client and per-operation level. -The azure-core [reference documentation][azure_core_ref_docs] -describes available configurations for retries, logging, transport protocols, and more. +Optional keyword arguments can be passed in at the client and per-operation level. The `azure-core` [reference documentation][azure_core_ref_docs] describes available configurations for retries, logging, transport protocols, and more. ## Next steps ### Additional documentation -For more extensive documentation on Azure Monitor Query, see the [Monitor Query documentation][python-query-product-docs] on docs.microsoft.com. +For more extensive documentation, see the [Azure Monitor Query documentation][python-query-product-docs]. ## Contributing + This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [cla.microsoft.com][cla]. -When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. +When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repositories using our CLA. This project has adopted the [Microsoft Open Source Code of Conduct][code_of_conduct]. For more information see the [Code of Conduct FAQ][coc_faq] or contact [opencode@microsoft.com][coc_contact] with any additional questions or comments. @@ -366,7 +357,7 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con [azure_core_exceptions]: https://aka.ms/azsdk/python/core/docs#module-azure.core.exceptions [python_logging]: https://docs.python.org/3/library/logging.html [azure_core_ref_docs]: https://aka.ms/azsdk/python/core/docs -[azure_subscription]: https://azure.microsoft.com/free/ +[azure_subscription]: https://azure.microsoft.com/free/python/ [cla]: https://cla.microsoft.com [code_of_conduct]: https://opensource.microsoft.com/codeofconduct/ From 3a523216f137aba16be4a0762bbf0462d9708ac1 Mon Sep 17 00:00:00 2001 From: Scott Addie Date: Wed, 28 Jul 2021 12:10:21 -0500 Subject: [PATCH 2/2] Relocate timeout section --- sdk/monitor/azure-monitor-query/README.md | 42 +++++++++++------------ 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/sdk/monitor/azure-monitor-query/README.md b/sdk/monitor/azure-monitor-query/README.md index 6776139db74b..d34c6b37881c 100644 --- a/sdk/monitor/azure-monitor-query/README.md +++ b/sdk/monitor/azure-monitor-query/README.md @@ -102,8 +102,8 @@ Each set of metric values is a time series with the following characteristics: - [Single logs query](#single-logs-query) - [Specify duration](#specify-duration) - [Query multiple workspaces](#query-multiple-workspaces) + - [Set logs query timeout](#set-logs-query-timeout) - [Batch logs query](#batch-logs-query) -- [Set logs query timeout](#set-logs-query-timeout) - [Query metrics](#query-metrics) - [Handle metrics response](#handle-metrics-response) - [Example of handling response](#example-of-handling-response) @@ -159,6 +159,26 @@ The `additional_workspaces` parameter can be used to pass a list of workspaces t ) ``` +#### Set logs query timeout + +The following example shows setting a server timeout in seconds. A gateway timeout is raised if the query takes more time than the mentioned timeout. The default is 180 seconds and can be set up to 10 minutes (600 seconds). + +```python +import os +import pandas as pd +from azure.monitor.query import LogsQueryClient +from azure.identity import DefaultAzureCredential + +credential = DefaultAzureCredential() +client = LogsQueryClient(credential) + +response = client.query( + os.environ['LOG_WORKSPACE_ID'], + "range x from 1 to 10000000000 step 1 | count", + server_timeout=1, + ) +``` + ### Batch logs query The following example demonstrates sending multiple queries at the same time using batch query API. The queries can either be represented as a list of `LogQueryRequest` objects or a dictionary. This example uses the former approach. @@ -203,26 +223,6 @@ for rsp in response: print(df) ``` -### Set logs query timeout - -The following example shows setting a server timeout in seconds. A gateway timeout is raised if the query takes more time than the mentioned timeout. The default is 180 seconds and can be set up to 10 minutes (600 seconds). - -```python -import os -import pandas as pd -from azure.monitor.query import LogsQueryClient -from azure.identity import DefaultAzureCredential - -credential = DefaultAzureCredential() -client = LogsQueryClient(credential) - -response = client.query( - os.environ['LOG_WORKSPACE_ID'], - "range x from 1 to 10000000000 step 1 | count", - server_timeout=1, - ) -``` - ### Query metrics The following example gets metrics for an Event Grid subscription. The resource URI is that of an event grid topic.