title | description | services | author | ms.author | ms.reviewer | ms.topic | ms.date | ms.custom |
---|---|---|---|---|---|---|---|---|
Ingest events from Azure Event Hubs into Azure Monitor Logs (Preview) |
Ingest logs from Event Hubs into Azure Monitor Logs |
azure-monitor |
guywi-ms |
guywild |
ilanawaitser |
tutorial |
12/28/2023 |
references_regions |
Azure Event Hubs is a big data streaming platform that collects events from multiple sources to be ingested by Azure and external services. This article explains how to ingest data directly from an event hub into a Log Analytics workspace.
In this tutorial, you learn how to:
[!div class="checklist"]
- Create a destination table for event hub data in your Log Analytics workspace
- Create a data collection endpoint
- Create a data collection rule
- Grant the data collection rule permissions to the event hub
- Associate the data collection rule with the event hub
To send events from Azure Event Hubs to Azure Monitor Logs, you need these resources, all in the same region:
- Log Analytics workspace where you have at least contributor rights.
- Your Log Analytics workspace needs to be linked to a dedicated cluster or to have a commitment tier.
- Event Hubs namespace that permits public network access. Private Link and Network Security Perimeters (NSP) are currently not supported.
- Event hub with events. You can send events to your event hub by following the steps in Send and receive events in Azure Event Hubs tutorials or by configuring the diagnostic settings of Azure resources.
Azure Monitor currently supports ingestion from Event Hubs in these regions:
Americas | Europe | Middle East | Africa | Asia Pacific |
---|---|---|---|---|
Brazil South | France Central | UAE North | South Africa North | Australia Central |
Brazil Southeast | North Europe | Australia East | ||
Canada Central | Norway East | Australia Southeast | ||
Canada East | Switzerland North | Central India | ||
East US | Switzerland West | East Asia | ||
East US 2 | UK South | Japan East | ||
South Central US | UK West | Jio India West | ||
West US | West Europe | Korea Central | ||
West US 3 | Southeast Asia |
You need your subscription ID, resource group name, workspace name, workspace resource ID, and event hub instance resource ID in subsequent steps:
-
Navigate to your workspace in the Log Analytics workspaces menu and select Properties and copy your Subscription ID, Resource group, and Workspace name. You'll need these details to create resources in this tutorial.
:::image type="content" source="media/ingest-logs-event-hub/create-custom-table-prepare.png" lightbox="media/ingest-logs-event-hub/create-custom-table-prepare.png" alt-text="Screenshot showing Log Analytics workspace overview screen with subscription ID, resource group name, and workspace name highlighted.":::
-
Select JSON to open the Resource JSON screen and copy the workspace's Resource ID. You'll need the workspace resource ID to create a data collection rule.
:::image type="content" source="media/ingest-logs-event-hub/log-analytics-workspace-id.png" lightbox="media/ingest-logs-event-hub/log-analytics-workspace-id.png" alt-text="Screenshot showing the Resource JSON screen with the workspace resource ID highlighted.":::
-
Navigate to your event hub instance, select JSON to open the Resource JSON screen, and copy the event hub instance's Resource ID. You'll need the event hub instance's resource ID to associate the data collection rule with the event hub.
:::image type="content" source="media/ingest-logs-event-hub/event-hub-resource-id.png" lightbox="media/ingest-logs-event-hub/event-hub-resource-id.png" alt-text="Screenshot showing the Resource JSON screen with the event hub resource ID highlighted.":::
Before you can ingest data, you need to set up a destination table. You can ingest data into custom tables and supported Azure tables.
To create a custom table into which to ingest events, in the Azure portal:
-
Select the Cloud Shell button and ensure the environment is set to PowerShell.
:::image type="content" source="media/ingest-logs-event-hub/create-custom-table-open-cloud-shell.png" lightbox="media/ingest-logs-event-hub/create-custom-table-open-cloud-shell.png" alt-text="Screenshot showing how to open Cloud Shell.":::
-
Run this PowerShell command to create the table, providing the table name (
<table_name>
) in the JSON (that too with suffix _CL in case of custom table), and setting the<subscription_id>
,<resource_group_name>
,<workspace_name>
, and<table_name>
values in theInvoke-AzRestMethod -Path
command:$tableParams = @' { "properties": { "schema": { "name": "<table_name>", "columns": [ { "name": "TimeGenerated", "type": "datetime", "description": "The time at which the data was ingested." }, { "name": "RawData", "type": "string", "description": "Body of the event." }, { "name": "Properties", "type": "dynamic", "description": "Additional message properties." } ] } } } '@ Invoke-AzRestMethod -Path "/subscriptions/<subscription_id>/resourcegroups/<resource_group_name>/providers/microsoft.operationalinsights/workspaces/<workspace_name>/tables/<table_name>?api-version=2021-12-01-preview" -Method PUT -payload $tableParams
Important
- Column names must start with a letter and can consist of up to 45 alphanumeric characters and underscores (
_
). _ResourceId
,id
,_ResourceId
,_SubscriptionId
,TenantId
,Type
,UniqueId
, andTitle
are reserved column names.- Column names are case-sensitive. Make sure to use the correct case in your data collection rule.
To collect data with a data collection rule, you need a data collection endpoint:
-
Create a data collection endpoint.
[!IMPORTANT] Create the data collection endpoint in the same region as your Log Analytics workspace.
-
From the data collection endpoint's Overview screen, select JSON View.
:::image type="content" source="media/ingest-logs-event-hub/data-collection-endpoint-details.png" lightbox="media/ingest-logs-event-hub/data-collection-rule-details.png" alt-text="Screenshot that shows the data collection endpoint Overview screen.":::
-
Copy the Resource ID for the data collection rule. You'll use this information in the next step.
:::image type="content" source="media/ingest-logs-event-hub/data-collection-rule-json-view.png" lightbox="media/ingest-logs-event-hub/data-collection-rule-json-view.png" alt-text="Screenshot that shows the data collection endpoint JSON view.":::
Azure Monitor uses data collection rules to define which data to collect, how to transform that data, and where to send the data.
To create a data collection rule in the Azure portal:
-
In the portal's search box, type in template and then select Deploy a custom template.
:::image type="content" source="media/tutorial-workspace-transformations-api/deploy-custom-template.png" lightbox="media/tutorial-workspace-transformations-api/deploy-custom-template.png" alt-text="Screenshot to deploy custom template.":::
-
Select Build your own template in the editor.
:::image type="content" source="media/tutorial-workspace-transformations-api/build-custom-template.png" lightbox="media/tutorial-workspace-transformations-api/build-custom-template.png" alt-text="Screenshot to build template in the editor.":::
-
Paste the Resource Manager template below into the editor and then select Save.
:::image type="content" source="media/tutorial-workspace-transformations-api/edit-template.png" lightbox="media/tutorial-workspace-transformations-api/edit-template.png" alt-text="Screenshot to edit Resource Manager template.":::
Notice the following details in the data collection rule below:
-
identity
- Defines which type of managed identity to use. In our example, we use system-assigned identity. You can also configure user-assigned managed identity. -
dataCollectionEndpointId
- Resource ID of the data collection endpoint. -
streamDeclarations
- Defines which data to ingest from the event hub (incoming data). The stream declaration can't be modified.TimeGenerated
- The time at which the data was ingested from event hub to Azure Monitor Logs.RawData
- Body of the event. For more information, see Read events.Properties
- User properties from the event. For more information, see Read events.
-
datasources
- Specifies the event hub consumer group and the stream to which you ingest the data. -
destinations
- Specifies all of the destinations where the data will be sent. You can ingest data to one or more Log Analytics workspaces. -
dataFlows
- Matches the stream with the destination workspace and specifies the transformation query and the destination table. In our example, we ingest data to the custom table we created previously. You can also ingest into a supported Azure table. -
transformKql
- Specifies a transformation to apply to the incoming data (stream declaration) before it's sent to the workspace. In our example, we settransformKql
tosource
, which doesn't modify the data from the source in any way, because we're mapping incoming data to a custom table we've created specifically with the corresponding schema. If you're ingesting data to a table with a different schema or to filter data before ingestion, define a data collection transformation.
{ "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "dataCollectionRuleName": { "type": "string", "metadata": { "description": "Specifies the name of the data collection Rule to create." } }, "workspaceResourceId": { "type": "string", "metadata": { "description": "Specifies the Azure resource ID of the Log Analytics workspace to use." } }, "endpointResourceId": { "type": "string", "metadata": { "description": "Specifies the Azure resource ID of the data collection endpoint to use." } }, "tableName": { "type": "string", "metadata": { "description": "Specifies the name of the table in the workspace." } }, "consumerGroup": { "type": "string", "metadata": { "description": "Specifies the consumer group of event hub." }, "defaultValue": "$Default" } }, "resources": [ { "type": "Microsoft.Insights/dataCollectionRules", "name": "[parameters('dataCollectionRuleName')]", "location": "[resourceGroup().location]", "apiVersion": "2022-06-01", "identity": { "type": "systemAssigned" }, "properties": { "dataCollectionEndpointId": "[parameters('endpointResourceId')]", "streamDeclarations": { "Custom-MyEventHubStream": { "columns": [ { "name": "TimeGenerated", "type": "datetime" }, { "name": "RawData", "type": "string" }, { "name": "Properties", "type": "dynamic" } ] } }, "dataSources": { "dataImports": { "eventHub": { "consumerGroup": "[parameters('consumerGroup')]", "stream": "Custom-MyEventHubStream", "name": "myEventHubDataSource1" } } }, "destinations": { "logAnalytics": [ { "workspaceResourceId": "[parameters('workspaceResourceId')]", "name": "MyDestination" } ] }, "dataFlows": [ { "streams": [ "Custom-MyEventHubStream" ], "destinations": [ "MyDestination" ], "transformKql": "source", "outputStream": "[concat('Custom-', parameters('tableName'))]" } ] } } ] }
-
-
On the Custom deployment screen, specify a Subscription and Resource group to store the data collection rule and then provide values for the parameters defined in the template, including:
- Region - Region for the data collection rule. Populated automatically based on the resource group you select.
- Data Collection Rule Name - Give the rule a name.
- Workspace Resource ID - See Collect required information.
- Endpoint Resource ID - Generated when you create the data collection endpoint.
- Table Name - The name of the destination table. In our example, and whenever you use a custom table, the table name must end with the suffix _CL. If you're ingesting data to an Azure table, enter the table name - for example,
Syslog
- without the suffix. - Consumer Group - By default, the consumer group is set to
$Default
. If needed, change the value to a different event hub consumer group.
:::image type="content" source="media/ingest-logs-event-hub/data-collection-rule-custom-template-deployment.png" lightbox="media/ingest-logs-event-hub/data-collection-rule-custom-template-deployment.png" alt-text="Screenshot showing the Custom Template Deployment screen with the deployment values for the data collection rule set up in this tutorial.":::
-
Select Review + create and then Create when you review the details.
-
When the deployment is complete, expand the Deployment details box, and select your data collection rule to view its details. Select JSON View.
:::image type="content" source="media/ingest-logs-event-hub/data-collection-rule-details.png" lightbox="media/ingest-logs-event-hub/data-collection-rule-details.png" alt-text="Screenshot that shows the Data Collection Rule Overview screen.":::
-
Copy the Resource ID for the data collection rule. You'll use this information in the next step.
:::image type="content" source="media/ingest-logs-event-hub/data-collection-rule-json-view.png" lightbox="media/ingest-logs-event-hub/data-collection-rule-json-view.png" alt-text="Screenshot that shows the data collection rule JSON view.":::
To configure your data collection rule to support user-assigned identity, in the example above, replace:
"identity": {
"type": "systemAssigned"
},
with:
"identity": {
"type": "userAssigned",
"userAssignedIdentities": {
"<identity_resource_Id>": {
}
}
},
To find the <identity_resource_Id>
value, navigate to your user-assigned managed identity resource in the Azure portal, select JSON to open the Resource JSON screen and copy the managed identity's Resource ID.
:::image type="content" source="media/ingest-logs-event-hub/managed-identity-resource-id.png" lightbox="media/ingest-logs-event-hub/managed-identity-resource-id.png" alt-text="Screenshot showing Resource JSON screen with the managed identity resource ID highlighted.":::
To ingest data into a supported Azure table:
-
In the data collection rule, change
outputStream
:From:
"outputStream": "[concat('Custom-', parameters('tableName'))]"
To:
"outputStream": "outputStream": "[concat(Microsoft-', parameters('tableName'))]"
-
In
transformKql
, define a transformation that sends the ingested data into the target columns in the destination Azure table.
With managed identity, you can give any event hub, or Event Hubs namespace, permission to send events to the data collection rule and data collection endpoint you created. When you grant the permissions to the Event Hubs namespace, all event hubs within the namespace inherit the permissions.
-
From the event hub or Event Hubs namespace in the Azure portal, select Access Control (IAM) > Add role assignment.
:::image type="content" source="media/ingest-logs-event-hub/event-hub-add-role-assignment.png" lightbox="media/ingest-logs-event-hub/event-hub-add-role-assignment.png" alt-text="Screenshot that shows the Access control screen for the data collection rule.":::
-
Select Azure Event Hubs Data Receiver and select Next.
:::image type="content" source="media/ingest-logs-event-hub/event-hub-data-receiver-role-assignment.png" lightbox="media/ingest-logs-event-hub/event-hub-data-receiver-role-assignment.png" alt-text="Screenshot that shows the Add Role Assignment screen for the event hub with the Azure Event Hubs Data Receiver role highlighted.":::
-
Select Managed identity for Assign access to and click Select members. Select Data collection rule, search for your data collection rule by name, and click Select.
:::image type="content" source="media/ingest-logs-event-hub/assign-access-to-managed-identity.png" lightbox="media/ingest-logs-event-hub/assign-access-to-managed-identity.png" alt-text="Screenshot that shows how to assign access to managed identity.":::
-
Select Review + assign and verify the details before saving your role assignment.
:::image type="content" source="media/ingest-logs-event-hub/event-hub-add-role-assignment-save.png" lightbox="media/ingest-logs-event-hub/event-hub-add-role-assignment-save.png" alt-text="Screenshot that shows the Review and Assign tab of the Add Role Assignment screen.":::
The final step is to associate the data collection rule to the event hub from which you want to collect events.
You can associate a single data collection rule with multiple event hubs that share the same consumer group and ingest data to the same stream. Alternatively, you can associate a unique data collection rule to each event hub.
Important
You must associate at least one data collection rule to the event hub to ingest data from an event hub. When you delete all data collection rule associations related to the event hub, you'll stop ingesting data from the event hub.
To create a data collection rule association in the Azure portal:
-
In the Azure portal's search box, type in template and then select Deploy a custom template.
-
Select Build your own template in the editor.
-
Paste the Resource Manager template below into the editor and then select Save.
{ "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "eventHubResourceID": { "type": "string", "metadata": { "description": "Specifies the Azure resource ID of the event hub to use." } }, "associationName": { "type": "string", "metadata": { "description": "The name of the association." } }, "dataCollectionRuleID": { "type": "string", "metadata": { "description": "The resource ID of the data collection rule." } } }, "resources": [ { "type": "Microsoft.Insights/dataCollectionRuleAssociations", "apiVersion": "2021-09-01-preview", "scope": "[parameters('eventHubResourceId')]", "name": "[parameters('associationName')]", "properties": { "description": "Association of data collection rule. Deleting this association will break the data collection for this event hub.", "dataCollectionRuleId": "[parameters('dataCollectionRuleId')]" } } ] }
-
On the Custom deployment screen, specify a Subscription and Resource group to store the data collection rule association and then provide values for the parameters defined in the template, including:
- Region - Populated automatically based on the resource group you select.
- Event Hub Instance Resource ID - See Collect required information.
- Association Name - Give the association a name.
- Data Collection Rule ID - Generated when you create the data collection rule.
:::image type="content" source="media/ingest-logs-event-hub/data-collection-rule-association-custom-template-deployment.png" lightbox="media/ingest-logs-event-hub/data-collection-rule-association-custom-template-deployment.png" alt-text="Screenshot showing the Custom Template Deployment screen with the deployment values for the data collection rule association set up in this tutorial.":::
-
Select Review + create and then Create when you review the details.
Now that you've associated the data collection rule with your event hub, Azure Monitor Logs will ingest all existing events whose retention period hasn't expired and all new events.
To check your destination table for ingested events:
-
Navigate to your workspace and select Logs.
-
Write a simple query in the query editor and select Run:
<table_name>
You should see events from your event hub.
:::image type="content" source="media/ingest-logs-event-hub/log-analytics-query-results-with-events.png" lightbox="media/ingest-logs-event-hub/log-analytics-query-results-with-events.png" alt-text="Screenshot showing the results of a simple query on a custom table. The results consist of events ingested from an event hub.":::
In this tutorial, you created the following resources:
- Custom table
- Data collection endpoint
- Data collection rule
- Data collection rule association
Evaluate whether you still need these resources. Delete the resources you don't need individually, or delete all of these resources at once by deleting the resource group. Resources you leave running can cost you money.
To stop ingesting data from the event hub, delete all data collection rule associations related to the event hub, or delete the data collection rules themselves. These actions also reset event hub checkpointing.
- If you transfer a subscription between Microsoft Entra directories, you need to follow the steps described in Known issues with managed identities for Azure resources to continue ingesting data.
- You can ingest messages of up to 64 KB from Event Hubs to Azure Monitor Logs.
Learn more about to: