The developer at Mystique Unicorn are looking for a way to alert them based on the logs their app is generating. Their app creates an log entry that has the following fields.
{
"request_id": "f5570d0f-5389-4bce-98c3-7ed5380eefb2",
"event_type": "inventory_event",
"store_id": 6,
"cust_id": 206,
"category": "Camera",
"sku": 100657,
"price": 2.52,
"qty": 18,
"discount": 8.2,
"gift_wrap": true,
"variant": "red",
"priority_shipping": true,
"TimeGenerated": "2021-04-25T12:52:40.938937",
"contact_me": "github.com/miztiik"
}
They would like to be alerted when there are suspicious activity. For example, consider this, If there is an deeply disocunted(>90%
) order and the number of quantites ordered is higher then (>x
) and priority shipping had been requested, then they would like to notified for further analysis.
Can you show them how the can get started?
We will use Azure Monitor Agent1 to collect the application logs from a specified location(for exampe: /var/log/miztiik*.json
) and push them to a Log Analytics Workspace3. To receive the logs in the correct format we need to define a custom table5,6 and a data collection rule. The rule will collect logs from our source(VM App), transform the logs using Kusto Query Language(KQL) to a structure defined in the custom table and store them in the sink(log analytics workspace). Later we will deploy a KQL query to setup an alert that meets the developer requirements.
This repository holds the bicep code to deploy the necessary infrastructure for this demo,
- Virtual Machine(RedHat) inside a Vnet with public access.
- Bootstrapped with custom libs using
customData
script.
- Bootstrapped with custom libs using
- Log Analytics Workspace ( Analytics tier)
- Custom Table
- Data Collection Endpoint - With public access
- Data Collection Rule
- Configured with the log location, KQL query and custom table
- Azure Monitort Alert Rule using KQL
-
This demo, instructions, scripts and bicep template is designed to be run in
westeurope
. With few or no modifications you can try it out in other regions as well(Not covered here).- π Azure CLI Installed & Configured - Get help here
- π Bicep Installed & Configured - Get help here
- π VS Code & Bicep Extenstions - Get help here
-
-
Get the application code
https://github.com/miztiik/azure-alert-on-custom-metrics cd azure-alert-on-custom-metrics
-
-
Let check you have Azure Cli working with
# You should have azure cli preinstalled az account show
You should see an output like this,
{ "environmentName": "AzureCloud", "homeTenantId": "16b30820b6d3", "id": "1ac6fdbff37cd9e3", "isDefault": true, "managedByTenants": [], "name": "YOUR-SUBS-NAME", "state": "Enabled", "tenantId": "16b30820b6d3", "user": { "name": "miztiik@", "type": "user" } }
-
Let us walk through each of the stacks,
-
Stack: Main Bicep The params required for the modules are in
params.json
. Do modify them to suit your need.(Especially theadminPassword.secureString
for the VM. You are strongly encouraged to Just-In-Time access or use SSH key instead of password based authentication). The helper deployment scriptdeploy.sh
will deploy themain.bicep
file. This will inturn call the modules for Resource Group(RG), VNet & Virtual Machine, Log analytics workspace, Custom table, Data collection endpoint, Data collection rule(DCR) and finally attach the DCR to the VM.Alert Module We will create a KQL query to identify order events that match the given criteria -
lumberyardTable001_CL | where discount > 90 and qty >1 and priority_shipping==true
We can also refine it to aggregate only the events that had been received in the last
5
minutes like this,lumberyardTable001_CL | where discount > 90 and qty > 1 and priority_shipping == true | where Timestamp > ago(5m)
But since have the option of choosing an evaluation frequency within the alert configuration, we will us that. Remember
evaluationFrequency
is smaller than theaggregation
granularity, this will result in sliding window evaluation. The code is configured forevaluationFrequency
of1
minute and aggregationwindowSize
of5
minutes, resulting in creating a sliding window. This may also mean we can trigger the alarm multiple times for the event. It is a conscious choice, to balance signal-to-noise towards more noise(aka false-positives) when we are trying to catch suspicious activity.sh deploy.sh
This stack will create almost all of the resoureces. The RG is created as a deployment at the subscription level. The other resources are deployments at resource group level.
-
Virtual Machine A Redhat vm is bootstrapped using
customData
to install python3, git. We will use a bash scriptgenerate_data.sh
. It will generate a json conformant log in this location/var/log/miztiik-yyyy-MM-dd.json
for example/var/log/miztiik-2023-04-03.json
. All new entries are appended to the file withnew-line
being the delimitter as expected by Azure Custom Logs.{"request_id": "08b6b258-c6d6-44ff-802e-69d52422095a", "event_type": "inventory_event", "store_id": 45, "cust_id": 4416, "category": "Camera", "sku": 17293, "price": 44.61, "qty": 15 , "discount": 73.2, "gift_wrap": 0, "variant": "MystiqueAutomatoon", "priority_shipping": 1, "TimeGenerated": "2023-04-03T10:54:23", "contact_me": "github.com/miztiik" } {"request_id": "60051c39-6817-4f52-bd8f-59cc9a38be0f", "event_type": "inventory_event", "store_id": 64, "cust_id": 4081, "category": "Camera", "sku": 17815, "price": 50.92, "qty": 16 , "discount": 29.2, "gift_wrap": 1, "variant": "MystiqueAutomatoon", "priority_shipping": 0, "TimeGenerated": "2023-04-03T10:54:24", "contact_me": "github.com/miztiik" } {"request_id": "f340f343-fc0f-40a1-b9f5-2abdd14b929d", "event_type": "inventory_event", "store_id": 35, "cust_id": 338, "category": "Camera", "sku": 16158, "price": 53.76, "qty": 13, "discount": 34.3, "gift_wrap": 1, "variant": "MystiqueAutomatoon", "priority_shipping": 0, "TimeGenerated": "2023-04-03T10:54:25", "contact_me": "github.com/miztiik" }
Formatting a single log line entry properly, will look like this,
{ "request_id": "f5570d0f-5389-4bce-98c3-7ed5380eefb2", "event_type": "inventory_event", "store_id": 6, "cust_id": 206, "category": "Camera", "sku": 100657, "price": 2.52, "qty": 18, "discount": 8.2, "gift_wrap": true, "variant": "red", "priority_shipping": true, "TimeGenerated": "2021-04-25T12:52:40.938937", "contact_me": "github.com/miztiik" }
After successfully deploying the stack, Check the
Resource Groups/Deployments
section for the resources. -
-
-
Connect to the VM
Connect to the VM using Just-In-Time access. This should allow your IP/IP list to securelly connect to your VM using private IP address. Use the password you updated in the
params.json
ssh miztiik@publicIP
-
Generate Log Data
For some reason the customData, bootstrap script fails, will fix it later. Until then manually copy the
generate_data.sh
from this repo to the server. _You may need elevated privileges to create log file in/var/log
.sudo su cd /var/log/ # Copy generate_data.sh to the server sh generate_data.sh &
The script by default is configured to write one log entry every
5
seconds and generate a total100000
log lines to/var/log/miztiik*.json
-
Query in Azure Log Analytics Workspace
Login to Azure Log Anaytics portal. You should be able to find it under
/Miztiik_Enterprises_Custom_Logs_To_Monitor_xxx/providers/Microsoft.OperationalInsights/workspaces/lumberyard-payGTier-xxx",
. Navigate to logs, select the custom table(lumberyardTablexxx_CL
) &Run
-
-
Troubleshooting Azure Monitor Agent
-
In case if you are facing issues, log into the VM and check the logs
/var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/
for potential causes. -
From the portal, you can query to find if the agent is reporting to Azure Monitor
Heartbeat | where Category == "Azure Monitor Agent" and Computer == "<computer-name>" | take 10
Check if the AMA is running,
systemctl status azuremonitoragent
-
-
Here we have demonstrated how to configure alerts based on custom logs. You can extend the solution to setup alerts actions groups & receive email notifications. An example of email notification is showin below,
If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well
- Resources created during Deploying The Application
- Any other custom resources, you have created for this demo
# Delete from resource group
az group delete --name Miztiik_Enterprises_xxx --yes
# Follow any on-screen prompt
This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.
This repository aims to show how to Bicep to new developers, Solution Architects & Ops Engineers in Azure.
Thank you for your interest in contributing to our project. Whether it is a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here
Buy me a coffee β.
- Azure Docs: Azure Monitor Agent Overview
- Azure Docs: Azure Monitor Agents -vs- Other Agatnes Overview
- Azure Docs: Collect text logs with Azure Monitor Agent
- Azure Docs: Collect text logs with Log Analytics Agent
- Azure Docs: Custom Table
- Azure Docs: Workspace Table Creation
- Azure Docs: Log Data Plan Basic-vs-Analytics Comparison
- Azure Docs: Custom Log Ingestion using python SDK
- Azure Docs: Custom Metrics to Azure Monitor
- Azure Docs: Check Logs in Azure Logs
- Azure Docs: Troubleshoot Azure Monitor Agent
Level: 200