Bug: Get-UAL collection crashes because of a duplicate key

[0xffr]

Hi,

I attempted to collect UAL logs from my tenant using the Get-UAL cmdlet, but this fails with the following error:

PS ____________ > Get-UAL -Output SOF-ELK
=== Starting Unified Audit Log Collection ===
[...]
[INFO] Found 38606 audit logs between 2024-08-03T12:28:20Z and 2024-10-05T05:27:20Z
[...]
[INFO] Retrieved 3606 records (Total: 38606 / 38606)
ConvertFrom-Json : Cannot convert the JSON string because a dictionary that was converted from the string contains the duplicated keys 'EventName' and 'eventName'.
At C:\Program Files\WindowsPowerShell\Modules\Microsoft-Extractor-Suite\3.0.0\Scripts\Get-UAL.ps1:533 char:43
+ ...                        $_.AuditData = $_.AuditData | ConvertFrom-Json
+                                                          ~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [ConvertFrom-Json], InvalidOperationException
    + FullyQualifiedErrorId : DuplicateKeysInJsonString,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

I actually found the AuditData blob, which actually causes this error:

{
	"ApplicationId": 12260,
	"ApplicationName": "Microsoft Azure",
	"ActivityLogData": {
		"EventTime": "2024-10-02T19:32:39",
		"ActionType": "Write StorageAccounts",
		"Application": "Microsoft Azure",
		"ApplicationId": 12260,
		"AppInstanceId": 0,
		"ActivityType": "Basic",
		"ObjectType": "Resource",
		"ObjectId": "/subscriptions/REMOVED/resourcegroups/REMOVED/providers/Microsoft.Storage/storageAccounts/testfiles",
		"HiddenObjectSearchColumn": null,
		"RawActivityData": {
			"AccountMoniker": "MdsResourceStackRPFGermanyWC",
			"AccountMonikerLocation": "germanywestcentral",
			"EventEnvironment": "diagnosticsprod",
			"EventId": 400,
			"EventName": "EventServiceEntries",
			"EventNamespace": "csmGermanyWCRPF",
			"EventVersion": "Ver27v0",
			"GDSEnqueueTimeUtc": "2024-10-02 19:34:03Z",
			"Level": 4,
			"OriginalEnqueueTimeUtc": "2024-10-02 19:34:01Z",
			"Pid": 10208,
			"PreciseTimeStamp": "2024-10-02T19:32:39",
			"ProviderName": "Microsoft-WindowsAzure-Frontdoor",
			"ReleaseVersion": "6.2024.37.6+13089b7.release_2024w37",
			"Role": "FrontdoorWorker",
			"RoleInstance": "FrontdoorWorker-vmss-fdworker2_4",
			"RoleLocation": "Germany West Central",
			"SDSQueryExecutionTimeUtc": "2024-10-02T19:34:05",
			"Stamp": "FDWorker2",
			"TIMESTAMP": "2024-10-02T19:32:39",
			"TaskName": "EventServiceEntry",
			"Tid": 9968,
			"applicationId": "REMOVED",
			"armServiceRequestId": "REMOVED",
			"audience": "https://management.core.windows.net/",
			"customerOperationName": "PUT/SUBSCRIPTIONS/RESOURCEGROUPS/PROVIDERS/MICROSOFT.STORAGE/STORAGEACCOUNTS/",
			"dataBoundary": "Global",
			"eventCategory": "Administrative",
			"eventName": "EndRequest",
			"eventSource": "Microsoft.Resources",
			"eventTimestamp": "2024-10-02T19:32:39",
			"operationName": "Microsoft.Storage/storageAccounts/write",
			"principalPuid": "100320010EFB5039",
			"properties": "{\"statusCode\":\"Accepted\",\"serviceRequestId\":null,\"eventCategory\":\"Administrative\",\"entity\":\"/subscriptions/REMOVED/resourcegroups/REMOVED/providers/Microsoft.Storage/storageAccounts/REMOVED\",\"message\":\"Microsoft.Storage/storageAccounts/write\",\"hierarchy\":\"REMOVED\REMOVED\"}",
			"resourceProvider": "Microsoft.Storage",
			"resourceUri": "/subscriptions/REMOVED/resourcegroups/REMOVED/providers/Microsoft.Storage/storageAccounts/REMOVED",
			"status": "Accepted",
			"subStatus": "Accepted",
		},
		"AccountType": "Regular",
		"_trackingPB": {
			"REMOVED":"REMOVED"
		},
		"ThreatIndicators": [],
		"AuditSource": "Defender for Cloud Apps app connector",
		"SessionData": null,
		"OAuthAppId": "REMOVED",
		"TextClassification": {},
		"Mda_AppstanceId": "12260",
		"Mda_UserGroupId": "",
		"Dlp_AdminUnits": "",
		"BitwiseProducts": 2,
		"Mda_RiskCategory": "",
		"Mda_DiscoveryStreamId ": ""
	},
	"UserId": "REMOVED",
	"ClientIP": "REMOVED",
	"Id": "REMOVED",
	"RecordType": 215,
	"CreationTime": "2024-10-02T19:32:39",
	"Operation": "Azure_WriteStorageAccounts",
	"OrganizationId": "REMOVED",
	"UserType": 0,
	"UserKey": "REMOVED",
	"Workload": "MDADataSecurityService",
	"Version": 1
}

This Blob contains the 'EventName' and 'eventName' keys which ConvertFrom-Json (from here) can't process.
According to the Microsoft documentation, this can be fixed using the -AsHashTable argument. But that argument is only supported by Powershell 6.0 and newer.

Do you see any other generic solution to fix this?

[JoeyInvictus]

Interesting! Why would Microsoft put duplicate keys in their logs..? I've never had this issue before (we never use SOF-ELK), so I've only used the flag in our testing environments. I tried running it a couple of times but unfortunately, I can't reproduce the error.

By the way, looking at your log blob, it doesn't seem like a normal UAL entry but more like Azure to me. No Google hits on the workload MDADataSecurityService, but I assume part of Defender for Cloud Apps app connector.

I guess we could do something dirty like this (not tested), but I'm not sure how often it happens or if there are other fields with the same issue.

elseif ($Output -eq "SOF-ELK") {
    # Encoding is hard-coded to UTF8 as UTF16 causes problems when importing the data into SOF-ELK
    foreach ($item in $allResults) {
        try {
            $item.AuditData | ConvertTo-Json -Compress -Depth 100 | 
                Out-File -Append "$OutputDir/UAL-$sessionID.json" -Encoding UTF8
        }
        catch [System.InvalidOperationException] {
            if ($_.Exception.Message -like "*duplicated keys*") {
                ($item.AuditData -replace '"EventName":', '"EventName_Upper":' -replace '"eventName":', '"eventName_Lower":') |
                    Out-File -Append "$OutputDir/UAL-$sessionID.json" -Encoding UTF8
            }
            else {
                $item.AuditData | Out-File -Append "$OutputDir/UAL-$sessionID.json" -Encoding UTF8
            }
        }
    }
}

I added the blob you provided to $blob and did run a few tests:

$test = $blob | ConvertFrom-Json
ConvertFrom-Json : Cannot convert the JSON string because a dictionary that was converted from the string contains the duplicated keys 'EventName' and 'eventName'.
At line:1 char:17
+ $test = $blob | ConvertFrom-Json
+                 ~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [ConvertFrom-Json], InvalidOperationException
    + FullyQualifiedErrorId : DuplicateKeysInJsonString,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

Replacing the field names with different names seems to get rid of the error.

PS C:\Users\Joey-IR\Documents\GitHub\Microsoft-Extractor-Suite> $modifiedJson = $blob -replace '"EventName":', '"EventName_Upper":' -replace '"eventName":', '"eventName_Lower":'
PS C:\Users\Joey-IR\Documents\GitHub\Microsoft-Extractor-Suite> $test = $modifiedJson | ConvertFrom-Json

Doesn't feel like a good solution but yea..