Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DomainEntity_EmailUrlInfo TI detection creates memory issues on large data sets #11340

Open
MSJosh opened this issue Oct 25, 2024 · 1 comment
Assignees

Comments

@MSJosh
Copy link

MSJosh commented Oct 25, 2024

KQL Query for DomainEntity_EmailUrlInfo is not optimized for larger data sets leading to memory issues in LAW.

Source- https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/Threat%20Intelligence/Analytic%20Rules/DomainEntity_EmailUrlInfo.yaml

To Reproduce
Run Existing query against data set that is TB a day in email traffic. Analysts will get excessive memory alerts and in cases for larger deployments
SEM00023 errors as materialize exceeds the limit and fails the query.
Image

Expected behavior
Utilize improved query performance by dropping materialize and split data up more to improve performance.

Here is a fix which improves overall performance of query by over 90 percent on same data sets with same results.

https://github.com/MSJosh/documentation/blob/main/Sentinel/Misc.%20KQL/TI/DomainEntity_EmailUrlInfo.yaml

@v-rusraut
Copy link
Contributor

Hi @MSJosh , thanks for flagging this issue, we will investigate this issue and get back to you with some updates. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants