x-pack/filebeat/module/microsoft: drop empty api responses by efd6 · Pull Request #34893 · elastic/beats

efd6 · 2023-03-22T03:37:14Z

What does this PR do?

Allows dropping of responses that hold no event data.

This is a port of elastic/integrations#5164.

Why is it important?

Empty responses waste space and do not conform to the expected format of the datastream.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Fixes Filebeat module for microsoft needs to drop empty api responses #34786

Use cases

Screenshots

Logs

This is a port of elastic/integrations#5164.

elasticmachine · 2023-03-22T04:50:29Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-03-28T00:08:19.700+0000
Duration: 74 min 46 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2875
Skipped	172
Total	3047

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-03-22T08:10:11Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh

LGTM. If there's not a sample log test case it would be nice to have one (but actually now that I think about this, I have no idea if the test harness would support verifying drops).

andrewkroh · 2023-03-27T23:28:30Z


 response.split:
  target: body.value
+  ignore_empty_value: true


Do we have any test cases in our logs that have an empty value field?

We do in the integration, but not here. I looked into adding one, but it's not as easy as it is in the integrations.

I've come up with an approach that serves to test this. It's not pretty, but it works.

efd6 · 2023-03-28T00:08:52Z

 {"id":"da637291048912199236_1126926584","incidentId":11,"investigationId":7,"assignedTo":null,"severity":"Medium","status":"New","classification":null,"determination":null,"investigationState":"TerminatedByUser","detectionSource":"WindowsDefenderAtp","category":"DefenseEvasion","threatFamilyName":null,"title":"Suspicious process injection observed","description":"A process abnormally injected code into another process, As a result, unexpected code may be running in the target process memory. Injection is often used to hide malicious code execution within a trusted process. \nAs a result, the target process may exhibit abnormal behaviors such as opening a listening port or connecting to a command and control server.","alertCreationTime":"2020-06-30T09:08:11.1084877Z","firstEventTime":"2020-06-30T09:04:56.8490679Z","lastEventTime":"2020-06-30T09:45:39.5484377Z","lastUpdateTime":"2020-06-30T15:29:44.7733333Z","resolvedTime":null,"machineId":"543bc5a964f417c11f6277d5bf9489f0d","computerDnsName":"testserver4","rbacGroupName":null,"aadTenantId":"123543-d66c-4c7e-9e30-40034eb7c6f3","relatedUser":{"userName":"administrator1","domainName":"TestServer4"},"comments":[],"evidence":{"entityType":"Process","sha1":"b6d237154f2e528f0b503b58b025862d66b02b73","sha256":"a92056d772260b39a876d01552496b2f8b4610a0b1e084952fe1176784e2ce77","fileName":"notepad.exe","filePath":"C:\\Windows\\System32","processId":4104,"processCommandLine":"\"notepad.exe\"","processCreationTime":"2020-06-30T09:45:38.9784654Z","parentProcessId":6012,"parentProcessCreationTime":"2020-06-30T09:04:51.487396Z","ipAddress":null,"url":null,"accountName":null,"domainName":null,"userSid":null,"aadUserId":null,"userPrincipalName":null}}
 {"id":"da637291048912199236_1126926584","incidentId":11,"investigationId":7,"assignedTo":null,"severity":"Medium","status":"New","classification":null,"determination":null,"investigationState":"TerminatedByUser","detectionSource":"WindowsDefenderAtp","category":"DefenseEvasion","threatFamilyName":null,"title":"Suspicious process injection observed","description":"A process abnormally injected code into another process, As a result, unexpected code may be running in the target process memory. Injection is often used to hide malicious code execution within a trusted process. \nAs a result, the target process may exhibit abnormal behaviors such as opening a listening port or connecting to a command and control server.","alertCreationTime":"2020-06-30T09:08:11.1084877Z","firstEventTime":"2020-06-30T09:04:56.8490679Z","lastEventTime":"2020-06-30T09:45:39.5484377Z","lastUpdateTime":"2020-06-30T15:29:44.7733333Z","resolvedTime":null,"machineId":"53425a964f417c11f6277d5bf9489f0d","computerDnsName":"testserver4","rbacGroupName":null,"aadTenantId":"43521344-d66c-4c7e-9e30-40034eb7c6f3","relatedUser":{"userName":"administrator1","domainName":"TestServer4"},"comments":[],"evidence":{"entityType":"User","sha1":null,"sha256":null,"fileName":null,"filePath":null,"processId":null,"processCommandLine":null,"processCreationTime":null,"parentProcessId":null,"parentProcessCreationTime":null,"ipAddress":null,"url":null,"accountName":"administrator1","domainName":"TestServer4","userSid":"S-1-5-21-46152456-1367606905-4031241297-500","aadUserId":null,"userPrincipalName":null}}
 {"id":"da637291063515066999_-2102938302","incidentId":12,"investigationId":9,"assignedTo":"Automation","severity":"Informational","status":"Resolved","classification":null,"determination":null,"investigationState":"Benign","detectionSource":"WindowsDefenderAv","category":"Malware","threatFamilyName":null,"title":"'Mountsi' malware was detected","description":"Malware and unwanted software are undesirable applications that perform annoying, disruptive, or harmful actions on affected machines. Some of these undesirable applications can replicate and spread from one machine to another. Others are able to receive commands from remote attackers and perform activities associated with cyber attacks.\n\nThis detection might indicate that the malware was stopped from delivering its payload. However, it is prudent to check the machine for signs of infection.","alertCreationTime":"2020-06-30T09:32:31.4579225Z","firstEventTime":"2020-06-30T09:31:22.5729558Z","lastEventTime":"2020-06-30T09:46:15.0876676Z","lastUpdateTime":"2020-06-30T11:13:12.9Z","resolvedTime":"2020-06-30T11:13:12.2680434Z","machineId":"t4563234bc5a964f417c11f6277d5bf9489f0d","computerDnsName":"testserver4","rbacGroupName":null,"aadTenantId":"1234543-d66c-4c7e-9e30-40034eb7c6f3","relatedUser":null,"comments":[],"evidence":{"entityType":"File","sha1":"ffb1670c6c6a9c5b4c5cea8b6b8e68d62e7ff281","sha256":"fd46705c4f67a8ef16e76259ca6d6253241e51a1f8952223145f92aa1907d356","fileName":"amsistream-1D89ECED25A52AB98B76FF619B7BA07A","filePath":null,"processId":null,"processCommandLine":null,"processCreationTime":null,"parentProcessId":null,"parentProcessCreationTime":null,"ipAddress":null,"url":null,"accountName":null,"domainName":null,"userSid":null,"aadUserId":null,"userPrincipalName":null}}
+{"value":[],"note":"THIS MESSAGE SHOULD NOT END UP IN THE EXPECTS JSON FILE."}


If this ends up in the result we error out. This is the case if the pipeline drop processors are not in place.

if you would like here is a sample of an empty event for your test case

{"@timestamp":"2023-03-31T23:06:13.507Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.4.3","pipeline":"filebeat-8.4.3-microsoft-defender_atp-pipeline"},"agent":{"type":"filebeat ","version":"8.4.3","ephemeral_id":"382fa7a9-285e-47b9-bedc-badbd225347e","id":"e20495e9-3358-4155-b4c9-9a519775d234","name":"collector-01"},"fileset":{"name":"defender_atp"},"service":{"typ e":"microsoft"},"ecs":{"version":"1.12.0"},"input":{"type":"httpjson"},"organization":{"name":"acme"},"json":{"@odata.context":"https://api.securitycenter.windows.com/api/$metadata#Alerts"," value":[]},"event":{"created":"2023-03-31T23:06:13.507Z","dataset":"microsoft.defender_atp","module":"microsoft"},"message":"{\"@odata.context\":\"https://api.securitycenter.windows.com/api/ $metadata#Alerts\",\"value\":[]}","tags":["acme","defender-atp","forwarded"]}

This is a port of elastic/integrations#5164. (cherry picked from commit 429bdeb)

…34947) This is a port of elastic/integrations#5164. (cherry picked from commit 429bdeb) Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

This is a port of elastic/integrations#5164.

efd6 added Filebeat Filebeat Team:Security-External Integrations 8.8-candidate bugfix backport-v8.7.0 Automated backport with mergify labels Mar 22, 2023

efd6 self-assigned this Mar 22, 2023

botelastic Bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 22, 2023

x-pack/filebeat/module/microsoft: drop empty api responses

140d22c

This is a port of elastic/integrations#5164.

efd6 force-pushed the 34786-microsoft branch from 85d1e7f to 140d22c Compare March 22, 2023 03:37

efd6 marked this pull request as ready for review March 22, 2023 08:10

efd6 requested a review from a team as a code owner March 22, 2023 08:10

andrewkroh approved these changes Mar 27, 2023

View reviewed changes

add tests

689f89c

efd6 commented Mar 28, 2023

View reviewed changes

efd6 merged commit 429bdeb into elastic:main Mar 28, 2023

mergify Bot pushed a commit that referenced this pull request Mar 28, 2023

x-pack/filebeat/module/microsoft: drop empty api responses (#34893)

6e9543e

This is a port of elastic/integrations#5164. (cherry picked from commit 429bdeb)

mergify Bot mentioned this pull request Mar 28, 2023

[8.7](backport #34893) x-pack/filebeat/module/microsoft: drop empty api responses #34947

Merged

CyberTaoFlow mentioned this pull request Mar 31, 2023

Filebeat module for microsoft needs to drop empty api responses #34786

Closed

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

x-pack/filebeat/module/microsoft: drop empty api responses (#34893)

8d86d3d

This is a port of elastic/integrations#5164.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x-pack/filebeat/module/microsoft: drop empty api responses#34893

x-pack/filebeat/module/microsoft: drop empty api responses#34893
efd6 merged 2 commits intoelastic:mainfrom
efd6:34786-microsoft

efd6 commented Mar 22, 2023 •

edited

Loading

Uh oh!

elasticmachine commented Mar 22, 2023 •

edited by jenkins-beats-ci Bot

Loading

Build stats

Test stats 🧪

Uh oh!

elasticmachine commented Mar 22, 2023

Uh oh!

andrewkroh left a comment

Uh oh!

andrewkroh Mar 27, 2023

Uh oh!

efd6 Mar 27, 2023 •

edited

Loading

Uh oh!

efd6 Mar 28, 2023 •

edited

Loading

Uh oh!

efd6 Mar 28, 2023

Uh oh!

CyberTaoFlow Mar 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

efd6 commented Mar 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

elasticmachine commented Mar 22, 2023 • edited by jenkins-beats-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

elasticmachine commented Mar 22, 2023

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh Mar 27, 2023

Choose a reason for hiding this comment

Uh oh!

efd6 Mar 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 Mar 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 Mar 28, 2023

Choose a reason for hiding this comment

Uh oh!

CyberTaoFlow Mar 31, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

efd6 commented Mar 22, 2023 •

edited

Loading

elasticmachine commented Mar 22, 2023 •

edited by jenkins-beats-ci Bot

Loading

efd6 Mar 27, 2023 •

edited

Loading

efd6 Mar 28, 2023 •

edited

Loading