[SIEM] Endgame Row Renderers: DNS, File (FIM), Network, Security (Authentication), Process

[andrew-goldstein]

[SIEM] Endgame Row Renderers: DNS, File (FIM), Network, Security (Authentication), Process

This PR renders Endgame events via row renderers in the Timeline, per the following screenshot:

The following Endgame event types / subtypes will be rendered via row renderers in the Timeline:

• DNS (dns_event)
  • request_event
• File (FIM) (file_event)
  • file_create_event
  • file_delete_event
• Network (network_event)
  • ipv4_connection_accept_event
  • ipv6_connection_accept_event
  • ipv4_disconnect_received_event
  • ipv6_disconnect_received_event
• Security (Authentication) (security_event)
  • user_logon
  • admin_logon
  • explicit_user_logon
  • user_logoff
• Process (process_event)
  • creation_event
  • termination_event

This PR also adds row rendering support for some non-Endgame events that conform to the Elastic Common Schema (ECS):

• DNS requests
• FIM file creation events
• FIM file deletion events

RELEASE NOTE: To view Endgame events in existing SIEM deployments, you must manually add endgame-* to the SIEM index pattern in Kibana Management > Advanced Settings > SIEM > Elasticsearch indices.

DNS Request events

Endgame DNS events with the following event type and subtype will be rendered in the Timeline via row renderers:

endgame.event_type_full: dns_event and endgame.event_subtype_full: request_event

To view these Endgame DNS events in a timeline, add endgame-* to the SIEM > Elasticsearch indices setting in Kibana Advanced Settings, then paste the query above into a timeline to view events.

Runtime matching criteria

All DNS events, including Endgame and non-Endgame DNS events matching the following criteria will be rendered:

dns.question.type: * and dns.question.name: *

The query above can be executed in a timeline to view all data that will be rendered via the (new) DNS event row renderer.

Sample rendered DNS event

Each field with this formatting will be draggable (to pivot a search) in the row-rendered event:

Arun \ Anvi-Acer @ HD-obe-8bf77f54 asked for clients4.google.com with question type A, which resolved to 10.58.197.78 (response code: NOERROR) via chrome.exe (11620) [ 3008]

Fields in a DNS event

The following fields will be used to render a DNS event:

user.name \ user.domain @ host.name asked for dns.question.name with question type dns.question.type, which resolved to dns.resolved_ip (resp code: dns.response_code) via process.name (process.pid) [ event.code | winlog.event_id]

Note: At the time of this writing, Endgame DNS events do not populate dns.response_code. Row renderers are designed to still render partial results when fields are missing. In this case the following text:

(resp code: dns.response_code)

will NOT be rendered, but the other (populated) fields in the DNS event will be rendered.

Additional Rendering of DNS events by the Netflow row renderer

In addition to being rendered by the new DNS renderer described above, DNS events will also be rendered by the Netflow row renderer.

The Neflow row renderer shows the directionality, protocol, and flow of data between a source and destination

Non-Endgame DNS events

The following screenshot shows a DNS event from packetbeat rendered by the new DNS row renderer:

A non-Endgame DNS event that conforms to ECS

File (FIM) Creation events

Endgame File (FIM) Creation events with the following event type and subtype will be rendered in the Timeline via row renderers:

endgame.event_type_full: file_event and endgame.event_subtype_full: file_create_event

Runtime matching criteria

All file creation events, including Endgame and non-Endgame events matching the following criteria will be rendered:

(event.category: file and event.action: file_create_event) or (event.dataset: file and event.action: created)

Sample rendered File (FIM) Creation event

Arun \ Anvi-Acer @ HD-obe-8bf77f54 created file the-real-index~RFa99cd75.TMP in C:\Users\Arun\AppData\Local\Google\Chrome\User Data\Default\Service Worker\CacheStorage\579544fd7d0441717f082c9eb123588966aa57ac\d81a98b1-59b9-43b2-a228-b3daf7da56df\index-dir\the-real-index~RFa99cd75.TMP via chrome.exe (11620)

Fields in a File (FIM) Creation event

user.name \ user.domain @ host.name created file file.name | endgame.file_name in file.path | endgame.file_path via process.name | endgame.process_name (process.pid | endgame.pid)

File (FIM) Deletion events

Endgame File (FIM) Deletion events with the following event type and subtype will be rendered in the Timeline via row renderers:

endgame.event_type_full: file_event and endgame.event_subtype_full: file_delete_event

Runtime matching criteria

All file deletion events, including Endgame and non-Endgame events matching the following criteria will be rendered:

(event.category: file and event.action: file_delete_event) or (event.dataset: file and event.action: deleted)

Sample rendered File (FIM) Deletion event

SYSTEM \ NT AUTHORITY @ HD-v1s-d2118419 deleted file tmp0000031a in C:\Windows\TEMP\tmp00000404\tmp0000031a via AmSvc.exe (1084)

Fields in a File (FIM) Deletion event

user.name \ user.domain @ host.name deleted file file.name | endgame.file_name in file.path | endgame.file_path via process.name | endgame.process_name (process.pid | endgame.pid)

Network Connection Accepted events

Endgame Network Connection Accepted events with the following event type and subtype will be rendered in the Timeline via row renderers:

(endgame.event_type_full: network_event and endgame.event_subtype_full: ipv4_connection_accept_event) or (endgame.event_type_full: network_event and endgame.event_subtype_full: ipv6_connection_accept_event)

Runtime matching criteria

All Endgame Connection Accepted events, and existing "socket opened" events matching the following criteria will be rendered:

event.action: ipv4_connection_accept_event or event.action: ipv6_connection_accept_event or event.action: socket_opened

Sample rendered Network Connection Accepted event

SYSTEM \ NT AUTHORITY @ HD-gqf-0af7b4fe accepted a connection via AmSvc.exe (1084)

Network Connection Accepted events are also be rendered with the Netflow row renderer, like the event.action: socket_opened events are rendered today. The Network Connection Accepted row renderer displays information about the principal actors in the event (i.e. user.name, host.name, process.name), and the Netflow row renderer displays information about the directionality, source / destination, protocol, etc.

Fields in a Network Connection Accepted event

user.name \ user.domain @ host.name accepted a connection via process.name (process.pid)

Network Disconnect Received events

Endgame Network Disconnect Received events with the following event type and subtype will be rendered in the Timeline via row renderers:

(endgame.event_type_full: network_event and endgame.event_subtype_full: ipv4_disconnect_received_event) or (endgame.event_type_full: network_event and endgame.event_subtype_full: ipv6_disconnect_received_event)

Runtime matching criteria

All Endgame Network Disconnect Received events, and existing "socket closed" events matching the following criteria will be rendered:

event.action: ipv4_disconnect_received_event or event.action: ipv6_disconnect_received_event or event.action: socket_closed

Sample rendered Network Disconnect Received event

SYSTEM \ NT AUTHORITY @ HD-gqf-0af7b4fe disconnected via AmSvc.exe (1084)

The existing row renderer for event.action: socket_closed will be enhanced to display additional fields:

• user.domain
• process.pid

Network Disconnect Received events will also be rendered with the Netflow row renderer, like the event.action: socket_closed events are rendered today. The Network Connection Accepted row renderer displays information about the principal actors in the event (i.e. user.name, host.name, process.name), and the Netflow row renderer displays information about the directionality, source / destination, protocol, etc.

Fields in a Network Disconnect Received event

user.name \ user.domain @ host.name disconnected via process.name (process.pid)

Security (Authentication) User Logon events

Endgame Security (Authentication) User Logon events with the following event type and subtype will be rendered in the Timeline via row renderers:

endgame.event_type_full: security_event and endgame.event_subtype_full: user_logon

Runtime matching criteria

Security (Authentication) User Logon events matching the following criteria will be rendered:

event.category: authentication and event.action: user_logon

Sample rendered Security (Authentication) User Logon event

SYSTEM \ NT AUTHORITY @ HD-v1s-d2118419 successfully logged in using logon type 5 - Service (target logon ID 0x3e7) via C:\Windows\System32\services.exe (432) as requested by subject WIN-Q3DOP1UKA81$ \ WORKGROUP (source logon ID 0x3e7) [ 4624]

Fields in an Security (Authentication) User Logon event

user.name \ user.domain @ host.name successfully logged in using logon type endgame.logon_type (target logon ID endgame.target_logon_id) via process.name | process.executable (process.pid) as requested by subject endgame.subject_user_name \ endgame.subject_domain_name (subject logon ID endgame.subject_logon_id) [ event.code | winlog.event_id]

Reference: LogonType Enumerations

The following enumerated values will humanize the numeric endgame.logon_type field:

2 - Interactive
3 - Network
4 - Batch
5 - Service
7 - Unlock
8 - Network Cleartext
9 - New Credentials
10 - Remote Interactive
11 - Cached Interactive

Security (Authentication) Admin Logon events

Endgame Security (Authentication) Admin Logon events with the following event type and subtype will be rendered in the Timeline via row renderers:

endgame.event_type_full: security_event and endgame.event_subtype_full: admin_logon

Runtime matching criteria

Security (Authentication) Admin Logon events matching the following criteria will be rendered:

event.category: authentication and event.action: admin_logon

Sample rendered Security (Authentication) Admin Logon event

With special privileges, SYSTEM \ NT AUTHORITY @ HD-v1s-d2118419 successfully logged in via C:\Windows\System32\services.exe (964) as requested by subject SYSTEM \ NT AUTHORITY (subject logon ID 0x3e7) [ 4672]

Fields in a Security (Authentication) Admin Logon event

With special privileges, user.name \ user.domain @ host.name successfully logged in via process.name | process.executable (process.pid) as requested by subject endgame.subject_user_name \ endgame.subject_domain_name (subject logon ID endgame.subject_logon_id) [ event.code | winlog.event_id]

Security (Authentication) Explicit User Logon events

Endgame Security (Authentication) Explicit User Logon events with the following event type and subtype will be rendered in the Timeline via row renderers:

endgame.event_type_full: security_event and endgame.event_subtype_full: explicit_user_logon

Runtime matching criteria

Security (Authentication) Explicit User Logon events matching the following criteria will be rendered:

event.category: authentication and event.action: explicit_user_logon

Sample rendered Security (Authentication) Explicit User Logon event

A login was attempted using explicit credentials Arun \ Anvi-Acer to HD-v1s-d2118419 via C:\Windows\System32\services.exe (1736) as requested by subject ANVI-ACER$ \ WORKGROUP (subject logon ID 0x3e7) [ 4648]

Fields in an Security (Authentication) Explicit User Logon event

A login was attempted using explicit credentials endgame.target_user_name \ endgame.target_domain_name to host.name via process.name | process.executable (process.pid) as requested by subject endgame.subject_user_name \ endgame.subject_domain_name (subject logon ID endgame.subject_logon_id) [ event.code | winlog.event_id]

Security (Authentication) User Logoff events

Endgame Security (Authentication) User Logoff events with the following event type and subtype will be rendered in the Timeline via row renderers:

endgame.event_type_full: security_event and endgame.event_subtype_full: user_logoff

Runtime matching criteria

Security (Authentication) User Logoff events matching the following criteria will be rendered:

event.category: authentication and event.action: user_logoff

Sample rendered Security (Authentication) User Logoff event

Arun \ Anvi-Acer @ HD-55b-3ec87f66 logged off using logon type 2 - Interactive (target logon ID 0x16db41e) via C:\Windows\System32\services.exe (964) [ 4634 ]

Fields in Security (Authentication) User Logoff event

endgame.target_user_name \ endgame.target_domain_name @ host.name logged off using logon type endgame.logon_type (target logon ID endgame.target_logon_id) via process.name | process.executable (process.pid) [ event.code | winlog.event_id]

Process Creation events

Endgame Process Creation events with the following event type and subtype will be rendered in the Timeline via row renderers:

endgame.event_type_full: process_event and endgame.event_subtype_full: creation_event

Runtime matching criteria

Process Creation events matching the following criteria will be rendered:

event.category: process and event.action: creation_event

Sample rendered Process Creation event

Arun \ Anvi-Acer @ HD-obe-8bf77f54 started process Microsoft.Photos.exe (441684) -ServerName:App.AppXzst44mncqdg84v7sv6p7yznqwssy6f7f.mca via parent process svchost.exe (8)

sha256 d4c97ed46046893141652e2ec0056a698f6445109949d7fcabbce331146889ee

sha1 12563599116157778a22600d2a163d8112aed845

md5 62d06d7235b37895b68de56687895743

Fields in a Process Creation event

The following fields will be used to render a Process Creation event:

user.name \ user.domain @ host.name started process process.name (process.pid) process.args via parent process endgame.parent_process_name (process.ppid)

process.hash.sha256

process.hash.sha1

process.hash.md5

Process Termination events

Endgame Process Termination events with the following event type and subtype will be rendered in the Timeline via row renderers:

endgame.event_type_full: process_event and endgame.event_subtype_full: termination_event

Runtime matching criteria

Process Termination events matching the following criteria will be rendered:

event.category: process and event.action: termination_event

Sample rendered Process Termination event

Arun \ Anvi-Acer @ HD-obe-8bf77f54 terminated process RuntimeBroker.exe (442384) with exit code 0

sha256 87976f3430cc99bc939e0694247c0759961a49832b87218f4313d6fc0bc3a776

sha1 797255e72d5ed5c058d4785950eba7abaa057653

md5 bd4401441a21bf1abce6404f4231db4d

Fields in a Process Termination event

The following fields will be used to render a Process Termination event:

user.name \ user.domain @ host.name terminated process process.name (process.pid) with exit code endgame.exit_code

process.hash.sha256

process.hash.sha1

process.hash.md5

Testing

Desk tested in:

• Dark / light mode
• Chrome 77.0.3865.90
• Firefox 69.0.3
• Safari 13.0.1
• NOT tested in IE11 (due to current blocker)

https://github.com/elastic/ecs-dev/issues/178

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[webmat]

Read through all of the descriptions and this looks great! The logic looks sound as well :-)

I can't review the actual code, but this has my thumbs up otherwise 👍

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 2b1cec7adf343e0eeac40b4cb250e83516fe995b

[XavierM]

That's just a question, I am wondering if our draggable item in the row render should match with the one in the left side when we go over them.
Row render->

left side table ->

[XavierM]

LGTM ->
I played with all the row render locally, and for some good reason, they match with the PR description.

Thank you so much for all this work, it is great to see the endpoint data in the timeline.

Not for this PR, but I have an idea how to make all these props more dynamic and also to get their associated value like here https://github.com/elastic/kibana/pull/48277/files#diff-f67a2211c98a5d3581c4219dc2d5133bR31

[andrew-goldstein]

That's just a question, I am wondering if our draggable item in the row render should match with the one in the left side when we go over them.
Row render->

left side table ->

Thanks for noting the differences in hover states.

I think they are styled slightly different at the moment because text rendered in tables on the left hand side of the SIEM app, (i.e. the All hosts table), doesn't own it's own closure until the text is in a hover state. (In its hover state, the draggable text has a blue fill, per the screenshot you provided above.)

For reference (to others reading this PR), here's (in context) table text in a non-hover state:

This is in contrast to the "badge" style drawn around each (floating) draggable in the timeline row renderers, i.e. Arun in the example provided by @XavierM above.

That said, as you suggested, the fill colors could converge. (In a future PR, of course. 😉)

CC @MichaelMarcialis

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 8659d73

[MichaelMarcialis]

That said, as you suggested, the fill colors could converge. (In a future PR, of course. 😉)

Yeah, once my priorities aren't so focused on the detection engine, I'm hoping Andrew and I can put our heads together for future ideation on the row renderers and their drag handle styles.