[ML] Update Security_Linux and Security_Windows Modules to V3 #123000

dishadasgupta · 2022-01-13T20:39:07Z

Summary

Adds files for new + refactored v3 jobs for both security_linux and security_windows - for use within the Security app.

Files/Job Artifacts:

2 updated manifest .json files - for both linux and windows
Updated/new ML Job configurations for 26 jobs - each with associated datafeed configuration files:
- security_linux: 14 jobs
  - v3_linux_anomalous_network_activity
  - v3_linux_anomalous_network_port_activity_ecs
  - v3_linux_anomalous_process_all_hosts_ecs
  - v3_linux_anomalous_user_name_ecs
  - v3_linux_network_configuration_discovery
  - v3_linux_network_connection_discovery
  - v3_linux_rare_metadata_process
  - v3_linux_rare_metadata_user
  - v3_linux_rare_sudo_user
  - v3_linux_rare_user_compiler
  - v3_linux_system_information_discovery
  - v3_linux_system_process_discovery
  - v3_linux_system_user_discovery
  - v3_rare_process_by_host_linux_ecs
- security_windows: 12 jobs
  - v3_rare_process_by_host_windows_ecs
  - v3_windows_anomalous_network_activity_ecs
  - v3_windows_anomalous_path_activity_ecs
  - v3_windows_anomalous_process_all_hosts_ecs
  - v3_windows_anomalous_process_creation
  - v3_windows_anomalous_script
  - v3_windows_anomalous_service
  - v3_windows_anomalous_user_name_ecs
  - v3_windows_rare_metadata_process
  - v3_windows_rare_metadata_user
  - v3_windows_rare_user_runas_event
  - v3_windows_rare_user_type10_remote_login

Tests:

Individual job test tracking stats available here: https://docs.google.com/spreadsheets/d/1JOUIVsitaMdEdhM3WT2Eag4ELI-rI2Jec7bXildJsdQ/edit#gid=0

Screenshot of an integration test done in a local Kibana instance is provided below. Tried to make the descriptions as informative as possible (because of this note I read from a previous release), please let us know next steps for that.

@randomuserid to also post more updates as needed to this issue + regarding tests, thanks

elasticmachine · 2022-01-13T20:39:10Z

Pinging @elastic/ml-ui (:ml)

kibana-ci · 2022-01-13T21:45:33Z

💔 Build Failed

Failed CI Steps

Test Failures

[job] [logs] Default CI Group #18 / apis Machine Learning modules get_module lists all modules
[job] [logs] Default CI Group #18 / apis Machine Learning modules get_module lists all modules

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @randomuserid @dishadasgupta

szabosteve

Thanks for these changes!
I left some suggestions that could decrease the word count. Please take or leave them.

szabosteve · 2022-01-14T14:54:54Z

x-pack/plugins/ml/server/models/data_recognizer/modules/security_linux/manifest.json

-  "description": "Detect suspicious activity using ECS Linux events. Tested with Auditbeat and the Elastic agent.",
+  "id": "security_linux_v3",
+  "title": "Security: Linux v3",
+  "description": "Security: Linux v3. Version 3 of the Linux Security Machine Learning Module, released in 2022, this module is a replacement for the v2 Linux module. This module contains all shipping ML jobs for Linux host based threat hunting and detection. Any ECS (Elastic Common Schema) compatable Linux events can be used by this module.",


As the title is Security: Linux v3, the description might be a bit shorter.

Suggested change

"description": "Security: Linux v3. Version 3 of the Linux Security Machine Learning Module, released in 2022, this module is a replacement for the v2 Linux module. This module contains all shipping ML jobs for Linux host based threat hunting and detection. Any ECS (Elastic Common Schema) compatable Linux events can be used by this module.",

"description": "Contains all shipping ML jobs for Linux host-based threat hunting and detection. Any ECS-compatible Linux events can be used by the jobs.",

szabosteve · 2022-01-14T15:01:17Z

...dels/data_recognizer/modules/security_linux/ml/v3_linux_network_configuration_discovery.json

@@ -0,0 +1,55 @@
+{
+    "job_type": "anomaly_detector",
+    "description": "Security: Linux - Looks for commands related to system network configuration discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used by a threat actor to engage in system network configuration discovery in order to increase their understanding of connected networks and hosts. This information may be used to shape follow-up behaviors such as lateral movement or additional discovery.",


Suggested change

"description": "Security: Linux - Looks for commands related to system network configuration discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used by a threat actor to engage in system network configuration discovery in order to increase their understanding of connected networks and hosts. This information may be used to shape follow-up behaviors such as lateral movement or additional discovery.",

"description": "Security: Linux - Looks for commands related to system network configuration discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used by a threat actor to engage in system network configuration discovery to increase their understanding of connected networks and hosts. This information may be used to shape follow-up behaviors such as lateral movement or additional discovery.",

szabosteve · 2022-01-14T15:02:47Z

.../models/data_recognizer/modules/security_linux/ml/v3_linux_network_connection_discovery.json

@@ -0,0 +1,55 @@
+{
+    "job_type": "anomaly_detector",
+    "description": "Security: Linux - Looks for commands related to system network connection discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used by a threat actor to engage in system network connection discovery in order to increase their understanding of connected services and systems. This information may be used to shape follow-up behaviors such as lateral movement or additional discovery.",


Suggested change

"description": "Security: Linux - Looks for commands related to system network connection discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used by a threat actor to engage in system network connection discovery in order to increase their understanding of connected services and systems. This information may be used to shape follow-up behaviors such as lateral movement or additional discovery.",

"description": "Security: Linux - Looks for commands related to system network connection discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used by a threat actor to engage in system network connection discovery to increase their understanding of connected services and systems. This information may be used to shape follow-up behaviors such as lateral movement or additional discovery.",

szabosteve · 2022-01-14T15:04:46Z

.../models/data_recognizer/modules/security_linux/ml/v3_linux_system_information_discovery.json

@@ -0,0 +1,55 @@
+{
+    "job_type": "anomaly_detector",
+    "description": "Security: Linux - Looks for commands related to system information discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used to engage in system information discovery in order to gather detailed information about system configuration and software versions. This may be a precursor to selection of a persistence mechanism or a method of privilege elevation.",


Suggested change

"description": "Security: Linux - Looks for commands related to system information discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used to engage in system information discovery in order to gather detailed information about system configuration and software versions. This may be a precursor to selection of a persistence mechanism or a method of privilege elevation.",

"description": "Security: Linux - Looks for commands related to system information discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used to engage in system information discovery to gather detailed information about system configuration and software versions. This may be a precursor to the selection of a persistence mechanism or a method of privilege elevation.",

szabosteve · 2022-01-14T15:05:27Z

...rver/models/data_recognizer/modules/security_linux/ml/v3_linux_system_process_discovery.json

@@ -0,0 +1,55 @@
+{
+    "job_type": "anomaly_detector",
+    "description": "Security: Linux - Looks for commands related to system process discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used to engage in system process discovery in order to increase their understanding of software applications running on a target host or network. This may be a precursor to selection of a persistence mechanism or a method of privilege elevation.",


Suggested change

"description": "Security: Linux - Looks for commands related to system process discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used to engage in system process discovery in order to increase their understanding of software applications running on a target host or network. This may be a precursor to selection of a persistence mechanism or a method of privilege elevation.",

"description": "Security: Linux - Looks for commands related to system process discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used to engage in system process discovery to increase their understanding of software applications running on a target host or network. This may be a precursor to the selection of a persistence mechanism or a method of privilege elevation.",

szabosteve · 2022-01-14T15:07:28Z

.../server/models/data_recognizer/modules/security_linux/ml/v3_linux_system_user_discovery.json

@@ -0,0 +1,55 @@
+{
+    "job_type": "anomaly_detector",
+    "description": "Security: Linux - Looks for commands related to system user or owner discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used to engage in system owner or user discovery in order to identify currently active or primary users of a system. This may be a precursor to additional discovery, credential dumping or privilege elevation activity.",


Suggested change

"description": "Security: Linux - Looks for commands related to system user or owner discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used to engage in system owner or user discovery in order to identify currently active or primary users of a system. This may be a precursor to additional discovery, credential dumping or privilege elevation activity.",

"description": "Security: Linux - Looks for commands related to system user or owner discovery from an unusual user context. This can be due to uncommon troubleshooting activity or due to a compromised account. A compromised account may be used to engage in system owner or user discovery to identify currently active or primary users of a system. This may be a precursor to additional discovery, credential dumping, or privilege elevation activity.",

szabosteve · 2022-01-14T15:09:15Z

x-pack/plugins/ml/server/models/data_recognizer/modules/security_windows/manifest.json

-  "description": "Detects suspicious activity using ECS Windows events. Tested with Winlogbeat and the Elastic agent.",
+  "id": "security_windows_v3",
+  "title": "Security: Windows v3",
+  "description": "Security: Linux v3. Version 3 of the Windows Security Machine Learning Module, released in 2022, this module is a replacement for the v2 Windows module. This module contains all shipping ML jobs for Windows host based threat hunting and detection. Any ECS (Elastic Common Schema) compatable Windows events can be used by this module.",


As the title is Security: Windows v3, the description might be a bit shorter.

Suggested change

"description": "Security: Linux v3. Version 3 of the Windows Security Machine Learning Module, released in 2022, this module is a replacement for the v2 Windows module. This module contains all shipping ML jobs for Windows host based threat hunting and detection. Any ECS (Elastic Common Schema) compatable Windows events can be used by this module.",

"description": "Contains all shipping ML jobs for Windows host-based threat hunting and detection. Any ECS-compatible Windows events can be used by the jobs.",

randomuserid

Looks like the v2 modules were accidentally overwritten with the v3 modules? Let's close this and start a new PR adding the v3_security_linux and v3_security_windows modules. I'll leave the tests failing here for the time being to ensure we don't merge this.

Updating linux + windows to v3

dd2bce6

dishadasgupta added release_note:enhancement :ml Feature:Anomaly Detection ML anomaly detection v8.1.0 labels Jan 13, 2022

dishadasgupta requested a review from a team as a code owner January 13, 2022 20:39

dishadasgupta assigned randomuserid and dishadasgupta Jan 13, 2022

szabosteve reviewed Jan 14, 2022

View reviewed changes

randomuserid suggested changes Jan 14, 2022

View reviewed changes

dishadasgupta mentioned this pull request Jan 18, 2022

[ML] Adding V3 modules for Security_Linux and Security_Windows #123274

Closed

bfilar closed this Apr 28, 2022

spalger deleted the ml-81 branch May 8, 2022 22:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ML] Update Security_Linux and Security_Windows Modules to V3 #123000

[ML] Update Security_Linux and Security_Windows Modules to V3 #123000

Uh oh!

dishadasgupta commented Jan 13, 2022

Uh oh!

elasticmachine commented Jan 13, 2022

Uh oh!

kibana-ci commented Jan 13, 2022 •

edited

Loading

Uh oh!

szabosteve left a comment

Uh oh!

szabosteve Jan 14, 2022 •

edited

Loading

Uh oh!

szabosteve Jan 14, 2022

Uh oh!

szabosteve Jan 14, 2022

Uh oh!

szabosteve Jan 14, 2022

Uh oh!

szabosteve Jan 14, 2022

Uh oh!

szabosteve Jan 14, 2022

Uh oh!

szabosteve Jan 14, 2022 •

edited

Loading

Uh oh!

randomuserid left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

	"description": "Security: Linux v3. Version 3 of the Linux Security Machine Learning Module, released in 2022, this module is a replacement for the v2 Linux module. This module contains all shipping ML jobs for Linux host based threat hunting and detection. Any ECS (Elastic Common Schema) compatable Linux events can be used by this module.",
	"description": "Contains all shipping ML jobs for Linux host-based threat hunting and detection. Any ECS-compatible Linux events can be used by the jobs.",

[ML] Update Security_Linux and Security_Windows Modules to V3 #123000

[ML] Update Security_Linux and Security_Windows Modules to V3 #123000

Uh oh!

Conversation

dishadasgupta commented Jan 13, 2022

Summary

Files/Job Artifacts:

Tests:

Uh oh!

elasticmachine commented Jan 13, 2022

Uh oh!

kibana-ci commented Jan 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

szabosteve left a comment

Choose a reason for hiding this comment

Uh oh!

szabosteve Jan 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szabosteve Jan 14, 2022

Choose a reason for hiding this comment

Uh oh!

szabosteve Jan 14, 2022

Choose a reason for hiding this comment

Uh oh!

szabosteve Jan 14, 2022

Choose a reason for hiding this comment

Uh oh!

szabosteve Jan 14, 2022

Choose a reason for hiding this comment

Uh oh!

szabosteve Jan 14, 2022

Choose a reason for hiding this comment

Uh oh!

szabosteve Jan 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

randomuserid left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kibana-ci commented Jan 13, 2022 •

edited

Loading

szabosteve Jan 14, 2022 •

edited

Loading

szabosteve Jan 14, 2022 •

edited

Loading