Add new ML detection rules for Privileged Access Detection by sodhikirti07 · Pull Request #4516 · elastic/detection-rules

sodhikirti07 · 2025-03-04T15:49:39Z

Pull Request

Issue link(s):

https://github.com/elastic/security-ml/issues/507

Summary - What I changed

Added detection rules corresponding to AD jobs in the Privileged Access Detection (PAD) package. You could see the jobs config here.
Note: Both updates are scheduled for release in version 8.18.0.
Excluded pad rules from react tests kick off.
Built manifest and schema for PAD tests

❯ python -m detection_rules dev integrations build-manifests -o --prerelease -i pad
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

loading rules to determine all integration tags
loaded pad manifests from the following package versions: ['0.0.1']
final integrations manifests dumped: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/integration-manifests.json.gz
(.venv) 
detection-rules on  add-new-ml-detection-rules [$!?] is 📦 v0.4.22 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ python -m detection_rules dev integrations build-schemas -o -i pad
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Building integration schemas...
processing pad
final integrations manifests dumped: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/integration-schemas.json.gz
(.venv) 
detection-rules on  add-new-ml-detection-rules [$!?] is 📦 v0.4.22 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 4s 
❯

Build Manifest and Schema for sysmon_linux

❯ python -m detection_rules dev integrations build-manifests -o -i sysmon_linux       
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

loading rules to determine all integration tags
loaded sysmon_linux manifests from the following package versions: ['1.8.0', '1.7.2', '1.7.1', '1.7.0', '1.6.3', '1.6.2', '1.6.1', '1.6.0', '1.5.1', '1.5.0', '1.4.0', '1.3.0', '1.2.0', '1.1.0', '1.0.0']
final integrations manifests dumped: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/integration-manifests.json.gz
(.venv) 
detection-rules on  add-new-ml-detection-rules [$!?] is 📦 v0.4.22 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ python -m detection_rules dev integrations build-schemas -o -i sysmon_linux              
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Building integration schemas...
processing sysmon_linux
final integrations manifests dumped: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/integration-schemas.json.gz
(.venv) 
detection-rules on  add-new-ml-detection-rules [$!?] is 📦 v0.4.22 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 45s 
❯

How To Test

Checklist

Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
Added the meta:rapid-merge label if planning to merge within 24 hours
Secret and sensitive material has been managed correctly
Automated testing was updated or added to match the most common scenarios
Documentation and comments were added for features that require explanation

Contributor checklist

Have you signed the contributor license agreement?
Have you followed the contributor guidelines?

tradebot-elastic · 2025-03-04T15:50:01Z

⛔️ Tests failed:

❌ Spike in Group Privilege Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Unusual Source IP for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in Special Logon Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ High Command Line Entropy Detected for Privileged Commands (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in Group Membership Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in User Lifecycle Management Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Unusual Privilege Type assigned to a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Unusual Host Name for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in Group Application Assignment Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in User Account Management Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Unusual Process Detected for Privileged Commands by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in Special Privilege Use Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in Group Management Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Unusual Host Name for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Unusual Spike in Concurrent Active Sessions by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Unusual Region Name for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in Group Lifecycle Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Decline in host-based traffic (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in Privileged Command Execution by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Unusual Region Name for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Unusual Source IP for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events
❌ Spike in host-based traffic (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
- events_validation_missing: Not tested with events

shashank-elastic · 2025-03-04T16:36:42Z

Quick Review note

PAD seems to be a new integration that we are adding.
This would mean we need to pull in manifests and schemas for the same!
When are targeting this we have a huge big PR for Prep work and a release scheduled this week once we are unblocked by the ML packages having 9.0.0.
From the integrations part I can help you generate for PAD and we can sync on this

shashank-elastic · 2025-03-04T16:50:14Z

Post Syncing with @sodhikirti07
The Package is not published in the EPR - https://epr.elastic.co/search?package=pad
This is scheduled to release in Timeline of 8.18 and we will be able to pull the rule dev work and integrations only after having at-least a beta version of the package and the same has been communicated.

terrancedejesus · 2025-03-04T16:53:42Z

@shashank-elastic - If I remember correctly, we still need to validate ML job IDs in packages upstream. Thus this PR will not be able to merge until that the ML package is in EPR. We could manually add them to bypass this for now, but need to pull later when available

tradebot-elastic · 2025-03-04T19:31:10Z

⛔️ Tests failed:

sodhikirti07 · 2025-03-04T19:31:15Z

@shashank-elastic Started a PR for Security:Host module here : #4519

shashank-elastic · 2025-03-06T14:19:30Z

Update

For this PR we have new integration PAD, and we wait for the Package to release add the package to our list of MACHINE_LEARNING packages here and then allow it pass through the normal dev cycle.

tradebot-elastic · 2025-03-12T18:18:36Z

⛔️ Tests failed:

❌ Spike in Group Privilege Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Source IP for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Special Logon Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ High Command Line Entropy Detected for Privileged Commands (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Membership Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in User Lifecycle Management Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Privilege Type assigned to a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Host Name for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Application Assignment Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in User Account Management Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Process Detected for Privileged Commands by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Special Privilege Use Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Management Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Host Name for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Spike in Concurrent Active Sessions by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Region Name for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Lifecycle Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Privileged Command Execution by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Region Name for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Source IP for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

sodhikirti07 · 2025-03-12T18:20:34Z

@shashank-elastic The Privileged Access Detection package is now available as a beta-release: https://epr.elastic.co/package/pad/0.0.1/. Could you help adding the detection-rules for this integration? This package is available for both 8.18 and 9.0.

tradebot-elastic · 2025-03-13T06:16:33Z

⛔️ Tests failed:

❌ Spike in Group Privilege Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Source IP for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Special Logon Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ High Command Line Entropy Detected for Privileged Commands (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Membership Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in User Lifecycle Management Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Privilege Type assigned to a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Host Name for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Application Assignment Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in User Account Management Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Process Detected for Privileged Commands by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Special Privilege Use Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Management Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Host Name for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Spike in Concurrent Active Sessions by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Region Name for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Lifecycle Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Privileged Command Execution by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Region Name for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Source IP for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

tradebot-elastic · 2025-03-13T06:32:06Z

⛔️ Tests failed:

❌ Spike in Group Privilege Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Source IP for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Special Logon Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ High Command Line Entropy Detected for Privileged Commands (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Membership Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in User Lifecycle Management Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Privilege Type assigned to a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Host Name for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Application Assignment Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in User Account Management Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Process Detected for Privileged Commands by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Special Privilege Use Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Management Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Host Name for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Spike in Concurrent Active Sessions by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Region Name for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Group Lifecycle Change Events (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Spike in Privileged Command Execution by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Region Name for Windows Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Group Name Accessed by a User (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Unusual Source IP for Okta Privileged Operations Detected (-)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

github-actions · 2025-03-13T06:45:01Z

shashank-elastic · 2025-03-13T07:01:50Z

March 13 Update

Some small changes in tactic is required I have added the header for discovery in commit, this is missing a technique / sub- tech might wanna check that @sodhikirti07
Ignore react test kick off for pad rules updated
Added PAD as machine learning package and updated the manifest and schema
Once the schema is added, one of the rule is failing with the below error. Please check the valid machine learning job_id for the said rule.

E           AssertionError: The following (1) rules are missing a valid `machine_learning_job_id`:
E           2bca4fcd-5228-4472-9071-148903a31057 - Unusual Host Name for Windows Privileged Operations Detected -> machine_learning_job_id `pad_windows_rare_host_name_by_user` not found in version `0.0.1` of `pad` integration. existing jobs: ['pad_windows_high_count_special_logon_events', 'pad_windows_high_count_special_privilege_use_events', 'pad_windows_high_count_group_management_events', 'pad_windows_high_count_user_account_management_events', 'pad_windows_rare_privilege_assigned_to_user', 'pad_windows_rare_group_name_by_user', 'pad_windows_rare_device_by_user', 'pad_windows_rare_source_ip_by_user', 'pad_windows_rare_region_name_by_user', 'pad_linux_high_count_privileged_process_events_by_user', 'pad_linux_rare_process_executed_by_user', 'pad_linux_high_median_process_command_line_entropy_by_user', 'pad_okta_spike_in_group_membership_changes', 'pad_okta_spike_in_user_lifecycle_management_changes', 'pad_okta_spike_in_group_privilege_changes', 'pad_okta_spike_in_group_application_assignment_changes', 'pad_okta_spike_in_group_lifecycle_changes', 'pad_okta_high_sum_concurrent_sessions_by_user', 'pad_okta_rare_source_ip_by_user', 'pad_okta_rare_region_name_by_user', 'pad_okta_rare_host_name_by_user']

Now after adding PAD we noticed a new integration sysmon_linux in the PAD rules. Now this integration does not have a 9.0.0 version compatible package, now it may pass for 8.18 but it definitely will block us in creating 9.0 package.
Even before we merge this its good to reach out to the said team in the owner filed "elastic/sec-linux-platform" for a 9.0.0 compatible package. I have raised similar requests, if you need help there please let me know

cc @sodhikirti07 @Mikaayenson

shashank-elastic · 2025-03-14T06:48:55Z

Fri 14 Update

Update Manifest and schema for sysmon_linux. The package has merged its 9.0.0 version support
Update patch version versions for code path changes.

sodhikirti07 · 2025-03-14T15:59:17Z

@terrancedejesus @eric-forte-elastic @Mikaayenson Can someone please approve this PR?

Mikaayenson

@sodhikirti07 can you add investigation guides to these rules?

shashank-elastic · 2025-03-18T15:01:47Z

@sodhikirti07 can you add investigation guides to these rules?

@Mikaayenson added the investigation guides

eric-forte-elastic

LGTM 👍 But would wait in merging until @Mikaayenson 👍 investigation guides.

susan-shu-c

Looked over the rules, great work Kirti, Shashank!

…4516)" This reverts commit 2ff8d1b.

…4516)" (#4548) This reverts commit 2ff8d1b.

* Temporaily Disable Changed FIles Workflow (elastic#4538) * Temporaily Disable Changed FIles Workflow * bump version * Add new ML detection rules for Privileged Access Detection (elastic#4516) Add detection-rules for privileged access detection integration * Revert "Add new ML detection rules for Privileged Access Detection (elastic#4516)" (elastic#4548) This reverts commit 2ff8d1b. * Min stack rules from 4516 (elastic#4549) * Update defense_evasion_posh_assembly_load.toml (elastic#4543) Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com> * Change description and name of problemchild ML detection-rules (elastic#4545) Changed description and name of problemchild ML detection-rules * Prep for Release 9.0 (elastic#4550) * [ci] Add new docs-builder automation. (elastic#4507) * Add new docs automation * Add path-pattern filters for documentation folders * Update .github/workflows/docs-build.yml Co-authored-by: Jan Calanog <nejcalanog@gmail.com> --------- Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com> Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com> Co-authored-by: Jan Calanog <nejcalanog@gmail.com> Co-authored-by: Sergey Polzunov <traut@users.noreply.github.com> * [FR] Bump changed-files Version to Patched Version (elastic#4542) * Bump changed-files Version to Patched Version * patch bump * reenable workflow * Use full length commit hash * Bump 44 to 46 --------- Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com> * [New Rule] Adding Coverage for DynamoDB Exfiltration Behaviors (elastic#4535) * new rules for AWS DynamoDB data exfiltration * bumping patch version * adjusting investigation guide * updating patch version * updating patch version * updating patch version --------- Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com> * fix: removing outdated code in Kibana client auth (elastic#4495) * Simplify kibana session management * Drop removed options from `kibana_args` set * Style fix * Patch version bump * Bumping kibana lib version * Relax CLI requirement, making `api_key` optional, to allow `help` to run * Create new detection rule set documentation to be included in the new docs. (elastic#4508) * move docs folder to docs-dev * Add new docs folder * update docset.yml to reflect latest usage * Add rules_building_block folder * revert changes to docs-dev/experimental-machine-learning/url-spoof.md * bump patch versions * revert bump --------- Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com> Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com> * fixing double header in investigation notes (elastic#4490) * [Bug] Update Custom Rules Markdown Location (elastic#4565) * Update to custom-rules markdown location * bump version * Update link reference * Prep main for 9.1 (elastic#4555) * Prep for Release 9.1 * Update Patch Version * Update Patch version * Update Patch version * [Rule Tuning] Added OWA (outlook for web) new AppID (elastic#4568) * Added OWA (outlook for web) new AppID **Title:** Add new Outlook for Web AppID to abnormal Microsoft 365 ClientAppID rule **Description:** This pull request updates the `initial_access_microsoft_365_abnormal_clientappid` rule to include the newly introduced Outlook for Web AppID: - **New AppID**: `9199bf20-a13f-4107-85dc-02114787ef48` ### Context Outlook for Web (OWA) is migrating to a new authentication platform using MSAL and a Single Page Application (SPA) auth model. As part of this backend change, Microsoft is replacing the existing OWA AppID with a new one. This change is being rolled out during the first half of calendar year 2024, with full deployment expected by Q4 2024. - **Old OWA AppID**: `00000002-0000-0ff1-ce00-000000000000` - **New OWA AppID**: `9199bf20-a13f-4107-85dc-02114787ef48` Although no action is required for tenant administrators, this new AppID may show up in logs and should be accounted for in detections relying on known legitimate ClientAppIDs. ### Why this change? The rule `initial_access_microsoft_365_abnormal_clientappid` flags potentially suspicious or unauthorized client applications accessing Microsoft 365 services. To prevent false positives caused by this official change from Microsoft, this PR adds the new OWA AppID to the allowlist. ### References - Microsoft 365 Message Center notice (ref: MC715025) - [MSAL documentation](https://learn.microsoft.com/en-us/azure/active-directory/develop/msal-overview) * Update initial_access_microsoft_365_abnormal_clientappid.toml Updated updated_date * Update Max signals value to supported limits (elastic#4556) * Update ATT&CK coverage URL(s) in docs-dev/ATT&CK-coverage.md (elastic#4571) * deprecating 'Azure Virtual Network Device Modified or Deleted' (elastic#4559) * tuning 'Azure Conditional Access Policy Modified' (elastic#4558) * [Rule Tuning] Tuning Illicit Grant Consent Detections in Azure and M365 (elastic#4557) * tuning Azure rule for illicit grant activity; creating new rule for M365 * Update rules/integrations/o365/initial_access_microsoft_365_illicit_consent_grant_via_registered_application.toml * Update rules/integrations/azure/initial_access_entra_illicit_consent_grant_via_registered_application.toml * adjusted tags * Update rules/integrations/azure/initial_access_entra_illicit_consent_grant_via_registered_application.toml * Update defense_evasion_microsoft_defender_tampering.toml (elastic#4573) Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com> * [Rule Tuning] O365 Exchange Suspicious Mailbox Right Delegation (elastic#4589) * Update docset.yml (elastic#4590) Remove diagnostic hint * [New] Unusual Network Connection to Suspicious Web Service (elastic#4569) * [New] Unusual Network Connection to Suspicious Web Service * Update rule threat order --------- Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com> Co-authored-by: eric-forte-elastic <eric.forte@elastic.co> Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com> * [New] Unusual Network Connection to Suspicious Top Level Domain (elastic#4563) * [Rule Tuning] Add Host Metadata to ES|QL Aggregation Rules (elastic#4592) Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com> * Add investigation guides (elastic#4600) * [Rule Tuning] Suspicious Execution via Scheduled Task (elastic#4599) * Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (elastic#4601) * [FN Tuning] Shared Object Created or Changed by Previously Unknown Pr… (elastic#4529) * [FN Tuning] Shared Object Created or Changed by Previously Unknown Process * Update process exclusions in TOML file --------- Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com> Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com> Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com> Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com> * [D4C Conversion] Converting Compatible D4C Rules to DR (elastic#4532) * [D4C Conversion] Converting Compatible D4C Rules to DR * added host.os.type * Rename * Update rules/linux/execution_container_management_binary_launched_inside_container.toml Co-authored-by: Isai <59296946+imays11@users.noreply.github.com> * Update rules/linux/privilege_escalation_debugfs_launched_inside_container.toml Co-authored-by: Isai <59296946+imays11@users.noreply.github.com> * Update rules/linux/privilege_escalation_debugfs_launched_inside_container.toml Co-authored-by: Isai <59296946+imays11@users.noreply.github.com> * Update rules/linux/privilege_escalation_mount_launched_inside_container.toml Co-authored-by: Isai <59296946+imays11@users.noreply.github.com> * Update rules/linux/privilege_escalation_mount_launched_inside_container.toml Co-authored-by: Isai <59296946+imays11@users.noreply.github.com> --------- Co-authored-by: Isai <59296946+imays11@users.noreply.github.com> Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com> Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com> Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com> * Remove Task List reference (elastic#4605) * [FR] Update Detection Rules MITRE Workflow to SHA Pin (elastic#4581) * Update to pinned hash * version bump * [FR] Add Kibana Action Connector Error to Exception List Workaround (elastic#4583) * Add error catch for workaround * Switch to set for efficiency * Patch version bump --------- Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com> * [Rule Tuning] SSH Authorized Keys File Deletion (elastic#4591) Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com> * [Rule Tuning] Suspicious WMI Event Subscription Created (elastic#4618) * [Rule Tuning] Suspicious Execution via Scheduled Task * [Rule Tuning] Suspicious WMI Event Subscription Created * [Rule Tuning] Adjusting `Microsoft Entra ID Rare Authentication Requirement for Principal User` (elastic#4562) * tuning 'Microsoft Entra ID Rare Authentication Requirement for Principal User' * updated MITRE ATT&CK mappings * updated index target * updated patch version * updating patch version * bumping patch version * updating patch version * [Rule Tuning] Tuning `Azure Service Principal Credentials Added` (elastic#4570) * tuning 'Azure Service Principal Credentials Added' * updated patch version * added investigation guide * updating patch version * updating patch version * [FR] Add Support for Local Dates Flag (elastic#4582) * Add support for local dates flag * Use two variables * Add support for import-rules-to-repo * Revert arg formatting * Update comment * Pass Rule Path as Path Object * Update to rule loader function * Streamline metadata function * Also support dictionaries * Bump patch version * Reduce complexity * Add if path exists check * Fix version bump * Feature exclude tactic name (elastic#4593) * Added new cli flag to exclude tactic name in rule file name * added a shortcut for the flag and adjusted CLI readme * Add no tactic flag also to import to prevent warnings * Added info about unit test * version bump * Added no_tactic_filename as config option + fixed linting * pyproject version bump --------- Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com> Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com> --------- Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com> Co-authored-by: Kirti Sodhi <109447885+sodhikirti07@users.noreply.github.com> Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com> Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com> Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com> Co-authored-by: Martijn Laarman <Mpdreamz@gmail.com> Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com> Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com> Co-authored-by: Jan Calanog <nejcalanog@gmail.com> Co-authored-by: Sergey Polzunov <traut@users.noreply.github.com> Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com> Co-authored-by: M. Visser <BugOrFeature@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: eric-forte-elastic <eric.forte@elastic.co> Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com> Co-authored-by: Isai <59296946+imays11@users.noreply.github.com> Co-authored-by: Frederik Berg <83548283+frederikb96@users.noreply.github.com>

sodhikirti07 and others added 3 commits February 19, 2025 15:14

Add detection-rules for privileged access detection integration

0f05d7a

Add detection-rules for security host module

ab62af1

Issue #661 - Checking PAD job descriptions, tactics and techniques

0c182f2

Remove security host traffic rules from this branch

e948fe4

sodhikirti07 changed the title ~~Add new ML detection rules for Host traffic detection and Privileged Access Detection~~ Add new ML detection rules for Privileged Access Detection Mar 4, 2025

add updated_date

295beb5

Merge branch 'main' into add-new-ml-detection-rules

3ddb7f9

Fix duplicate Tactic error in Rule

1e11b61

Ignore react tests for PAD and add PAD in machine learning definitions

94f069a

shashank-elastic assigned sodhikirti07 Mar 13, 2025

shashank-elastic requested review from Mikaayenson, eric-forte-elastic, shashank-elastic, terrancedejesus and traut March 13, 2025 06:42

shashank-elastic added the enhancement New feature or request label Mar 13, 2025

Update PAD manifest and schema

607c7f5

github-actions bot added the backport: auto label Mar 13, 2025

Update sysmon_linux manifest and schema

6a70f97

botelastic bot added python Internal python for the repository schema labels Mar 14, 2025

shashank-elastic added the patch label Mar 14, 2025

Update patch version

d32d9ea

shashank-elastic approved these changes Mar 14, 2025

View reviewed changes

shashank-elastic and others added 2 commits March 14, 2025 21:32

Merge branch 'main' into add-new-ml-detection-rules

91f6f16

Merge branch 'main' into add-new-ml-detection-rules

8e98fe3

traut approved these changes Mar 17, 2025

View reviewed changes

Update Patch Version

d8b9268

Mikaayenson reviewed Mar 18, 2025

View reviewed changes

Add investigation guides

8c6cd9a

shashank-elastic requested a review from Mikaayenson March 18, 2025 15:01

eric-forte-elastic approved these changes Mar 18, 2025

View reviewed changes

sodhikirti07 requested review from jmcarlock, mgarzon and susan-shu-c March 19, 2025 12:49

susan-shu-c approved these changes Mar 19, 2025

View reviewed changes

mgarzon approved these changes Mar 19, 2025

View reviewed changes

sodhikirti07 merged commit 2ff8d1b into main Mar 19, 2025
12 checks passed

sodhikirti07 deleted the add-new-ml-detection-rules branch March 19, 2025 15:02

eric-forte-elastic added a commit that referenced this pull request Mar 20, 2025

Revert "Add new ML detection rules for Privileged Access Detection (#…

d558cea

…4516)" This reverts commit 2ff8d1b.

eric-forte-elastic mentioned this pull request Mar 20, 2025

[Revert] "Add new ML detection rules for Privileged Access Detection (#4516)" #4548

Merged

5 tasks

eric-forte-elastic added a commit that referenced this pull request Mar 20, 2025

Revert "Add new ML detection rules for Privileged Access Detection (#…

5b3dc4a

…4516)" (#4548) This reverts commit 2ff8d1b.

eric-forte-elastic mentioned this pull request Mar 20, 2025

[New Rules] Add new ML detection rules for Privileged Access Detection with Min Stack #4549

Merged

5 tasks

Conversation

sodhikirti07 commented Mar 4, 2025 • edited by shashank-elastic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request

Summary - What I changed

How To Test

Checklist

Contributor checklist

Uh oh!

tradebot-elastic commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shashank-elastic commented Mar 4, 2025

Uh oh!

shashank-elastic commented Mar 4, 2025

Uh oh!

terrancedejesus commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tradebot-elastic commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sodhikirti07 commented Mar 4, 2025

Uh oh!

shashank-elastic commented Mar 6, 2025

Uh oh!

tradebot-elastic commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sodhikirti07 commented Mar 12, 2025

Uh oh!

tradebot-elastic commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tradebot-elastic commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025

Enhancement - Guidelines

Documentation and Context

Code Standards and Practices

Testing

Additional Checks

Uh oh!

shashank-elastic commented Mar 13, 2025

Uh oh!

shashank-elastic commented Mar 14, 2025

Uh oh!

sodhikirti07 commented Mar 14, 2025

Uh oh!

Mikaayenson left a comment

Choose a reason for hiding this comment

Uh oh!

shashank-elastic commented Mar 18, 2025

Uh oh!

eric-forte-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

susan-shu-c left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

sodhikirti07 commented Mar 4, 2025 •

edited by shashank-elastic

Loading

tradebot-elastic commented Mar 4, 2025 •

edited

Loading

terrancedejesus commented Mar 4, 2025 •

edited

Loading

tradebot-elastic commented Mar 4, 2025 •

edited

Loading

tradebot-elastic commented Mar 12, 2025 •

edited

Loading

tradebot-elastic commented Mar 13, 2025 •

edited

Loading

tradebot-elastic commented Mar 13, 2025 •

edited

Loading

susan-shu-c left a comment •

edited

Loading