[Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 4 (DRAFT)

[banderror]

Epic: https://github.com/elastic/security-team/issues/1974 (internal)
Milestones: << • >>

Status: Draft.

Summary

Milestone 4: Improve prebuilt rule customization, upgrade, and installation UX.

This meta ticket is created to simplify tracking of various tickets related to the epic, and to make this public information so our users can track the progress.

Useful info:

• RFC: Prebuilt Rules Customization - a new technical design document describing in detail how prebuilt rule customization should work and affect other existing features.
• Prebuilt Rules Customization Technical Design - an older document from Milestone 1.

Product and UX improvements

Rule customization UX

Beta Give feedback

1. Show customized prebuilt rules in the Rules table. A customized rule should be marked with some indicator.
2. Rule Details page: show what fields are customized and what are these customizations exactly (e.g. show a diff between the base and the current versions)
3. Ability to reset customizations and revert back to the base version
4. [Security Solution] Include namespace of source indices in index patterns of prebuilt rules #183616
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement needs product +6
5. Add BE-side validation for ESQL query field
6. Add BE-side validation for EQL query field
7. Add BE-side validation for KQL query field

Rule installation and upgrade UX

Beta Give feedback

1. [Security Solution] Show callouts when new prebuilt rules or rule updates are available #160270
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +4
2. [Security Solution] Implement filtering of Rule Installation and Rule Upgrade tables by Elastic Rules and Customized Elastic rules (DRAFT) #180396
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp triage_needed +5
3. [Security Solution] Inform users about missing base versions during prebuilt rule upgrade #186880
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +5
4. Implement a preview of the applied changes in the Rule Upgrade Flyout and show it before calling the upgrade/_perform API endpoint
5. [Security Solution] Indicate conflicting rules in the Rule Upgrade table (DRAFT) #190500
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs product +5
   

Rule upgrade, diff algorithms

Beta Give feedback

1. [Security Solution] Implement array of objects diff algorithm #180161
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +5
   
2. [Security Solution] Implement MITRE ATT&CK® field diff algorithm #187660
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +5
   
3. [Security Solution] Implement query filters diff algorithm #190241
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +5
   
4. [Security Solution] Assign proper diff algorithms to all rule fields #148191
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +4
   

"Last Updated" field in the UI

Beta Give feedback

1. [FR] Add source_updated_at to Rule Schema as a Build Time Field detection-rules#2826
   backlog enhancement v8.10 +3
2. [Security Solution] Add source_updated_at field to PrebuiltRuleAsset #176286
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +4
3. [Security Solution] Add source_updated_at field to RuleResponse via ResponseFields #174740
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +4
4. [Security Solution] [Prebuilt Rules Customization] Add "Last Updated" column in the Add Elastic Rules and Rule Updates tables #174767
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management +3

Bugs

Bugs: rule installation and upgrade

Beta Give feedback

1. [Security Solution] Installing prebuilt rules by 2 users at the same time causes rules to be duplicated (DRAFT) #180196
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug triage_needed +6
2. [Security Solution] Rule Update failure on 8.13 from 7.17.18 #177852
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:medium +6
   
3. [Security Solution] Error when upgrading a rule that has an action referencing a deleted connector #198771
   Feature:Prebuilt Detection Rules Feature:Rule Actions Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:medium needs product sdh-linked +9
4. [Security Solution] Rule is not updated and is followed by 'Rule failed to update' message when user attempts to upgrade a rule linked to a deleted shared exception list #198845
   Feature:Prebuilt Detection Rules Feature:Rule Exceptions Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:medium +7

Bugs: rule import and export

Beta Give feedback

1. [Security Solution] Rule import creates extra rules while importing a large number of rules #176207
   Feature:Rule Import/Export Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:high +7
   
2. [Security Solution] Prevent creation of rules with the same rule_id when importing rules concurrently #177283
   Feature:Rule Import/Export Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:high +6
3. [Security Solution] Error when exporting a rule that has an action referencing a deleted connector #178221
   Feature:Rule Actions Feature:Rule Import/Export Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:medium needs product +8
4. https://github.com/elastic/security-team/issues/8644

Bugs: misc

Beta Give feedback

1. [Security Solution] Large number of related integrations can impact performance on rule creation/edit form #183607
   Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp bug impact:medium performance +9
   
2. [Security Solution] See what's new in Prebuilt Security Detection Rules link on Add Elastic Rules page works with clicking anywhere on the blank space available on the right side of the page. #194275
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug impact:low +6

Technical improvements and debt

Schema migration from immutable to rule_source

Beta Give feedback

1. [Security Solution] Detection rule migration mechanism #187651
   Feature:Alerting/RulesFramework Feature:Rule Management Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp Team:ResponseOps +7
2. [Security Solution] Migrate security rules encrypted saved objects to new schema with ruleSource field (BLOCKED) #184113
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp blocked +5
3. [Security Solution] Update rule searching to search by rule_source (BLOCKED) #180126
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp blocked +5
4. [Security Solution] Mark immutable as optional in the internal rule schema and stop writing it to rules (BLOCKED) #182573
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp blocked +5
5. [Security Solution] Deprecate the immutable field (DRAFT)(BLOCKED) #180269
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp triage_needed +5

Fleet package with prebuilt rules

Beta Give feedback

1. [Security Solution] Splitting the package with prebuilt rules #187648
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs product +5
2. [Security Solution] Alternative mechanism for distributing prebuilt rules #187649
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs product +5
3. [Fleet] Stream-based programmatic API for installing packages #187646
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet +5
   
4. Support "content-only" integrations package-spec#351
   Team:Ecosystem discuss +2

Refactoring

Beta Give feedback

1. [Security Solution] DetectionRulesClient refactoring. Part 3 #187656
   Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring +5
2. [Security Solution] Implement a Zod transformation from snake_case to camelCase and vice versa #180121
   Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt +4
3. [Security Solution] Improve rule diff algorithm performance by splitting diff calculation into async chunks #180164
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp performance refactoring +6
   

Tests

Beta Give feedback

1. [Security Solution] Move existing test plans of Rules Management team from Google docs to Kibana repo #180451
   Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test-plan +6
   
2. [Security Solution] Write tests for filtering, sorting, pagination in the Rule Installation and Upgrade tables #166215
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test-coverage +6

Performance

Beta Give feedback

1. [Security Solution] /upgrade/_perform performance improvements #199101
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp performance +5

Misc

Beta Give feedback

No tasks being tracked yet.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)