[RAC][Security Solution] Register Security Detection Rules with Rule Registry

[spong]

Summary

This PR starts the migration of the Security Solution rules to use the rule-registry introduced in #95903. This is a pathfinding effort in porting over the existing Security Solution rules, and may include some temporary reference rules for testing out different paradigms as we move the rules over. See #95735 for details

Enable via the following feature flags in your kibana.dev.yml:

# Security Solution Rules on Rule Registry
xpack.ruleRegistry.index: '.kibana-[USERNAME]-alerts' # Only necessary to scope from other devs testing, if not specified defaults to `.alerts-security-solution`
xpack.securitySolution.enableExperimental: ['ruleRegistryEnabled']

Note: if setting a custom xpack.ruleRegistry.index, for the time being you must also update the DEFAULT_ALERTS_INDEX in order for the UI to display alerts within the alerts table.

---

Three reference rule types have been added (query, eql, threshold), along with scripts for creating them located in:

x-pack/plugins/security_solution/server/lib/detection_engine/reference_rules/scripts/

Main Detection page TGrid queries have been short-circuited to query .alerts-security-solution* for displaying alerts from the new alerts as data indices.

To test, checkout, enable the above feature flag(s), and run one of the scripts from the above directory, e.g. ./create_reference_rule_query.sh (ensure your ENV vars as set! :)

Alerts as data within the main Detection Page 🎉

cc @madirey @dgieselaar @pmuellr @yctercero @dhurley14 @marshallmain

[madirey]

After playing around in here, I'd suggest a couple of new rule type factories: EventLogRuleType (or PersistenceRuleType? ... which allows for writing multiple signals) and ThresholdRuleType (which encapsulates rule state to persist signals written on overlapping rule intervals so that dupes can be mitigated across buckets... this could mirror how dupe mitigation is performed for the DE threshold rules, except we'd store the state on the rule itself, rather than on the generated signals).

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[justinkambic]

Unsure why @elastic/uptime was pinged, but I have built this locally and all appears well with our products, and none of the files in the diff are owned by my team.

LGTM

[smith]

👍🏻 from me once the APM change is reverted.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 2a3cbfd
• Storybooks Preview

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
ruleRegistry	41	43	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	6.9MB	6.9MB	+3.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	60.8KB	61.1KB	+264.0B

Unknown metric groups

API count

id	before	after	diff
ruleRegistry	41	43	+2

References to deprecated APIs

id	before	after	diff
canvas	29	25	-4
crossClusterReplication	8	6	-2
fleet	22	20	-2
globalSearch	4	2	-2
lens	67	45	-22
lists	239	236	-3
ml	121	115	-6
monitoring	109	56	-53
securitySolution	342	346	+4
total			-90

History

• 💔 Build #127805 failed 5201667
• 💔 Build #127771 failed 91bede9
• 💚 Build #127466 succeeded 670ce49
• 💚 Build #126907 succeeded 05c6a3b
• 💚 Build #126417 succeeded 222a52a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @madirey @spong

[kibanamachine]

💔 Backport failed

Status	Branch	Result
❌	7.x	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 96015