[Security Solution] Alternative mechanism for distributing prebuilt rules

[banderror]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168

NOTE: This might be converted to an epic when we have more understanding of the scope and requirements.

Summary

Recently we had an incident in Serverless where Kibana instances would crash with an OOM because of an installation of the security_detection_engine Fleet package that Security Solution uses to distribute prebuilt detection rules. Fleet loads whole packages into memory before installing their assets, and this package had become too big for that. The incident has been mitigated by temporarily decreasing the number of assets in the package by ~50%.

Mid-term measures for the 8.16 release cycle will be stream-based package installation and smart limits for it.

As a long-term measure, we should consider moving away from Fleet as a mechanism for distributing prebuilt rules. Such a mechanism should support Out-Of-Band rule updates decoupled from Kibana releases.

Ideas

• A dedicated microservice for publishing and serving rule versions.
• Git as a storage.
• Git as a service.
• Should be used by the TRADE team in https://github.com/elastic/detection-rules
• Should be integrated with Detections as Code
• We'll need to keep the Fleet-based mechanism in Kibana for backward compatibility
• How do we support air-gapped environments?

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)