[Security Solution] [Detections] Adds support for system actions (and cases action) to detection rules

[dhurley14]

Summary

Usually, connectors are used to communicate with external services like PagerDuty or ServiceNow. Automated actions like opening a case or running an OsQuery do not need any configuration. These types of actions are known as "system actions" and represent ways to automate functionality between different kibana features.

This PR allows detection rules to take advantage of system actions, like the cases action, to expedite tasks which our security customers might already undertake, while providing support for future system actions to be added later. You can read more about the cases action here: #153837

Testing

The cases system action is only available for Platinum licensing. To test all we need is to create a rule and in the Actions tab, select Cases, then turn the rule on and have it generate alerts. When the alerts are generated, a case (or cases) will be created and can be viewed in the Cases page.

The configuration options shown in the above screenshot are explained in more detail in this RFC, specifically Diagram 1, but to summarize:

1. Group by field determines how the alerts are attached to a specific or other cases
2. Time window is for whenever a case is older than the given time window, and the rule finds more alerts, the case system action will generate a new case after the "time window" period has elapsed.
3. Reopen - This will reopen the case if the rule discovers alerts after the case has been closed.

The grouping pieces can be better understood via the following example:

If I select agent.type in the group by field for the cases action, and have a rule that generates alerts from packetbeat-* and from auditbeat-* then that singular rule will generate two separate cases. One for alerts grouped by agent.type: packetbeat and another case for auditbeat

And we can still use system actions in conjunction with other action connectors.

Checklist

Delete any items that are not applicable to this PR.

• Documentation was added for features that require explanation or tutorials: Security Solution docs ticket here
• Unit or functional tests were updated or added to match the most common scenarios (Todo)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Impact of failing system action on other actions associated with the rule	Low	High	Ensure testing system actions and third party connectors, mock a failure in cases api endpoints

[dhurley14]

/ci

[dhurley14]

For any questions on this, I pulled this from the system actions RFC:

All rules APIs will not accept the systemActions in their requests. The system will deduct if an action is a system action automatically using the actions’ client. As system actions are not allowed to have a group, the property will be optional (at the moment it is required) in all APIs. The APIs will convert the actions to the schema needed by the rules client. Finally, the APIs will merge the system actions with the default actions before sending the response back to the client.

Ref: https://docs.google.com/document/d/1mYStNLxrafnyxR6v_uB4ONw_iFHTg1pmE3EAQYgKFSY/edit?pli=1

[elasticmachine]

⏳ Build in-progress

• Buildkite Build
• Commit: 604a755
• Cloud Deployment
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-183937-604a755628d7
• Security Deployment

History

• 💚 Build #222247 succeeded 374309c
• 💛 Build #222212 was flaky 374309c
• 💔 Build #222186 failed 991e10a
• 💔 Build #221920 failed cc33a01

cc @dhurley14

[dhurley14]

Here is a list of test cases I tried manually. All passed:

1. Can I update a rule’s query without modifying or deleting system action? ✅
2. Can I update a single rule’s system action?✅
3. Can I duplicate a single rule and have the system action be present?✅
4. Can I update a single rule’s action without modifying / deleting the system action?✅
5. Can I create a rule with two non-system actions?✅
6. Can I create a rule with two non-system actions and the cases action?✅
7. Can I update a rule's non-system action?✅
8. Can I use the bulk add action feature to add a system action?✅
9. Can I use the bulk add action feature to add a non-system action and ensure it did not remove a system action already present on a rule?✅
10. Can I use the bulk add action feature to add the cases system action and ensure it did not remove a non-system action from a rule?✅
11. Can I use the bulk add action feature to add the cases action and overwrite other rules actions?✅
12. Can I use the bulk add action feature to add a non-system action and overwrite the other rules actions, including overwriting a system action?✅
13. Can I import and export a rule with system actions?✅
14. Will a case be created if an alert does not have a field?✅ (displays 'unknown' as the group by value and attaches the correct number of alerts)

[jpdjere]

Hey @dhurley14. Thanks for addressing my points. Rechecked the bulk adding of system actions and everything seems to be working as expected now 👍

Left a couple of nits, but approving ✅

(I did cause a conflict in x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/converters/internal_rule_to_api_response.ts when I merged #188631, sorry about that)

[vitaliidm]

Thanks for implementing this feature and addressing comments.

Approving it, only one test to add left: #183937 (comment)

[cnasikas]

I did a high-level review and testing. LGTM! I noticed that the rule description does not show how the system actions are executed. System actions always work with alert summaries and they are executed on each rule run if the rule matches (triggers). Should we add something to describe that for all system actions?

In the stack management or the o11y we show

Also, unrelated to the PR, I noticed that the ITSM connector is not working as expected. cc @js-jankisalvi

[cnasikas]

nit: There is already a function to find cases in x-pack/test/cases_api_integration/common/lib/api/index.ts? What do you think of using that instead? Feel free to extend it to match your needs.

[dhurley14]

I did investigate this however I could not find a clean way to go about doing this. The options all seemed to introduce too much complexity into the findCases function for a single security solution test case. The problems stem from the use of the auth parameter in the findCases function, whereas we use a supertest that has the auth added to the header to support both ESS and serverless test environments. To update the findCases function to support our use-case would be introducing more maintenance overhead than it was worth. At least that's what I found to be the situation. Thank you for the review!

[js-jankisalvi]

Also, unrelated to the PR, I noticed that the ITSM connector is not working as expected. cc @js-jankisalvi

@cnasikas Verified it happens in main as well. Created an issue.

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 256b6ea
• Cloud Deployment
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-183937-256b6eaa2d03
• Security Deployment

Failed CI Steps

• FTR Configs #20

Test Failures

• [job] [logs] FTR Configs #20 / serverless search UI Serverless Playground Overview setup Page with existing indices can select index from dropdown and load chat page

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	20.4MB	20.4MB	+3.0KB

History

• 💛 Build #224094 was flaky 872477e
• 💛 Build #223689 was flaky 739a6a3
• 💔 Build #223647 failed 8d517b4
• 💔 Build #223520 failed 8de96fa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @dhurley14