[AI4DSOC] Fix issue with filtering by integrations#216574
Merged
PhilippeOberti merged 4 commits intoelastic:mainfrom Apr 4, 2025
Merged
[AI4DSOC] Fix issue with filtering by integrations#216574PhilippeOberti merged 4 commits intoelastic:mainfrom
PhilippeOberti merged 4 commits intoelastic:mainfrom
Conversation
Contributor
|
Pinging @elastic/security-threat-hunting-investigations (Team:Threat Hunting:Investigations) |
d403ab1 to
3cfdfcc
Compare
3cfdfcc to
6ce0b4f
Compare
2 tasks
christineweng
approved these changes
Apr 3, 2025
Contributor
Author
Contributor
💚 Build Succeeded
Metrics [docs]Async chunks
History
|
PhilippeOberti
added a commit
to PhilippeOberti/kibana
that referenced
this pull request
May 30, 2025
## Summary This PR fixes an issue with the logic implemented in [this previous PR](elastic#215586): _In the AI for SOC effort, each integration is bundled with a single rule. This means that deselecting a source from the Source filter button is equivalent to adding a filter to the search bar to exclude all alerts with the kibana.alert.rule.name property having the value of that integration._ The problem with the previous logic above is the value in the `kibana.alert.rule.name` field can be overridden (see `Rule name override [here](https://www.elastic.co/guide/en/security/current/rules-ui-create.html)). Therefore filtering alerts by this value does not guarantee that all the alerts generated by the rule will be correctly filtered out. The new logic uses the `rule.id` instead of the `rule.name`, which we then use to filter using the `signal.rule.id` field instead of `kibana.alert.rule.name` ### Example: There are following 2 integrations installed: ```typescript [ { id: 'splunk', name: 'splunk', status: installationStatuses.Installed, title: 'Splunk', version: '', }, { id: 'google_secops', name: 'google_secops', status: installationStatuses.Installed, title: 'Google SecOps', version: '', }, ] ``` This means that - in theory - there are the following 2 rules installed and running: ```typescript [ { related_integrations: [{ package: 'splunk' }], id: 'splunk_rule_id', }, { related_integrations: [{ package: 'google_secops' }], id: 'google_secops_rule_id', }, ] ``` In this case, the `Sources` button would show 2 entries, as follow: ```typescript [ { checked: 'on', key: 'splunk_rule_id', label: 'Splunk', }, { checked: 'on', key: 'google_secops_rule_id', label: 'Splunk', }, ] ``` This PR also fixes a small miss in [the prior PR](elastic#215585) that implemented the KPI section, where I had forgotten to pass the KQL filters to the charts. #### Before https://github.com/user-attachments/assets/77e583c6-718f-46d9-96b4-42ee9976161b #### After https://github.com/user-attachments/assets/50e8e541-5798-4906-b7cc-4f9756dbdefc ## How to test This needs to be ran in Serverless: - `yarn es serverless --projectType security` - `yarn serverless-security --no-base-path` You also need to enable the AI for SOC tier, by adding the following to your `serverless.security.dev.yaml` file: ``` xpack.securitySolutionServerless.productTypes: [ { product_line: 'ai_soc', product_tier: 'search_ai_lake' }, ] ``` Use one of these Serverless users: - `platform_engineer` - `endpoint_operations_analyst` - `endpoint_policy_manager` - `admin` - `system_indices_superuser` ### Notes - generate data: `yarn test:generate:serverless-dev` - create 4 catch all rules, each with a name of a AI for SOC integration (`google_secops`, `microsoft_sentinel`,, `sentinel_one` and `crowdstrike`) - change [this line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_fetch_integrations.ts#L73) to `installedPackages: availablePackages` to force having some packages installed - change [this line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_integrations.ts#L63) to `r.name === p.name` to make sure there will be matches between integrations and rules ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios elastic/security-team#11956 (cherry picked from commit 2ed4266)
PhilippeOberti
added a commit
to PhilippeOberti/kibana
that referenced
this pull request
May 30, 2025
## Summary This PR fixes an issue with the logic implemented in [this previous PR](elastic#215586): _In the AI for SOC effort, each integration is bundled with a single rule. This means that deselecting a source from the Source filter button is equivalent to adding a filter to the search bar to exclude all alerts with the kibana.alert.rule.name property having the value of that integration._ The problem with the previous logic above is the value in the `kibana.alert.rule.name` field can be overridden (see `Rule name override [here](https://www.elastic.co/guide/en/security/current/rules-ui-create.html)). Therefore filtering alerts by this value does not guarantee that all the alerts generated by the rule will be correctly filtered out. The new logic uses the `rule.id` instead of the `rule.name`, which we then use to filter using the `signal.rule.id` field instead of `kibana.alert.rule.name` ### Example: There are following 2 integrations installed: ```typescript [ { id: 'splunk', name: 'splunk', status: installationStatuses.Installed, title: 'Splunk', version: '', }, { id: 'google_secops', name: 'google_secops', status: installationStatuses.Installed, title: 'Google SecOps', version: '', }, ] ``` This means that - in theory - there are the following 2 rules installed and running: ```typescript [ { related_integrations: [{ package: 'splunk' }], id: 'splunk_rule_id', }, { related_integrations: [{ package: 'google_secops' }], id: 'google_secops_rule_id', }, ] ``` In this case, the `Sources` button would show 2 entries, as follow: ```typescript [ { checked: 'on', key: 'splunk_rule_id', label: 'Splunk', }, { checked: 'on', key: 'google_secops_rule_id', label: 'Splunk', }, ] ``` This PR also fixes a small miss in [the prior PR](elastic#215585) that implemented the KPI section, where I had forgotten to pass the KQL filters to the charts. #### Before https://github.com/user-attachments/assets/77e583c6-718f-46d9-96b4-42ee9976161b #### After https://github.com/user-attachments/assets/50e8e541-5798-4906-b7cc-4f9756dbdefc ## How to test This needs to be ran in Serverless: - `yarn es serverless --projectType security` - `yarn serverless-security --no-base-path` You also need to enable the AI for SOC tier, by adding the following to your `serverless.security.dev.yaml` file: ``` xpack.securitySolutionServerless.productTypes: [ { product_line: 'ai_soc', product_tier: 'search_ai_lake' }, ] ``` Use one of these Serverless users: - `platform_engineer` - `endpoint_operations_analyst` - `endpoint_policy_manager` - `admin` - `system_indices_superuser` ### Notes - generate data: `yarn test:generate:serverless-dev` - create 4 catch all rules, each with a name of a AI for SOC integration (`google_secops`, `microsoft_sentinel`,, `sentinel_one` and `crowdstrike`) - change [this line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_fetch_integrations.ts#L73) to `installedPackages: availablePackages` to force having some packages installed - change [this line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_integrations.ts#L63) to `r.name === p.name` to make sure there will be matches between integrations and rules ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios elastic/security-team#11956 (cherry picked from commit 2ed4266)
PhilippeOberti
added a commit
to PhilippeOberti/kibana
that referenced
this pull request
Jun 4, 2025
## Summary This PR fixes an issue with the logic implemented in [this previous PR](elastic#215586): _In the AI for SOC effort, each integration is bundled with a single rule. This means that deselecting a source from the Source filter button is equivalent to adding a filter to the search bar to exclude all alerts with the kibana.alert.rule.name property having the value of that integration._ The problem with the previous logic above is the value in the `kibana.alert.rule.name` field can be overridden (see `Rule name override [here](https://www.elastic.co/guide/en/security/current/rules-ui-create.html)). Therefore filtering alerts by this value does not guarantee that all the alerts generated by the rule will be correctly filtered out. The new logic uses the `rule.id` instead of the `rule.name`, which we then use to filter using the `signal.rule.id` field instead of `kibana.alert.rule.name` ### Example: There are following 2 integrations installed: ```typescript [ { id: 'splunk', name: 'splunk', status: installationStatuses.Installed, title: 'Splunk', version: '', }, { id: 'google_secops', name: 'google_secops', status: installationStatuses.Installed, title: 'Google SecOps', version: '', }, ] ``` This means that - in theory - there are the following 2 rules installed and running: ```typescript [ { related_integrations: [{ package: 'splunk' }], id: 'splunk_rule_id', }, { related_integrations: [{ package: 'google_secops' }], id: 'google_secops_rule_id', }, ] ``` In this case, the `Sources` button would show 2 entries, as follow: ```typescript [ { checked: 'on', key: 'splunk_rule_id', label: 'Splunk', }, { checked: 'on', key: 'google_secops_rule_id', label: 'Splunk', }, ] ``` This PR also fixes a small miss in [the prior PR](elastic#215585) that implemented the KPI section, where I had forgotten to pass the KQL filters to the charts. #### Before https://github.com/user-attachments/assets/77e583c6-718f-46d9-96b4-42ee9976161b #### After https://github.com/user-attachments/assets/50e8e541-5798-4906-b7cc-4f9756dbdefc ## How to test This needs to be ran in Serverless: - `yarn es serverless --projectType security` - `yarn serverless-security --no-base-path` You also need to enable the AI for SOC tier, by adding the following to your `serverless.security.dev.yaml` file: ``` xpack.securitySolutionServerless.productTypes: [ { product_line: 'ai_soc', product_tier: 'search_ai_lake' }, ] ``` Use one of these Serverless users: - `platform_engineer` - `endpoint_operations_analyst` - `endpoint_policy_manager` - `admin` - `system_indices_superuser` ### Notes - generate data: `yarn test:generate:serverless-dev` - create 4 catch all rules, each with a name of a AI for SOC integration (`google_secops`, `microsoft_sentinel`,, `sentinel_one` and `crowdstrike`) - change [this line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_fetch_integrations.ts#L73) to `installedPackages: availablePackages` to force having some packages installed - change [this line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_integrations.ts#L63) to `r.name === p.name` to make sure there will be matches between integrations and rules ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios elastic/security-team#11956 (cherry picked from commit 2ed4266)
PhilippeOberti
added a commit
that referenced
this pull request
Jun 4, 2025
…) (#222074) # Backport This will backport the following commits from `main` to `8.19`: - [[AI4DSOC] Alert summary page routing and initialization (#214889)](#214889) - [[AI4DSOC] Alert summary landing page (#215246)](#215246) - [[AI4DSOC] Alert summary dataview (#215265)](#215265) - [[AI4DSOC] Alert summary KQL bar [#215586]](#215586) - [[AI4DSOC] Alert summary KPI charts [#215585]](#215585) - [[AI4DSOR] Alert summary integrations section [#215266]](#215266) - [[AI4DSOC] Fix issue with filtering by integrations [#216574]](#216574) - [[AI4DSOC] Alert summary table setup [#216744]](#216744) - [Alerty summary table flyout setup [#217421]](#217421) - [[AI4DSOC] Alert summary alert actions in table and flyout [#217696]](#217696) - [[AI4DSOC] Alert summary table custom cell renderers [#217124]](#217124) - [[AI4DSOC] Alert summary table and flyout ai assistant [#217744]](#217744) - [[AI4DSOC] Alert summary page performance improvements [#218632]](#218632) - [[AI4DSOC] Change the Attack Discovery page to use the AI for SOC alerts table [#218736]](#218736) - [[AI4DSOC] Change the Cases page to use the AI for SOC alerts table [#218742]](#218742) - [[AI4DSOC] Fix spacing issue on alert summary landing page integration card [#218868]](#218868) - [[AI4DSOC][ResponseOps] Fix alerts table not handling undefined maintenanceWindow capability [#218999]](#218999) - [[AI4DSOC] Fix link to the new integrations page [#219030]](#219030) - [[AI4DSOC] Disable CellActions and PreviewLinks on the Attack discovery page [#219033]](#219033) - [[AI4DSOC] Add cell renderer for datetime fields to the alert summary table [#219126]](#219126) - [[AI4DSOC] Remove Assistant icon from row action in alert summary table [#219141]](#219141) - [[AI4DSOC] Add checkboxes to the alert summary table [#219169]](#219169) - [[Security Solution][AI4DSOC] Fix table not applying alert tags for Attack discovery and Cases pages in AI4DSOC [#219410]](#219410) - [[AI4DSOC] Fix logic that renders the group title when grouping by integrations [#219430]](#219430) - [[AI4DSOC] Alert summary table truncates long values and display the field/value pair in tooltip [#219438]](#219438) - [[Security Solution] Fix alerts table potentially not applying alert assignees [#219460]](#219460) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
This PR fixes an issue with the logic implemented in this previous PR:
In the AI for SOC effort, each integration is bundled with a single rule. This means that deselecting a source from the Source filter button is equivalent to adding a filter to the search bar to exclude all alerts with the kibana.alert.rule.name property having the value of that integration.
The problem with the previous logic above is the value in the
kibana.alert.rule.namefield can be overridden (see `Rule name override here). Therefore filtering alerts by this value does not guarantee that all the alerts generated by the rule will be correctly filtered out.The new logic uses the
rule.idinstead of therule.name, which we then use to filter using thesignal.rule.idfield instead ofkibana.alert.rule.nameExample:
There are following 2 integrations installed:
This means that - in theory - there are the following 2 rules installed and running:
In this case, the
Sourcesbutton would show 2 entries, as follow:This PR also fixes a small miss in the prior PR that implemented the KPI section, where I had forgotten to pass the KQL filters to the charts.
Before
Screen.Recording.2025-04-01.at.1.30.03.PM.mov
After
Screen.Recording.2025-04-01.at.1.31.07.PM.mov
How to test
This needs to be ran in Serverless:
yarn es serverless --projectType securityyarn serverless-security --no-base-pathYou also need to enable the AI for SOC tier, by adding the following to your
serverless.security.dev.yamlfile:Use one of these Serverless users:
platform_engineerendpoint_operations_analystendpoint_policy_manageradminsystem_indices_superuserNotes
yarn test:generate:serverless-devgoogle_secops,microsoft_sentinel,,sentinel_oneandcrowdstrike)installedPackages: availablePackagesto force having some packages installedr.name === p.nameto make sure there will be matches between integrations and rulesChecklist
https://github.com/elastic/security-team/issues/11956