Skip to content

[AI4DSOC] Fix issue with filtering by integrations#216574

Merged
PhilippeOberti merged 4 commits intoelastic:mainfrom
PhilippeOberti:alert-summary-source-filter-fix
Apr 4, 2025
Merged

[AI4DSOC] Fix issue with filtering by integrations#216574
PhilippeOberti merged 4 commits intoelastic:mainfrom
PhilippeOberti:alert-summary-source-filter-fix

Conversation

@PhilippeOberti
Copy link
Contributor

@PhilippeOberti PhilippeOberti commented Apr 1, 2025

Summary

This PR fixes an issue with the logic implemented in this previous PR:
In the AI for SOC effort, each integration is bundled with a single rule. This means that deselecting a source from the Source filter button is equivalent to adding a filter to the search bar to exclude all alerts with the kibana.alert.rule.name property having the value of that integration.

The problem with the previous logic above is the value in the kibana.alert.rule.name field can be overridden (see `Rule name override here). Therefore filtering alerts by this value does not guarantee that all the alerts generated by the rule will be correctly filtered out.

The new logic uses the rule.id instead of the rule.name, which we then use to filter using the signal.rule.id field instead of kibana.alert.rule.name

Example:

There are following 2 integrations installed:

[
  {
    id: 'splunk',
    name: 'splunk',
    status: installationStatuses.Installed,
    title: 'Splunk',
    version: '',
  },
  {
    id: 'google_secops',
    name: 'google_secops',
    status: installationStatuses.Installed,
    title: 'Google SecOps',
    version: '',
  },
]

This means that - in theory - there are the following 2 rules installed and running:

[
  {
    related_integrations: [{ package: 'splunk' }],
    id: 'splunk_rule_id',
  },
  {
    related_integrations: [{ package: 'google_secops' }],
    id: 'google_secops_rule_id',
  },
]

In this case, the Sources button would show 2 entries, as follow:

[
  {
    checked: 'on',
    key: 'splunk_rule_id',
    label: 'Splunk',
  },
  {
    checked: 'on',
    key: 'google_secops_rule_id',
    label: 'Splunk',
  },
]

This PR also fixes a small miss in the prior PR that implemented the KPI section, where I had forgotten to pass the KQL filters to the charts.

Before

Screen.Recording.2025-04-01.at.1.30.03.PM.mov

After

Screen.Recording.2025-04-01.at.1.31.07.PM.mov

How to test

This needs to be ran in Serverless:

  • yarn es serverless --projectType security
  • yarn serverless-security --no-base-path

You also need to enable the AI for SOC tier, by adding the following to your serverless.security.dev.yaml file:

xpack.securitySolutionServerless.productTypes:
  [
    { product_line: 'ai_soc', product_tier: 'search_ai_lake' },
  ]

Use one of these Serverless users:

  • platform_engineer
  • endpoint_operations_analyst
  • endpoint_policy_manager
  • admin
  • system_indices_superuser

Notes

  • generate data: yarn test:generate:serverless-dev
  • create 4 catch all rules, each with a name of a AI for SOC integration (google_secops, microsoft_sentinel,, sentinel_one and crowdstrike)
  • change this line to installedPackages: availablePackages to force having some packages installed
  • change this line to r.name === p.name to make sure there will be matches between integrations and rules

Checklist

https://github.com/elastic/security-team/issues/11956

@PhilippeOberti PhilippeOberti added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:Threat Hunting:Investigations Security Solution Threat Hunting Investigations Team Team:Security Generative AI Security Generative AI v9.1.0 labels Apr 1, 2025
@PhilippeOberti PhilippeOberti requested review from a team as code owners April 1, 2025 03:39
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting-investigations (Team:Threat Hunting:Investigations)

@PhilippeOberti PhilippeOberti force-pushed the alert-summary-source-filter-fix branch from d403ab1 to 3cfdfcc Compare April 1, 2025 18:24
@PhilippeOberti PhilippeOberti force-pushed the alert-summary-source-filter-fix branch from 3cfdfcc to 6ce0b4f Compare April 1, 2025 18:34
Copy link
Contributor

@christineweng christineweng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code changes LGTM.

Do you know the difference between rule.id and rule_id? I'm seeing different values in my local
image

@PhilippeOberti
Copy link
Contributor Author

Code changes LGTM.

Do you know the difference between rule.id and rule_id? I'm seeing different values in my local image

I think this is because a rule object has both an id field and a rule_id field. After doing some checks the one I need is id. Not sure what the rule_id actually is 😆

@PhilippeOberti PhilippeOberti enabled auto-merge (squash) April 3, 2025 19:38
@PhilippeOberti PhilippeOberti merged commit 2ed4266 into elastic:main Apr 4, 2025
9 checks passed
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 8.9MB 8.9MB +92.0B

History

@PhilippeOberti PhilippeOberti deleted the alert-summary-source-filter-fix branch April 4, 2025 19:10
PhilippeOberti added a commit to PhilippeOberti/kibana that referenced this pull request May 30, 2025
## Summary

This PR fixes an issue with the logic implemented in [this previous
PR](elastic#215586):
_In the AI for SOC effort, each integration is bundled with a single
rule. This means that deselecting a source from the Source filter button
is equivalent to adding a filter to the search bar to exclude all alerts
with the kibana.alert.rule.name property having the value of that
integration._

The problem with the previous logic above is the value in the
`kibana.alert.rule.name` field can be overridden (see `Rule name
override
[here](https://www.elastic.co/guide/en/security/current/rules-ui-create.html)).
Therefore filtering alerts by this value does not guarantee that all the
alerts generated by the rule will be correctly filtered out.

The new logic uses the `rule.id` instead of the `rule.name`, which we
then use to filter using the `signal.rule.id` field instead of
`kibana.alert.rule.name`

### Example:

 There are following 2 integrations installed:
```typescript
[
  {
    id: 'splunk',
    name: 'splunk',
    status: installationStatuses.Installed,
    title: 'Splunk',
    version: '',
  },
  {
    id: 'google_secops',
    name: 'google_secops',
    status: installationStatuses.Installed,
    title: 'Google SecOps',
    version: '',
  },
]
```

This means that - in theory - there are the following 2 rules installed
and running:
```typescript
[
  {
    related_integrations: [{ package: 'splunk' }],
    id: 'splunk_rule_id',
  },
  {
    related_integrations: [{ package: 'google_secops' }],
    id: 'google_secops_rule_id',
  },
]
```

In this case, the `Sources` button would show 2 entries, as follow:
```typescript
[
  {
    checked: 'on',
    key: 'splunk_rule_id',
    label: 'Splunk',
  },
  {
    checked: 'on',
    key: 'google_secops_rule_id',
    label: 'Splunk',
  },
]
```

This PR also fixes a small miss in [the prior
PR](elastic#215585) that implemented the
KPI section, where I had forgotten to pass the KQL filters to the
charts.

#### Before

https://github.com/user-attachments/assets/77e583c6-718f-46d9-96b4-42ee9976161b

#### After

https://github.com/user-attachments/assets/50e8e541-5798-4906-b7cc-4f9756dbdefc

## How to test

This needs to be ran in Serverless:
- `yarn es serverless --projectType security`
- `yarn serverless-security --no-base-path`

You also need to enable the AI for SOC tier, by adding the following to
your `serverless.security.dev.yaml` file:
```
xpack.securitySolutionServerless.productTypes:
  [
    { product_line: 'ai_soc', product_tier: 'search_ai_lake' },
  ]
```

Use one of these Serverless users:
- `platform_engineer`
- `endpoint_operations_analyst`
- `endpoint_policy_manager`
- `admin`
- `system_indices_superuser`

### Notes

- generate data: `yarn test:generate:serverless-dev`
- create 4 catch all rules, each with a name of a AI for SOC integration
(`google_secops`, `microsoft_sentinel`,, `sentinel_one` and
`crowdstrike`)
- change [this
line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_fetch_integrations.ts#L73)
to `installedPackages: availablePackages` to force having some packages
installed
- change [this
line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_integrations.ts#L63)
to `r.name === p.name` to make sure there will be matches between
integrations and rules

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

elastic/security-team#11956
(cherry picked from commit 2ed4266)
PhilippeOberti added a commit to PhilippeOberti/kibana that referenced this pull request May 30, 2025
## Summary

This PR fixes an issue with the logic implemented in [this previous
PR](elastic#215586):
_In the AI for SOC effort, each integration is bundled with a single
rule. This means that deselecting a source from the Source filter button
is equivalent to adding a filter to the search bar to exclude all alerts
with the kibana.alert.rule.name property having the value of that
integration._

The problem with the previous logic above is the value in the
`kibana.alert.rule.name` field can be overridden (see `Rule name
override
[here](https://www.elastic.co/guide/en/security/current/rules-ui-create.html)).
Therefore filtering alerts by this value does not guarantee that all the
alerts generated by the rule will be correctly filtered out.

The new logic uses the `rule.id` instead of the `rule.name`, which we
then use to filter using the `signal.rule.id` field instead of
`kibana.alert.rule.name`

### Example:

 There are following 2 integrations installed:
```typescript
[
  {
    id: 'splunk',
    name: 'splunk',
    status: installationStatuses.Installed,
    title: 'Splunk',
    version: '',
  },
  {
    id: 'google_secops',
    name: 'google_secops',
    status: installationStatuses.Installed,
    title: 'Google SecOps',
    version: '',
  },
]
```

This means that - in theory - there are the following 2 rules installed
and running:
```typescript
[
  {
    related_integrations: [{ package: 'splunk' }],
    id: 'splunk_rule_id',
  },
  {
    related_integrations: [{ package: 'google_secops' }],
    id: 'google_secops_rule_id',
  },
]
```

In this case, the `Sources` button would show 2 entries, as follow:
```typescript
[
  {
    checked: 'on',
    key: 'splunk_rule_id',
    label: 'Splunk',
  },
  {
    checked: 'on',
    key: 'google_secops_rule_id',
    label: 'Splunk',
  },
]
```

This PR also fixes a small miss in [the prior
PR](elastic#215585) that implemented the
KPI section, where I had forgotten to pass the KQL filters to the
charts.

#### Before

https://github.com/user-attachments/assets/77e583c6-718f-46d9-96b4-42ee9976161b

#### After

https://github.com/user-attachments/assets/50e8e541-5798-4906-b7cc-4f9756dbdefc

## How to test

This needs to be ran in Serverless:
- `yarn es serverless --projectType security`
- `yarn serverless-security --no-base-path`

You also need to enable the AI for SOC tier, by adding the following to
your `serverless.security.dev.yaml` file:
```
xpack.securitySolutionServerless.productTypes:
  [
    { product_line: 'ai_soc', product_tier: 'search_ai_lake' },
  ]
```

Use one of these Serverless users:
- `platform_engineer`
- `endpoint_operations_analyst`
- `endpoint_policy_manager`
- `admin`
- `system_indices_superuser`

### Notes

- generate data: `yarn test:generate:serverless-dev`
- create 4 catch all rules, each with a name of a AI for SOC integration
(`google_secops`, `microsoft_sentinel`,, `sentinel_one` and
`crowdstrike`)
- change [this
line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_fetch_integrations.ts#L73)
to `installedPackages: availablePackages` to force having some packages
installed
- change [this
line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_integrations.ts#L63)
to `r.name === p.name` to make sure there will be matches between
integrations and rules

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

elastic/security-team#11956
(cherry picked from commit 2ed4266)
PhilippeOberti added a commit to PhilippeOberti/kibana that referenced this pull request Jun 4, 2025
## Summary

This PR fixes an issue with the logic implemented in [this previous
PR](elastic#215586):
_In the AI for SOC effort, each integration is bundled with a single
rule. This means that deselecting a source from the Source filter button
is equivalent to adding a filter to the search bar to exclude all alerts
with the kibana.alert.rule.name property having the value of that
integration._

The problem with the previous logic above is the value in the
`kibana.alert.rule.name` field can be overridden (see `Rule name
override
[here](https://www.elastic.co/guide/en/security/current/rules-ui-create.html)).
Therefore filtering alerts by this value does not guarantee that all the
alerts generated by the rule will be correctly filtered out.

The new logic uses the `rule.id` instead of the `rule.name`, which we
then use to filter using the `signal.rule.id` field instead of
`kibana.alert.rule.name`

### Example:

 There are following 2 integrations installed:
```typescript
[
  {
    id: 'splunk',
    name: 'splunk',
    status: installationStatuses.Installed,
    title: 'Splunk',
    version: '',
  },
  {
    id: 'google_secops',
    name: 'google_secops',
    status: installationStatuses.Installed,
    title: 'Google SecOps',
    version: '',
  },
]
```

This means that - in theory - there are the following 2 rules installed
and running:
```typescript
[
  {
    related_integrations: [{ package: 'splunk' }],
    id: 'splunk_rule_id',
  },
  {
    related_integrations: [{ package: 'google_secops' }],
    id: 'google_secops_rule_id',
  },
]
```

In this case, the `Sources` button would show 2 entries, as follow:
```typescript
[
  {
    checked: 'on',
    key: 'splunk_rule_id',
    label: 'Splunk',
  },
  {
    checked: 'on',
    key: 'google_secops_rule_id',
    label: 'Splunk',
  },
]
```

This PR also fixes a small miss in [the prior
PR](elastic#215585) that implemented the
KPI section, where I had forgotten to pass the KQL filters to the
charts.

#### Before

https://github.com/user-attachments/assets/77e583c6-718f-46d9-96b4-42ee9976161b

#### After

https://github.com/user-attachments/assets/50e8e541-5798-4906-b7cc-4f9756dbdefc

## How to test

This needs to be ran in Serverless:
- `yarn es serverless --projectType security`
- `yarn serverless-security --no-base-path`

You also need to enable the AI for SOC tier, by adding the following to
your `serverless.security.dev.yaml` file:
```
xpack.securitySolutionServerless.productTypes:
  [
    { product_line: 'ai_soc', product_tier: 'search_ai_lake' },
  ]
```

Use one of these Serverless users:
- `platform_engineer`
- `endpoint_operations_analyst`
- `endpoint_policy_manager`
- `admin`
- `system_indices_superuser`

### Notes

- generate data: `yarn test:generate:serverless-dev`
- create 4 catch all rules, each with a name of a AI for SOC integration
(`google_secops`, `microsoft_sentinel`,, `sentinel_one` and
`crowdstrike`)
- change [this
line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_fetch_integrations.ts#L73)
to `installedPackages: availablePackages` to force having some packages
installed
- change [this
line](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detections/hooks/alert_summary/use_integrations.ts#L63)
to `r.name === p.name` to make sure there will be matches between
integrations and rules

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

elastic/security-team#11956
(cherry picked from commit 2ed4266)
PhilippeOberti added a commit that referenced this pull request Jun 4, 2025
…) (#222074)

# Backport

This will backport the following commits from `main` to `8.19`:
- [[AI4DSOC] Alert summary page routing and initialization
(#214889)](#214889)
- [[AI4DSOC] Alert summary landing page
(#215246)](#215246)
- [[AI4DSOC] Alert summary dataview
(#215265)](#215265)
- [[AI4DSOC] Alert summary KQL bar
[#215586]](#215586)
- [[AI4DSOC] Alert summary KPI charts
[#215585]](#215585)
- [[AI4DSOR] Alert summary integrations section
[#215266]](#215266)
- [[AI4DSOC] Fix issue with filtering by integrations
[#216574]](#216574)
- [[AI4DSOC] Alert summary table setup
[#216744]](#216744)
- [Alerty summary table flyout setup
[#217421]](#217421)
- [[AI4DSOC] Alert summary alert actions in table and flyout
[#217696]](#217696)
- [[AI4DSOC] Alert summary table custom cell renderers
[#217124]](#217124)
- [[AI4DSOC] Alert summary table and flyout ai assistant
[#217744]](#217744)
- [[AI4DSOC] Alert summary page performance improvements
[#218632]](#218632)
- [[AI4DSOC] Change the Attack Discovery page to use the AI for SOC
alerts table [#218736]](#218736)
- [[AI4DSOC] Change the Cases page to use the AI for SOC alerts table
[#218742]](#218742)
- [[AI4DSOC] Fix spacing issue on alert summary landing page integration
card [#218868]](#218868)
- [[AI4DSOC][ResponseOps] Fix alerts table not handling undefined
maintenanceWindow capability
[#218999]](#218999)
- [[AI4DSOC] Fix link to the new integrations page
[#219030]](#219030)
- [[AI4DSOC] Disable CellActions and PreviewLinks on the Attack
discovery page [#219033]](#219033)
- [[AI4DSOC] Add cell renderer for datetime fields to the alert summary
table [#219126]](#219126)
- [[AI4DSOC] Remove Assistant icon from row action in alert summary
table [#219141]](#219141)
- [[AI4DSOC] Add checkboxes to the alert summary table
[#219169]](#219169)
- [[Security Solution][AI4DSOC] Fix table not applying alert tags for
Attack discovery and Cases pages in AI4DSOC
[#219410]](#219410)
- [[AI4DSOC] Fix logic that renders the group title when grouping by
integrations [#219430]](#219430)
- [[AI4DSOC] Alert summary table truncates long values and display the
field/value pair in tooltip
[#219438]](#219438)
- [[Security Solution] Fix alerts table potentially not applying alert
assignees [#219460]](#219460)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:Security Generative AI Security Generative AI Team:Threat Hunting:Investigations Security Solution Threat Hunting Investigations Team v8.19.0 v9.1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants