[Investigations] - Improve useFieldBrowser performance by michaelolo24 · Pull Request #225859 · elastic/kibana

michaelolo24 · 2025-06-30T14:50:32Z

Summary

This PR looks to improve the performance of the newly introduced useBrowserFields implementation. The currently used DataViewSpec.fields isn't performant at scale and is not cached. Existing code is as follows: DataView.toSpec call as seen here, here, and finally here. While this may not be a significant issue at a few thousand fields, this does not scale well as the number of fields reaches the tens to hundreds of thousands.

Separate PR that moves away from spec usage to pure DataView usage

The useFieldsBrowser hook is improved by relying directly on the DataView rather than the DataViewSpec. And there are improvements to the original sourcerer fieldBrowser calculation. Currently the use of memoizeOne doesn't actually work as the fields object passed to it is an object and not an array.

A new fieldBrowserManager was introduced that caches based on the scope. Originally investigated caching via a WeakMap to avoid the scenario where all 4 (current) scopes cache the exact same DataView, but it looked like references to previously selected dataViews retained a strong reference in memory, which often led to all N number of selected dataViews being retained in the cache. The current approach is safer as the number of cached entries are capped at DataViewManagerScopeName.length

** Relevant configurations **
xpack.securitySolution.enableExperimental: ['newDataViewPickerEnabled']

elasticmachine · 2025-07-02T19:04:18Z

Pinging @elastic/security-threat-hunting-investigations (Team:Threat Hunting:Investigations)

PhilippeOberti · 2025-07-03T12:17:16Z

...c/attack_discovery/pages/results/attack_discovery_panel/tabs/alerts_tab/ai_for_soc/table.tsx

  const { browserFields } = useMemo(
-    () => getDataViewStateFromIndexFields('', dataViewSpec.fields),
-    [dataViewSpec.fields]
+    () => browserFieldsManager.getBrowserFields(dataView, DataViewManagerScopeName.detections),


I feel like we should encourage developers to use the useBrowserFields hook instead of using the browserFieldsManager directly, unless there is a reason not to?

I think we'll make sure a bit more required once we move all the content of the data_view_manager folder into a package, but I think we should start using the correct files now if we can?

Yea, only problem is: useBrowserFields currently only properly works when the flag is on. useDataView doesn't return anything when the flag is off 😅 , so this would break everything. I'd rather not add custom logic to useBrowserFields, but can add a comment to these to replace them when the flag is defaulted on?

Is it better to now add custom logic to useBrowserFields or to add logic that will have to be removed every where it's being used? 😄

PhilippeOberti

LGTM! Left a few non-blocker comments, mostly cleanup and somewhat subjective so feel free to ignore if you disagree with them :)

...c/attack_discovery/pages/results/attack_discovery_panel/tabs/alerts_tab/ai_for_soc/table.tsx

x-pack/solutions/security/plugins/security_solution/public/common/containers/source/index.tsx

...ions/security/plugins/security_solution/public/data_view_manager/hooks/use_browser_fields.ts

.../plugins/security_solution/public/data_view_manager/utils/security_browser_fields_manager.ts

elasticmachine · 2025-07-07T19:14:10Z

💚 Build Succeeded

Buildkite Build
Commit: d2b9fa3

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	7853	7854	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	9.8MB	9.8MB	+1.8KB

Unknown metric groups

References to deprecated APIs

id	before	after	diff
`securitySolution`	345	348	+3

History

💛 Build #315968 was flaky 461fa85
💔 Build #315237 failed 7623bc5
💔 Build #315182 failed 030082b
💔 Build #315106 failed 01816ca
💔 Build #314875 failed 0c82d98

andrew-goldstein

Thanks for this improvement @michaelolo24 🙏
✅ Desk tested locally
LGTM 🚀

kibanamachine · 2025-07-07T20:23:29Z

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/16127071487

kibanamachine · 2025-07-07T20:23:43Z

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/16127075971

kibanamachine · 2025-07-07T20:29:52Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts You might need to backport the following PRs to 8.19: - [Security Solution][AI4DSOC] Fix table not applying alert tags for Attack discovery and Cases pages in AI4DSOC (#219410) - [AI4DSOC] Add checkboxes to the alert summary table (#219169)
❌	9.1	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 225859

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-07-07T20:30:11Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts You might need to backport the following PRs to 8.19: - [Security Solution][AI4DSOC] Fix table not applying alert tags for Attack discovery and Cases pages in AI4DSOC (#219410) - [AI4DSOC] Add checkboxes to the alert summary table (#219169)
❌	9.1	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 225859

Questions ?

Please refer to the Backport tool documentation

## Summary This PR looks to improve the performance of the newly introduced `useBrowserFields` implementation. The currently used `DataViewSpec.fields` isn't performant at scale and is not cached. Existing code is as follows: [DataView.toSpec](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/data_views/data_view.ts#L147) call as seen [here](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/data_views/abstract_data_views.ts#L391-L422), [here](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/fields/field_list.ts#L157-L159), and finally [here](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/fields/data_view_field.ts#L438-L440). While this may not be a significant issue at a few thousand fields, this does not scale well as the number of fields reaches the tens to hundreds of thousands. [Separate PR](elastic#225726) that moves away from spec usage to pure `DataView` usage The `useFieldsBrowser` hook is improved by relying directly on the DataView rather than the DataViewSpec. And there are improvements to the original sourcerer fieldBrowser calculation. Currently the use of `memoizeOne` doesn't actually work as the fields object passed to it is an object and not an array. A new fieldBrowserManager was introduced that caches based on the scope. Originally investigated caching via a `WeakMap` to avoid the scenario where all 4 (current) scopes cache the exact same DataView, but it looked like references to previously selected dataViews retained a strong reference in memory, which often led to all N number of selected dataViews being retained in the cache. The current approach is safer as the number of cached entries are capped at `DataViewManagerScopeName.length` ** Relevant configurations ** `xpack.securitySolution.enableExperimental: ['newDataViewPickerEnabled']` (cherry picked from commit 9a18757) # Conflicts: # x-pack/solutions/security/plugins/security_solution/public/cases/components/ai_for_soc/table.tsx

michaelolo24 · 2025-07-08T12:27:38Z

💚 All backports created successfully

Status	Branch	Result
✅	9.1
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

## Summary This PR looks to improve the performance of the newly introduced `useBrowserFields` implementation. The currently used `DataViewSpec.fields` isn't performant at scale and is not cached. Existing code is as follows: [DataView.toSpec](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/data_views/data_view.ts#L147) call as seen [here](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/data_views/abstract_data_views.ts#L391-L422), [here](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/fields/field_list.ts#L157-L159), and finally [here](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/fields/data_view_field.ts#L438-L440). While this may not be a significant issue at a few thousand fields, this does not scale well as the number of fields reaches the tens to hundreds of thousands. [Separate PR](elastic#225726) that moves away from spec usage to pure `DataView` usage The `useFieldsBrowser` hook is improved by relying directly on the DataView rather than the DataViewSpec. And there are improvements to the original sourcerer fieldBrowser calculation. Currently the use of `memoizeOne` doesn't actually work as the fields object passed to it is an object and not an array. A new fieldBrowserManager was introduced that caches based on the scope. Originally investigated caching via a `WeakMap` to avoid the scenario where all 4 (current) scopes cache the exact same DataView, but it looked like references to previously selected dataViews retained a strong reference in memory, which often led to all N number of selected dataViews being retained in the cache. The current approach is safer as the number of cached entries are capped at `DataViewManagerScopeName.length` ** Relevant configurations ** `xpack.securitySolution.enableExperimental: ['newDataViewPickerEnabled']` (cherry picked from commit 9a18757) # Conflicts: # x-pack/solutions/security/plugins/security_solution/public/cases/components/ai_for_soc/table.tsx

kibanamachine · 2025-07-08T20:48:46Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @michaelolo24

kibanamachine · 2025-07-09T20:49:19Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @michaelolo24

… (#227009) # Backport This will backport the following commits from `main` to `8.19`: - [[Investigations] - Improve useFieldBrowser performance (#225859)](#225859)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

#227004) # Backport This will backport the following commits from `main` to `9.1`: - [[Investigations] - Improve useFieldBrowser performance (#225859)](#225859)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

## Summary This PR looks to improve the performance of the newly introduced `useBrowserFields` implementation. The currently used `DataViewSpec.fields` isn't performant at scale and is not cached. Existing code is as follows: [DataView.toSpec](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/data_views/data_view.ts#L147) call as seen [here](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/data_views/abstract_data_views.ts#L391-L422), [here](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/fields/field_list.ts#L157-L159), and finally [here](https://github.com/elastic/kibana/blob/27183690142a5590b4ad72d060c43cad869f3f3c/src/platform/plugins/shared/data_views/common/fields/data_view_field.ts#L438-L440). While this may not be a significant issue at a few thousand fields, this does not scale well as the number of fields reaches the tens to hundreds of thousands. [Separate PR](elastic#225726) that moves away from spec usage to pure `DataView` usage The `useFieldsBrowser` hook is improved by relying directly on the DataView rather than the DataViewSpec. And there are improvements to the original sourcerer fieldBrowser calculation. Currently the use of `memoizeOne` doesn't actually work as the fields object passed to it is an object and not an array. A new fieldBrowserManager was introduced that caches based on the scope. Originally investigated caching via a `WeakMap` to avoid the scenario where all 4 (current) scopes cache the exact same DataView, but it looked like references to previously selected dataViews retained a strong reference in memory, which often led to all N number of selected dataViews being retained in the cache. The current approach is safer as the number of cached entries are capped at `DataViewManagerScopeName.length` ** Relevant configurations ** `xpack.securitySolution.enableExperimental: ['newDataViewPickerEnabled']`

michaelolo24 force-pushed the use-browser-fields-performance branch from e23f816 to 22fd768 Compare June 30, 2025 17:26

michaelolo24 added release_note:skip Skip the PR/issue when compiling release notes Team:Threat Hunting:Investigations Security Solution Threat Hunting Investigations Team backport:version Backport to applied version labels v9.1.0 v8.19.0 labels Jun 30, 2025

michaelolo24 force-pushed the use-browser-fields-performance branch 4 times, most recently from 01816ca to 030082b Compare July 2, 2025 18:57

michaelolo24 marked this pull request as ready for review July 2, 2025 19:04

michaelolo24 requested review from a team as code owners July 2, 2025 19:04

michaelolo24 requested a review from umbopepato July 2, 2025 19:04

michaelolo24 force-pushed the use-browser-fields-performance branch from 6407cdd to aa88062 Compare July 3, 2025 12:44

PhilippeOberti reviewed Jul 3, 2025

View reviewed changes

michaelolo24 force-pushed the use-browser-fields-performance branch from aa88062 to 461fa85 Compare July 4, 2025 23:53

PhilippeOberti approved these changes Jul 7, 2025

View reviewed changes

lgestc approved these changes Jul 7, 2025

View reviewed changes

michaelolo24 added 5 commits July 7, 2025 12:33

improve useFieldBrowser performance

8a56145

fix lint error

591f10c

fix tests and improve caching

8c99001

use with useBrowserFields

dd840aa

update comments and documentation

d2b9fa3

michaelolo24 force-pushed the use-browser-fields-performance branch from 461fa85 to d2b9fa3 Compare July 7, 2025 17:32

andrew-goldstein approved these changes Jul 7, 2025

View reviewed changes

michaelolo24 merged commit 9a18757 into elastic:main Jul 7, 2025
12 checks passed

kibanamachine added the v9.2.0 label Jul 7, 2025

michaelolo24 mentioned this pull request Jul 8, 2025

[9.1] [Investigations] - Improve useFieldBrowser performance (#225859) #227004

Merged

michaelolo24 mentioned this pull request Jul 8, 2025

[8.19] [Investigations] - Improve useFieldBrowser performance (#225859) #227009

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jul 8, 2025

kibanamachine mentioned this pull request Jul 9, 2025

[Security Solution][Sourcerer] Add dedicated scope for explore pages #226545

Merged

2 tasks

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jul 10, 2025

Conversation

michaelolo24 commented Jun 30, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

elasticmachine commented Jul 2, 2025

Uh oh!

PhilippeOberti Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

PhilippeOberti Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

michaelolo24 Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

PhilippeOberti Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

PhilippeOberti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jul 7, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

References to deprecated APIs

History

Uh oh!

andrew-goldstein left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Jul 7, 2025

Uh oh!

kibanamachine commented Jul 7, 2025

Uh oh!

kibanamachine commented Jul 7, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

kibanamachine commented Jul 7, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

michaelolo24 commented Jul 8, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Jul 8, 2025

Uh oh!

kibanamachine commented Jul 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

michaelolo24 commented Jun 30, 2025 •

edited by kibanamachine

Loading