[SecuritySolution] [Bug] Fix Manage data sources page index import by machadoum · Pull Request #226099 · elastic/kibana

machadoum · 2025-07-02T07:46:42Z

Fix the Manage Data Sources page index import by implementing the creation, updating, deletion, and retrieval of entity source configurations. It includes updates to the TypeScript definitions, API schema, and client-side API methods to support these operations.

After this change, the user will be able to create only one data source using the UI.

API and types changes

Added new types (CreateMonitoringEntitySource, UpdatedMonitoringEntitySource, MonitoringEntitySource) with expanded fields for entity source configurations, such as indexPattern, enabled, error, and matchers. [1] [2] [3]
Updated the API schema (monitoring_entity_source.schema.yaml) to reflect new entity source operations, including CreateEntitySource, UpdateEntitySource, DeleteEntitySource, GetEntitySource, and ListEntitySources. Added support for query parameters and refined response schema definitions. [1] [2] [3] [4] [5] [6] [7]
Add a uniqueness assertion for the data source name.
Create the data source 'PUT' API that allows updates
Force the managed to be consistently defined by default
Update list data sources API to support filter params

UI changes

Extract IndexImportManageDataSource component to its file.
Stop calling the init API on updates
Implement the edit mode instead of creating a new data source on every save
- Load the already added indices when editing
Fix the displayed added indices message to show the right value

Video

Screen.Recording.2025-07-02.at.10.16.15.mov

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

elasticmachine · 2025-07-02T11:33:06Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2025-07-02T11:33:07Z

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

CAWilson94 · 2025-07-04T08:32:01Z

...erver/lib/entity_analytics/privilege_monitoring/monitoring_entity_source_data_client.test.ts


      defaultOpts.soClient.update.mockResolvedValue({
-        id: `temp-id`, // TODO: https://github.com/elastic/security-team/issues/12851
+        id: 'entity-analytics-monitoring-entity-source-test-namespace-test-type-test-index-pattern', // TODO: https://github.com/elastic/security-team/issues/12851


We can move this comment out with the id here now - thank you for this Pablo 🙇

Sure. Just to be 100% clear. This PR doesn't fix the issue described here https://github.com/elastic/security-team/issues/12851

It just ensures the create and update methods use the same ID. And the if is still based on the initial index pattern, which is weird since the user can update the index. Therefore, we need to work on improving the ID generation process.

CAWilson94

Code looks good, nice test additions too 🚀 Thank you!

tiansivive

LGTM ✅
Left a question about a comment

...analytics/components/privileged_user_monitoring_onboarding/components/select_index_modal.tsx

...n/server/lib/entity_analytics/privilege_monitoring/saved_objects/monitoring_entity_source.ts

elasticmachine · 2025-07-07T09:20:57Z

💔 Build Failed

Buildkite Build
Commit: 49c0e5d

Failed CI Steps

Test Failures

[job] [logs] Jest Tests #3 / MonitoringEntitySourceDataClient init should initialize Monitoring Entity Source Sync Config Successfully
[job] [logs] Jest Tests #3 / MonitoringEntitySourceDataClient init should initialize Monitoring Entity Source Sync Config Successfully
[job] [logs] Jest Tests #3 / MonitoringEntitySourceDataClient init should not create Monitoring Entity Source Sync Config when a SO already exist with the same name
[job] [logs] Jest Tests #3 / MonitoringEntitySourceDataClient init should not create Monitoring Entity Source Sync Config when a SO already exist with the same name
[job] [logs] Jest Tests #3 / MonitoringEntitySourceDataClient init should update Monitoring Entity Source Sync Config Successfully when calling init when the SO already exists
[job] [logs] Jest Tests #3 / MonitoringEntitySourceDataClient init should update Monitoring Entity Source Sync Config Successfully when calling init when the SO already exists
[job] [logs] Jest Tests #3 / MonitoringEntitySourceDataClient update should update Monitoring Entity Source Sync Config Successfully
[job] [logs] Jest Tests #3 / MonitoringEntitySourceDataClient update should update Monitoring Entity Source Sync Config Successfully

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	7819	7821	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	9.8MB	9.8MB	+1.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	94.2KB	94.2KB	+1.0B

History

💚 Build #315744 succeeded a440abb
💔 Build #314993 failed 45c16e6
💔 Build #314864 failed ec89f28
💔 Build #314767 failed f87b49f

cc @machadoum

kibanamachine · 2025-07-07T12:39:29Z

Starting backport for target branches: 9.1

https://github.com/elastic/kibana/actions/runs/16117178051

kibanamachine · 2025-07-07T12:39:41Z

Starting backport for target branches: 9.1

https://github.com/elastic/kibana/actions/runs/16117183088

kibanamachine · 2025-07-07T12:45:41Z

💔 All backports failed

Status	Branch	Result
❌	9.1	Backport failed because of merge conflicts You might need to backport the following PRs to 9.1: - Add API Key Management and Enable Index Sync via Scheduled Task (#225551)

Manual backport

To create the backport manually run:

node scripts/backport --pr 226099

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-07-07T12:45:50Z

💔 All backports failed

Status	Branch	Result
❌	9.1	Backport failed because of merge conflicts You might need to backport the following PRs to 9.1: - Add API Key Management and Enable Index Sync via Scheduled Task (#225551)

Manual backport

To create the backport manually run:

node scripts/backport --pr 226099

Questions ?

Please refer to the Backport tool documentation

machadoum · 2025-07-08T07:10:05Z

💚 All backports created successfully

Status	Branch	Result
✅	9.1

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…lastic#226099) Fix the Manage Data Sources page index import by implementing the creation, updating, deletion, and retrieval of entity source configurations. It includes updates to the TypeScript definitions, API schema, and client-side API methods to support these operations. After this change, the user will be able to create only one data source using the UI. ### API and types changes * Added new types (`CreateMonitoringEntitySource`, `UpdatedMonitoringEntitySource`, `MonitoringEntitySource`) with expanded fields for entity source configurations, such as `indexPattern`, `enabled`, `error`, and `matchers`. [[1]](diffhunk://#diff-b8c44f34825a1f7fa2c3f70fdda00f38b07e19ef8600e1a2e937660346d95c72R18-R21) [[2]](diffhunk://#diff-b8c44f34825a1f7fa2c3f70fdda00f38b07e19ef8600e1a2e937660346d95c72L39-R64) [[3]](diffhunk://#diff-b8c44f34825a1f7fa2c3f70fdda00f38b07e19ef8600e1a2e937660346d95c72R76-R122) * Updated the API schema (`monitoring_entity_source.schema.yaml`) to reflect new entity source operations, including `CreateEntitySource`, `UpdateEntitySource`, `DeleteEntitySource`, `GetEntitySource`, and `ListEntitySources`. Added support for query parameters and refined response schema definitions. [[1]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L10-R30) [[2]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L42-R48) [[3]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L58-R72) [[4]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L78-R102) [[5]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L88-R114) [[6]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L128-R189) [[7]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1R219-R220) * Add a uniqueness assertion for the data source name. * Create the data source 'PUT' API that allows updates * Force the `managed` to be consistently defined by default * Update list data sources API to support filter params ### UI changes * Extract `IndexImportManageDataSource` component to its file. * Stop calling the `init` API on updates * Implement the edit mode instead of creating a new data source on every save * Load the already added indices when editing * Fix the displayed added indices message to show the right value ### Video https://github.com/user-attachments/assets/9d1c1ad1-09b9-49f0-8cea-1b640a26e954 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. (cherry picked from commit 8453edc)

…ort (#226099) (#226921) # Backport This will backport the following commits from `main` to `9.1`: - [[SecuritySolution] [Bug] Fix Manage data sources page index import (#226099)](#226099)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…lastic#226099) Fix the Manage Data Sources page index import by implementing the creation, updating, deletion, and retrieval of entity source configurations. It includes updates to the TypeScript definitions, API schema, and client-side API methods to support these operations. After this change, the user will be able to create only one data source using the UI. ### API and types changes * Added new types (`CreateMonitoringEntitySource`, `UpdatedMonitoringEntitySource`, `MonitoringEntitySource`) with expanded fields for entity source configurations, such as `indexPattern`, `enabled`, `error`, and `matchers`. [[1]](diffhunk://#diff-b8c44f34825a1f7fa2c3f70fdda00f38b07e19ef8600e1a2e937660346d95c72R18-R21) [[2]](diffhunk://#diff-b8c44f34825a1f7fa2c3f70fdda00f38b07e19ef8600e1a2e937660346d95c72L39-R64) [[3]](diffhunk://#diff-b8c44f34825a1f7fa2c3f70fdda00f38b07e19ef8600e1a2e937660346d95c72R76-R122) * Updated the API schema (`monitoring_entity_source.schema.yaml`) to reflect new entity source operations, including `CreateEntitySource`, `UpdateEntitySource`, `DeleteEntitySource`, `GetEntitySource`, and `ListEntitySources`. Added support for query parameters and refined response schema definitions. [[1]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L10-R30) [[2]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L42-R48) [[3]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L58-R72) [[4]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L78-R102) [[5]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L88-R114) [[6]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1L128-R189) [[7]](diffhunk://#diff-e7f2de573ae15c6a3c3781b2a97b9875299c0674f81dd673d2bf9b3a6cf397f1R219-R220) * Add a uniqueness assertion for the data source name. * Create the data source 'PUT' API that allows updates * Force the `managed` to be consistently defined by default * Update list data sources API to support filter params ### UI changes * Extract `IndexImportManageDataSource` component to its file. * Stop calling the `init` API on updates * Implement the edit mode instead of creating a new data source on every save * Load the already added indices when editing * Fix the displayed added indices message to show the right value ### Video https://github.com/user-attachments/assets/9d1c1ad1-09b9-49f0-8cea-1b640a26e954 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels.

Fix Manage data sources page

f87b49f

machadoum force-pushed the siem-ea-12915 branch from 580a579 to f87b49f Compare July 2, 2025 07:55

machadoum changed the title ~~Siem ea 12915~~ [SecuritySolution] [Bug] Fix Manage data sources page index import Jul 2, 2025

machadoum self-assigned this Jul 2, 2025

machadoum added 2 commits July 2, 2025 10:33

Self code review

442c17e

Improve tests

ec89f28

machadoum marked this pull request as ready for review July 2, 2025 11:33

machadoum requested review from a team as code owners July 2, 2025 11:33

machadoum requested a review from tiansivive July 2, 2025 11:33

machadoum added 3 commits July 2, 2025 15:58

Fix types

45c16e6

Fix unit tests

622ba90

Merge branch 'main' into siem-ea-12915

7dba912

CAWilson94 reviewed Jul 4, 2025

View reviewed changes

CAWilson94 approved these changes Jul 4, 2025

View reviewed changes

Remove comment

a440abb

tiansivive approved these changes Jul 7, 2025

View reviewed changes

...analytics/components/privileged_user_monitoring_onboarding/components/select_index_modal.tsx Show resolved Hide resolved

...n/server/lib/entity_analytics/privilege_monitoring/saved_objects/monitoring_entity_source.ts Show resolved Hide resolved

Merge remote-tracking branch 'kibana/main' into siem-ea-12915

49c0e5d

machadoum enabled auto-merge (squash) July 7, 2025 08:05

Fix merge conflicts

b2cb2bf

machadoum merged commit 8453edc into elastic:main Jul 7, 2025
13 checks passed

kibanamachine added the v9.2.0 label Jul 7, 2025

machadoum mentioned this pull request Jul 8, 2025

[9.1] [SecuritySolution] [Bug] Fix Manage data sources page index import (#226099) #226921

Merged

Conversation

machadoum commented Jul 2, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

API and types changes

UI changes

Video

Checklist

Uh oh!

elasticmachine commented Jul 2, 2025

Uh oh!

elasticmachine commented Jul 2, 2025

Uh oh!

CAWilson94 Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

machadoum Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

CAWilson94 left a comment

Choose a reason for hiding this comment

Uh oh!

tiansivive left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

Uh oh!

Uh oh!

kibanamachine commented Jul 7, 2025

Uh oh!

kibanamachine commented Jul 7, 2025

Uh oh!

kibanamachine commented Jul 7, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

kibanamachine commented Jul 7, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

machadoum commented Jul 8, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

machadoum commented Jul 2, 2025 •

edited by kibanamachine

Loading

elasticmachine commented Jul 7, 2025 •

edited

Loading