Saved Object Mapping for Entity Source Sync Configuration by CAWilson94 · Pull Request #219680 · elastic/kibana

CAWilson94 · 2025-04-30T08:58:01Z

Summary

This PR introduces a new Saved Object mapping for Monitoring Entity Source Configurations to support Privileged User Monitoring.

✅ Implemented

Saved object mapping schema aligned with the current spec
DescriptorClient to handle saved object methods
OpenAPI schema and generated types
Implemented Data Client - SO operations exposed via GET and POST API
Created route and registered with privmonRoutes
Data Client Unit Testing
Rename typename of "monitoring_entity_source-sync" -> "entity-analytics-monitoring-entity-source"

Figma - currently working in the pink box

Testing Steps:

privilegeMonitoringEnabled: true - set this flag
From Dev tools - create a saved object via POST (below) and view this saved object via GET (also below):

POST kbn:/api/entity_analytics/monitoring/entity_source
{
  "type": "some-type-here",
  "name": "name-here",
  "managed": false,
  "indexPattern": "logs-*",
  "enabled": true,
  "integrationName": "Okta",
 "matchers": [
    {
      "fields": ["user.role"],
      "values": ["admin"]
    }
  ],
  "filter": {}
}

Expected output:

{
  "type": "some-type-here",
  "name": "name-here",
  "managed": false,
  "indexPattern": "logs-*",
  "enabled": true,
  "integrationName": "Okta",
  "matchers": [
    {
      "fields": [
        "user.role"
      ],
      "values": [
        "admin"
      ]
    }
  ],
  "filter": {}
}

GET Example

GET kbn:/api/entity_analytics/monitoring/entity_source

Output for get, expected to be the same as that for POST.

prodsecmachine · 2025-05-01T07:47:48Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

Copilot

Pull Request Overview

This PR introduces a new saved object mapping for monitoring entity source configurations to support privileged user monitoring. It defines the saved object mapping schema with key fields and implements a descriptor client to manage these objects while also adding an OpenAPI-generated Zod schema for API type validation.

Defined mappings for the new saved object type with fields such as type, name, managed, and matchers.
Implemented a descriptor client for initializing, updating, and querying the monitoring entity source object.
Added a generated API file for validating the monitoring entity source descriptor.

Reviewed Changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File	Description
x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source_type.ts	Created a new saved object type with mappings and a model version update.
x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source.ts	Added a descriptor client for managing the monitoring entity source object.
x-pack/solutions/security/plugins/security_solution/common/api/entity_analytics/privilege_monitoring/monitoring_entity_source/monitoring_entity_source.gen.ts	Generated Zod schema for type validation of the monitoring entity source descriptor.

Files not reviewed (1)

x-pack/solutions/security/plugins/security_solution/common/api/entity_analytics/privilege_monitoring/monitoring_entity_source/monitoring_entity_source.schema.yaml: Language not supported

…s; wired in get and create SO via routes and data client; added to context factory and types in secsol

Copilot

Pull Request Overview

This PR introduces a new saved object mapping and supporting infrastructure (data client and routes) for Monitoring Entity Source Sync Configuration to support Privileged User Monitoring.

Adds saved object schema, client, and API routes
Integrates the new data client into the request context and registers it with the existing privilege monitoring routes
Provides OpenAPI schema and generated types for the configuration

Reviewed Changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
x-pack/solutions/security/plugins/security_solution/server/types.ts	Added import and context method for the new data client
x-pack/solutions/security/plugins/security_solution/server/saved_objects.ts	Registered a new saved object type mapping
x-pack/solutions/security/plugins/security_solution/server/request_context_factory.ts	Added memoized method to instantiate the new data client
x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source_type.ts	Defined the saved object mapping for the monitoring entity source
x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source.ts	Implemented the saved object client for monitoring entity source operations
x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/privilege_monitoring/routes/monitoring_entity_source_sync.ts	Added POST and GET API routes for sync configuration
x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/privilege_monitoring/monitoring_entity_source_sync_data_client.ts	Provided the data client that wraps saved object operations
x-pack/solutions/security/plugins/security_solution/common/api/entity_analytics/privilege_monitoring/monitoring_entity_source/monitoring_entity_source.gen.ts	Introduced auto-generated OpenAPI schema and types

Files not reviewed (1)

x-pack/solutions/security/plugins/security_solution/common/api/entity_analytics/privilege_monitoring/monitoring_entity_source/monitoring_entity_source.schema.yaml: Language not supported

Comments suppressed due to low confidence (1)

x-pack/solutions/security/plugins/security_solution/server/request_context_factory.ts:268

Consider renaming this function to 'getMonitoringEntitySourceSyncDataClient' to match the naming of the underlying class and reflect the intended naming conventions mentioned in the PR.

getMonitoringEntitySourceDataClient: memoize(() => {

…e to move blockers while debugging joi issues

… src/core/server/integration_tests/ci_checks'

kibanamachine · 2025-06-02T16:20:07Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#8315

[❌] x-pack/test_serverless/functional/test_suites/observability/config.ts: 69/100 tests passed.

see run history

… src/core/server/integration_tests/ci_checks'

…-fix'

afharo · 2025-06-03T18:50:55Z

+        "type": "object"
+      },
+      "indexPattern": {
+        "index": false,


Using index: false is discouraged. Refer to https://docs.elastic.dev/kibana-dev-docs/tutorials/saved-objects to know more:

Never use enabled: false or index: false in your mappings. Elasticsearch does not support toggling these mapping options, so if your plugin ever needs to query the data, you will not be able to do so. Since these fields cannot be queried, they would require migrating to a new field and making associated code changes. Instead, use dynamic: false which provides the same flexibility while maintaining the future ability to query fields if necessary.

Thanks!

I was trying to change these to using dynamic: false but I am unsure of this use case --

properties": { "fields": { "index": false, "type": "keyword" }, "values": { "index": false, "type": "keyword" } }, "type": "object" },

My aim here is to make both matchers.find and matchers.values required and non searchable. To achieve this, can I change this to something like this where I rely on the top level - the child attributes should pick be non searchable from here?

properties": { "dynamic": false "fields": { "type": "keyword" }, "values": { "type": "keyword" } }, "type": "object" },

ok I think I have my answer from the docs, this should be good according to the example below:

> export const dashboardVisualization: SavedObjectsType = { > name: 'dashboard_visualization', > ... > mappings: { > properties: { > dynamic: false, // ✅ Do this instead > metadata: { > // dynamic: false gets inherited from above > properties: { > // `created_by` can now be stored but won't be queryable > } > }, > // `description` can now be stored but won't be queryable > } > } > };

@hop-dev I have updated to dynamic now :D but quick reference for the mappings - properties I was a bit blocked with. Time to merge? 🥳

hop-dev

🚀 Just the dynamic: true question to resolve and lets get this merged!

afharo · 2025-06-04T16:18:42Z

+    },
+    indexPattern: {
+      type: 'keyword',
+      dynamic: false,


CI is failing with the following error:

info [o.e.c.m.MetadataCreateIndexService] [node-01] failed on parsing mappings on index creation [.kibana_security_solution_9.1.0_001] org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping: unknown parameter [dynamic] on mapper [indexPattern] of type [keyword]

AFAIK, dynamic: false is only compatible with objects, not final fields. Since the parent object already has dynamic: false, I'd recommend that you don't declare the field indexPattern if you don't need to index it for now.
You can always add it later if needed.

The same recommendation applies to the other fields that used to have index: false. If you don't plan on using them to query/aggregate, just don't declare them in the mappings (and make sure that dynamic: false is in the parent object).

Thanks for explaining that, that makes more sense knowing we can leave it out if we aren't searching on it.

afharo

SO changes LGTM. Thanks for iterating and bearing with me.

elasticmachine · 2025-06-05T10:51:04Z

💔 Build Failed

Buildkite Build
Commit: 6cdf28c

Failed CI Steps

Checks

History

💔 Build #305678 failed 23fec8f
💔 Build #305254 failed 21b3e4a
💔 Build #305230 failed d5e1b93
💛 Build #304701 was flaky ba4646d
💔 Build #304684 failed 6332210

cc @CAWilson94

… src/core/server/integration_tests/ci_checks'

elasticmachine · 2025-06-05T16:02:04Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 7d11fa7

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #29 / EQL execution logic API @ess @serverless @serverlessQA EQL type rules uses the provided timestamp_field

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`securitySolution`	128	129	+1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`securitySolution`	35	36	+1

Unknown metric groups

API count

id	before	after	diff
`securitySolution`	197	198	+1

ESLint disabled in files

id	before	after	diff
`securitySolution`	94	95	+1

Total ESLint disabled count

id	before	after	diff
`securitySolution`	703	704	+1

History

💔 Build #305690 failed 6cdf28c
💔 Build #305678 failed 23fec8f
💔 Build #305254 failed 21b3e4a
💔 Build #305230 failed d5e1b93
💛 Build #304701 was flaky ba4646d

cc @CAWilson94

…9680) ## Summary This PR introduces a new Saved Object mapping for Monitoring Entity Source Configurations to support Privileged User Monitoring. ### ✅ Implemented * Saved object mapping schema aligned with the [current spec](https://docs.google.com/document/d/1-c6A82p2CCjYz8Mb6tL6R7BXXemQ4mpWZliKqki0SWo/edit?tab=t.0#heading=h.4axxr154e1lc) * DescriptorClient to handle saved object methods * OpenAPI schema and generated types * Implemented Data Client - SO operations exposed via GET and POST API * Created route and registered with privmonRoutes * Data Client Unit Testing * Rename typename of "monitoring_entity_source-sync" -> "entity-analytics-monitoring-entity-source" [Figma ](https://www.figma.com/board/yBr1pBDGu4JqNxb5ZrULtk/MonEntitySourceSyncTask?node-id=0-1&p=f&t=4hQaGsdvOpghi4QS-0)- currently working in the pink box ## Testing Steps: 1. privilegeMonitoringEnabled: true - set this flag 2. From Dev tools - create a saved object via POST (below) and view this saved object via GET (also below): ``` POST kbn:/api/entity_analytics/monitoring/entity_source { "type": "some-type-here", "name": "name-here", "managed": false, "indexPattern": "logs-*", "enabled": true, "integrationName": "Okta", "matchers": [ { "fields": ["user.role"], "values": ["admin"] } ], "filter": {} } ``` **Expected output:** ``` { "type": "some-type-here", "name": "name-here", "managed": false, "indexPattern": "logs-*", "enabled": true, "integrationName": "Okta", "matchers": [ { "fields": [ "user.role" ], "values": [ "admin" ] } ], "filter": {} } ``` GET Example ``` GET kbn:/api/entity_analytics/monitoring/entity_source ``` Output for get, expected to be the same as that for POST. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

CAWilson94 added 2 commits April 29, 2025 17:56

saved object for engine monitoring config, type started WiP

5596af3

add all schema items to SOMapping for SavedObjectsTyoe

990132a

hop-dev reviewed Apr 30, 2025

View reviewed changes

Comment thread ...rver/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source_type.ts Outdated

hop-dev reviewed Apr 30, 2025

View reviewed changes

Comment thread ...rver/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source_type.ts Outdated

hop-dev reviewed Apr 30, 2025

View reviewed changes

Comment thread ...rver/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source_type.ts Outdated

hop-dev reviewed Apr 30, 2025

View reviewed changes

Comment thread ...rver/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source_type.ts

PR comments addressed: savedObjectMapping initial pass

2bc807b

CAWilson94 added 2 commits May 1, 2025 09:14

matchers, dynamic false to stop uncontrolled field additions

e4c5383

schema for monitoring eneity source; descriptor client for savedObject

25e346f

CAWilson94 requested a review from Copilot May 1, 2025 09:35

Copilot AI reviewed May 1, 2025

View reviewed changes

Comment thread ...rver/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source_type.ts Outdated

Comment thread ...on/server/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source.ts Outdated

hop-dev reviewed May 1, 2025

View reviewed changes

Comment thread ...rver/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source_type.ts Outdated

CAWilson94 added 3 commits May 1, 2025 14:12

typo, add SO to saved objectsin secsol server

6a17bac

schema updated - removed detectRemovals, matcher leafs all array type…

1253a07

…s; wired in get and create SO via routes and data client; added to context factory and types in secsol

remove list endpoint, not required

4771fde

CAWilson94 requested a review from Copilot May 1, 2025 20:23

Copilot AI reviewed May 1, 2025

View reviewed changes

CAWilson94 self-assigned this May 1, 2025

CAWilson94 commented May 2, 2025

View reviewed changes

Comment thread ...analytics/privilege_monitoring/monitoring_entity_source/monitoring_entity_source.schema.yaml Outdated

fields and values string to arrays

6bd8d77

CAWilson94 commented May 7, 2025

View reviewed changes

Comment thread ...on/server/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source.ts

CAWilson94 commented May 7, 2025

View reviewed changes

Comment thread ...rver/lib/entity_analytics/privilege_monitoring/saved_object/monitoring_entity_source_type.ts Outdated

data client jest test - WiP issues with joi locally, testing buildkit…

44c176f

…e to move blockers while debugging joi issues

CAWilson94 marked this pull request as ready for review May 7, 2025 13:14

CAWilson94 requested review from a team as code owners May 7, 2025 13:14

CAWilson94 requested a review from hop-dev May 7, 2025 13:14

Merge branch 'main' into ea-12283-monitoring-entity-source-config

bf92e49

CAWilson94 requested a review from Copilot May 7, 2025 13:15

CAWilson94 and others added 2 commits June 2, 2025 10:30

Merge branch 'main' into ea-12283-monitoring-entity-source-config

59a5312

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

f00f4d0

… src/core/server/integration_tests/ci_checks'

CAWilson94 and others added 6 commits June 3, 2025 11:22

Merge branch 'main' into ea-12283-monitoring-entity-source-config

1efef05

Merge branch 'main' into ea-12283-monitoring-entity-source-config

0f23d39

matchers reverted to specific properties

1dc32ca

remove unused mappings addition

5313ebb

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

6332210

… src/core/server/integration_tests/ci_checks'

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

ba4646d

…-fix'

afharo reviewed Jun 3, 2025

View reviewed changes

CAWilson94 requested a review from hop-dev June 4, 2025 09:04

hop-dev approved these changes Jun 4, 2025

View reviewed changes

CAWilson94 and others added 2 commits June 4, 2025 14:16

dynamic false added - index false out

25e005c

Merge branch 'main' into ea-12283-monitoring-entity-source-config

d5e1b93

CAWilson94 requested a review from afharo June 4, 2025 13:37

Merge branch 'main' into ea-12283-monitoring-entity-source-config

21b3e4a

afharo reviewed Jun 4, 2025

View reviewed changes

CAWilson94 and others added 3 commits June 5, 2025 10:37

Remove index pattern - don't need to query this.

a364ec8

Merge branch 'main' into ea-12283-monitoring-entity-source-config

23fec8f

Updated matchers for non searchable with kibana core guidance

6cdf28c

afharo approved these changes Jun 5, 2025

View reviewed changes

CAWilson94 and others added 3 commits June 5, 2025 14:59

saved objects count update

616ff78

Merge branch 'main' into ea-12283-monitoring-entity-source-config

cd27928

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

7d11fa7

… src/core/server/integration_tests/ci_checks'

CAWilson94 merged commit a3de6d8 into elastic:main Jun 5, 2025
10 checks passed

gsoldevila mentioned this pull request Jun 19, 2025

[Fleet] Save package policy previous revision on package upgrade #222779

Merged

2 tasks

Conversation

CAWilson94 commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

✅ Implemented

Testing Steps:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

prodsecmachine commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎉 Snyk checks have passed. No issues have been found so far.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kibanamachine commented Jun 2, 2025

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#8315

Uh oh!

afharo Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CAWilson94 Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

CAWilson94 Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

CAWilson94 Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hop-dev Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

hop-dev left a comment

Choose a reason for hiding this comment

Uh oh!

afharo Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

CAWilson94 Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

afharo left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

elasticmachine commented Jun 5, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

ESLint disabled in files

Total ESLint disabled count

CAWilson94 commented Apr 30, 2025 •

edited

Loading

prodsecmachine commented May 1, 2025 •

edited

Loading

afharo Jun 3, 2025 •

edited

Loading

CAWilson94 Jun 4, 2025 •

edited

Loading

elasticmachine commented Jun 5, 2025 •

edited

Loading