[Automatic migrations] Dashboard v/s Rule Migration generic logic by logeekal · Pull Request #232526 · elastic/kibana

logeekal · 2025-08-21T17:17:45Z

Summary

This PR refactors complete Automatic Migrations current data clients and services for Dashboards and Rules Migrations.

Only objective of this PR is share logic between rules and dashboards migrations.

Impact

This PR is intended to keep the functionality exactly the same.

Code Owners Summary

elastic/kibana-core

src/platform/packages/shared/kbn-utility-types/index.ts

elastic/security-threat-hunting-investigations

x-pack/solutions/security/plugins/security_solution/public/onboarding/components/onboarding_body/cards/siem_migrations/start_migration/start_migration_card.test.tsx

...solutions/security/plugins/security_solution/common/siem_migrations/model/common.schema.yaml

x-pack/solutions/security/plugins/security_solution/common/siem_migrations/resources/types.ts

...ons/security/plugins/security_solution/server/lib/siem_migrations/common/data/dsl_queries.ts

logeekal · 2025-08-22T12:38:43Z

.../security_solution/server/lib/siem_migrations/common/task/util/nullify_missing_properties.ts

+}
+export const nullifyMissingProperties = <T extends object = object>(
+  params: NullifyMissingProperties<T>
+): T => {


Suggested change

): T => {

): Nullable<T> => {

logeekal · 2025-08-22T12:40:38Z

.../security_solution/server/lib/siem_migrations/common/task/util/nullify_missing_properties.ts

+ */
+
+interface NullifyMissingProperties<T> {
+  source?: T;


Probably needs some explanation about source and target

logeekal · 2025-08-25T11:23:44Z

x-pack/solutions/security/plugins/security_solution/common/siem_migrations/constants.ts

 export const SIEM_MIGRATIONS_PATH = '/internal/siem_migrations' as const;
+
+// TODO: Move `SIEM_RULE_MIGRATIONS_PATH` and composed paths to rules/constants.ts
 export const SIEM_RULE_MIGRATIONS_PATH = `${SIEM_MIGRATIONS_PATH}/rules` as const;


Will be done in separate PR. A sub folder of rules/ will be good but will unncessary increases the number of files in this PR.

logeekal · 2025-08-25T11:34:56Z

...security/plugins/security_solution/server/lib/siem_migrations/common/data/__mocks__/mocks.ts

+import type { SiemMigrationsDataResourcesClient } from '../siem_migrations_data_resources_client';
+import type { SiemMigrationsDataItemClient } from '../siem_migrations_data_item_client';
+
+// Rule migrations data items client


Need to remove this.

...ions/security/plugins/security_solution/server/lib/siem_migrations/rules/data/dsl_queries.ts

...ons/security/plugins/security_solution/server/lib/siem_migrations/common/data/dsl_queries.ts

logeekal · 2025-08-26T14:35:02Z

@semd Here are the Evaluator changes.

3d91676

...plugins/security_solution/server/lib/siem_migrations/common/task/util/actions_client_chat.ts

x-pack/solutions/security/plugins/security_solution/server/lib/siem_migrations/common/types.ts

...olution/server/lib/siem_migrations/dashboards/data/dashboard_migrations_dashboards_client.ts

semd

LGTM! 🚀🚀🚀

…fix'

dmlemeshko · 2025-08-29T12:21:43Z

x-pack/test/api_integration/services/security_solution_api.gen.ts

@@ -0,0 +1,2626 @@
+/*


This file was relocated as part of #233112

We are removing x-pack/test directory quite soon.

Thanks. yes I saw that. At least for this PR , there was no impact since we're not using auto generated client in integration tests.

Sorry, i just realized that i added it by mistake. Let me remove it.

@dmlemeshko , I have removed it now.

afharo · 2025-08-29T14:01:03Z

src/platform/packages/shared/kbn-utility-types/index.ts

+/**
+ * Utility type for making all properties of an object nullable.
+ */
+export type Nullable<T extends object> = { [K in keyof T]: T[K] | null };


nit: Can we use a different name? Nullable sounds like the opposite of NonNullable, but they are not directly the opposite.

Maybe something like NullableProps?

I agree with your suggestion but It looks like we had removed its usages and so i have for now removed the type. I will add it in further PRs.

elasticmachine · 2025-08-29T15:33:07Z

💚 Build Succeeded

Buildkite Build
Commit: bffeaa4
Cloud Deployment
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-232526-bffeaa45fb8c
Security Deployment

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	10.2MB	10.2MB	-839.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
`_data_stream_timestamp`	1	-	-1
`_doc_count`	1	-	-1
`_ignored_source`	1	-	-1
`_index_mode`	1	-	-1
`_inference_fields`	1	-	-1
`_tier`	1	-	-1
`apm-custom-dashboards`	5	-	-5
`apm-server-schema`	2	-	-2
`apm-service-group`	5	-	-5
`application_usage_daily`	2	-	-2
`config`	2	-	-2
`config-global`	2	-	-2
`coreMigrationVersion`	1	-	-1
`created_at`	1	-	-1
`created_by`	1	-	-1
`entity-definition`	9	-	-9
`entity-discovery-api-key`	2	-	-2
`event_loop_delays_daily`	2	-	-2
`favorites`	4	-	-4
`file`	11	-	-11
`file-upload-usage-collection-telemetry`	3	-	-3
`fileShare`	5	-	-5
`infra-custom-dashboards`	4	-	-4
`infrastructure-monitoring-log-view`	2	-	-2
`intercept_trigger_record`	5	-	-5
`legacy-url-alias`	7	-	-7
`managed`	1	-	-1
`ml-job`	6	-	-6
`ml-module`	13	-	-13
`ml-trained-model`	7	-	-7
`monitoring-telemetry`	2	-	-2
`namespace`	1	-	-1
`namespaces`	1	-	-1
`observability-onboarding-state`	2	-	-2
`originId`	1	-	-1
`product-doc-install-status`	7	-	-7
`references`	4	-	-4
`sample-data-telemetry`	3	-	-3
`security-ai-prompt`	8	-	-8
`slo`	11	-	-11
`space`	5	-	-5
`synthetics-monitor`	34	-	-34
`synthetics-monitor-multi-space`	34	-	-34
`tag`	4	-	-4
`type`	1	-	-1
`typeMigrationVersion`	1	-	-1
`ui-metric`	2	-	-2
`updated_at`	1	-	-1
`updated_by`	1	-	-1
`upgrade-assistant-ml-upgrade-operation`	3	-	-3
`upgrade-assistant-reindex-operation`	3	-	-3
`uptime-synthetics-api-key`	2	-	-2
`url`	5	-	-5
`usage-counters`	2	-	-2
total			-246

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	679	681	+2

Total ESLint disabled count

id	before	after	diff
`securitySolution`	779	781	+2

History

💛 Build #334068 was flaky 965265d
💛 Build #333963 was flaky 1e579fc
💚 Build #333689 succeeded 1c448c6
💔 Build #333539 failed 88469ab

…astic#232526) ## Summary This PR refactors complete Automatic Migrations current data clients and services for Dashboards and Rules Migrations. Only objective of this PR is share logic between rules and dashboards migrations. ### Impact This PR is intended to keep the functionality exactly the same. ### Code Owners Summary #### elastic/kibana-core * src/platform/packages/shared/kbn-utility-types/index.ts #### elastic/security-threat-hunting-investigations * x-pack/solutions/security/plugins/security_solution/public/onboarding/components/onboarding_body/cards/siem_migrations/start_migration/start_migration_card.test.tsx </details> --------- Co-authored-by: Sergi Massaneda <sergi.massaneda@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

semd and others added 13 commits August 21, 2025 13:37

abstract task and data clients

e5e9bd7

data service changes

94d839d

filter queries

eba0139

dsl queries

d6b68b6

agent tools extracted

c040df4

dashboards agent graph initial implementation

d64a9c6

graph executing

901abbe

fix selectIndexPatterns

2f3b84d

fix: removed dashboard changes

dee3f06

fix: types

16dd7ee

fix: types after rebase

69edfde

fix: resource retriever types

62a2d60

fix: integration tests

5eaf603

logeekal commented Aug 21, 2025

View reviewed changes

...solutions/security/plugins/security_solution/common/siem_migrations/model/common.schema.yaml Show resolved Hide resolved

logeekal commented Aug 21, 2025

View reviewed changes

x-pack/solutions/security/plugins/security_solution/common/siem_migrations/resources/types.ts Outdated Show resolved Hide resolved

logeekal commented Aug 21, 2025

View reviewed changes

...ons/security/plugins/security_solution/server/lib/siem_migrations/common/data/dsl_queries.ts Show resolved Hide resolved

Rename RuleTranslationResult --> MigrationTranslationResult

9f516c9

logeekal commented Aug 22, 2025

View reviewed changes

logeekal changed the title ~~Automatic migrations generic task~~ [Automatic migrations] Dashboard v/s Rule Migration generic logic Aug 25, 2025

logeekal mentioned this pull request Aug 25, 2025

[Security Solution] Automatic dashboard migration agent initial implementation #231853

Closed

[CI] Auto-commit changed files from 'yarn openapi:generate'

c8e1575

logeekal commented Aug 25, 2025

View reviewed changes

fix: types

f66a46c

semd reviewed Aug 26, 2025

View reviewed changes

...ions/security/plugins/security_solution/server/lib/siem_migrations/rules/data/dsl_queries.ts Outdated Show resolved Hide resolved

semd reviewed Aug 26, 2025

View reviewed changes

...ons/security/plugins/security_solution/server/lib/siem_migrations/common/data/dsl_queries.ts Outdated Show resolved Hide resolved

fix: Task runner as dependency of Evaluator

3d91676

fix: evaluator types

77f8000

fix: pr review

21061c3

logeekal commented Aug 28, 2025

View reviewed changes

...plugins/security_solution/server/lib/siem_migrations/common/task/util/actions_client_chat.ts Outdated Show resolved Hide resolved

logeekal commented Aug 28, 2025

View reviewed changes

x-pack/solutions/security/plugins/security_solution/server/lib/siem_migrations/common/types.ts Outdated Show resolved Hide resolved

[CI] Auto-commit changed files from 'yarn openapi:generate'

26c2408

logeekal commented Aug 28, 2025

View reviewed changes

...olution/server/lib/siem_migrations/dashboards/data/dashboard_migrations_dashboards_client.ts Outdated Show resolved Hide resolved

chores: pr review

88469ab

logeekal added ci:cloud-deploy Create or update a Cloud deployment ci:cloud-persist-deployment Persist cloud deployment indefinitely and removed ci:project-deploy-security Create a Security Serverless Project ci:project-persist-deployment Persist project deployment indefinitely labels Aug 28, 2025

Merge branch 'main' into automatic_migrations_generic_task

1c448c6

semd approved these changes Aug 29, 2025

View reviewed changes

semd mentioned this pull request Aug 29, 2025

[Automatic Migrations][API] Implemented installation logic as dashboards endpoint #233378

Merged

fix: action client chat

1e579fc

logeekal added ci:project-deploy-security Create a Security Serverless Project ci:project-persist-deployment Persist project deployment indefinitely labels Aug 29, 2025

logeekal and others added 2 commits August 29, 2025 13:40

merge main --> current branch

52f7c49

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

d0f9250

…fix'

elastic-vault-github-plugin-prod bot requested a review from a team as a code owner August 29, 2025 11:51

[CI] Auto-commit changed files from 'yarn openapi:generate'

0be7403

dmlemeshko reviewed Aug 29, 2025

View reviewed changes

logeekal added 2 commits August 29, 2025 14:28

fix: merge mistakes

4ff87b1

fix: tsconfig

965265d

afharo reviewed Aug 29, 2025

View reviewed changes

fix: remove unnecesary type

bffeaa4

logeekal enabled auto-merge (squash) August 29, 2025 14:41

logeekal merged commit 0e2a3bd into elastic:main Aug 29, 2025
12 checks passed

Conversation

logeekal commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Impact

Code Owners Summary

elastic/kibana-core

elastic/security-threat-hunting-investigations

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

logeekal commented Aug 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

semd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Metrics [docs]

Async chunks

Saved Objects .kibana field count

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

logeekal commented Aug 21, 2025 •

edited

Loading

elasticmachine commented Aug 29, 2025 •

edited

Loading