[Siem Migrations] Adds separate migration index to store migration metadata by logeekal · Pull Request #216164 · elastic/kibana

logeekal · 2025-03-27T12:37:43Z

Summary

Fixes: https://github.com/elastic/security-team/issues/12233

This PR simply adds a new migration index to store the migration data with migration_id as the only property for now.

The APIs remain unchanged.

Below are the mapping for new index .kibana-siem-rule-migrations-migrations-default based on the pattern ..kibana-siem-rule-migrations-<indexAdapterId>-<spaceName>

{
  ".kibana-siem-rule-migrations-migrations-default": {
    "mappings": {
      "dynamic": "false",
      "_meta": {
        "namespace": "default",
        "kibana": {
          "version": "9.1.0"
        },
        "managed": true
      },
      "properties": {
        "created_at": {
          "type": "date"
        },
        "created_by": {
          "type": "keyword"
        },
        "id": {
          "type": "keyword"
        }
      }
    }
  }
}

Below is how a sample document looks like:

      {
        "_index": ".kibana-siem-rule-migrations-migrations-default",
        "_id": "C7oi15UBS6DCfB3qd4_l",
        "_score": 1,
        "_source": {
      
          "created_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
          "created_at": "2025-03-27T10:25:15.232Z"
        }
      }

elasticmachine · 2025-03-27T14:33:43Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

...rity_solution/server/lib/siem_migrations/rules/data/rule_migrations_data_migration_client.ts

x-pack/solutions/security/plugins/security_solution/server/lib/siem_migrations/rules/types.ts

...gins/security_solution/server/lib/siem_migrations/rules/data/rule_migrations_data_service.ts

...rity_solution/server/lib/siem_migrations/rules/data/rule_migrations_data_migration_client.ts

.../solutions/security/plugins/security_solution/server/lib/siem_migrations/rules/api/create.ts

…fix'

...rity_solution/server/lib/siem_migrations/rules/data/rule_migrations_data_migration_client.ts

.../solutions/security/plugins/security_solution/server/lib/siem_migrations/rules/api/create.ts

...rity_solution/server/lib/siem_migrations/rules/data/rule_migrations_data_migration_client.ts

semd

LGTM!
Thanks @logeekal, great job 💯

kibanamachine · 2025-04-04T09:25:31Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/14262267537

elasticmachine · 2025-04-04T09:26:20Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: e532e46

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #78 / InfraOps App Metrics UI Infrastructure Source Configuration with metrics present renders the waffle map

Metrics [docs]

✅ unchanged

History

💚 Build #289873 succeeded 9d3cc9a
💔 Build #289767 failed 43b91c5
💔 Build #289378 failed 60e8450
💛 Build #288975 was flaky a9e6c44
💔 Build #288928 failed c6ff3c7
💔 Build #288546 failed 90e0ac7

kibanamachine · 2025-04-04T09:31:41Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [Security Solution] Use static declaration for navigation hierarchy (#215969)

Manual backport

To create the backport manually run:

node scripts/backport --pr 216164

Questions ?

Please refer to the Backport tool documentation

logeekal · 2025-04-08T08:42:03Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…tadata (elastic#216164) ## Summary Fixes: elastic/security-team#12233 This PR simply adds a new `migration` index to store the migration data with `migration_id` as the only property for now. The APIs remain unchanged. Below are the mapping for new index `.kibana-siem-rule-migrations-migrations-default` based on the pattern `..kibana-siem-rule-migrations-<indexAdapterId>-<spaceName>` ``` { ".kibana-siem-rule-migrations-migrations-default": { "mappings": { "dynamic": "false", "_meta": { "namespace": "default", "kibana": { "version": "9.1.0" }, "managed": true }, "properties": { "created_at": { "type": "date" }, "created_by": { "type": "keyword" }, "id": { "type": "keyword" } } } } } ``` Below is how a sample document looks like: ```json { "_index": ".kibana-siem-rule-migrations-migrations-default", "_id": "C7oi15UBS6DCfB3qd4_l", "_score": 1, "_source": { "created_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", "created_at": "2025-03-27T10:25:15.232Z" } } ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 5846a58) # Conflicts: # x-pack/solutions/security/plugins/security_solution/tsconfig.json

kibanamachine · 2025-04-08T09:48:41Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

…ion metadata (#216164) (#217454) # Backport This will backport the following commits from `main` to `8.x`: - [[Siem Migrations] Adds separate migration index to store migration metadata (#216164)](#216164)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

## Summary Resolves elastic/security-team#12483 This PR changes REST API Endpoints scheme to align with elastic/security-team#12483. Below is the summary of changes done. ### API Scheme changes The REST API scheme has been changed to reflect elastic/security-team#12483. This is pretty much self explanatory as defined in below openapi schema yaml : - [x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml](https://github.com/elastic/kibana/pull/219597/files#diff-3025af9eca156f3308474e2b42808da1531423457b7791daf6660db95a53b978) ### Introduction of Delete Migration API This PR also adds `DELETE` method on route `/rules/siem_migrations/{migration_id}` for deleting a migrations. Deleting a migration does below operations: - Stops a migration if it is running - Deletes the rules, resources related to migration and migration document itself. ### File Reorganizations Directly structure has been changed a little bit to reflect the endpoint. There is a sub-directory called `rules` which deals with only `rules` of the migration and the root directly only contains the endpoints related to the migration. #### Before ``` //siem_migrations/rules/api ├── create.ts ├── get.ts ├── start.ts ├── update.ts ├── ├── ├── ``` #### After ``` //siem_migrations/rules/api ├── create.ts ├── delete.ts ├── get.ts ├── rules │ ├── add.ts │ ├── get.ts │ └── update.ts ├── ├── ├── ``` ## Migration Strategy ### TL,DR; ```mermaid flowchart TD StartM[Start Migration] --> isMigExists{Does Migration Index Exists} isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs] isMigExists -->|No|CreateMIndex[Create MigrationIndex] CreateMIndex --> FetchMDoc FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index] FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index} FilterMigration --> |is Empty|END[END] FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs] CreateMDocs --> END ``` At the time of merging this PR, the Migration indices can be in 3 states: ### There are migrations created after #216164 and this means that there are `some` migrations existing in `.kibana-siem-migrations-migrations-<space_id>` and migrations created before above mentioned PR will only exist in `.kibana-siem-migrations-rules-<space_id>`. In this case `migrateRuleMigrationIndex` will create migration in below steps: 1. Look for **all** migration Documents in `.kibana-siem-migrations-migrations-<space_id>` 2. Get **all** Migrations stats from `.kibana-siem-migrations-rules-<space_id>` which includes below properties - migration_id : will help in reconciling the migration id in .kibana-siem-migrations-migrations-<space_id>` index - created_at : Date on which migration_id was created. - created_by: User who created the migrations. 3. A new document with above migration will be added to `.kibana-siem-migrations-migrations-<space_id>`. 4. Now both `.kibana-siem-migrations-migrations-<space_id>` and `.kibana-siem-migrations-rules-<space_id>` will be in sync. ### Alternatively, there are no migration created after #216164. In that case, there is a possibility that `.kibana-siem-migrations-migrations-<space_id>`, will not even exist. In this case `migrateRuleMigrationIndex` will create migration in below steps: 1. Create the `.kibana-siem-migrations-migrations-<space_id>` index. 2. Do steps mentioned in above scenario. ### Once the migrations has been run successfully, both `.kibana-siem-migrations-migrations-<space_id>` index and `.kibana-siem-migrations-rules-<space_id>` will be in sync. 1. In this case, migration will not run, since it tries to filter the migrations by `id` which exist in `kibana-siem-migrations-rules-<space_id>` but do not exist in `kibana-siem-migrations-migrations-<space_id>` ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

## Summary Resolves elastic/security-team#12483 This PR changes REST API Endpoints scheme to align with elastic/security-team#12483. Below is the summary of changes done. ### API Scheme changes The REST API scheme has been changed to reflect elastic/security-team#12483. This is pretty much self explanatory as defined in below openapi schema yaml : - [x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml](https://github.com/elastic/kibana/pull/219597/files#diff-3025af9eca156f3308474e2b42808da1531423457b7791daf6660db95a53b978) ### Introduction of Delete Migration API This PR also adds `DELETE` method on route `/rules/siem_migrations/{migration_id}` for deleting a migrations. Deleting a migration does below operations: - Stops a migration if it is running - Deletes the rules, resources related to migration and migration document itself. ### File Reorganizations Directly structure has been changed a little bit to reflect the endpoint. There is a sub-directory called `rules` which deals with only `rules` of the migration and the root directly only contains the endpoints related to the migration. #### Before ``` //siem_migrations/rules/api ├── create.ts ├── get.ts ├── start.ts ├── update.ts ├── ├── ├── ``` #### After ``` //siem_migrations/rules/api ├── create.ts ├── delete.ts ├── get.ts ├── rules │ ├── add.ts │ ├── get.ts │ └── update.ts ├── ├── ├── ``` ## Migration Strategy ### TL,DR; ```mermaid flowchart TD StartM[Start Migration] --> isMigExists{Does Migration Index Exists} isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs] isMigExists -->|No|CreateMIndex[Create MigrationIndex] CreateMIndex --> FetchMDoc FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index] FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index} FilterMigration --> |is Empty|END[END] FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs] CreateMDocs --> END ``` At the time of merging this PR, the Migration indices can be in 3 states: ### There are migrations created after elastic#216164 and this means that there are `some` migrations existing in `.kibana-siem-migrations-migrations-<space_id>` and migrations created before above mentioned PR will only exist in `.kibana-siem-migrations-rules-<space_id>`. In this case `migrateRuleMigrationIndex` will create migration in below steps: 1. Look for **all** migration Documents in `.kibana-siem-migrations-migrations-<space_id>` 2. Get **all** Migrations stats from `.kibana-siem-migrations-rules-<space_id>` which includes below properties - migration_id : will help in reconciling the migration id in .kibana-siem-migrations-migrations-<space_id>` index - created_at : Date on which migration_id was created. - created_by: User who created the migrations. 3. A new document with above migration will be added to `.kibana-siem-migrations-migrations-<space_id>`. 4. Now both `.kibana-siem-migrations-migrations-<space_id>` and `.kibana-siem-migrations-rules-<space_id>` will be in sync. ### Alternatively, there are no migration created after elastic#216164. In that case, there is a possibility that `.kibana-siem-migrations-migrations-<space_id>`, will not even exist. In this case `migrateRuleMigrationIndex` will create migration in below steps: 1. Create the `.kibana-siem-migrations-migrations-<space_id>` index. 2. Do steps mentioned in above scenario. ### Once the migrations has been run successfully, both `.kibana-siem-migrations-migrations-<space_id>` index and `.kibana-siem-migrations-rules-<space_id>` will be in sync. 1. In this case, migration will not run, since it tries to filter the migrations by `id` which exist in `kibana-siem-migrations-rules-<space_id>` but do not exist in `kibana-siem-migrations-migrations-<space_id>` ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit bf642b0) # Conflicts: # x-pack/test/security_solution_cypress/cypress/urls/navigation.ts

…221536) # Backport This will backport the following commits from `main` to `8.19`: - [[Automatic Migrations] Siem migrations new endpoint (#219597)](#219597)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  isMigExists{Does Migration Index Exists}\n isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs]\n isMigExists -->|No|CreateMIndex[Create MigrationIndex]\n CreateMIndex --> FetchMDoc\n FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index]\n FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index}\n FilterMigration --> |is Empty|END[END]\n FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs]\n CreateMDocs --> END\n```\n\n\n\nAt the time of merging this PR, the Migration indices can be in 3\nstates:\n\n### There are migrations created after\nhttps://github.com//pull/216164 and this means that there\nare `some` migrations existing in\n`.kibana-siem-migrations-migrations-<space_id>` and migrations created\nbefore above mentioned PR will only exist in\n`.kibana-siem-migrations-rules-<space_id>`.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Look for **all** migration Documents in\n`.kibana-siem-migrations-migrations-<space_id>`\n2. Get **all** Migrations stats from\n`.kibana-siem-migrations-rules-<space_id>` which includes below\nproperties\n- migration_id : will help in reconciling the migration id in\n.kibana-siem-migrations-migrations-<space_id>` index\n - created_at : Date on which migration_id was created.\n - created_by: User who created the migrations.\n3. A new document with above migration will be added to\n`.kibana-siem-migrations-migrations-<space_id>`.\n4. Now both `.kibana-siem-migrations-migrations-<space_id>` and\n`.kibana-siem-migrations-rules-<space_id>` will be in sync.\n\n### Alternatively, there are no migration created after\nhttps://github.com//pull/216164. In that case, there is a\npossibility that `.kibana-siem-migrations-migrations-<space_id>`, will\nnot even exist.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Create the `.kibana-siem-migrations-migrations-<space_id>` index.\n2. Do steps mentioned in above scenario.\n\n### Once the migrations has been run successfully, both\n`.kibana-siem-migrations-migrations-<space_id>` index and\n`.kibana-siem-migrations-rules-<space_id>` will be in sync.\n\n1. In this case, migration will not run, since it tries to filter the\nmigrations by `id` which exist in\n`kibana-siem-migrations-rules-<space_id>` but do not exist in\n`kibana-siem-migrations-migrations-<space_id>`\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n### Identify risks\n\nDoes this PR introduce any risks? For example, consider risks like hard\nto test bugs, performance regression, potential of data loss.\n\nDescribe the risk, its severity, and mitigation for each identified\nrisk. Invite stakeholders and evaluate how to proceed before merging.\n\n- [ ] [See some risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n- [ ] ...\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bf642b00395079ef0e2f334b4fca65d2ff15d94c","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport missing","release_note:feature","backport:version","v9.1.0","v8.19.0"],"title":"[Automatic Migrations] Siem migrations new endpoint scheme","number":219597,"url":"https://github.com/elastic/kibana/pull/219597","mergeCommit":{"message":"[Automatic Migrations] Siem migrations new endpoint (#219597)\n\n## Summary\n\nResolves https://github.com/elastic/security-team/issues/12483\n\nThis PR changes REST API Endpoints scheme to align with\nhttps://github.com/elastic/security-team/issues/12483. Below is the\nsummary of changes done.\n\n### API Scheme changes\n\nThe REST API scheme has been changed to reflect\nhttps://github.com/elastic/security-team/issues/12483. This is pretty\nmuch self explanatory as defined in below openapi schema yaml :\n\n-\n[x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml](https://github.com/elastic/kibana/pull/219597/files#diff-3025af9eca156f3308474e2b42808da1531423457b7791daf6660db95a53b978)\n\n### Introduction of Delete Migration API\n\nThis PR also adds `DELETE` method on route\n`/rules/siem_migrations/{migration_id}` for deleting a migrations.\nDeleting a migration does below operations:\n\n- Stops a migration if it is running\n- Deletes the rules, resources related to migration and migration\ndocument itself.\n\n### File Reorganizations\n\nDirectly structure has been changed a little bit to reflect the\nendpoint. There is a sub-directory called `rules` which deals with only\n`rules` of the migration and the root directly only contains the\nendpoints related to the migration.\n\n#### Before\n\n```\n//siem_migrations/rules/api\n\n├── create.ts\n├── get.ts\n├── start.ts\n├── update.ts\n├── \n├── \n├── \n```\n\n#### After\n\n```\n//siem_migrations/rules/api\n\n\n├── create.ts\n├── delete.ts\n├── get.ts\n├── rules\n│ ├── add.ts\n│ ├── get.ts\n│ └── update.ts\n├── \n├── \n├── \n```\n\n## Migration Strategy\n\n### TL,DR; \n```mermaid\nflowchart TD\n StartM[Start Migration] --> isMigExists{Does Migration Index Exists}\n isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs]\n isMigExists -->|No|CreateMIndex[Create MigrationIndex]\n CreateMIndex --> FetchMDoc\n FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index]\n FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index}\n FilterMigration --> |is Empty|END[END]\n FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs]\n CreateMDocs --> END\n```\n\n\n\nAt the time of merging this PR, the Migration indices can be in 3\nstates:\n\n### There are migrations created after\nhttps://github.com//pull/216164 and this means that there\nare `some` migrations existing in\n`.kibana-siem-migrations-migrations-<space_id>` and migrations created\nbefore above mentioned PR will only exist in\n`.kibana-siem-migrations-rules-<space_id>`.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Look for **all** migration Documents in\n`.kibana-siem-migrations-migrations-<space_id>`\n2. Get **all** Migrations stats from\n`.kibana-siem-migrations-rules-<space_id>` which includes below\nproperties\n- migration_id : will help in reconciling the migration id in\n.kibana-siem-migrations-migrations-<space_id>` index\n - created_at : Date on which migration_id was created.\n - created_by: User who created the migrations.\n3. A new document with above migration will be added to\n`.kibana-siem-migrations-migrations-<space_id>`.\n4. Now both `.kibana-siem-migrations-migrations-<space_id>` and\n`.kibana-siem-migrations-rules-<space_id>` will be in sync.\n\n### Alternatively, there are no migration created after\nhttps://github.com//pull/216164. In that case, there is a\npossibility that `.kibana-siem-migrations-migrations-<space_id>`, will\nnot even exist.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Create the `.kibana-siem-migrations-migrations-<space_id>` index.\n2. Do steps mentioned in above scenario.\n\n### Once the migrations has been run successfully, both\n`.kibana-siem-migrations-migrations-<space_id>` index and\n`.kibana-siem-migrations-rules-<space_id>` will be in sync.\n\n1. In this case, migration will not run, since it tries to filter the\nmigrations by `id` which exist in\n`kibana-siem-migrations-rules-<space_id>` but do not exist in\n`kibana-siem-migrations-migrations-<space_id>`\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n### Identify risks\n\nDoes this PR introduce any risks? For example, consider risks like hard\nto test bugs, performance regression, potential of data loss.\n\nDescribe the risk, its severity, and mitigation for each identified\nrisk. Invite stakeholders and evaluate how to proceed before merging.\n\n- [ ] [See some risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n- [ ] ...\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bf642b00395079ef0e2f334b4fca65d2ff15d94c"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/219597","number":219597,"mergeCommit":{"message":"[Automatic Migrations] Siem migrations new endpoint (#219597)\n\n## Summary\n\nResolves https://github.com/elastic/security-team/issues/12483\n\nThis PR changes REST API Endpoints scheme to align with\nhttps://github.com/elastic/security-team/issues/12483. Below is the\nsummary of changes done.\n\n### API Scheme changes\n\nThe REST API scheme has been changed to reflect\nhttps://github.com/elastic/security-team/issues/12483. This is pretty\nmuch self explanatory as defined in below openapi schema yaml :\n\n-\n[x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml](https://github.com/elastic/kibana/pull/219597/files#diff-3025af9eca156f3308474e2b42808da1531423457b7791daf6660db95a53b978)\n\n### Introduction of Delete Migration API\n\nThis PR also adds `DELETE` method on route\n`/rules/siem_migrations/{migration_id}` for deleting a migrations.\nDeleting a migration does below operations:\n\n- Stops a migration if it is running\n- Deletes the rules, resources related to migration and migration\ndocument itself.\n\n### File Reorganizations\n\nDirectly structure has been changed a little bit to reflect the\nendpoint. There is a sub-directory called `rules` which deals with only\n`rules` of the migration and the root directly only contains the\nendpoints related to the migration.\n\n#### Before\n\n```\n//siem_migrations/rules/api\n\n├── create.ts\n├── get.ts\n├── start.ts\n├── update.ts\n├── \n├── \n├── \n```\n\n#### After\n\n```\n//siem_migrations/rules/api\n\n\n├── create.ts\n├── delete.ts\n├── get.ts\n├── rules\n│ ├── add.ts\n│ ├── get.ts\n│ └── update.ts\n├── \n├── \n├── \n```\n\n## Migration Strategy\n\n### TL,DR; \n```mermaid\nflowchart TD\n StartM[Start Migration] --> isMigExists{Does Migration Index Exists}\n isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs]\n isMigExists -->|No|CreateMIndex[Create MigrationIndex]\n CreateMIndex --> FetchMDoc\n FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index]\n FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index}\n FilterMigration --> |is Empty|END[END]\n FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs]\n CreateMDocs --> END\n```\n\n\n\nAt the time of merging this PR, the Migration indices can be in 3\nstates:\n\n### There are migrations created after\nhttps://github.com//pull/216164 and this means that there\nare `some` migrations existing in\n`.kibana-siem-migrations-migrations-<space_id>` and migrations created\nbefore above mentioned PR will only exist in\n`.kibana-siem-migrations-rules-<space_id>`.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Look for **all** migration Documents in\n`.kibana-siem-migrations-migrations-<space_id>`\n2. Get **all** Migrations stats from\n`.kibana-siem-migrations-rules-<space_id>` which includes below\nproperties\n- migration_id : will help in reconciling the migration id in\n.kibana-siem-migrations-migrations-<space_id>` index\n - created_at : Date on which migration_id was created.\n - created_by: User who created the migrations.\n3. A new document with above migration will be added to\n`.kibana-siem-migrations-migrations-<space_id>`.\n4. Now both `.kibana-siem-migrations-migrations-<space_id>` and\n`.kibana-siem-migrations-rules-<space_id>` will be in sync.\n\n### Alternatively, there are no migration created after\nhttps://github.com//pull/216164. In that case, there is a\npossibility that `.kibana-siem-migrations-migrations-<space_id>`, will\nnot even exist.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Create the `.kibana-siem-migrations-migrations-<space_id>` index.\n2. Do steps mentioned in above scenario.\n\n### Once the migrations has been run successfully, both\n`.kibana-siem-migrations-migrations-<space_id>` index and\n`.kibana-siem-migrations-rules-<space_id>` will be in sync.\n\n1. In this case, migration will not run, since it tries to filter the\nmigrations by `id` which exist in\n`kibana-siem-migrations-rules-<space_id>` but do not exist in\n`kibana-siem-migrations-migrations-<space_id>`\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n### Identify risks\n\nDoes this PR introduce any risks? For example, consider risks like hard\nto test bugs, performance regression, potential of data loss.\n\nDescribe the risk, its severity, and mitigation for each identified\nrisk. Invite stakeholders and evaluate how to proceed before merging.\n\n- [ ] [See some risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n- [ ] ...\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bf642b00395079ef0e2f334b4fca65d2ff15d94c"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->

## Summary Resolves elastic/security-team#12483 This PR changes REST API Endpoints scheme to align with elastic/security-team#12483. Below is the summary of changes done. ### API Scheme changes The REST API scheme has been changed to reflect elastic/security-team#12483. This is pretty much self explanatory as defined in below openapi schema yaml : - [x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml](https://github.com/elastic/kibana/pull/219597/files#diff-3025af9eca156f3308474e2b42808da1531423457b7791daf6660db95a53b978) ### Introduction of Delete Migration API This PR also adds `DELETE` method on route `/rules/siem_migrations/{migration_id}` for deleting a migrations. Deleting a migration does below operations: - Stops a migration if it is running - Deletes the rules, resources related to migration and migration document itself. ### File Reorganizations Directly structure has been changed a little bit to reflect the endpoint. There is a sub-directory called `rules` which deals with only `rules` of the migration and the root directly only contains the endpoints related to the migration. #### Before ``` //siem_migrations/rules/api ├── create.ts ├── get.ts ├── start.ts ├── update.ts ├── ├── ├── ``` #### After ``` //siem_migrations/rules/api ├── create.ts ├── delete.ts ├── get.ts ├── rules │ ├── add.ts │ ├── get.ts │ └── update.ts ├── ├── ├── ``` ## Migration Strategy ### TL,DR; ```mermaid flowchart TD StartM[Start Migration] --> isMigExists{Does Migration Index Exists} isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs] isMigExists -->|No|CreateMIndex[Create MigrationIndex] CreateMIndex --> FetchMDoc FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index] FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index} FilterMigration --> |is Empty|END[END] FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs] CreateMDocs --> END ``` At the time of merging this PR, the Migration indices can be in 3 states: ### There are migrations created after elastic#216164 and this means that there are `some` migrations existing in `.kibana-siem-migrations-migrations-<space_id>` and migrations created before above mentioned PR will only exist in `.kibana-siem-migrations-rules-<space_id>`. In this case `migrateRuleMigrationIndex` will create migration in below steps: 1. Look for **all** migration Documents in `.kibana-siem-migrations-migrations-<space_id>` 2. Get **all** Migrations stats from `.kibana-siem-migrations-rules-<space_id>` which includes below properties - migration_id : will help in reconciling the migration id in .kibana-siem-migrations-migrations-<space_id>` index - created_at : Date on which migration_id was created. - created_by: User who created the migrations. 3. A new document with above migration will be added to `.kibana-siem-migrations-migrations-<space_id>`. 4. Now both `.kibana-siem-migrations-migrations-<space_id>` and `.kibana-siem-migrations-rules-<space_id>` will be in sync. ### Alternatively, there are no migration created after elastic#216164. In that case, there is a possibility that `.kibana-siem-migrations-migrations-<space_id>`, will not even exist. In this case `migrateRuleMigrationIndex` will create migration in below steps: 1. Create the `.kibana-siem-migrations-migrations-<space_id>` index. 2. Do steps mentioned in above scenario. ### Once the migrations has been run successfully, both `.kibana-siem-migrations-migrations-<space_id>` index and `.kibana-siem-migrations-rules-<space_id>` will be in sync. 1. In this case, migration will not run, since it tries to filter the migrations by `id` which exist in `kibana-siem-migrations-rules-<space_id>` but do not exist in `kibana-siem-migrations-migrations-<space_id>` ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

logeekal added 2 commits March 27, 2025 12:16

initial commit

51b03c7

types: housekeeping

5ae7eeb

logeekal changed the title ~~[Siem Migrations] Adds migration index.~~ [Siem Migrations] Adds migration index Mar 27, 2025

fix: types

fc80d53

logeekal changed the title ~~[Siem Migrations] Adds migration index~~ [Siem Migrations] Adds separate migration index to store migration metadata Mar 27, 2025

logeekal added 2 commits March 27, 2025 14:14

more types fixes

c24ddc0

fix: test types

c1e75b1

logeekal marked this pull request as ready for review March 27, 2025 14:33

logeekal requested a review from a team as a code owner March 27, 2025 14:33