[8.19] [Automatic Migrations] Siem migrations new endpoint (#219597)

[logeekal]

Backport

This will backport the following commits from main to 8.19:

• [Automatic Migrations] Siem migrations new endpoint (#219597)

Questions ?

Please refer to the Backport tool documentation

isMigExists{Does Migration Index Exists}\n isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs]\n isMigExists -->|No|CreateMIndex[Create MigrationIndex]\n CreateMIndex --> FetchMDoc\n FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index]\n FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index}\n FilterMigration --> |is Empty|END[END]\n FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs]\n CreateMDocs --> END\n```\n\n\n\nAt the time of merging this PR, the Migration indices can be in 3\nstates:\n\n### There are migrations created after\nhttps://github.com//pull/216164 and this means that there\nare `some` migrations existing in\n`.kibana-siem-migrations-migrations-` and migrations created\nbefore above mentioned PR will only exist in\n`.kibana-siem-migrations-rules-`.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Look for **all** migration Documents in\n`.kibana-siem-migrations-migrations-`\n2. Get **all** Migrations stats from\n`.kibana-siem-migrations-rules-` which includes below\nproperties\n- migration_id : will help in reconciling the migration id in\n.kibana-siem-migrations-migrations-` index\n - created_at : Date on which migration_id was created.\n - created_by: User who created the migrations.\n3. A new document with above migration will be added to\n`.kibana-siem-migrations-migrations-`.\n4. Now both `.kibana-siem-migrations-migrations-` and\n`.kibana-siem-migrations-rules-` will be in sync.\n\n### Alternatively, there are no migration created after\nhttps://github.com//pull/216164. In that case, there is a\npossibility that `.kibana-siem-migrations-migrations-`, will\nnot even exist.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Create the `.kibana-siem-migrations-migrations-` index.\n2. Do steps mentioned in above scenario.\n\n### Once the migrations has been run successfully, both\n`.kibana-siem-migrations-migrations-` index and\n`.kibana-siem-migrations-rules-` will be in sync.\n\n1. In this case, migration will not run, since it tries to filter the\nmigrations by `id` which exist in\n`kibana-siem-migrations-rules-` but do not exist in\n`kibana-siem-migrations-migrations-`\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n### Identify risks\n\nDoes this PR introduce any risks? For example, consider risks like hard\nto test bugs, performance regression, potential of data loss.\n\nDescribe the risk, its severity, and mitigation for each identified\nrisk. Invite stakeholders and evaluate how to proceed before merging.\n\n- [ ] [See some risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n- [ ] ...\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bf642b00395079ef0e2f334b4fca65d2ff15d94c","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:feature","backport:version","v9.1.0","v8.19.0"],"title":"[Automatic Migrations] Siem migrations new endpoint scheme","number":219597,"url":"https://github.com//pull/219597","mergeCommit":{"message":"[Automatic Migrations] Siem migrations new endpoint (#219597)\n\n## Summary\n\nResolves https://github.com/elastic/security-team/issues/12483\n\nThis PR changes REST API Endpoints scheme to align with\nhttps://github.com/elastic/security-team/issues/12483. Below is the\nsummary of changes done.\n\n### API Scheme changes\n\nThe REST API scheme has been changed to reflect\nhttps://github.com/elastic/security-team/issues/12483. This is pretty\nmuch self explanatory as defined in below openapi schema yaml :\n\n-\n[x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml](https://github.com//pull/219597/files#diff-3025af9eca156f3308474e2b42808da1531423457b7791daf6660db95a53b978)\n\n### Introduction of Delete Migration API\n\nThis PR also adds `DELETE` method on route\n`/rules/siem_migrations/{migration_id}` for deleting a migrations.\nDeleting a migration does below operations:\n\n- Stops a migration if it is running\n- Deletes the rules, resources related to migration and migration\ndocument itself.\n\n### File Reorganizations\n\nDirectly structure has been changed a little bit to reflect the\nendpoint. There is a sub-directory called `rules` which deals with only\n`rules` of the migration and the root directly only contains the\nendpoints related to the migration.\n\n#### Before\n\n```\n//siem_migrations/rules/api\n\n├── create.ts\n├── get.ts\n├── start.ts\n├── update.ts\n├── \n├── \n├── \n```\n\n#### After\n\n```\n//siem_migrations/rules/api\n\n\n├── create.ts\n├── delete.ts\n├── get.ts\n├── rules\n│   ├── add.ts\n│   ├── get.ts\n│   └── update.ts\n├── \n├── \n├── \n```\n\n## Migration Strategy\n\n### TL,DR; \n```mermaid\nflowchart TD\n StartM[Start Migration] --> isMigExists{Does Migration Index Exists}\n isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs]\n isMigExists -->|No|CreateMIndex[Create MigrationIndex]\n CreateMIndex --> FetchMDoc\n FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index]\n FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index}\n FilterMigration --> |is Empty|END[END]\n FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs]\n CreateMDocs --> END\n```\n\n\n\nAt the time of merging this PR, the Migration indices can be in 3\nstates:\n\n### There are migrations created after\nhttps://github.com//pull/216164 and this means that there\nare `some` migrations existing in\n`.kibana-siem-migrations-migrations-` and migrations created\nbefore above mentioned PR will only exist in\n`.kibana-siem-migrations-rules-`.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Look for **all** migration Documents in\n`.kibana-siem-migrations-migrations-`\n2. Get **all** Migrations stats from\n`.kibana-siem-migrations-rules-` which includes below\nproperties\n- migration_id : will help in reconciling the migration id in\n.kibana-siem-migrations-migrations-` index\n - created_at : Date on which migration_id was created.\n - created_by: User who created the migrations.\n3. A new document with above migration will be added to\n`.kibana-siem-migrations-migrations-`.\n4. Now both `.kibana-siem-migrations-migrations-` and\n`.kibana-siem-migrations-rules-` will be in sync.\n\n### Alternatively, there are no migration created after\nhttps://github.com//pull/216164. In that case, there is a\npossibility that `.kibana-siem-migrations-migrations-`, will\nnot even exist.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Create the `.kibana-siem-migrations-migrations-` index.\n2. Do steps mentioned in above scenario.\n\n### Once the migrations has been run successfully, both\n`.kibana-siem-migrations-migrations-` index and\n`.kibana-siem-migrations-rules-` will be in sync.\n\n1. In this case, migration will not run, since it tries to filter the\nmigrations by `id` which exist in\n`kibana-siem-migrations-rules-` but do not exist in\n`kibana-siem-migrations-migrations-`\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n### Identify risks\n\nDoes this PR introduce any risks? For example, consider risks like hard\nto test bugs, performance regression, potential of data loss.\n\nDescribe the risk, its severity, and mitigation for each identified\nrisk. Invite stakeholders and evaluate how to proceed before merging.\n\n- [ ] [See some risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n- [ ] ...\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bf642b00395079ef0e2f334b4fca65d2ff15d94c"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com//pull/219597","number":219597,"mergeCommit":{"message":"[Automatic Migrations] Siem migrations new endpoint (#219597)\n\n## Summary\n\nResolves https://github.com/elastic/security-team/issues/12483\n\nThis PR changes REST API Endpoints scheme to align with\nhttps://github.com/elastic/security-team/issues/12483. Below is the\nsummary of changes done.\n\n### API Scheme changes\n\nThe REST API scheme has been changed to reflect\nhttps://github.com/elastic/security-team/issues/12483. This is pretty\nmuch self explanatory as defined in below openapi schema yaml :\n\n-\n[x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml](https://github.com//pull/219597/files#diff-3025af9eca156f3308474e2b42808da1531423457b7791daf6660db95a53b978)\n\n### Introduction of Delete Migration API\n\nThis PR also adds `DELETE` method on route\n`/rules/siem_migrations/{migration_id}` for deleting a migrations.\nDeleting a migration does below operations:\n\n- Stops a migration if it is running\n- Deletes the rules, resources related to migration and migration\ndocument itself.\n\n### File Reorganizations\n\nDirectly structure has been changed a little bit to reflect the\nendpoint. There is a sub-directory called `rules` which deals with only\n`rules` of the migration and the root directly only contains the\nendpoints related to the migration.\n\n#### Before\n\n```\n//siem_migrations/rules/api\n\n├── create.ts\n├── get.ts\n├── start.ts\n├── update.ts\n├── \n├── \n├── \n```\n\n#### After\n\n```\n//siem_migrations/rules/api\n\n\n├── create.ts\n├── delete.ts\n├── get.ts\n├── rules\n│   ├── add.ts\n│   ├── get.ts\n│   └── update.ts\n├── \n├── \n├── \n```\n\n## Migration Strategy\n\n### TL,DR; \n```mermaid\nflowchart TD\n StartM[Start Migration] --> isMigExists{Does Migration Index Exists}\n isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs]\n isMigExists -->|No|CreateMIndex[Create MigrationIndex]\n CreateMIndex --> FetchMDoc\n FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index]\n FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index}\n FilterMigration --> |is Empty|END[END]\n FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs]\n CreateMDocs --> END\n```\n\n\n\nAt the time of merging this PR, the Migration indices can be in 3\nstates:\n\n### There are migrations created after\nhttps://github.com//pull/216164 and this means that there\nare `some` migrations existing in\n`.kibana-siem-migrations-migrations-` and migrations created\nbefore above mentioned PR will only exist in\n`.kibana-siem-migrations-rules-`.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Look for **all** migration Documents in\n`.kibana-siem-migrations-migrations-`\n2. Get **all** Migrations stats from\n`.kibana-siem-migrations-rules-` which includes below\nproperties\n- migration_id : will help in reconciling the migration id in\n.kibana-siem-migrations-migrations-` index\n - created_at : Date on which migration_id was created.\n - created_by: User who created the migrations.\n3. A new document with above migration will be added to\n`.kibana-siem-migrations-migrations-`.\n4. Now both `.kibana-siem-migrations-migrations-` and\n`.kibana-siem-migrations-rules-` will be in sync.\n\n### Alternatively, there are no migration created after\nhttps://github.com//pull/216164. In that case, there is a\npossibility that `.kibana-siem-migrations-migrations-`, will\nnot even exist.\n\nIn this case `migrateRuleMigrationIndex` will create migration in below\nsteps:\n\n1. Create the `.kibana-siem-migrations-migrations-` index.\n2. Do steps mentioned in above scenario.\n\n### Once the migrations has been run successfully, both\n`.kibana-siem-migrations-migrations-` index and\n`.kibana-siem-migrations-rules-` will be in sync.\n\n1. In this case, migration will not run, since it tries to filter the\nmigrations by `id` which exist in\n`kibana-siem-migrations-rules-` but do not exist in\n`kibana-siem-migrations-migrations-`\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n\n### Identify risks\n\nDoes this PR introduce any risks? For example, consider risks like hard\nto test bugs, performance regression, potential of data loss.\n\nDescribe the risk, its severity, and mitigation for each identified\nrisk. Invite stakeholders and evaluate how to proceed before merging.\n\n- [ ] [See some risk\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)\n- [ ] ...\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bf642b00395079ef0e2f334b4fca65d2ff15d94c"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 67bf048

Failed CI Steps

• Linting
• Check Types

History