[Automatic Migrations] Siem migrations new endpoint scheme

[logeekal]

Summary

Resolves https://github.com/elastic/security-team/issues/12483

This PR changes REST API Endpoints scheme to align with https://github.com/elastic/security-team/issues/12483. Below is the summary of changes done.

API Scheme changes

The REST API scheme has been changed to reflect https://github.com/elastic/security-team/issues/12483. This is pretty much self explanatory as defined in below openapi schema yaml :

• x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/api/rules/rule_migration.schema.yaml

Introduction of Delete Migration API

This PR also adds DELETE method on route /rules/siem_migrations/{migration_id} for deleting a migrations. Deleting a migration does below operations:

• Stops a migration if it is running
• Deletes the rules, resources related to migration and migration document itself.

File Reorganizations

Directly structure has been changed a little bit to reflect the endpoint. There is a sub-directory called rules which deals with only rules of the migration and the root directly only contains the endpoints related to the migration.

Before

//siem_migrations/rules/api

├── create.ts
├── get.ts
├── start.ts
├── update.ts
├── 
├── 
├── 

After

//siem_migrations/rules/api


├── create.ts
├── delete.ts
├── get.ts
├── rules
│   ├── add.ts
│   ├── get.ts
│   └── update.ts
├── 
├── 
├── 

Migration Strategy

TL,DR;

flowchart TD
    StartM[Start Migration] --> isMigExists{Does Migration Index Exists}
    isMigExists -->|Yes|FetchMDoc[Fetch Migration Docs]
    isMigExists -->|No|CreateMIndex[Create MigrationIndex]
    CreateMIndex --> FetchMDoc
    FetchMDoc --> FetchMRules[Fetch Migration Stats Rules index]
    FetchMRules --> FilterMigration{Filter Migration Docs not in Migration Index}
    FilterMigration --> |is Empty|END[END]
    FilterMigration --> |is Not Empty| CreateMDocs[Create Migration Docs]
    CreateMDocs --> END

Loading

At the time of merging this PR, the Migration indices can be in 3 states:

There are migrations created after #216164 and this means that there are some migrations existing in .kibana-siem-migrations-migrations-<space_id> and migrations created before above mentioned PR will only exist in .kibana-siem-migrations-rules-<space_id>.

In this case migrateRuleMigrationIndex will create migration in below steps:

1. Look for all migration Documents in .kibana-siem-migrations-migrations-<space_id>
2. Get all Migrations stats from .kibana-siem-migrations-rules-<space_id> which includes below properties
   • migration_id : will help in reconciling the migration id in .kibana-siem-migrations-migrations-<space_id>` index
   • created_at : Date on which migration_id was created.
   • created_by: User who created the migrations.
3. A new document with above migration will be added to .kibana-siem-migrations-migrations-<space_id>.
4. Now both .kibana-siem-migrations-migrations-<space_id> and .kibana-siem-migrations-rules-<space_id> will be in sync.

Alternatively, there are no migration created after #216164. In that case, there is a possibility that .kibana-siem-migrations-migrations-<space_id>, will not even exist.

In this case migrateRuleMigrationIndex will create migration in below steps:

1. Create the .kibana-siem-migrations-migrations-<space_id> index.
2. Do steps mentioned in above scenario.

Once the migrations has been run successfully, both .kibana-siem-migrations-migrations-<space_id> index and .kibana-siem-migrations-rules-<space_id> will be in sync.

1. In this case, migration will not run, since it tries to filter the migrations by id which exist in kibana-siem-migrations-rules-<space_id> but do not exist in kibana-siem-migrations-migrations-<space_id>

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[prodsecmachine]

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

[kqualters-elastic]

talked through some of this during our syncs, and desk tested this most recent iteration locally and with frontend changes based on this branch, everything looks good to me 👍

[semd]

Tested locally, index migrators create the expected documents. Regular rule migration workflow works as usual. LGTM.
Thanks Jatin! 🚀

[kibanamachine]

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15190552779

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 4f740bb

Failed CI Steps

• FTR Configs #83

Test Failures

• [job] [logs] FTR Configs #83 / Synthetics API Tests SyncGlobalParams parsed params for previously added http monitors

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	9.2MB	9.2MB	+1.0KB

History

• 💔 Build #301570 failed 7a8b5e0
• 💔 Build #301568 failed a2f1f5e
• 💔 Build #301430 failed b629c21
• 💛 Build #301331 was flaky cae736d
• 💛 Build #300723 was flaky 6115d50

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 219597

Questions ?

Please refer to the Backport tool documentation

[logeekal]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @logeekal

[logeekal]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)