[Security Solution] Extend upgrade perform endpoint logic #191439
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/ci |
jpdjere
force-pushed
the
extend-upgrade-perform-endpoint-logic
branch
from
0c8ec85
to
710fc91
Compare
|
/ci |
1 task
jpdjere
force-pushed
the
extend-upgrade-perform-endpoint-logic
branch
4 times, most recently
from
396af3b
to
5784b19
Compare
jpdjere
changed the title
jpdjere
force-pushed
the
extend-upgrade-perform-endpoint-logic
branch
2 times, most recently
from
e5f8e33
to
6fa2ec1
Compare
jpdjere
added
release_note:enhancement
backport:skip
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
banderror
added
backport:prev-minor
jpdjere
force-pushed
the
extend-upgrade-perform-endpoint-logic
branch
from
329c06d
to
838120e
Compare
botelastic
bot
added
the
Team:Fleet
jpdjere
force-pushed
the
extend-upgrade-perform-endpoint-logic
branch
from
6ce32fc
to
5235856
Compare
jaredburgettelastic
approved these changes
Oct 15, 2024
|
Starting backport for target branches: 8.x https://github.com/elastic/kibana/actions/runs/11357418946 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 16, 2024
…1439) Fixes: elastic#166376 (main ticket) Fixes: elastic#186544 (handling of specific fields) Fixes: elastic#180195 (replace PATCH with PUT logic on rule upgrade) ## Summary - Enhances the `/upgrade/_perform` endpoint to upgrade rules in a way that works with prebuilt rules customized by users and resolve conflicts between user customizations and updates from Elastic. - Handles special fields under the hood (see below) - Replaces the update prebuilt rule logic to work with PUT instead of PATCH. ### Rough implementation plan - For each `upgradeableRule`, we attempt to build the payload necessary to pass to `upgradePrebuiltRules()`, which is of type `PrebuiltRuleAsset`. So we retrieve the field names from `FIELDS_PAYLOAD_BY_RULE_TYPE` and loop through them. - If any of those `field`s are non-upgreadable, (i.e. its value needs to be handled under the hood) we do so in `determineFieldUpgradeStatus`. - Otherwise, we continue to build a `FieldUpgradeSpecifier` for each field, which will help us determine if that field needs to be set to the base, current, target version, OR if it needs to be calculated as a MERGED value, or it is passed in the request payload as a RESOLVED value. - Notice that we are iterating over "flat" (non-grouped) fields which are part of the `PrebuiltRuleAsset` schema. This means that mapping is necessary between these flat fields and the diffable (grouped) fields that are used in the API contract, part of `DiffableRule`. For example, if we try to determine the value for the `query` field, we will need to look up for its value in the `eql_query` field if the target rule is `eql` or in `esql_query` if the target rule is `esql`. All these mappings can be found in `diffable_rule_fields_mappings.ts`. - Once a `FieldUpgradeSpecifier` has been retrieved for each field of the payload we are building, retrieve its actual value: either fetching it from the base, current or target versions of the rule, from the three way diff calculation, or retrieving it from the request payload if it resolved. - Do this for all upgreadable rules, and the pass the payload array into `upgradePrebuiltRules()`. - **IMPORTANT:** The upgrade prebuilt rules logic has been changed from PATCH to PUT. That means that if the next version of a rule removes a field, and the user updates to that target version, those fields will be undefined in the resulting rule. **Additional example:** a installs a rule, and creates a `timeline_id` for it rule by modifying it. If neither the next version (target version) still does not have a `timeline_id` field for it, and the user updates to that target version fully (without resolving the conflict), that field will not exist anymore in the resulting rule. ## Acceptance criteria - [x] Extend the contract of the API endpoint according to the [POC](elastic#144060): - [x] Add the ability to pick the `MERGED` version for rule upgrades. If the `MERGED` version is selected, the diffs are recalculated and the rule fields are updated to the result of the diff calculation. This is only possible if all field diffs return a `conflict` value of either `NO`. If any fields returns a value of `NON_SOLVABLE` or `SOLVABLE`, reject the request with an error specifying that there are conflicts, and that they must be resolved on a per-field basis. - [x] Calculate diffs inside this endpoint, when the value of `pick_version` is `MERGED`. - [x] Add the ability to specify rule field versions, to update specific fields to different `pick_versions`: `BASE' | 'CURRENT' | 'TARGET' | 'MERGED' | 'RESOLVED'` (See `FieldUpgradeRequest` in [PoC](elastic#144060) for details) ## Handling of special fields Specific fields are handled under the hood based on elastic#186544 See implementation in `x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`, which imports fields to handle under the hood: - `DiffableFieldsToOmit` - `FieldsToUpdateToCurrentVersion` ## Edge cases - [x] If target version of rule has a **rule type change**, check that all `pick_version`, at all levels, match `TARGET`. Otherwise, create new error and add to ruleErrors array. - [x] if a rule has a specific `targetVersion.type` (for example, EQL) and the user includes in its `fields` object of the request payload any fields which do not match that rule type (in this case, for example, sending in `machine_learning_job_id` as part of `fields`), throw an error for that rule. - [x] Calculation of field diffs: what happens if some fields have a conflict value of `NON_SOLVABLE`: - [x] If the whole rule is being updated to `MERGED`, and **ANY** fields return with a `NON_SOLVABLE` conflict, reject the whole update for that rule: create new error and add to ruleErrors array. - [x] **EXCEPTION** for case above: the whole rule is being updated to `MERGED`, and one or more of the fields return with a `NON_SOLVABLE` conflict, BUT those same fields have a specific `pick_version` for them in the `fields` object which **ARE NOT** `MERGED`. No error should be reported in this case. - [x] The whole rule is being updated to any `pick_version` other than MERGED, but any specific field in the `fields` object is set to upgrade to `MERGED`, and the diff for that fields returns a `NON_SOLVABLE` conflict. In that case, create new error and add to ruleErrors array. ### TODO - [[Security Solution] Add InvestigationFields and AlertSuppression fields to the upgrade workflow [elastic#190597]](elastic#190597): InvestigationFields is already working, but AlertSuppression is still currently handled under the hood to update to current version. ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Maxim Palenov <[email protected]> (cherry picked from commit 7c38873)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Oct 16, 2024
) (#196471) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Extend upgrade perform endpoint logic (#191439)](#191439) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Juan Pablo Djeredjian","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-16T01:51:25Z","message":"[Security Solution] Extend upgrade perform endpoint logic (#191439)\n\nFixes: #166376 (main ticket)\r\nFixes: #186544 (handling of\r\nspecific fields)\r\nFixes: #180195 (replace PATCH\r\nwith PUT logic on rule upgrade)\r\n\r\n## Summary\r\n\r\n- Enhances the `/upgrade/_perform` endpoint to upgrade rules in a way\r\nthat works with prebuilt rules customized by users and resolve conflicts\r\nbetween user customizations and updates from Elastic.\r\n- Handles special fields under the hood (see below)\r\n- Replaces the update prebuilt rule logic to work with PUT instead of\r\nPATCH.\r\n\r\n### Rough implementation plan\r\n- For each `upgradeableRule`, we attempt to build the payload necessary\r\nto pass to `upgradePrebuiltRules()`, which is of type\r\n`PrebuiltRuleAsset`. So we retrieve the field names from\r\n`FIELDS_PAYLOAD_BY_RULE_TYPE` and loop through them.\r\n- If any of those `field`s are non-upgreadable, (i.e. its value needs to\r\nbe handled under the hood) we do so in `determineFieldUpgradeStatus`.\r\n- Otherwise, we continue to build a `FieldUpgradeSpecifier` for each\r\nfield, which will help us determine if that field needs to be set to the\r\nbase, current, target version, OR if it needs to be calculated as a\r\nMERGED value, or it is passed in the request payload as a RESOLVED\r\nvalue.\r\n- Notice that we are iterating over \"flat\" (non-grouped) fields which\r\nare part of the `PrebuiltRuleAsset` schema. This means that mapping is\r\nnecessary between these flat fields and the diffable (grouped) fields\r\nthat are used in the API contract, part of `DiffableRule`. For example,\r\nif we try to determine the value for the `query` field, we will need to\r\nlook up for its value in the `eql_query` field if the target rule is\r\n`eql` or in `esql_query` if the target rule is `esql`. All these\r\nmappings can be found in `diffable_rule_fields_mappings.ts`.\r\n- Once a `FieldUpgradeSpecifier` has been retrieved for each field of\r\nthe payload we are building, retrieve its actual value: either fetching\r\nit from the base, current or target versions of the rule, from the three\r\nway diff calculation, or retrieving it from the request payload if it\r\nresolved.\r\n- Do this for all upgreadable rules, and the pass the payload array into\r\n`upgradePrebuiltRules()`.\r\n- **IMPORTANT:** The upgrade prebuilt rules logic has been changed from\r\nPATCH to PUT. That means that if the next version of a rule removes a\r\nfield, and the user updates to that target version, those fields will be\r\nundefined in the resulting rule. **Additional example:** a installs a\r\nrule, and creates a `timeline_id` for it rule by modifying it. If\r\nneither the next version (target version) still does not have a\r\n`timeline_id` field for it, and the user updates to that target version\r\nfully (without resolving the conflict), that field will not exist\r\nanymore in the resulting rule.\r\n\r\n## Acceptance criteria\r\n\r\n- [x] Extend the contract of the API endpoint according to the\r\n[POC](https://github.com/elastic/kibana/pull/144060):\r\n- [x] Add the ability to pick the `MERGED` version for rule upgrades. If\r\nthe `MERGED` version is selected, the diffs are recalculated and the\r\nrule fields are updated to the result of the diff calculation. This is\r\nonly possible if all field diffs return a `conflict` value of either\r\n`NO`. If any fields returns a value of `NON_SOLVABLE` or `SOLVABLE`,\r\nreject the request with an error specifying that there are conflicts,\r\nand that they must be resolved on a per-field basis.\r\n- [x] Calculate diffs inside this endpoint, when the value of\r\n`pick_version` is `MERGED`.\r\n- [x] Add the ability to specify rule field versions, to update specific\r\nfields to different `pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |\r\n'MERGED' | 'RESOLVED'` (See `FieldUpgradeRequest` in\r\n[PoC](#144060) for details)\r\n\r\n## Handling of special fields\r\n\r\nSpecific fields are handled under the hood based on\r\nhttps://github.com//issues/186544\r\n\r\nSee implementation in\r\n`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,\r\nwhich imports fields to handle under the hood:\r\n- `DiffableFieldsToOmit`\r\n- `FieldsToUpdateToCurrentVersion`\r\n\r\n## Edge cases\r\n\r\n- [x] If target version of rule has a **rule type change**, check that\r\nall `pick_version`, at all levels, match `TARGET`. Otherwise, create new\r\nerror and add to ruleErrors array.\r\n- [x] if a rule has a specific `targetVersion.type` (for example, EQL)\r\nand the user includes in its `fields` object of the request payload any\r\nfields which do not match that rule type (in this case, for example,\r\nsending in `machine_learning_job_id` as part of `fields`), throw an\r\nerror for that rule.\r\n- [x] Calculation of field diffs: what happens if some fields have a\r\nconflict value of `NON_SOLVABLE`:\r\n- [x] If the whole rule is being updated to `MERGED`, and **ANY** fields\r\nreturn with a `NON_SOLVABLE` conflict, reject the whole update for that\r\nrule: create new error and add to ruleErrors array.\r\n- [x] **EXCEPTION** for case above: the whole rule is being updated to\r\n`MERGED`, and one or more of the fields return with a `NON_SOLVABLE`\r\nconflict, BUT those same fields have a specific `pick_version` for them\r\nin the `fields` object which **ARE NOT** `MERGED`. No error should be\r\nreported in this case.\r\n- [x] The whole rule is being updated to any `pick_version` other than\r\nMERGED, but any specific field in the `fields` object is set to upgrade\r\nto `MERGED`, and the diff for that fields returns a `NON_SOLVABLE`\r\nconflict. In that case, create new error and add to ruleErrors array.\r\n\r\n### TODO\r\n\r\n- [[Security Solution] Add InvestigationFields and AlertSuppression\r\nfields to the upgrade workflow\r\n[#190597]](https://github.com/elastic/kibana/issues/190597):\r\nInvestigationFields is already working, but AlertSuppression is still\r\ncurrently handled under the hood to update to current version.\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: Maxim Palenov <[email protected]>","sha":"7c3887309cec54cc21e1abf8a2522afa49147712","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","Team:Fleet","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:prev-minor","v8.16.0"],"title":"[Security Solution] Extend upgrade perform endpoint logic","number":191439,"url":"https://github.com/elastic/kibana/pull/191439","mergeCommit":{"message":"[Security Solution] Extend upgrade perform endpoint logic (#191439)\n\nFixes: #166376 (main ticket)\r\nFixes: #186544 (handling of\r\nspecific fields)\r\nFixes: #180195 (replace PATCH\r\nwith PUT logic on rule upgrade)\r\n\r\n## Summary\r\n\r\n- Enhances the `/upgrade/_perform` endpoint to upgrade rules in a way\r\nthat works with prebuilt rules customized by users and resolve conflicts\r\nbetween user customizations and updates from Elastic.\r\n- Handles special fields under the hood (see below)\r\n- Replaces the update prebuilt rule logic to work with PUT instead of\r\nPATCH.\r\n\r\n### Rough implementation plan\r\n- For each `upgradeableRule`, we attempt to build the payload necessary\r\nto pass to `upgradePrebuiltRules()`, which is of type\r\n`PrebuiltRuleAsset`. So we retrieve the field names from\r\n`FIELDS_PAYLOAD_BY_RULE_TYPE` and loop through them.\r\n- If any of those `field`s are non-upgreadable, (i.e. its value needs to\r\nbe handled under the hood) we do so in `determineFieldUpgradeStatus`.\r\n- Otherwise, we continue to build a `FieldUpgradeSpecifier` for each\r\nfield, which will help us determine if that field needs to be set to the\r\nbase, current, target version, OR if it needs to be calculated as a\r\nMERGED value, or it is passed in the request payload as a RESOLVED\r\nvalue.\r\n- Notice that we are iterating over \"flat\" (non-grouped) fields which\r\nare part of the `PrebuiltRuleAsset` schema. This means that mapping is\r\nnecessary between these flat fields and the diffable (grouped) fields\r\nthat are used in the API contract, part of `DiffableRule`. For example,\r\nif we try to determine the value for the `query` field, we will need to\r\nlook up for its value in the `eql_query` field if the target rule is\r\n`eql` or in `esql_query` if the target rule is `esql`. All these\r\nmappings can be found in `diffable_rule_fields_mappings.ts`.\r\n- Once a `FieldUpgradeSpecifier` has been retrieved for each field of\r\nthe payload we are building, retrieve its actual value: either fetching\r\nit from the base, current or target versions of the rule, from the three\r\nway diff calculation, or retrieving it from the request payload if it\r\nresolved.\r\n- Do this for all upgreadable rules, and the pass the payload array into\r\n`upgradePrebuiltRules()`.\r\n- **IMPORTANT:** The upgrade prebuilt rules logic has been changed from\r\nPATCH to PUT. That means that if the next version of a rule removes a\r\nfield, and the user updates to that target version, those fields will be\r\nundefined in the resulting rule. **Additional example:** a installs a\r\nrule, and creates a `timeline_id` for it rule by modifying it. If\r\nneither the next version (target version) still does not have a\r\n`timeline_id` field for it, and the user updates to that target version\r\nfully (without resolving the conflict), that field will not exist\r\nanymore in the resulting rule.\r\n\r\n## Acceptance criteria\r\n\r\n- [x] Extend the contract of the API endpoint according to the\r\n[POC](https://github.com/elastic/kibana/pull/144060):\r\n- [x] Add the ability to pick the `MERGED` version for rule upgrades. If\r\nthe `MERGED` version is selected, the diffs are recalculated and the\r\nrule fields are updated to the result of the diff calculation. This is\r\nonly possible if all field diffs return a `conflict` value of either\r\n`NO`. If any fields returns a value of `NON_SOLVABLE` or `SOLVABLE`,\r\nreject the request with an error specifying that there are conflicts,\r\nand that they must be resolved on a per-field basis.\r\n- [x] Calculate diffs inside this endpoint, when the value of\r\n`pick_version` is `MERGED`.\r\n- [x] Add the ability to specify rule field versions, to update specific\r\nfields to different `pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |\r\n'MERGED' | 'RESOLVED'` (See `FieldUpgradeRequest` in\r\n[PoC](#144060) for details)\r\n\r\n## Handling of special fields\r\n\r\nSpecific fields are handled under the hood based on\r\nhttps://github.com//issues/186544\r\n\r\nSee implementation in\r\n`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,\r\nwhich imports fields to handle under the hood:\r\n- `DiffableFieldsToOmit`\r\n- `FieldsToUpdateToCurrentVersion`\r\n\r\n## Edge cases\r\n\r\n- [x] If target version of rule has a **rule type change**, check that\r\nall `pick_version`, at all levels, match `TARGET`. Otherwise, create new\r\nerror and add to ruleErrors array.\r\n- [x] if a rule has a specific `targetVersion.type` (for example, EQL)\r\nand the user includes in its `fields` object of the request payload any\r\nfields which do not match that rule type (in this case, for example,\r\nsending in `machine_learning_job_id` as part of `fields`), throw an\r\nerror for that rule.\r\n- [x] Calculation of field diffs: what happens if some fields have a\r\nconflict value of `NON_SOLVABLE`:\r\n- [x] If the whole rule is being updated to `MERGED`, and **ANY** fields\r\nreturn with a `NON_SOLVABLE` conflict, reject the whole update for that\r\nrule: create new error and add to ruleErrors array.\r\n- [x] **EXCEPTION** for case above: the whole rule is being updated to\r\n`MERGED`, and one or more of the fields return with a `NON_SOLVABLE`\r\nconflict, BUT those same fields have a specific `pick_version` for them\r\nin the `fields` object which **ARE NOT** `MERGED`. No error should be\r\nreported in this case.\r\n- [x] The whole rule is being updated to any `pick_version` other than\r\nMERGED, but any specific field in the `fields` object is set to upgrade\r\nto `MERGED`, and the diff for that fields returns a `NON_SOLVABLE`\r\nconflict. In that case, create new error and add to ruleErrors array.\r\n\r\n### TODO\r\n\r\n- [[Security Solution] Add InvestigationFields and AlertSuppression\r\nfields to the upgrade workflow\r\n[#190597]](https://github.com/elastic/kibana/issues/190597):\r\nInvestigationFields is already working, but AlertSuppression is still\r\ncurrently handled under the hood to update to current version.\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: Maxim Palenov <[email protected]>","sha":"7c3887309cec54cc21e1abf8a2522afa49147712"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191439","number":191439,"mergeCommit":{"message":"[Security Solution] Extend upgrade perform endpoint logic (#191439)\n\nFixes: #166376 (main ticket)\r\nFixes: #186544 (handling of\r\nspecific fields)\r\nFixes: #180195 (replace PATCH\r\nwith PUT logic on rule upgrade)\r\n\r\n## Summary\r\n\r\n- Enhances the `/upgrade/_perform` endpoint to upgrade rules in a way\r\nthat works with prebuilt rules customized by users and resolve conflicts\r\nbetween user customizations and updates from Elastic.\r\n- Handles special fields under the hood (see below)\r\n- Replaces the update prebuilt rule logic to work with PUT instead of\r\nPATCH.\r\n\r\n### Rough implementation plan\r\n- For each `upgradeableRule`, we attempt to build the payload necessary\r\nto pass to `upgradePrebuiltRules()`, which is of type\r\n`PrebuiltRuleAsset`. So we retrieve the field names from\r\n`FIELDS_PAYLOAD_BY_RULE_TYPE` and loop through them.\r\n- If any of those `field`s are non-upgreadable, (i.e. its value needs to\r\nbe handled under the hood) we do so in `determineFieldUpgradeStatus`.\r\n- Otherwise, we continue to build a `FieldUpgradeSpecifier` for each\r\nfield, which will help us determine if that field needs to be set to the\r\nbase, current, target version, OR if it needs to be calculated as a\r\nMERGED value, or it is passed in the request payload as a RESOLVED\r\nvalue.\r\n- Notice that we are iterating over \"flat\" (non-grouped) fields which\r\nare part of the `PrebuiltRuleAsset` schema. This means that mapping is\r\nnecessary between these flat fields and the diffable (grouped) fields\r\nthat are used in the API contract, part of `DiffableRule`. For example,\r\nif we try to determine the value for the `query` field, we will need to\r\nlook up for its value in the `eql_query` field if the target rule is\r\n`eql` or in `esql_query` if the target rule is `esql`. All these\r\nmappings can be found in `diffable_rule_fields_mappings.ts`.\r\n- Once a `FieldUpgradeSpecifier` has been retrieved for each field of\r\nthe payload we are building, retrieve its actual value: either fetching\r\nit from the base, current or target versions of the rule, from the three\r\nway diff calculation, or retrieving it from the request payload if it\r\nresolved.\r\n- Do this for all upgreadable rules, and the pass the payload array into\r\n`upgradePrebuiltRules()`.\r\n- **IMPORTANT:** The upgrade prebuilt rules logic has been changed from\r\nPATCH to PUT. That means that if the next version of a rule removes a\r\nfield, and the user updates to that target version, those fields will be\r\nundefined in the resulting rule. **Additional example:** a installs a\r\nrule, and creates a `timeline_id` for it rule by modifying it. If\r\nneither the next version (target version) still does not have a\r\n`timeline_id` field for it, and the user updates to that target version\r\nfully (without resolving the conflict), that field will not exist\r\nanymore in the resulting rule.\r\n\r\n## Acceptance criteria\r\n\r\n- [x] Extend the contract of the API endpoint according to the\r\n[POC](https://github.com/elastic/kibana/pull/144060):\r\n- [x] Add the ability to pick the `MERGED` version for rule upgrades. If\r\nthe `MERGED` version is selected, the diffs are recalculated and the\r\nrule fields are updated to the result of the diff calculation. This is\r\nonly possible if all field diffs return a `conflict` value of either\r\n`NO`. If any fields returns a value of `NON_SOLVABLE` or `SOLVABLE`,\r\nreject the request with an error specifying that there are conflicts,\r\nand that they must be resolved on a per-field basis.\r\n- [x] Calculate diffs inside this endpoint, when the value of\r\n`pick_version` is `MERGED`.\r\n- [x] Add the ability to specify rule field versions, to update specific\r\nfields to different `pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |\r\n'MERGED' | 'RESOLVED'` (See `FieldUpgradeRequest` in\r\n[PoC](#144060) for details)\r\n\r\n## Handling of special fields\r\n\r\nSpecific fields are handled under the hood based on\r\nhttps://github.com//issues/186544\r\n\r\nSee implementation in\r\n`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,\r\nwhich imports fields to handle under the hood:\r\n- `DiffableFieldsToOmit`\r\n- `FieldsToUpdateToCurrentVersion`\r\n\r\n## Edge cases\r\n\r\n- [x] If target version of rule has a **rule type change**, check that\r\nall `pick_version`, at all levels, match `TARGET`. Otherwise, create new\r\nerror and add to ruleErrors array.\r\n- [x] if a rule has a specific `targetVersion.type` (for example, EQL)\r\nand the user includes in its `fields` object of the request payload any\r\nfields which do not match that rule type (in this case, for example,\r\nsending in `machine_learning_job_id` as part of `fields`), throw an\r\nerror for that rule.\r\n- [x] Calculation of field diffs: what happens if some fields have a\r\nconflict value of `NON_SOLVABLE`:\r\n- [x] If the whole rule is being updated to `MERGED`, and **ANY** fields\r\nreturn with a `NON_SOLVABLE` conflict, reject the whole update for that\r\nrule: create new error and add to ruleErrors array.\r\n- [x] **EXCEPTION** for case above: the whole rule is being updated to\r\n`MERGED`, and one or more of the fields return with a `NON_SOLVABLE`\r\nconflict, BUT those same fields have a specific `pick_version` for them\r\nin the `fields` object which **ARE NOT** `MERGED`. No error should be\r\nreported in this case.\r\n- [x] The whole rule is being updated to any `pick_version` other than\r\nMERGED, but any specific field in the `fields` object is set to upgrade\r\nto `MERGED`, and the diff for that fields returns a `NON_SOLVABLE`\r\nconflict. In that case, create new error and add to ruleErrors array.\r\n\r\n### TODO\r\n\r\n- [[Security Solution] Add InvestigationFields and AlertSuppression\r\nfields to the upgrade workflow\r\n[#190597]](https://github.com/elastic/kibana/issues/190597):\r\nInvestigationFields is already working, but AlertSuppression is still\r\ncurrently handled under the hood to update to current version.\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: Maxim Palenov <[email protected]>","sha":"7c3887309cec54cc21e1abf8a2522afa49147712"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Juan Pablo Djeredjian <[email protected]>
4 tasks
jpdjere
added a commit
that referenced
this pull request
Oct 23, 2024
…rade with `MERGED` pick_version (#197262) ## Summary The PR #191439 enhanced the `/upgrade/_perform` API contract and functionality to allow the users of the endpoint to upgrade rules to their `MERGED` version. However, a bug slipped in, where the two different types of `DataSource` (`type: index_patterns` or `type: data_view_id`) weren't properly handled and would cause, in some cases, a rule payload to be created having both an `index` and `data_view` field, causing upgrade to fail. This PR fixes the issue by handling these two field in a specific way, checking what the `DataSource` diffable field's type is, and setting the other field to `undefined`. ### Checklist Delete any items that are not applicable to this PR. - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels) - [ ] This will appear in the **Release Notes** and follow the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 23, 2024
…rade with `MERGED` pick_version (elastic#197262) ## Summary The PR elastic#191439 enhanced the `/upgrade/_perform` API contract and functionality to allow the users of the endpoint to upgrade rules to their `MERGED` version. However, a bug slipped in, where the two different types of `DataSource` (`type: index_patterns` or `type: data_view_id`) weren't properly handled and would cause, in some cases, a rule payload to be created having both an `index` and `data_view` field, causing upgrade to fail. This PR fixes the issue by handling these two field in a specific way, checking what the `DataSource` diffable field's type is, and setting the other field to `undefined`. ### Checklist Delete any items that are not applicable to this PR. - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels) - [ ] This will appear in the **Release Notes** and follow the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 9656621)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 23, 2024
…rade with `MERGED` pick_version (elastic#197262) ## Summary The PR elastic#191439 enhanced the `/upgrade/_perform` API contract and functionality to allow the users of the endpoint to upgrade rules to their `MERGED` version. However, a bug slipped in, where the two different types of `DataSource` (`type: index_patterns` or `type: data_view_id`) weren't properly handled and would cause, in some cases, a rule payload to be created having both an `index` and `data_view` field, causing upgrade to fail. This PR fixes the issue by handling these two field in a specific way, checking what the `DataSource` diffable field's type is, and setting the other field to `undefined`. ### Checklist Delete any items that are not applicable to this PR. - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels) - [ ] This will appear in the **Release Notes** and follow the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 9656621)
kibanamachine
added a commit
that referenced
this pull request
Oct 23, 2024
… during rule upgrade with `MERGED` pick_version (#197262) (#197467) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version (#197262)](#197262) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Juan Pablo Djeredjian","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-23T14:44:13Z","message":"[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR #191439 enhanced the\r\n`/upgrade/_perform` API contract and functionality to allow the users of\r\nthe endpoint to upgrade rules to their `MERGED` version.\r\n\r\nHowever, a bug slipped in, where the two different types of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`) weren't properly\r\nhandled and would cause, in some cases, a rule payload to be created\r\nhaving both an `index` and `data_view` field, causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling these two field in a specific way,\r\nchecking what the `DataSource` diffable field's type is, and setting the\r\nother field to `undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n- [ ] This will appear in the **Release Notes** and follow the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","v8.16.0","backport:version","v8.17.0"],"title":"[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version","number":197262,"url":"https://github.com/elastic/kibana/pull/197262","mergeCommit":{"message":"[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR #191439 enhanced the\r\n`/upgrade/_perform` API contract and functionality to allow the users of\r\nthe endpoint to upgrade rules to their `MERGED` version.\r\n\r\nHowever, a bug slipped in, where the two different types of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`) weren't properly\r\nhandled and would cause, in some cases, a rule payload to be created\r\nhaving both an `index` and `data_view` field, causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling these two field in a specific way,\r\nchecking what the `DataSource` diffable field's type is, and setting the\r\nother field to `undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n- [ ] This will appear in the **Release Notes** and follow the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10"}},"sourceBranch":"main","suggestedTargetBranches":["8.16","8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197262","number":197262,"mergeCommit":{"message":"[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR #191439 enhanced the\r\n`/upgrade/_perform` API contract and functionality to allow the users of\r\nthe endpoint to upgrade rules to their `MERGED` version.\r\n\r\nHowever, a bug slipped in, where the two different types of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`) weren't properly\r\nhandled and would cause, in some cases, a rule payload to be created\r\nhaving both an `index` and `data_view` field, causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling these two field in a specific way,\r\nchecking what the `DataSource` diffable field's type is, and setting the\r\nother field to `undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n- [ ] This will appear in the **Release Notes** and follow the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Juan Pablo Djeredjian <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Oct 23, 2024
…n during rule upgrade with `MERGED` pick_version (#197262) (#197466) # Backport This will backport the following commits from `main` to `8.16`: - [[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version (#197262)](#197262) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Juan Pablo Djeredjian","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-23T14:44:13Z","message":"[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR #191439 enhanced the\r\n`/upgrade/_perform` API contract and functionality to allow the users of\r\nthe endpoint to upgrade rules to their `MERGED` version.\r\n\r\nHowever, a bug slipped in, where the two different types of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`) weren't properly\r\nhandled and would cause, in some cases, a rule payload to be created\r\nhaving both an `index` and `data_view` field, causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling these two field in a specific way,\r\nchecking what the `DataSource` diffable field's type is, and setting the\r\nother field to `undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n- [ ] This will appear in the **Release Notes** and follow the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","v8.16.0","backport:version","v8.17.0"],"title":"[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version","number":197262,"url":"https://github.com/elastic/kibana/pull/197262","mergeCommit":{"message":"[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR #191439 enhanced the\r\n`/upgrade/_perform` API contract and functionality to allow the users of\r\nthe endpoint to upgrade rules to their `MERGED` version.\r\n\r\nHowever, a bug slipped in, where the two different types of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`) weren't properly\r\nhandled and would cause, in some cases, a rule payload to be created\r\nhaving both an `index` and `data_view` field, causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling these two field in a specific way,\r\nchecking what the `DataSource` diffable field's type is, and setting the\r\nother field to `undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n- [ ] This will appear in the **Release Notes** and follow the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10"}},"sourceBranch":"main","suggestedTargetBranches":["8.16","8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197262","number":197262,"mergeCommit":{"message":"[Security Solution] Fix `DataSource` payload creation during rule upgrade with `MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR #191439 enhanced the\r\n`/upgrade/_perform` API contract and functionality to allow the users of\r\nthe endpoint to upgrade rules to their `MERGED` version.\r\n\r\nHowever, a bug slipped in, where the two different types of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`) weren't properly\r\nhandled and would cause, in some cases, a rule payload to be created\r\nhaving both an `index` and `data_view` field, causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling these two field in a specific way,\r\nchecking what the `DataSource` diffable field's type is, and setting the\r\nother field to `undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n- [ ] This will appear in the **Release Notes** and follow the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Juan Pablo Djeredjian <[email protected]>
banderror
added
release_note:skip
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: #166376 (main ticket)
Fixes: #186544 (handling of specific fields)
Fixes: #180195 (replace PATCH with PUT logic on rule upgrade)
Summary
/upgrade/_performendpoint to upgrade rules in a way that works with prebuilt rules customized by users and resolve conflicts between user customizations and updates from Elastic.Rough implementation plan
upgradeableRule, we attempt to build the payload necessary to pass toupgradePrebuiltRules(), which is of typePrebuiltRuleAsset. So we retrieve the field names fromFIELDS_PAYLOAD_BY_RULE_TYPEand loop through them.fields are non-upgreadable, (i.e. its value needs to be handled under the hood) we do so indetermineFieldUpgradeStatus.FieldUpgradeSpecifierfor each field, which will help us determine if that field needs to be set to the base, current, target version, OR if it needs to be calculated as a MERGED value, or it is passed in the request payload as a RESOLVED value.PrebuiltRuleAssetschema. This means that mapping is necessary between these flat fields and the diffable (grouped) fields that are used in the API contract, part ofDiffableRule. For example, if we try to determine the value for thequeryfield, we will need to look up for its value in theeql_queryfield if the target rule iseqlor inesql_queryif the target rule isesql. All these mappings can be found indiffable_rule_fields_mappings.ts.FieldUpgradeSpecifierhas been retrieved for each field of the payload we are building, retrieve its actual value: either fetching it from the base, current or target versions of the rule, from the three way diff calculation, or retrieving it from the request payload if it resolved.upgradePrebuiltRules().timeline_idfor it rule by modifying it. If neither the next version (target version) still does not have atimeline_idfield for it, and the user updates to that target version fully (without resolving the conflict), that field will not exist anymore in the resulting rule.Acceptance criteria
MERGEDversion for rule upgrades. If theMERGEDversion is selected, the diffs are recalculated and the rule fields are updated to the result of the diff calculation. This is only possible if all field diffs return aconflictvalue of eitherNO. If any fields returns a value ofNON_SOLVABLEorSOLVABLE, reject the request with an error specifying that there are conflicts, and that they must be resolved on a per-field basis.pick_versionisMERGED.pick_versions:BASE' | 'CURRENT' | 'TARGET' | 'MERGED' | 'RESOLVED'(SeeFieldUpgradeRequestin PoC for details)Handling of special fields
Specific fields are handled under the hood based on #186544
See implementation in
x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts, which imports fields to handle under the hood:DiffableFieldsToOmitFieldsToUpdateToCurrentVersionEdge cases
pick_version, at all levels, matchTARGET. Otherwise, create new error and add to ruleErrors array.targetVersion.type(for example, EQL) and the user includes in itsfieldsobject of the request payload any fields which do not match that rule type (in this case, for example, sending inmachine_learning_job_idas part offields), throw an error for that rule.NON_SOLVABLE:MERGED, and ANY fields return with aNON_SOLVABLEconflict, reject the whole update for that rule: create new error and add to ruleErrors array.MERGED, and one or more of the fields return with aNON_SOLVABLEconflict, BUT those same fields have a specificpick_versionfor them in thefieldsobject which ARE NOTMERGED. No error should be reported in this case.pick_versionother than MERGED, but any specific field in thefieldsobject is set to upgrade toMERGED, and the diff for that fields returns aNON_SOLVABLEconflict. In that case, create new error and add to ruleErrors array.TODO
[#190597]: InvestigationFields is already working, but AlertSuppression is still currently handled under the hood to update to current version.
For maintainers