diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade_review_algorithms.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade_review_algorithms.md index e73b976d0b44..26b01da20090 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade_review_algorithms.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade_review_algorithms.md @@ -24,14 +24,18 @@ Status: `in progress`. - [**Scenario: `ABB` - Rule field is any type**](#scenario-abb---rule-field-is-any-type) - [Rule field has an update and a custom value that are NOT the same - `ABC`](#rule-field-has-an-update-and-a-custom-value-that-are-not-the-same---abc) - [**Scenario: `ABC` - Rule field is a number or single line string**](#scenario-abc---rule-field-is-a-number-or-single-line-string) - - [**Scenario: `ABC` - Rule field is a mergable multi line string**](#scenario-abc---rule-field-is-a-mergable-multi-line-string) - - [**Scenario: `ABC` - Rule field is a non-mergable multi line string**](#scenario-abc---rule-field-is-a-non-mergable-multi-line-string) + - [**Scenario: `ABC` - Rule field is a mergeable multi line string**](#scenario-abc---rule-field-is-a-mergeable-multi-line-string) + - [**Scenario: `ABC` - Rule field is a non-mergeable multi line string**](#scenario-abc---rule-field-is-a-non-mergeable-multi-line-string) - [**Scenario: `ABC` - Rule field is an array of scalar values**](#scenario-abc---rule-field-is-an-array-of-scalar-values) + - [**Scenario: `ABC` - Rule field is a solvable `data_source` object**](#scenario-abc---rule-field-is-a-solvable-data_source-object) + - [**Scenario: `ABC` - Rule field is a non-solvable `data_source` object**](#scenario-abc---rule-field-is-a-non-solvable-data_source-object) - [Rule field has an update and a custom value that are the same and the rule base version doesn't exist - `-AA`](#rule-field-has-an-update-and-a-custom-value-that-are-the-same-and-the-rule-base-version-doesnt-exist----aa) - [**Scenario: `-AA` - Rule field is any type**](#scenario--aa---rule-field-is-any-type) - - [Rule field has an update and a custom value that are NOT the same and the rule base version doesn't exist - `-BC`](#rule-field-has-an-update-and-a-custom-value-that-are-not-the-same-and-the-rule-base-version-doesnt-exist----bc) - - [**Scenario: `-BC` - Rule field is a number or single line string**](#scenario--bc---rule-field-is-a-number-or-single-line-string) - - [**Scenario: `-BC` - Rule field is an array of scalar values**](#scenario--bc---rule-field-is-an-array-of-scalar-values) + - [Rule field has an update and a custom value that are NOT the same and the rule base version doesn't exist - `-AB`](#rule-field-has-an-update-and-a-custom-value-that-are-not-the-same-and-the-rule-base-version-doesnt-exist----ab) + - [**Scenario: `-AB` - Rule field is a number or single line string**](#scenario--ab---rule-field-is-a-number-or-single-line-string) + - [**Scenario: `-AB` - Rule field is an array of scalar values**](#scenario--ab---rule-field-is-an-array-of-scalar-values) + - [**Scenario: `-AB` - Rule field is a solvable `data_source` object**](#scenario--ab---rule-field-is-a-solvable-data_source-object) + - [**Scenario: `-AB` - Rule field is a non-solvable `data_source` object**](#scenario--ab---rule-field-is-a-non-solvable-data_source-object) ## Useful information @@ -52,6 +56,9 @@ Status: `in progress`. - **Merged version**: Also labeled as `merged_version`. This is the version of the rule that we determine via the various algorithms. It could contain a mix of all the rule versions on a per-field basis to create a singluar version of the rule containing all relevant updates and user changes to display to the user. +- **Grouped fields** + - `data_source`: an object that contains a `type` field with a value of `data_view_id` or `index_patterns` and another field that's either `data_view_id` of type string OR `index_patterns` of type string array + ### Assumptions - All scenarios will contain at least 1 prebuilt rule installed in Kibana. @@ -63,7 +70,7 @@ Status: `in progress`. #### **Scenario: `AAA` - Rule field is any type** -**Automation**: 4 integration tests with mock rules + a set of unit tests for each algorithm +**Automation**: 6 integration tests with mock rules + a set of unit tests for each algorithm ```Gherkin Given field is not customized by the user (current version == base version) @@ -73,18 +80,20 @@ And field should not be returned from the `upgrade/_review` API end And field should not be shown in the upgrade preview UI Examples: -| algorithm | field_name | base_version | current_version | target_version | merged_version | -| single line string | name | "A" | "A" | "A" | "A" | -| multi line string | description | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." | -| number | risk_score | 1 | 1 | 1 | 1 | -| array of scalars | tags | ["one", "two", "three"] | ["one", "three", "two"] | ["three", "one", "two"] | ["one", "three", "two"] | +| algorithm | field_name | base_version | current_version | target_version | merged_version | +| single line string | name | "A" | "A" | "A" | "A" | +| multi line string | description | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." | +| number | risk_score | 1 | 1 | 1 | 1 | +| array of scalars | tags | ["one", "two", "three"] | ["one", "three", "two"] | ["three", "one", "two"] | ["one", "three", "two"] | +| data_source | data_source | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | +| data_source | data_source | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | ``` ### Rule field doesn't have an update but has a custom value - `ABA` #### **Scenario: `ABA` - Rule field is any type** -**Automation**: 3 integration tests with mock rules + a set of unit tests for each algorithm +**Automation**: 6 integration tests with mock rules + a set of unit tests for each algorithm ```Gherkin Given field is customized by the user (current version != base version) @@ -94,18 +103,20 @@ And field should be returned from the `upgrade/_review` API endpoin And field should be shown in the upgrade preview UI Examples: -| algorithm | field_name | base_version | current_version | target_version | merged_version | -| single line string | name | "A" | "B" | "A" | "B" | -| multi line string | description | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | -| number | risk_score | 1 | 2 | 1 | 2 | -| array of scalars | tags | ["one", "two", "three"] | ["one", "two", "four"] | ["one", "two", "three"] | ["one", "two", "four"] | +| algorithm | field_name | base_version | current_version | target_version | merged_version | +| single line string | name | "A" | "B" | "A" | "B" | +| multi line string | description | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | +| number | risk_score | 1 | 2 | 1 | 2 | +| array of scalars | tags | ["one", "two", "three"] | ["one", "two", "four"] | ["one", "two", "three"] | ["one", "two", "four"] | +| data_source | data_source | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | +| data_source | data_source | {type: "data_view", "data_view_id": "A"} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | ``` ### Rule field has an update and doesn't have a custom value - `AAB` #### **Scenario: `AAB` - Rule field is any type** -**Automation**: 3 integration tests with mock rules + a set of unit tests for each algorithm +**Automation**: 6 integration tests with mock rules + a set of unit tests for each algorithm ```Gherkin Given field is not customized by the user (current version == base version) @@ -115,18 +126,20 @@ And field should be returned from the `upgrade/_review` API endpoin And field should be shown in the upgrade preview UI Examples: -| algorithm | field_name | base_version | current_version | target_version | merged_version | -| single line string | name | "A" | "A" | "B" | "B" | -| multi line string | description | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | -| number | risk_score | 1 | 1 | 2 | 2 | -| array of scalars | tags | ["one", "two", "three"] | ["one", "two", "three"] | ["one", "two", "four"] | ["one", "two", "four"] | +| algorithm | field_name | base_version | current_version | target_version | merged_version | +| single line string | name | "A" | "A" | "B" | "B" | +| multi line string | description | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | +| number | risk_score | 1 | 1 | 2 | 2 | +| array of scalars | tags | ["one", "two", "three"] | ["one", "two", "three"] | ["one", "two", "four"] | ["one", "two", "four"] | +| data_source | data_source | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | +| data_source | data_source | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | ``` ### Rule field has an update and a custom value that are the same - `ABB` #### **Scenario: `ABB` - Rule field is any type** -**Automation**: 3 integration tests with mock rules + a set of unit tests for each algorithm +**Automation**: 6 integration tests with mock rules + a set of unit tests for each algorithm ```Gherkin Given field is customized by the user (current version != base version) @@ -137,11 +150,13 @@ And field should be returned from the `upgrade/_review` API endpoin And field should be shown in the upgrade preview UI Examples: -| algorithm | field_name | base_version | current_version | target_version | merged_version | -| single line string | name | "A" | "B" | "B" | "B" | -| multi line string | description | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | -| number | risk_score | 1 | 2 | 2 | 2 | -| array of scalars | tags | ["one", "two", "three"] | ["one", "two", "four"] | ["one", "two", "four"] | ["one", "two", "four"] | +| algorithm | field_name | base_version | current_version | target_version | merged_version | +| single line string | name | "A" | "B" | "B" | "B" | +| multi line string | description | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | +| number | risk_score | 1 | 2 | 2 | 2 | +| array of scalars | tags | ["one", "two", "three"] | ["one", "two", "four"] | ["one", "two", "four"] | ["one", "two", "four"] | +| data_source | data_source | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | +| data_source | data_source | {type: "data_view", "data_view_id": "A"} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | ``` ### Rule field has an update and a custom value that are NOT the same - `ABC` @@ -164,7 +179,7 @@ Examples: | number | risk_score | 1 | 2 | 3 | 2 | ``` -#### **Scenario: `ABC` - Rule field is a mergable multi line string** +#### **Scenario: `ABC` - Rule field is a mergeable multi line string** **Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithms @@ -172,7 +187,7 @@ Examples: Given field is customized by the user (current version != base version) And field is updated by Elastic in this upgrade (target version != base version) And customized field is different than the Elastic update in this upgrade (current version != target version) -And the 3-way diff of fields are determined to be mergable +And the 3-way diff of fields are determined to be mergeable Then for field the diff algorithm should output a merged version as the merged one with a solvable conflict And field should be returned from the `upgrade/_review` API endpoint And field should be shown in the upgrade preview UI @@ -182,7 +197,7 @@ Examples: | multi line string | description | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My description.\nThis is a second line, now longer." | "My GREAT description.\nThis is a second line, now longer." | ``` -#### **Scenario: `ABC` - Rule field is a non-mergable multi line string** +#### **Scenario: `ABC` - Rule field is a non-mergeable multi line string** **Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithms @@ -190,7 +205,7 @@ Examples: Given field is customized by the user (current version != base version) And field is updated by Elastic in this upgrade (target version != base version) And customized field is different than the Elastic update in this upgrade (current version != target version) -And the 3-way diff of fields are determined to be unmergable +And the 3-way diff of fields are determined to be unmergeable Then for field the diff algorithm should output the current version as the merged one with a non-solvable conflict And field should be returned from the `upgrade/_review` API endpoint And field should be shown in the upgrade preview UI @@ -225,11 +240,55 @@ Examples: | array of scalars | index | ["logs-*"] | ["logs-*", "Logs-*"] | ["logs-*", "new-*"] | ["logs-*", "Logs-*", "new-*"] | ``` +#### **Scenario: `ABC` - Rule field is a solvable `data_source` object** + +**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm + +```Gherkin +Given data_source field is customized by the user (current version != base version) +And data_source field is updated by Elastic in this upgrade (target version != base version) +And customized data_source field is different than the Elastic update in this upgrade (current version != target version) +And both current version and target version are of type "index_patterns" in data_source +Then for data_source field the diff algorithm should output a custom merged version with a solvable conflict +And arrays should be deduplicated before comparison +And arrays should be compared sensitive of case +And arrays should be compared agnostic of order +And data_source field should be returned from the `upgrade/_review` API endpoint +And data_source field should be shown in the upgrade preview UI + +Examples: +| algorithm | base_version | current_version | target_version | merged_version | +| data_source | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["two", "one", "four"]} | {type: "index_patterns", "index_patterns": ["one", "two", "five"]} | {type: "index_patterns", "index_patterns": ["one", "two", "four", "five"]} | +| data_source | {type: "data_view", "data_view_id": "A"} | {type: "index_patterns", "index_patterns": ["one", "one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "five"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three", "five"]} | +``` + +#### **Scenario: `ABC` - Rule field is a non-solvable `data_source` object** + +**Automation**: 6 integration tests with mock rules + a set of unit tests for the algorithm + +```Gherkin +Given data_source field is customized by the user (current version != base version) +And data_source field is updated by Elastic in this upgrade (target version != base version) +And customized data_source field is different than the Elastic update in this upgrade (current version != target version) +Then for data_source field the diff algorithm should output the current version as the merged version with a non-solvable conflict +And data_source field should be returned from the `upgrade/_review` API endpoint +And data_source field should be shown in the upgrade preview UI + +Examples: +| algorithm | base_version | current_version | target_version | merged_version | +| data_source | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "B"} | {type: "data_view", "data_view_id": "C"} | {type: "data_view", "data_view_id": "B"} | +| data_source | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "B"} | {type: "data_view", "data_view_id": "A"} | +| data_source | {type: "data_view", "data_view_id": "A"} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "B"} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | +| data_source | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "B"} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "B"} | +| data_source | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "four"]} | {type: "data_view", "data_view_id": "C"} | {type: "index_patterns", "index_patterns": ["one", "two", "four"]} | +| data_source | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | {type: "index_patterns", "index_patterns": ["one", "two", "five"]} | {type: "data_view", "data_view_id": "A"} | +``` + ### Rule field has an update and a custom value that are the same and the rule base version doesn't exist - `-AA` #### **Scenario: `-AA` - Rule field is any type** -**Automation**: 3 integration tests with mock rules + a set of unit tests for each algorithm +**Automation**: 5 integration tests with mock rules + a set of unit tests for each algorithm ```Gherkin Given at least 1 installed prebuilt rule has a new version available @@ -240,18 +299,20 @@ And field should not be returned from the `upgrade/_review` API end And field should not be shown in the upgrade preview UI Examples: -| algorithm | field_name | base_version | current_version | target_version | merged_version | -| single line string | name | N/A | "A" | "A" | "A" | -| multi line string | description | N/A | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." | -| number | risk_score | N/A | 1 | 1 | 1 | -| array of scalars | tags | N/A | ["one", "three", "two"] | ["three", "one", "two"] | ["one", "three", "two"] | +| algorithm | field_name | base_version | current_version | target_version | merged_version | +| single line string | name | N/A | "A" | "A" | "A" | +| multi line string | description | N/A | "My description.\nThis is a second line." | "My description.\nThis is a second line." | "My description.\nThis is a second line." | +| number | risk_score | N/A | 1 | 1 | 1 | +| array of scalars | tags | N/A | ["one", "three", "two"] | ["three", "one", "two"] | ["one", "three", "two"] | +| data_source | data_source | N/A | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | +| data_source | data_source | N/A | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | ``` -### Rule field has an update and a custom value that are NOT the same and the rule base version doesn't exist - `-BC` +### Rule field has an update and a custom value that are NOT the same and the rule base version doesn't exist - `-AB` -#### **Scenario: `-BC` - Rule field is a number or single line string** +#### **Scenario: `-AB` - Rule field is a number or single line string** -**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithms +**Automation**: 4 integration tests with mock rules + a set of unit tests for the algorithms ```Gherkin Given at least 1 installed prebuilt rule has a new version available @@ -266,15 +327,16 @@ Examples: | single line string | name | N/A | "B" | "C" | "C" | | multi line string | description | N/A | "My description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | "My GREAT description.\nThis is a second line." | | number | risk_score | N/A | 2 | 3 | 3 | +| data_source | data_source | N/A | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "B"} | {type: "data_view", "data_view_id": "B"} | ``` -#### **Scenario: `-BC` - Rule field is an array of scalar values** +#### **Scenario: `-AB` - Rule field is an array of scalar values** **Automation**: 1 integration test with mock rules + a set of unit tests for the algorithm ```Gherkin -Given field is customized by the user (current version != base version) -And field is updated by Elastic in this upgrade (target version != base version) +Given at least 1 installed prebuilt rule has a new version available +And the base version of the rule cannot be determined And customized field is different than the Elastic update in this upgrade (current version != target version) Then for field the diff algorithm should output a custom merged version with a solvable conflict And arrays should be deduplicated before comparison @@ -288,3 +350,43 @@ Examples: | algorithm | field_name | base_version | current_version | target_version | merged_version | | array of scalars | tags | N/A | ["one", "two", "four"] | ["one", "two", "five"] | ["one", "two", "four", "five"] | ``` + +#### **Scenario: `-AB` - Rule field is a solvable `data_source` object** + +**Automation**: 1 integration test with mock rules + a set of unit tests for the algorithm + +```Gherkin +Given at least 1 installed prebuilt rule has a new version available +And the base version of the rule cannot be determined +And customized data_source field is different than the Elastic update in this upgrade (current version != target version) +And current version and target version are both array fields in data_source +Then for data_source field the diff algorithm should output a custom merged version with a solvable conflict +And arrays should be deduplicated before comparison +And arrays should be compared sensitive of case +And arrays should be compared agnostic of order +And data_source field should be returned from the `upgrade/_review` API endpoint +And data_source field should be shown in the upgrade preview UI + +Examples: +| algorithm | base_version | current_version | target_version | merged_version | +| data_source | N/A | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "index_patterns", "index_patterns": ["one", "two", "four"]} | {type: "index_patterns", "index_patterns": ["one", "two", "three", "four"]} | + +``` + +#### **Scenario: `-AB` - Rule field is a non-solvable `data_source` object** + +**Automation**: 1 integration test with mock rules + a set of unit tests for the algorithm + +```Gherkin +Given at least 1 installed prebuilt rule has a new version available +And the base version of the rule cannot be determined +And customized data_source field is different than the Elastic update in this upgrade (current version != target version) +And current version and target version are not both array fields in data_source +Then for data_source field the diff algorithm should output the target version as the merged version with a solvable conflict +And data_source field should be returned from the `upgrade/_review` API endpoint +And data_source field should be shown in the upgrade preview UI + +Examples: +| algorithm | base_version | current_version | target_version | merged_version | +| data_source | N/A | {type: "index_patterns", "index_patterns": ["one", "two", "three"]} | {type: "data_view", "data_view_id": "A"} | {type: "data_view", "data_view_id": "A"} | +```