Skip to content

Commit

Permalink
[Security Solution] Simple diff algorithm test plan (#184484)
Browse files Browse the repository at this point in the history 
## Summary

Adds test plan in accordance to
#180158

Adds test cases for simple diff algorithm to prebuilt rules'
`Installation and upgrade` test plan

---------

Co-authored-by: Kibana Machine <[email protected]>
  • Loading branch information
@dplumlee @kibanamachine
dplumlee and kibanamachine committed Jun 10, 2024
1 parent 2d3dc3f commit d014908
Showing 1 changed file with 146 additions and 0 deletions.
146 changes: 146 additions & 0 deletions ...esting/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,14 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule
- [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading)
- [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading)
- [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination)
- [Rule upgrade workflow: simple diff algorithm](#rule-upgrade-workflow-simple-diff-algorithm)
- [**Scenario: Rule field doesn't have an update and has no custom value**](#scenario-rule-field-doesnt-have-an-update-and-has-no-custom-value)
- [**Scenario: Rule field doesn't have an update but has a custom value**](#scenario-rule-field-doesnt-have-an-update-but-has-a-custom-value)
- [**Scenario: Rule field has an update and doesn't have a custom value**](#scenario-rule-field-has-an-update-and-doesnt-have-a-custom-value)
- [**Scenario: Rule field has an update and a custom value that are the same**](#scenario-rule-field-has-an-update-and-a-custom-value-that-are-the-same)
- [**Scenario: Rule field has an update and a custom value that are NOT the same**](#scenario-rule-field-has-an-update-and-a-custom-value-that-are-not-the-same)
- [**Scenario: Rule field has an update and a custom value that are NOT the same and the rule base version doesn't exist**](#scenario-rule-field-has-an-update-and-a-custom-value-that-are-not-the-same-and-the-rule-base-version-doesnt-exist)
- [**Scenario: Rule field has an update and a custom value that are the same and the rule base version doesn't exist**](#scenario-rule-field-has-an-update-and-a-custom-value-that-are-the-same-and-the-rule-base-version-doesnt-exist)
- [Rule upgrade workflow: viewing rule changes in JSON diff view](#rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view)
- [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view)
- [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade)
Expand Down Expand Up @@ -815,6 +823,144 @@ And the Investigation Guide tab should NOT be displayed

TODO: add scenarios https://github.com/elastic/kibana/issues/166215

### Rule upgrade workflow: simple diff algorithm

#### **Scenario: Rule field doesn't have an update and has no custom value**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is not customized by the user (current version == base version)
And <field_name> field is not updated by Elastic in this upgrade (target version == base version)
Then for <field_name> field the diff algorithm should output the current version as the merged one without a conflict
And <field_name> field should not be returned from the `upgrade/_review` API endpoint
And <field_name> field should not be shown in the upgrade preview UI
Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "A" | "A" |
| risk_score | 1 | 1 | 1 |
```

#### **Scenario: Rule field doesn't have an update but has a custom value**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is customized by the user (current version != base version)
And <field_name> field is not updated by Elastic in this upgrade (target version == base version)
Then for <field_name> field the diff algorithm should output the current version as the merged one without a conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI
Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "B" | "A" |
| risk_score | 1 | 2 | 1 |
```

#### **Scenario: Rule field has an update and doesn't have a custom value**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is not customized by the user (current version == base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
Then for <field_name> field the diff algorithm should output the target version as the merged one without a conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI
Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "A" | "B" |
| risk_score | 1 | 1 | 2 |
```

#### **Scenario: Rule field has an update and a custom value that are the same**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is customized by the user (current version != base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
And customized <field_name> field is the same as the Elastic update in this upgrade (current version == target version)
Then for <field_name> field the diff algorithm should output the current version as the merged one without a conflict
And <field_name> field should not be returned from the `upgrade/_review` API endpoint
And <field_name> field should not be shown in the upgrade preview UI
Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "B" | "B" |
| risk_score | 1 | 2 | 2 |
```

#### **Scenario: Rule field has an update and a custom value that are NOT the same**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is customized by the user (current version != base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
And customized <field_name> field is different than the Elastic update in this upgrade (current version != target version)
Then for <field_name> field the diff algorithm should output the current version as the merged one with a conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI
Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "B" | "C" |
| risk_score | 1 | 2 | 3 |
```

#### **Scenario: Rule field has an update and a custom value that are NOT the same and the rule base version doesn't exist**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And the base version of the rule cannot be determined
And customized <field_name> field is different than the Elastic update in this upgrade (current version != target version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI
Examples:
| field_name | base_version | current_version | target_version |
| name | N/A | "B" | "C" |
| risk_score | N/A | 2 | 3 |
```

#### **Scenario: Rule field has an update and a custom value that are the same and the rule base version doesn't exist**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And the base version of the rule cannot be determined
And customized <field_name> field is the same as the Elastic update in this upgrade (current version == target version)
Then for <field_name> field the diff algorithm should output the current version as the merged one without a conflict
And <field_name> field should not be returned from the `upgrade/_review` API endpoint
And <field_name> field should not be shown in the upgrade preview UI
Examples:
| field_name | base_version | current_version | target_version |
| name | N/A | "A" | "A" |
| risk_score | N/A | 1 | 1 |
```

### Rule upgrade workflow: viewing rule changes in JSON diff view

#### **Scenario: User can see changes in a side-by-side JSON diff view**
Expand Down

0 comments on commit d014908

Please sign in to comment.