Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Simple diff algorithm test plan #184484

Merged
merged 10 commits into from
Jun 10, 2024
Merged

[Security Solution] Simple diff algorithm test plan #184484

Changes from 7 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
765624f
adds initial cases
dplumlee May 29, 2024
4a2b9df
initial test plan
dplumlee May 30, 2024
48f1a42
adds more specific examples
dplumlee May 30, 2024
5dfb07b
addresses comments
dplumlee May 31, 2024
87f239a
Merge remote-tracking branch 'upstream/main' into single-line-field-d…
dplumlee May 31, 2024
6fd0820
addresses comments
dplumlee Jun 4, 2024
90c5bac
Merge remote-tracking branch 'upstream/main' into single-line-field-d…
dplumlee Jun 4, 2024
023c381
Merge remote-tracking branch 'upstream/main' into single-line-field-d…
dplumlee Jun 7, 2024
4c9137f
addresses comments
dplumlee Jun 7, 2024
b4ed0d0
Merge branch 'main' into single-line-field-diff-test-plan
kibanamachine Jun 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions ...esting/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,12 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule
- [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading)
- [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading)
- [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination)
- [Rule upgrade workflow: simple diff algorithm](#rule-upgrade-workflow-simple-diff-algorithm)
- [**Scenario: Rule field doesn't have an update and has no custom value**](#scenario-rule-field-doesnt-have-an-update-and-has-no-custom-value)
- [**Scenario: Rule field doesn't have an update but has a custom value**](#scenario-rule-field-doesnt-have-an-update-but-has-a-custom-value)
- [**Scenario: Rule field has an update and doesn't have a custom value**](#scenario-rule-field-has-an-update-and-doesnt-have-a-custom-value)
- [**Scenario: Rule field has an update and a custom value that are the same**](#scenario-rule-field-has-an-update-and-a-custom-value-that-are-the-same)
- [**Scenario: Rule field has an update and a custom value that are NOT the same**](#scenario-rule-field-has-an-update-and-a-custom-value-that-are-not-the-same)
dplumlee marked this conversation as resolved.
Show resolved Hide resolved
- [Rule upgrade workflow: viewing rule changes in JSON diff view](#rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view)
- [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view)
- [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade)
Expand Down Expand Up @@ -815,6 +821,143 @@ And the Investigation Guide tab should NOT be displayed

TODO: add scenarios https://github.com/elastic/kibana/issues/166215

### Rule upgrade workflow: simple diff algorithm

#### **Scenario: Rule field doesn't have an update and has no custom value**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is not customized by the user (current version == base version)
And <field_name> field is not updated by Elastic in this upgrade (target version == base version)
Then for <field_name> field the diff algorithm should output the current version as the merged one without a conflict
And <field_name> field should not be returned from the `upgrade/_review` API endpoint
And <field_name> field should not be shown in the upgrade preview UI

Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "A" | "A" |
| risk_score | 1 | 1 | 1 |
```

#### **Scenario: Rule field doesn't have an update but has a custom value**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is customized by the user (current version != base version)
And <field_name> field is not updated by Elastic in this upgrade (target version == base version)
Then for <field_name> field the diff algorithm should output the current version as the merged one without a conflict
And <field_name> field should not be returned from the `upgrade/_review` API endpoint
And <field_name> field should not be shown in the upgrade preview UI
banderror marked this conversation as resolved.
Show resolved Hide resolved

Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "B" | "A" |
| risk_score | 1 | 2 | 1 |
```

#### **Scenario: Rule field has an update and doesn't have a custom value**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is customized by the user (current version == base version)
And <field_name> field is not updated by Elastic in this upgrade (target version != base version)
dplumlee marked this conversation as resolved.
Show resolved Hide resolved
Then for <field_name> field the diff algorithm should output the target version as the merged one without a conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI

Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "A" | "B" |
| risk_score | 1 | 1 | 2 |
```

#### **Scenario: Rule field has an update and a custom value that are the same**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is customized by the user (current version != base version)
And <field_name> field is updated by Elastic in this upgrade (target version != base version)
And customized <field_name> field is the same as the Elastic update in this upgrade (current version == target version)
Then for <field_name> field the diff algorithm should output the current version as the merged one without a conflict
And <field_name> field should not be returned from the `upgrade/_review` API endpoint
And <field_name> field should not be shown in the upgrade preview UI

Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "B" | "B" |
| risk_score | 1 | 2 | 2 |
Comment on lines +896 to +902
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Just want to point out that this case is similar to the one described in #180154, and we might want to revisit it later. But for now let's keep it as is.

```

#### **Scenario: Rule field has an update and a custom value that are NOT the same**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And <field_name> field is customized by the user (current version != base version)
And <field_name> field is not updated by Elastic in this upgrade (target version != base version)
And customized <field_name> field is different than the Elastic update in this upgrade (current version != target version)
dplumlee marked this conversation as resolved.
Show resolved Hide resolved
Then for <field_name> field the diff algorithm should output the current version as the merged one with a conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI

Examples:
| field_name | base_version | current_version | target_version |
| name | "A" | "B" | "C" |
| risk_score | 1 | 2 | 3 |
```

#### **Scenario: Rule field has an update and a custom value that are NOT the same and the rule base version doesn't exist**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
And the base version of the rule cannot be determined
And customized <field_name> field is different than the Elastic update in this upgrade (current version != target version)
Then for <field_name> field the diff algorithm should output the target version as the merged one with a conflict
And <field_name> field should be returned from the `upgrade/_review` API endpoint
And <field_name> field should be shown in the upgrade preview UI

Examples:
| field_name | base_version | current_version | target_version |
| name | N/A | "B" | "C" |
| risk_score | N/A | 2 | 3 |
```

#### **Scenario: Rule field has an update and a custom value that are the same and the rule base version doesn't exist**

**Automation**: 2 integration tests with mock rules + a set of unit tests for the algorithm

```Gherkin
Given at least 1 prebuilt rule is installed in Kibana
And for this rule there is a new version available
dplumlee marked this conversation as resolved.
Show resolved Hide resolved
And customized <field_name> field is the same as the Elastic update in this upgrade (current version == target version)
Then for <field_name> field the diff algorithm should output the current version as the merged one without a conflict
And <field_name> field should not be returned from the `upgrade/_review` API endpoint
And <field_name> field should not be shown in the upgrade preview UI


Examples:
| field_name | base_version | current_version | target_version |
| name | N/A | "A" | "A" |
| risk_score | N/A | 1 | 1 |
```

### Rule upgrade workflow: viewing rule changes in JSON diff view

#### **Scenario: User can see changes in a side-by-side JSON diff view**
Expand Down