From 11550805dae4fc8958a954bbed5513a318e454e5 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Fri, 6 Dec 2024 16:23:00 -0300 Subject: [PATCH 01/18] Initial draft of test plan --- .../prebuilt_rules/installation.md | 559 ++++++++++++ .../package_installation_and_upgrade.md | 165 ++++ ...installation_and_upgrade.md => upgrade.md} | 804 ++++++------------ 3 files changed, 1007 insertions(+), 521 deletions(-) create mode 100644 x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md create mode 100644 x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md rename x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/{installation_and_upgrade.md => upgrade.md} (55%) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md new file mode 100644 index 0000000000000..dac2a0e8f1ac6 --- /dev/null +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md @@ -0,0 +1,559 @@ +# Installation of Prebuilt Rules + +This is a test plan for the workflows of installing prebuilt rules. + +Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). + +## Table of Contents + +- [Useful information](#useful-information) + - [Tickets](#tickets) + - [Terminology](#terminology) + - [Assumptions](#assumptions) + - [Non-functional requirements](#non-functional-requirements) + - [Functional requirements](#functional-requirements) +- [Scenarios](#scenarios) + - [Rule installation notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) + - [**Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets**](#scenario-user-is-not-notified-when-no-prebuilt-rules-are-installed-and-there-are-no-prebuilt-rules-assets) + - [**Scenario: User is NOT notified when all prebuilt rules are installed and up to date**](#scenario-user-is-not-notified-when-all-prebuilt-rules-are-installed-and-up-to-date) + - [**Scenario: User is notified when no prebuilt rules are installed and there are rules available to install**](#scenario-user-is-notified-when-no-prebuilt-rules-are-installed-and-there-are-rules-available-to-install) + - [**Scenario: User is notified when some prebuilt rules can be installed**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-installed) + - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) + - [**Scenario: User is notified after a prebuilt rule gets deleted**](#scenario-user-is-notified-after-a-prebuilt-rule-gets-deleted) + - [Rule installation workflow: base cases](#rule-installation-workflow-base-cases) + - [**Scenario: User can install prebuilt rules one by one**](#scenario-user-can-install-prebuilt-rules-one-by-one) + - [**Scenario: User can install multiple prebuilt rules selected on the page**](#scenario-user-can-install-multiple-prebuilt-rules-selected-on-the-page) + - [**Scenario: User can install all available prebuilt rules at once**](#scenario-user-can-install-all-available-prebuilt-rules-at-once) + - [**Scenario: Empty screen is shown when all prebuilt rules are installed**](#scenario-empty-screen-is-shown-when-all-prebuilt-rules-are-installed) + - [**Scenario: User can preview rules available for installation**](#scenario-user-can-preview-rules-available-for-installation) + - [**Scenario: User can install a rule using the rule preview**](#scenario-user-can-install-a-rule-using-the-rule-preview) + - [**Scenario: User can see correct rule information in preview before installing**](#scenario-user-can-see-correct-rule-information-in-preview-before-installing) + - [**Scenario: Tabs and sections without content should be hidden in preview before installing**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-installing) + - [Rule installation workflow: filtering, sorting, pagination](#rule-installation-workflow-filtering-sorting-pagination) + - [Rule installation workflow: misc cases](#rule-installation-workflow-misc-cases) + - [**Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed**](#scenario-user-opening-the-add-rules-page-sees-a-loading-skeleton-until-the-package-installation-is-completed) + - [**Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs**](#scenario-user-can-navigate-from-the-add-rules-page-to-the-rule-management-page-via-breadcrumbs) + - [Rule installation and upgrade via the Prebuilt rules API](#rule-installation-and-upgrade-via-the-prebuilt-rules-api) + - [**Scenario: API can install all prebuilt rules**](#scenario-api-can-install-all-prebuilt-rules) + - [**Scenario: API can install prebuilt rules that are not yet installed**](#scenario-api-can-install-prebuilt-rules-that-are-not-yet-installed) + - [**Scenario: API does not install prebuilt rules if they are up to date**](#scenario-api-does-not-installupgrade-prebuilt-rules-if-they-are-up-to-date) + - [Error handling](#error-handling) + - [**Scenario: Error is handled when any operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-operation-on-prebuilt-rules-fails) + - [Authorization / RBAC](#authorization--rbac) + - [**Scenario: User with read privileges on Security Solution cannot install prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-install-prebuilt-rules) + +## Useful information + +### Tickets + +- [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) + +**Milestone 3 - Prebuilt Rules Customization:** +- [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) +- [Tests for prebuilt rule upgrade workflow #202078](https://github.com/elastic/kibana/issues/202078) + +**Milestone 2:** +- [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) +- [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) + +### Terminology + +- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**. + +- **Package**: `security_detection_engine` Fleet package that we use to distribute prebuilt detection rules in the form of `security-rule` assets (saved objects). + +- **Real package**: actual latest stable package distributed and pulled from EPR via Fleet. + +- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly in the test setup, either by using the ES client _in integration tests_ or by an API request _in Cypress tests_. + +- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. + +- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. + - CTA to install prebuilt rules - at this moment, it's a link button with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. + - CTA to upgrade prebuilt rules - at this moment, it's a tab with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. + +### Assumptions + +- Below scenarios only apply to prebuilt detection rules. +- Users should be able to install prebuilt rules on the `Basic` license and higher. +- EPR is available for fetching the package unless explicitly indicated otherwise. +- Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored. + +### Non-functional requirements + +- Notifications, rule installation workflows should work: + - regardless of the package type: with historical rule versions or without; + - regardless of the package registry availability: i.e., they should also work in air-gapped environments. +- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet. +- Kibana should not crash with Out Of Memory exception during package installation. +- For test purposes, it should be possible to use detection rules package versions lower than the latest. + +### Functional requirements + +- User should be able to install prebuilt rules with and without previewing what exactly they would install (rule properties). +- If user chooses to preview a prebuilt rule to be installed/upgraded, we currently show this preview in a flyout. +- In the prebuilt rule preview a tab that doesn't have any sections should not be displayed and a section that doesn't have any properties also should not be displayed. + +Examples of rule properties we show in the prebuilt rule preview flyout: + +```Gherkin +Examples: +| rule_type | property | tab | section | +│ All rule types │ Author │ Overview │ About │ +│ All rule types │ Building block │ Overview │ About │ +│ All rule types │ Severity │ Overview │ About │ +│ All rule types │ Severity override │ Overview │ About │ +│ All rule types │ Risk score │ Overview │ About │ +│ All rule types │ Risk score override │ Overview │ About │ +│ All rule types │ Reference URLs │ Overview │ About │ +│ All rule types │ False positive examples │ Overview │ About │ +│ All rule types │ Custom highlighted fields │ Overview │ About │ +│ All rule types │ License │ Overview │ About │ +│ All rule types │ Rule name override │ Overview │ About │ +│ All rule types │ MITRE ATT&CK™ │ Overview │ About │ +│ All rule types │ Timestamp override │ Overview │ About │ +│ All rule types │ Tags │ Overview │ About │ +│ All rule types │ Type │ Overview │ Definition │ +│ All rule types │ Related integrations │ Overview │ Definition │ +│ All rule types │ Required fields │ Overview │ Definition │ +│ All rule types │ Timeline template │ Overview │ Definition │ +│ All rule types │ Runs every │ Overview │ Schedule │ +│ All rule types │ Additional look-back time │ Overview │ Schedule │ +│ All rule types │ Setup guide │ Overview │ Setup guide │ +│ All rule types │ Investigation guide │ Investigation guide │ Investigation guide │ +│ Custom Query │ Index patterns │ Overview │ Definition │ +│ Custom Query │ Data view ID │ Overview │ Definition │ +│ Custom Query │ Data view index pattern │ Overview │ Definition │ +│ Custom Query │ Custom query │ Overview │ Definition │ +│ Custom Query │ Filters │ Overview │ Definition │ +│ Custom Query │ Saved query name │ Overview │ Definition │ +│ Custom Query │ Saved query filters │ Overview │ Definition │ +│ Custom Query │ Saved query │ Overview │ Definition │ +│ Custom Query │ Suppress alerts by │ Overview │ Definition │ +│ Custom Query │ Suppress alerts for │ Overview │ Definition │ +│ Custom Query │ If a suppression field is missing │ Overview │ Definition │ +│ Machine Learning │ Anomaly score threshold │ Overview │ Definition │ +│ Machine Learning │ Machine Learning job │ Overview │ Definition │ +│ Threshold │ Threshold │ Overview │ Definition │ +│ Threshold │ Index patterns │ Overview │ Definition │ +│ Threshold │ Data view ID │ Overview │ Definition │ +│ Threshold │ Data view index pattern │ Overview │ Definition │ +│ Threshold │ Custom query │ Overview │ Definition │ +│ Threshold │ Filters │ Overview │ Definition │ +│ Event Correlation │ EQL query │ Overview │ Definition │ +│ Event Correlation │ Filters │ Overview │ Definition │ +│ Event Correlation │ Index patterns │ Overview │ Definition │ +│ Event Correlation │ Data view ID │ Overview │ Definition │ +│ Event Correlation │ Data view index pattern │ Overview │ Definition │ +│ Indicator Match │ Indicator index patterns │ Overview │ Definition │ +│ Indicator Match │ Indicator mapping │ Overview │ Definition │ +│ Indicator Match │ Indicator filters │ Overview │ Definition │ +│ Indicator Match │ Indicator index query │ Overview │ Definition │ +│ Indicator Match │ Index patterns │ Overview │ Definition │ +│ Indicator Match │ Data view ID │ Overview │ Definition │ +│ Indicator Match │ Data view index pattern │ Overview │ Definition │ +│ Indicator Match │ Custom query │ Overview │ Definition │ +│ Indicator Match │ Filters │ Overview │ Definition │ +│ New Terms │ Fields │ Overview │ Definition │ +│ New Terms │ History Window Size │ Overview │ Definition │ +│ New Terms │ Index patterns │ Overview │ Definition │ +│ New Terms │ Data view ID │ Overview │ Definition │ +│ New Terms │ Data view index pattern │ Overview │ Definition │ +│ New Terms │ Custom query │ Overview │ Definition │ +│ New Terms │ Filters │ Overview │ Definition │ +│ ESQL │ ESQL query │ Overview │ Definition │ +│ ESQL │ Suppress alerts by │ Overview │ Definition │ +│ ESQL │ Suppress alerts for │ Overview │ Definition │ +│ ESQL │ If a suppression field is missing │ Overview │ Definition │ +``` + +## Scenarios + +### Rule installation notifications on the Rule Management page + +#### **Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given no prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +When user opens the Rule Management page +Then user should NOT see a CTA to install prebuilt rules +And user should NOT see a number of rules available to install +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table +``` + +#### **Scenario: User is NOT notified when all prebuilt rules are installed and up to date** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given the latest prebuilt rule assets exist in Kibana +And all the latest prebuilt rules from those rule assets are installed +When user opens the Rule Management page +Then user should NOT see a CTA to install prebuilt rules +And user should NOT see a number of rules available to install +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table +``` + +#### **Scenario: User is notified when no prebuilt rules are installed and there are rules available to install** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +And there are X prebuilt rules available to install +When user opens the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see a number of rules available to install (X) +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table +``` + +#### **Scenario: User is notified when some prebuilt rules can be installed** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given Y prebuilt rule assets exist in Kibana +And X (where X < Y) prebuilt rules are installed +And there are Y more prebuilt rules available to install +And for all X installed rules there are no new versions available +When user opens the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install (Y) +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table +``` + +#### **Scenario: User is notified when both rules to install and upgrade are available** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given Y prebuilt rule assets exist in Kibana +And X (where X < Y) prebuilt rules are installed +And Z (where Z < X) installed rules have matching prebuilt rule assets with higher version available +And for Z of the installed rules there are new versions available +When user opens the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install (Y) +And user should see a CTA to upgrade prebuilt rules +And user should see the number of rules available to upgrade (Z) +``` + +#### **Scenario: User is notified after a prebuilt rule gets deleted** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And there are no more prebuilt rules available to install +When user opens the Rule Management page +And user deletes Y prebuilt rules +Then user should see a CTA to install prebuilt rules +And user should see rules available to install +``` + +### Rule installation workflow: base cases + +#### **Scenario: User can install prebuilt rules one by one** + +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. + +```Gherkin +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +When user opens the Add Rules page +Then prebuilt rules available for installation should be displayed in the table +When user installs one individual rule without previewing it +Then success message should be displayed after installation +And the installed rule should be removed from the table +When user navigates back to the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install decreased by 1 +``` + +#### **Scenario: User can install multiple prebuilt rules selected on the page** + +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. + +```Gherkin +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +When user opens the Add Rules page +Then prebuilt rules available for installation should be displayed in the table +When user selects rules +Then user should see a CTA to install number of rules +When user clicks the CTA +Then success message should be displayed after installation +And all the installed rules should be removed from the table +When user navigates back to the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install decreased by number of installed rules + +Examples: + | Y | + | a few rules on the page, e.g. 2 | + | all rules on the page, e.g. 12 | +``` + +#### **Scenario: User can install all available prebuilt rules at once** + +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. + +```Gherkin +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +When user opens the Add Rules page +Then prebuilt rules available for installation should be displayed in the table +When user installs all rules +Then success message should be displayed after installation +And all the rules should be removed from the table +And user should see a message indicating that all available rules have been installed +And user should see a CTA that leads to the Rule Management page +When user clicks on the CTA +Then user should be navigated back to Rule Management page +And user should NOT see a CTA to install prebuilt rules +And user should NOT see a number of rules available to install +``` + +#### **Scenario: Empty screen is shown when all prebuilt rules are installed** + +**Automation**: 1 e2e test with mock rules + 1 integration test. + +```Gherkin +Given all the available prebuilt rules are installed in Kibana +When user opens the Add Rules page +Then user should see a message indicating that all available rules have been installed +And user should see a CTA that leads to the Rule Management page +``` + +#### **Scenario: User can preview rules available for installation** + +**Automation**: 1 e2e test + +```Gherkin +Given 2 prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +When user opens the Add Rules page +Then all rules available for installation should be displayed in the table +When user opens the rule preview for the 1st rule +Then the preview should open +When user closes the preview +Then it should disappear +``` + +#### **Scenario: User can install a rule using the rule preview** + +**Automation**: 1 e2e test + +```Gherkin +Given 2 prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +When user opens the Add Rules page +Then all rules available for installation should be displayed in the table +When user opens the rule preview for the rule +Then the preview should open +When user installs the rule using a CTA in the rule preview +Then the rule should be installed +And a success message should be displayed after installation +And the rule should be removed from the Add Rules table +When user navigates back to the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install as initial number minus 1 +``` + +#### **Scenario: User can see correct rule information in preview before installing** + +**Automation**: 1 e2e test + +```Gherkin +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +When user opens the Add Rules page +Then all X rules available for installation should be displayed in the table +When user opens a rule preview for any rule +Then the preview should appear +And all properties of a rule should be displayed in the correct tab and section of the preview (see examples of rule properties above) +``` + +#### **Scenario: Tabs and sections without content should be hidden in preview before installing** + +**Automation**: 1 e2e test + +```Gherkin +Given 1 prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +And this rule has neither Setup guide nor Investigation guide +When user opens the Add Rules page +Then all rules available for installation should be displayed in the table +When user opens the rule preview for this rule +Then the preview should open +And the Setup Guide section should NOT be displayed in the Overview tab +And the Investigation Guide tab should NOT be displayed +``` + +### Rule installation workflow: filtering, sorting, pagination + +TODO: add scenarios https://github.com/elastic/kibana/issues/166215 + +### Rule installation workflow: misc cases + +#### **Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed** + +**Automation**: unit tests. + +```Gherkin +Given prebuilt rules package is not installed +When user opens the Add Rules page +Then user should see a loading skeleton until the package installation is completed +``` + +#### **Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs** + +**Automation**: 1 e2e test. + +```Gherkin +Given user is on the Add Rules page +When user navigates to the Rule Management page via breadcrumbs +Then the Rule Management page should be displayed +``` + +### Rule installation via the Prebuilt rules API + +There's a legacy prebuilt rules API and a new one. Both should be tested against two types of the package: with and without historical rule versions. + +#### **Scenario: API can install all prebuilt rules** + +**Automation**: 8 integration tests with mock rules: 4 examples below \* 2 (we split checking API response and installed rules into two different tests). + +```Gherkin +Given the package is installed +And the package contains N rules +When user installs all rules via install +Then the endpoint should return success response (HTTP 200 code) with +And N rule objects should be created +And each rule object should have correct id and version + +Examples: + | package_type | api | install_response | + | with historical versions | legacy | installed: N, updated: 0 | + | w/o historical versions | legacy | installed: N, updated: 0 | + | with historical versions | new | total: N, succeeded: N | + | w/o historical versions | new | total: N, succeeded: N | +``` + +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + +#### **Scenario: API can install prebuilt rules that are not yet installed** + +**Automation**: 4 integration tests with mock rules. + +```Gherkin +Given the package is installed +And the package contains N rules +When user installs all rules via install +And deletes one of the installed rules +And gets prebuilt rules status via status +Then the endpoint should return successful response (HTTP 200 code) with +When user installs all rules via install again +Then the endpoint should return successful response (HTTP 200 code) with + +Examples: + | package_type | api | status_response | install_response | + | with historical versions | legacy | not_installed: 1 | installed: 1, updated: 0 | + | w/o historical versions | legacy | not_installed: 1 | installed: 1, updated: 0 | + | with historical versions | new | to_install: 1 | total: 1, succeeded: 1 | + | w/o historical versions | new | to_install: 1 | total: 1, succeeded: 1 | +``` + +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` + - status: `GET /api/detection_engine/rules/prepackaged/_status` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + - status: `GET /internal/detection_engine/prebuilt_rules/status` + + +#### **Scenario: API does not install prebuilt rules if they are up to date** + +**Automation**: 4 integration tests with mock rules. + +```Gherkin +Given the package is installed +And the package contains N rules +When user installs all rules via install +And user gets prebuilt rules status via status +Then the endpoint should return successful response (HTTP 200 code) with +When user calls install +Then the endpoint should return successful response (HTTP 200 code) with +When user calls upgrade +Then the endpoint should return successful response (HTTP 200 code) with + +Examples: + | package_type | api | status_response | install_response | upgrade_response | + | with historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | + | w/o historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | + | with historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | + | w/o historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | +``` + +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` + - upgrade: `PUT /api/detection_engine/rules/prepackaged` + - status: `GET /api/detection_engine/rules/prepackaged/_status` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` + - status: `GET /internal/detection_engine/prebuilt_rules/status` + +### Error handling + +#### **Scenario: Error is handled when any installation operation on prebuilt rules fails** + +**Automation**: e2e test with mock rules + +```Gherkin +When user is prebuilt rules +And this operation fails +Then user should see an error message + +Examples: + | operation | + | installing all | + | installing selected | + | installing individual | +``` + +### Authorization / RBAC + +#### **Scenario: User with read privileges on Security Solution cannot install prebuilt rules** + +**Automation**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and installation endpoints. + +```Gherkin +Given user with "Security: read" privileges on Security Solution +And prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +When user opens the Add Rules page +Then user should see prebuilt rules available to install +But user should not be able to install them +``` \ No newline at end of file diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md new file mode 100644 index 0000000000000..57e16684facca --- /dev/null +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md @@ -0,0 +1,165 @@ +# Prebuilt Rules Package Installation and Upgrade + +This is a test plan for the workflows of installing and updating the Prebuilt Rules package. + +> For the test plans for installing and upgrading prebuilt rules, see [Installation of Prebuilt Rules](./installation.md) and [Upgrade of Prebuilt Rules](./upgrade.md). + +Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). + +## Table of Contents + +- [Useful information](#useful-information) + - [Tickets](#tickets) + - [Terminology](#terminology) + - [Assumptions](#assumptions) + - [Non-functional requirements](#non-functional-requirements) + - [Functional requirements](#functional-requirements) +- [Scenarios](#scenarios) + - [Package installation](#package-installation) + - [**Scenario: Package is installed via Fleet**](#scenario-package-is-installed-via-fleet) + - [**Scenario: Package is installed via bundled Fleet package in Kibana**](#scenario-package-is-installed-via-bundled-fleet-package-in-kibana) + - [**Scenario: Large package can be installed on a small Kibana instance**](#scenario-large-package-can-be-installed-on-a-small-kibana-instance) + - [Scenarios for the real package](#scenarios-for-the-real-package) + - [**Scenario: User can install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package**](#scenario-user-can-install-prebuilt-rules-from-scratch-then-install-new-rules-and-upgrade-existing-rules-from-the-new-package) + - [Kibana upgrade](#kibana-upgrade) + - [**Scenario: User can use prebuilt rules after upgrading Kibana from version A to B**](#scenario-user-can-use-prebuilt-rules-after-upgrading-kibana-from-version-a-to-b) + +## Useful information + +### Tickets + +- [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) + +**Milestone 3 - Prebuilt Rules Customization:** +- [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) +- [Tests for prebuilt rule upgrade workflow #202078](https://github.com/elastic/kibana/issues/202078) + +**Milestone 2:** +- [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) +- [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) + +### Terminology + +- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**. + +- **Package**: `security_detection_engine` Fleet package that we use to distribute prebuilt detection rules in the form of `security-rule` assets (saved objects). + +- **Real package**: actual latest stable package distributed and pulled from EPR via Fleet. + +- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly in the test setup, either by using the ES client _in integration tests_ or by an API request _in Cypress tests_. + +- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. + +- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. + - CTA to install prebuilt rules - at this moment, it's a link button with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. + - CTA to upgrade prebuilt rules - at this moment, it's a tab with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. + +### Assumptions + +- Below scenarios only apply to prebuilt detection rules. +- Users should be able to install and upgrade prebuilt rules on the `Basic` license and higher. +- EPR is available for fetching the package unless explicitly indicated otherwise. +- Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored. + +### Non-functional requirements + +- Package installation, rule installation and rule upgrade workflows should work: + - regardless of the package type: with historical rule versions or without; + - regardless of the package registry availability: i.e., they should also work in air-gapped environments. +- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet. +- Kibana should not crash with Out Of Memory exception during package installation. +- For test purposes, it should be possible to use detection rules package versions lower than the latest. + +### Functional requirements + +- User should be able to install prebuilt rules with and without previewing what exactly they would install (rule properties). +- User should be able to upgrade prebuilt rules with and without previewing what updates they would apply (rule properties of target rule versions). + +## Scenarios + +### Package installation + +#### **Scenario: Package is installed via Fleet** + +**Automation**: 2 e2e tests that install the real package. + +```Gherkin +Given the prebuilt rules package is not installed +When user opens any Security Solution page +Then the package gets installed in the background from EPR +``` + +#### **Scenario: Package is installed via bundled Fleet package in Kibana** + +**Automation**: 2 integration tests. + +```Gherkin +Given the package is not installed +And user is in an air-gapped environment +When user opens any Security Solution page +Then the package gets installed in the background from packages bundled into Kibana +``` + +#### **Scenario: Large package can be installed on a memory restricted Kibana instance** + +**Automation**: 1 integration test. + +```Gherkin +Given the package is not installed +And the package contains the largest amount of historical rule versions (15000) +And the Kibana instance has a memory heap size of 700 Mb (see note below) +When user opens any Security Solution page +Then the package is installed without Kibana crashing with an Out Of Memory error +``` + +**Note**: 600 Mb seems to always crash Kibana with an OOM error. 700 Mb runs with no issues in the Flaky test runner with 100 iterations: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2215. + +### Scenarios for the real package + +#### **Scenario: User can install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package** + +**Automation**: 1 integration test with real packages. + +```Gherkin +Given there are two package versions: A and B where A < B +And the package of A version is installed +When user calls the status endpoint +Then it should return a 200 response with some number of rules to install and 0 rules to upgrade +When user calls the installation/_review endpoint +Then it should return a 200 response matching the response of the status endpoint +When user calls the installation/_perform_ endpoint +Then it should return a 200 response matching the response of the status endpoint +And rules returned in this response should exist as alert saved objects +When user installs version B of the package +Then it should be installed successfully +When user calls the status endpoint +Then it should return a 200 response with some number of new rules to install and some number of rules to upgrade +When user calls the installation/_review endpoint +Then it should return a 200 response matching the response of the status endpoint +When user calls the installation/_perform_ endpoint +Then rules returned in this response should exist as alert saved objects +When user calls the upgrade/_review endpoint +Then it should return a 200 response matching the response of the status endpoint +When user calls the upgrade/_perform_ endpoint +Then rules returned in this response should exist as alert saved objects +``` + +### Kibana upgrade + +#### **Scenario: User can use prebuilt rules after upgrading Kibana from version A to B** + +**Automation**: not automated, manual testing required. + +```Gherkin +Given user is upgrading Kibana from version to version +And the version Kibana instance has already installed prebuilt rules +When the Kibana upgrade is complete +Then user should be able to install new prebuilt rules +And delete installed prebuilt rules +And upgrade installed prebuilt rules that have newer versions in Kibana version + +Examples: + | A | B | + | 8.7 | 8.9.0 | + | 7.17.x | 8.9.0 | +``` diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md similarity index 55% rename from x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md rename to x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index e5479ec502865..20520ae1cab0d 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -1,8 +1,8 @@ -# Installation and Upgrade of Prebuilt Rules +# Upgrade of Prebuilt Rules -This is a test plan for the workflows of installing and upgrading prebuilt rules. +This is a test plan for the workflow of upgrading prebuilt rules. -Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic. It does not cover any past functionality that was removed or functionality to be implemented in the future. The plan is about to change in the future Milestones. +Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). ## Table of Contents @@ -12,84 +12,73 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule - [Assumptions](#assumptions) - [Non-functional requirements](#non-functional-requirements) - [Functional requirements](#functional-requirements) -- [Scenarios](#scenarios) - - - [Package installation](#package-installation) - - [**Scenario: Package is installed via Fleet**](#scenario-package-is-installed-via-fleet) - - [**Scenario: Package is installed via bundled Fleet package in Kibana**](#scenario-package-is-installed-via-bundled-fleet-package-in-kibana) - - [**Scenario: Large package can be installed on a small Kibana instance**](#scenario-large-package-can-be-installed-on-a-small-kibana-instance) - - [Rule installation and upgrade via the Prebuilt rules API](#rule-installation-and-upgrade-via-the-prebuilt-rules-api) - - [**Scenario: API can install all prebuilt rules**](#scenario-api-can-install-all-prebuilt-rules) - - [**Scenario: API can install prebuilt rules that are not yet installed**](#scenario-api-can-install-prebuilt-rules-that-are-not-yet-installed) - - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) - - [**Scenario: API does not install or upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-install-or-upgrade-prebuilt-rules-if-they-are-up-to-date) - - [Scenarios for the real package](#scenarios-for-the-real-package) - - [**Scenario: User can install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package**](#scenario-user-can-install-prebuilt-rules-from-scratch-then-install-new-rules-and-upgrade-existing-rules-from-the-new-package) - - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) - - [**Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets**](#scenario-user-is-not-notified-when-no-prebuilt-rules-are-installed-and-there-are-no-prebuilt-rules-assets) - - [**Scenario: User is NOT notified when all prebuilt rules are installed and up to date**](#scenario-user-is-not-notified-when-all-prebuilt-rules-are-installed-and-up-to-date) - - [**Scenario: User is notified when no prebuilt rules are installed and there are rules available to install**](#scenario-user-is-notified-when-no-prebuilt-rules-are-installed-and-there-are-rules-available-to-install) - - [**Scenario: User is notified when some prebuilt rules can be installed**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-installed) - - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) - - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) - - [**Scenario: User is notified after a prebuilt rule gets deleted**](#scenario-user-is-notified-after-a-prebuilt-rule-gets-deleted) - - [Rule installation workflow: base cases](#rule-installation-workflow-base-cases) - - [**Scenario: User can install prebuilt rules one by one**](#scenario-user-can-install-prebuilt-rules-one-by-one) - - [**Scenario: User can install multiple prebuilt rules selected on the page**](#scenario-user-can-install-multiple-prebuilt-rules-selected-on-the-page) - - [**Scenario: User can install all available prebuilt rules at once**](#scenario-user-can-install-all-available-prebuilt-rules-at-once) - - [**Scenario: Empty screen is shown when all prebuilt rules are installed**](#scenario-empty-screen-is-shown-when-all-prebuilt-rules-are-installed) - - [**Scenario: User can preview rules available for installation**](#scenario-user-can-preview-rules-available-for-installation) - - [**Scenario: User can install a rule using the rule preview**](#scenario-user-can-install-a-rule-using-the-rule-preview) - - [**Scenario: User can see correct rule information in preview before installing**](#scenario-user-can-see-correct-rule-information-in-preview-before-installing) - - [**Scenario: Tabs and sections without content should be hidden in preview before installing**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-installing) - - [Rule installation workflow: filtering, sorting, pagination](#rule-installation-workflow-filtering-sorting-pagination) - - [Rule installation workflow: misc cases](#rule-installation-workflow-misc-cases) - - [**Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed**](#scenario-user-opening-the-add-rules-page-sees-a-loading-skeleton-until-the-package-installation-is-completed) - - [**Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs**](#scenario-user-can-navigate-from-the-add-rules-page-to-the-rule-management-page-via-breadcrumbs) - - [Rule upgrade workflow: base cases](#rule-upgrade-workflow-base-cases) - - [**Scenario: User can upgrade prebuilt rules one by one**](#scenario-user-can-upgrade-prebuilt-rules-one-by-one) - - [**Scenario: User can upgrade multiple prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-prebuilt-rules-selected-on-the-page) - - [**Scenario: User can upgrade all available prebuilt rules at once**](#scenario-user-can-upgrade-all-available-prebuilt-rules-at-once) - - [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade) - - [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview) - - [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading) - - [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading) - - [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination) - - [Rule upgrade workflow: viewing rule changes in JSON diff view](#rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view) - - [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view) - - [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade) - - [**Scenario: Rule actions and exception lists should not be shown as modified**](#scenario-rule-actions-and-exception-lists-should-not-be-shown-as-modified) - - [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview) - - [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview) - - [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified) - - [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default) - - [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically) - - [Rule upgrade workflow: viewing rule changes in per-field diff view](#rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view) - - [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view) - - [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel) - - [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs) - - [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview) - - [Rule upgrade workflow: preserving rule bound data](#rule-upgrade-workflow-preserving-rule-bound-data) - - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-the-same-rule-type) - - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-a-different-rule-type) - - [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases) - - [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed) - - [Error handling](#error-handling) - - [**Scenario: Error is handled when any operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-operation-on-prebuilt-rules-fails) - - [Authorization / RBAC](#authorization--rbac) - - [**Scenario: User with read privileges on Security Solution cannot install prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-install-prebuilt-rules) - - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) - - [Kibana upgrade](#kibana-upgrade) - - [**Scenario: User can use prebuilt rules after upgrading Kibana from version A to B**](#scenario-user-can-use-prebuilt-rules-after-upgrading-kibana-from-version-a-to-b) + - [Scenarios](#scenarios) + - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) + - [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date) + - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) + - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) + - [Rule upgrade workflow: individual upgrade from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) + - [**Scenario: User can upgrade conflict-free prebuilt rules one by one**](#scenario-user-can-upgrade-conflict-free-prebuilt-rules-one-by-one) + - [**Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts**](#scenario-user-cannot-upgrade-prebuilt-rules-one-by-one-from-rules-update-table-if-they-have-conflicts) + - [Rule upgrade workflow: bulk upgrade from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) + - [**Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-conflict-free-prebuilt-rules-selected-on-the-page) + - [**Scenario: User cannot upgrade multiple prebuilt rules selected on the page when they have upgrade conflicts**](#scenario-user-cannot-upgrade-multiple-prebuilt-rules-selected-on-the-page-when-they-have-upgrade-conflicts) + - [**Scenario: User can upgrade all available conflict-free prebuilt rules at once**](#scenario-user-can-upgrade-all-available-conflict-free-prebuilt-rules-at-once) + - [**Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts**](#scenario-user-cannot-upgrade-all-prebuilt-rules-at-once-if-they-have-upgrade-conflicts) + - [**Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade in the Rules Table**](#scenario-user-can-upgrade-only-conflict-free-rules-when-a-mix-of-rules-with-and-without-conflicts-are-selected-for-upgrade-in-the-rules-table) + - [**Scenario: User can upgrade only conflict-free rules when user attempts to upgrade all rules and only a subset contains upgrade conflicts**](#scenario-user-can-upgrade-only-conflict-free-rules-when-user-attempts-to-upgrade-all-rules-and-only-a-subset-contains-upgrade-conflicts) + - [Rule upgrade workflow: upgrading rules with rule type change](#rule-upgrade-workflow-upgrading-rules-with-rule-type-change) + - [**Scenario: User can upgrade rule with rule type change individually**](#scenario-user-can-upgrade-rule-with-rule-type-change-individually) + - [**Scenario: User can bulk upgrade selected rules with rule type changes**](#scenario-user-can-bulk-upgrade-selected-rules-with-rule-type-changes) + - [**Scenario: User can bulk upgrade all rules with rule type changes**](#scenario-user-can-bulk-upgrade-all-rules-with-rule-type-changes) + - [Rule upgrade workflow: rule previews](#rule-upgrade-workflow-rule-previews) + - [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade) + - [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview) + - [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading) + - [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading) + - [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination) + - [MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in JSON diff view](#milestone-2-legacy---rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view) + - [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view) + - [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade) + - [**Scenario: Rule actions and exception lists should not be shown as modified**](#scenario-rule-actions-and-exception-lists-should-not-be-shown-as-modified) + - [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview) + - [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview) + - [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified) + - [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default) + - [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically) + - [MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in per-field diff view](#milestone-2-legacy---rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view) + - [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view) + - [**Scenario: User can see changes when updated rule is a different rule type**](#scenario-user-can-see-changes-when-updated-rule-is-a-different-rule-type) + - [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel) + - [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs) + - [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview) + - [Rule upgrade workflow: preserving rule bound data](#rule-upgrade-workflow-preserving-rule-bound-data) + - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-the-same-rule-type) + - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-a-different-rule-type) + - [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases) + - [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed) + - [Error handling](#error-handling) + - [**Scenario: Error is handled when any upgrade operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-upgrade-operation-on-prebuilt-rules-fails) + - [Rule upgrade via the Prebuilt rules API](#rule-upgrade-via-the-prebuilt-rules-api) + - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) + - [**Scenario: API does not upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-upgrade-prebuilt-rules-if-they-are-up-to-date) + - [Authorization / RBAC](#authorization-rbac) + - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) + ## Useful information ### Tickets - [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic + +**Milestone 3 - Prebuilt Rules Customization:** +- [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) +- [Tests for prebuilt rule upgrade workflow #202078](https://github.com/elastic/kibana/issues/202078) + +**Milestone 2:** - [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) - [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) -- [Document the new UI for installing and upgrading prebuilt detection rules](https://github.com/elastic/security-docs/issues/3496) ### Terminology @@ -205,266 +194,15 @@ Examples: ## Scenarios -### Package installation - -#### **Scenario: Package is installed via Fleet** - -**Automation**: 2 e2e tests that install the real package. - -```Gherkin -Given the package is not installed -When user opens the Rule Management page -Then the package gets installed in the background from EPR -``` - -#### **Scenario: Package is installed via bundled Fleet package in Kibana** - -**Automation**: 2 integration tests. - -```Gherkin -Given the package is not installed -And user is in an air-gapped environment -When user opens the Rule Management page -Then the package gets installed in the background from packages bundled into Kibana -``` - -#### **Scenario: Large package can be installed on a small Kibana instance** - -**Automation**: 1 integration test. - -```Gherkin -Given the package is not installed -And the package contains the largest amount of historical rule versions (15000) -And the Kibana instance has a memory heap size of 700 Mb (see note below) -When user opens the Rule Management page -Then the package is installed without Kibana crashing with an Out Of Memory error -``` - -**Note**: 600 Mb seems to always crash Kibana with an OOM error. 700 Mb runs with no issues in the Flaky test runner with 100 iterations: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2215. - -### Rule installation and upgrade via the Prebuilt rules API - -There's a legacy prebuilt rules API and a new one. Both should be tested against two types of the package: with and without historical rule versions. - -#### **Scenario: API can install all prebuilt rules** - -**Automation**: 8 integration tests with mock rules: 4 examples below \* 2 (we split checking API response and installed rules into two different tests). - -```Gherkin -Given the package is installed -And the package contains N rules -When user installs all rules via install -Then the endpoint should return 200 with -And N rule objects should be created -And each rule object should have correct id and version - -Examples: - | package_type | api | install_response | - | with historical versions | legacy | installed: N, updated: 0 | - | w/o historical versions | legacy | installed: N, updated: 0 | - | with historical versions | new | total: N, succeeded: N | - | w/o historical versions | new | total: N, succeeded: N | -``` - -Notes: - -- Legacy API: - - install: `PUT /api/detection_engine/rules/prepackaged` -- New API: - - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` - -#### **Scenario: API can install prebuilt rules that are not yet installed** - -**Automation**: 4 integration tests with mock rules. - -```Gherkin -Given the package is installed -And the package contains N rules -When user installs all rules via install -And deletes one of the installed rules -And gets prebuilt rules status via status -Then the endpoint should return 200 with -When user installs all rules via install again -Then the endpoint should return 200 with - -Examples: - | package_type | api | status_response | install_response | - | with historical versions | legacy | not_installed: 1 | installed: 1, updated: 0 | - | w/o historical versions | legacy | not_installed: 1 | installed: 1, updated: 0 | - | with historical versions | new | to_install: 1 | total: 1, succeeded: 1 | - | w/o historical versions | new | to_install: 1 | total: 1, succeeded: 1 | -``` - -Notes: - -- Legacy API: - - install: `PUT /api/detection_engine/rules/prepackaged` - - status: `GET /api/detection_engine/rules/prepackaged/_status` -- New API: - - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` - - status: `GET /internal/detection_engine/prebuilt_rules/status` - -#### **Scenario: API can upgrade prebuilt rules that are outdated** - -**Automation**: 4 integration tests with mock rules. - -```Gherkin -Given the package is installed -And the package contains N rules -When user installs all rules via install -And new X+1 version of a rule asset -And user gets prebuilt rules status via status -Then the endpoint should return 200 with -When user upgrades all rules via upgrade -Then the endpoint should return 200 with - -Examples: - | package_type | api | assets_update | status_response | upgrade_response | - | with historical versions | legacy | gets added | not_updated: 1 | installed: 0, updated: 1 | - | w/o historical versions | legacy | replaces X | not_updated: 1 | installed: 0, updated: 1 | - | with historical versions | new | gets added | to_upgrade: 1 | total: 1, succeeded: 1 | - | w/o historical versions | new | replaces X | to_upgrade: 1 | total: 1, succeeded: 1 | -``` - -TODO: Check why for the legacy API Dmitrii has added 2 integration tests for `rule package with historical versions` instead of 1: - -- `should update outdated prebuilt rules when previous historical versions available` -- `should update outdated prebuilt rules when previous historical versions unavailable` - -(NOTE: the second scenario tests that, if a new version of a rule is released, it can upgrade the current instance of that rule even if the historical versions of that rule are no longer in the package) - -Notes: - -- Legacy API: - - install: `PUT /api/detection_engine/rules/prepackaged` - - upgrade: `PUT /api/detection_engine/rules/prepackaged` - - status: `GET /api/detection_engine/rules/prepackaged/_status` -- New API: - - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` - - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` - - status: `GET /internal/detection_engine/prebuilt_rules/status` - -#### **Scenario: API does not install or upgrade prebuilt rules if they are up to date** - -**Automation**: 4 integration tests with mock rules. - -```Gherkin -Given the package is installed -And the package contains N rules -When user installs all rules via install -And user gets prebuilt rules status via status -Then the endpoint should return 200 with -When user calls install -Then the endpoint should return 200 with -When user calls upgrade -Then the endpoint should return 200 with - -Examples: - | package_type | api | status_response | install_response | upgrade_response | - | with historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | - | w/o historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | - | with historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | - | w/o historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | -``` - -Notes: - -- Legacy API: - - install: `PUT /api/detection_engine/rules/prepackaged` - - upgrade: `PUT /api/detection_engine/rules/prepackaged` - - status: `GET /api/detection_engine/rules/prepackaged/_status` -- New API: - - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` - - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` - - status: `GET /internal/detection_engine/prebuilt_rules/status` - -### Scenarios for the real package - -#### **Scenario: User can install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package** - -**Automation**: 1 integration test with real packages. - -```Gherkin -Given there are two package versions: N-1 and N -And the package of N-1 version is installed -When user calls the status endpoint -Then it should return a 200 response with some number of rules to install and 0 rules to upgrade -When user calls the installation/_review endpoint -Then it should return a 200 response matching the response of the status endpoint -When user calls the installation/_perform_ endpoint -Then it should return a 200 response matching the response of the status endpoint -And rules returned in this response should exist as alert saved objects -When user installs the package of N version -Then it should be installed successfully -When user calls the status endpoint -Then it should return a 200 response with some number of new rules to install and some number of rules to upgrade -When user calls the installation/_review endpoint -Then it should return a 200 response matching the response of the status endpoint -When user calls the installation/_perform_ endpoint -Then rules returned in this response should exist as alert saved objects -When user calls the upgrade/_review endpoint -Then it should return a 200 response matching the response of the status endpoint -When user calls the upgrade/_perform_ endpoint -Then rules returned in this response should exist as alert saved objects -``` - ### Rule installation and upgrade notifications on the Rule Management page -#### **Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets** - -**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. - -```Gherkin -Given no prebuilt rules are installed in Kibana -And no prebuilt rule assets exist -When user opens the Rule Management page -Then user should NOT see a CTA to install prebuilt rules -And user should NOT see a number of rules available to install -And user should NOT see a CTA to upgrade prebuilt rules -And user should NOT see a number of rules available to upgrade -And user should NOT see the Rule Updates table -``` - -#### **Scenario: User is NOT notified when all prebuilt rules are installed and up to date** +#### **Scenario: User is NOT notified when all installed prebuilt rules are up to date** **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin Given all the latest prebuilt rules are installed in Kibana When user opens the Rule Management page -Then user should NOT see a CTA to install prebuilt rules -And user should NOT see a number of rules available to install -And user should NOT see a CTA to upgrade prebuilt rules -And user should NOT see a number of rules available to upgrade -And user should NOT see the Rule Updates table -``` - -#### **Scenario: User is notified when no prebuilt rules are installed and there are rules available to install** - -**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. - -```Gherkin -Given no prebuilt rules are installed in Kibana -And there are X prebuilt rules available to install -When user opens the Rule Management page -Then user should see a CTA to install prebuilt rules -And user should see a number of rules available to install (X) -And user should NOT see a CTA to upgrade prebuilt rules -And user should NOT see a number of rules available to upgrade -And user should NOT see the Rule Updates table -``` - -#### **Scenario: User is notified when some prebuilt rules can be installed** - -**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. - -```Gherkin -Given X prebuilt rules are installed in Kibana -And there are Y more prebuilt rules available to install -And for all X installed rules there are no new versions available -When user opens the Rule Management page -Then user should see a CTA to install prebuilt rules -And user should see the number of rules available to install (Y) And user should NOT see a CTA to upgrade prebuilt rules And user should NOT see a number of rules available to upgrade And user should NOT see the Rule Updates table @@ -500,242 +238,219 @@ And user should see a CTA to upgrade prebuilt rules And user should see the number of rules available to upgrade (Z) ``` -#### **Scenario: User is notified after a prebuilt rule gets deleted** +### Rule upgrade workflow: individual updates from Rule Updates table -**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. +#### **Scenario: User can upgrade conflict-free prebuilt rules one by one** + +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. ```Gherkin Given X prebuilt rules are installed in Kibana -And there are no more prebuilt rules available to install -When user opens the Rule Management page -And user deletes Y prebuilt rules -Then user should see a CTA to install prebuilt rules -And user should see the number of rules available to install (Y) +And for Y of the installed rules there are new versions available +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user upgrades one individual rule without previewing it +Then success message should be displayed after upgrade +And the upgraded rule should be removed from the table +And user should see the number of rules available to upgrade decreased by 1 ``` -### Rule installation workflow: base cases - -#### **Scenario: User can install prebuilt rules one by one** +#### **Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts** -**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. +**Automation**: 1 e2e test with mock rules ```Gherkin -Given no prebuilt rules are installed in Kibana -And there are X prebuilt rules available to install -When user opens the Add Rules page -Then prebuilt rules available for installation should be displayed in the table -When user installs one individual rule without previewing it -Then success message should be displayed after installation -And the installed rule should be removed from the table -When user navigates back to the Rule Management page -Then user should see a CTA to install prebuilt rules -And user should see the number of rules available to install decreased by 1 +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And user is on the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +And for Z (where Z < Y) of the rules with upgrades there are upgrade conflicts +Then for those Z rules the Upgrade Rule button should be disabled +And the user should not be able to upgrade them directly from the table +And there should be a message/tooltip indicating why the rule cannot be upgraded directly ``` -#### **Scenario: User can install multiple prebuilt rules selected on the page** +### Rule upgrade workflow: bulk updates from Rule Updates table -**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. +#### **Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page** + +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. ```Gherkin -Given no prebuilt rules are installed in Kibana -And there are X prebuilt rules available to install -When user opens the Add Rules page -Then prebuilt rules available for installation should be displayed in the table -When user selects rules -Then user should see a CTA to install number of rules +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user selects Z (where Z < Y) rules, which have no upgrade conflicts +Then user should see a CTA to upgrade rules When user clicks the CTA -Then success message should be displayed after installation -And all the installed rules should be removed from the table -When user navigates back to the Rule Management page -Then user should see a CTA to install prebuilt rules -And user should see the number of rules available to install decreased by number of installed rules +Then success message should be displayed after upgrade +And all the upgraded rules should be removed from the table +And user should see the number of rules available to upgrade decreased by number of upgraded rules Examples: - | Y | + | Z | | a few rules on the page, e.g. 2 | | all rules on the page, e.g. 12 | ``` -#### **Scenario: User can install all available prebuilt rules at once** +#### **Scenario: User cannot upgrade selected prebuilt rules with conflicts** -**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. +**Automation**: 1 e2e test with mock rules ```Gherkin -Given no prebuilt rules are installed in Kibana -And there are X prebuilt rules available to install -When user opens the Add Rules page -Then prebuilt rules available for installation should be displayed in the table -When user installs all rules -Then success message should be displayed after installation -And all the rules should be removed from the table -And user should see a message indicating that all available rules have been installed -And user should see a CTA that leads to the Rule Management page -When user clicks on the CTA -Then user should be navigated back to Rule Management page -And user should NOT see a CTA to install prebuilt rules -And user should NOT see a number of rules available to install -``` - -#### **Scenario: Empty screen is shown when all prebuilt rules are installed** - -**Automation**: 1 e2e test with mock rules + 1 integration test. +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And all of those Y new versions have conflicts with the current versions +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user selects rules, all of which have upgrade conflicts +Then user should see a CTA to upgrade number of rules, which should be disabled +When user hovers on the CTA to upgrade multiple rules +Then a message should be displayed that informs the user why the rules cannot be updated -```Gherkin -Given all the available prebuilt rules are installed in Kibana -When user opens the Add Rules page -Then user should see a message indicating that all available rules have been installed -And user should see a CTA that leads to the Rule Management page +Examples: + | Z | + | a few rules on the page, e.g. 2 | + | all rules on the page, e.g. 12 | ``` -#### **Scenario: User can preview rules available for installation** +#### **Scenario: User can upgrade all available conflict-free prebuilt rules at once** -**Automation**: 1 e2e test +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. ```Gherkin -Given no prebuilt rules are installed in Kibana -And there are 2 rules available to install -When user opens the Add Rules page -Then all rules available for installation should be displayed in the table -When user opens the rule preview for the 1st rule -Then the preview should open -When user closes the preview -Then it should disappear +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And those Y new versions don't have conflicts with the current versions +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user upgrades all rules +Then success message should be displayed after upgrade +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade ``` -#### **Scenario: User can install a rule using the rule preview** +#### **Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts** -**Automation**: 1 e2e test +**Automation**: 1 e2e test with mock rules ```Gherkin -Given no prebuilt rules are installed in Kibana -And there are 2 rules available to install -When user opens the Add Rules page -Then all rules available for installation should be displayed in the table -When user opens the rule preview for the rule -Then the preview should open -When user installs the rule using a CTA in the rule preview -Then the rule should be installed -And a success message should be displayed after installation -And the rule should be removed from the Add Rules table -When user navigates back to the Rule Management page -Then user should see a CTA to install prebuilt rules -And user should see the number of rules available to install as initial number minus 1 +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And all Y new versions have conflicts with the current versions +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +Then user should see a CTA to upgrade all rules +And the CTA to upgrade all rules should be disabled +When user hovers on the CTA to upgrade all rules +Then a message should be displayed that informs the user why the rules cannot be updated ``` -#### **Scenario: User can see correct rule information in preview before installing** +#### **Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade in the Rules Table** -**Automation**: 1 e2e test +**Automation**: 1 e2e test with mock rules ```Gherkin -Given no prebuilt rules are installed in Kibana -And there are X prebuilt rules of all types available to install -When user opens the Add Rules page -Then all X rules available for installation should be displayed in the table -When user opens a rule preview for any rule -Then the preview should appear -And all properties of a rule should be displayed in the correct tab and section of the preview (see examples of rule properties above) -``` - -#### **Scenario: Tabs and sections without content should be hidden in preview before installing** - -**Automation**: 1 e2e test +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And a subset Z of the rules have conflicts with the current versions +And user is on the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +And user selects rules, which is a mixture of rules with and without upgrade conflicts +Then user should see a CTA to upgrade number of rules, which is enabled +When user clicks the CTA +A modal window should inform the user that only W rules without conflicts will be upgraded +When user confirms the action in the modal +Then success message should be displayed after upgrade informing that W rules were updated +And the W upgraded rules should be removed from the table +And the remaining Z - W rules should still be present in the table +And user should see the number of rules available to upgrade decreased by W number of upgraded rules -```Gherkin -Given no prebuilt rules are installed in Kibana -And there is at least 1 rule available to install -And this rule has neither Setup guide nor Investigation guide -When user opens the Add Rules page -Then all rules available for installation should be displayed in the table -When user opens the rule preview for this rule -Then the preview should open -And the Setup Guide section should NOT be displayed in the Overview tab -And the Investigation Guide tab should NOT be displayed +Examples: + | Z | + | a few rules on the page, e.g. 2 | + | all rules on the page, e.g. 12 | ``` -### Rule installation workflow: filtering, sorting, pagination - -TODO: add scenarios https://github.com/elastic/kibana/issues/166215 - -### Rule installation workflow: misc cases - -#### **Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed** +#### **Scenario: User can upgrade only conflict-free rules when attempting to upgrade all rules** -**Automation**: unit tests. +**Automation**: 1 e2e test with mock rules ```Gherkin -Given prebuilt rules package is not installed -When user opens the Add Rules page -Then user should see a loading skeleton until the package installation is completed +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And Z (where Z < Y) rules have conflicts with the current versions +And user is on the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +Then user should see an enabled CTA to upgrade all rules +When user clicks the CTA +A modal window should inform the user that only K (where K < Y) rules without conflicts will be upgraded +When user confirms the action in the modal +Then success message should be displayed after upgrade informing that K rules were updated +And the K upgraded rules should be removed from the table +And the remaining M = Y - K rules should still be present in the table +And user should see the number of rules available to upgrade decreased by K number of upgraded rules ``` -#### **Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs** -**Automation**: 1 e2e test. +### Rule upgrade workflow: upgrading rules with rule type change -```Gherkin -Given user is on the Add Rules page -When user navigates to the Rule Management page via breadcrumbs -Then the Rule Management page should be displayed -``` - -### Rule upgrade workflow: base cases - -#### **Scenario: User can upgrade prebuilt rules one by one** +#### **Scenario: User can upgrade rule with rule type change individually** -**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. +**Automation**: 1 e2e test with mock rules ```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And user is on the Rule Management page +Given a prebuilt rule is installed in Kibana +And this rule has an update available that changes its rule type When user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -When user upgrades one individual rule without previewing it +Then this rule should be displayed in the table +And the Upgrade Rule button should be enabled +When user upgrades the rule Then success message should be displayed after upgrade And the upgraded rule should be removed from the table And user should see the number of rules available to upgrade decreased by 1 ``` -#### **Scenario: User can upgrade multiple prebuilt rules selected on the page** +#### **Scenario: User can bulk upgrade selected rules with rule type changes** -**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. + +**Automation**: 1 e2e test with mock rules ```Gherkin Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And user is on the Rule Management page +And Y of these rules have updates available that change their rule types When user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -When user selects rules -Then user should see a CTA to upgrade number of rules +Then Y rules should be displayed in the table +When user selects Z rules (where Z < Y) with rule type changes +Then user should see an enabled CTA to upgrade Z rules When user clicks the CTA Then success message should be displayed after upgrade -And all the upgraded rules should be removed from the table -And user should see the number of rules available to upgrade decreased by number of upgraded rules - -Examples: - | Z | - | a few rules on the page, e.g. 2 | - | all rules on the page, e.g. 12 | +And all Z upgraded rules should be removed from the table +And user should see the number of rules available to upgrade decreased by Z ``` -#### **Scenario: User can upgrade all available prebuilt rules at once** +#### **Scenario: User can bulk upgrade all rules with rule type changes** -**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. +**Automation**: 1 e2e test with mock rules ```Gherkin Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And user is on the Rule Management page +And all X rules have updates available that change their rule types When user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -When user upgrades all rules +Then X rules should be displayed in the table +And user should see an enabled CTA to upgrade all rules +When user clicks the CTA to upgrade all rules Then success message should be displayed after upgrade -And user should NOT see a CTA to upgrade prebuilt rules -And user should NOT see a number of rules available to upgrade -And user should NOT see the Rule Updates table +And all X upgraded rules should be removed from the table ``` +### Rule upgrade workflow: rule previews + #### **Scenario: User can preview rules available for upgrade** ```Gherkin @@ -807,7 +522,9 @@ And the Investigation Guide tab should NOT be displayed TODO: add scenarios https://github.com/elastic/kibana/issues/166215 -### Rule upgrade workflow: viewing rule changes in JSON diff view +### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in JSON diff view + +> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. #### **Scenario: User can see changes in a side-by-side JSON diff view** @@ -953,7 +670,9 @@ When a user expands all hidden sections Then all properties of the rule should be sorted alphabetically ``` -### Rule upgrade workflow: viewing rule changes in per-field diff view +### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in per-field diff view + +> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. #### **Scenario: User can see changes in a side-by-side per-field diff view** @@ -1091,7 +810,7 @@ Then user should NOT see the Rule Updates tab until the package installation is ### Error handling -#### **Scenario: Error is handled when any operation on prebuilt rules fails** +#### **Scenario: Error is handled when any upgrade operation on prebuilt rules fails** **Automation**: e2e test with mock rules @@ -1102,29 +821,92 @@ Then user should see an error message Examples: | operation | - | installing all | - | installing selected | - | installing individual | | upgrading all | | upgrading selected | | upgrading individual | ``` -### Authorization / RBAC -#### **Scenario: User with read privileges on Security Solution cannot install prebuilt rules** +### Rule upgrade via the Prebuilt rules API + +There's a legacy prebuilt rules API and a new one. Both should be tested against two types of the package: with and without historical rule versions. + +#### **Scenario: API can upgrade prebuilt rules that are outdated** -**Automation**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and installation endpoints. +**Automation**: 4 integration tests with mock rules. ```Gherkin -Given user with "Security: read" privileges on Security Solution -And no prebuilt rules are installed in Kibana -And there are prebuilt rules available to install -When user opens the Add Rules page -Then user should see prebuilt rules available to install -But user should not be able to install them +Given the package is installed +And the package contains N rules +When user installs all rules via install +And new X+1 version of a rule asset +And user gets prebuilt rules status via status +Then the endpoint should return 200 with +When user upgrades all rules via upgrade +Then the endpoint should return 200 with + +Examples: + | package_type | api | assets_update | status_response | upgrade_response | + | with historical versions | legacy | gets added | not_updated: 1 | installed: 0, updated: 1 | + | w/o historical versions | legacy | replaces X | not_updated: 1 | installed: 0, updated: 1 | + | with historical versions | new | gets added | to_upgrade: 1 | total: 1, succeeded: 1 | + | w/o historical versions | new | replaces X | to_upgrade: 1 | total: 1, succeeded: 1 | +``` + +TODO: Check why for the legacy API Dmitrii has added 2 integration tests for `rule package with historical versions` instead of 1: + +- `should update outdated prebuilt rules when previous historical versions available` +- `should update outdated prebuilt rules when previous historical versions unavailable` + +(NOTE: the second scenario tests that, if a new version of a rule is released, it can upgrade the current instance of that rule even if the historical versions of that rule are no longer in the package) + +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` + - upgrade: `PUT /api/detection_engine/rules/prepackaged` + - status: `GET /api/detection_engine/rules/prepackaged/_status` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` + - status: `GET /internal/detection_engine/prebuilt_rules/status` + +#### **Scenario: API does not upgrade prebuilt rules if they are up to date** + +**Automation**: 4 integration tests with mock rules. + +```Gherkin +Given the package is installed +And the package contains N rules +When user installs all rules via install +And user gets prebuilt rules status via status +Then the endpoint should return 200 with +When user calls install +Then the endpoint should return 200 with +When user calls upgrade +Then the endpoint should return 200 with + +Examples: + | package_type | api | status_response | install_response | upgrade_response | + | with historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | + | w/o historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | + | with historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | + | w/o historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | ``` +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` + - upgrade: `PUT /api/detection_engine/rules/prepackaged` + - status: `GET /api/detection_engine/rules/prepackaged/_status` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` + - status: `GET /internal/detection_engine/prebuilt_rules/status` + +### Authorization / RBAC + #### **Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules** **Automation**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and upgrade endpoints. @@ -1137,24 +919,4 @@ When user opens the Rule Management page And user opens the Rule Updates table Then user should see prebuilt rules available to upgrade But user should not be able to upgrade them -``` - -### Kibana upgrade - -#### **Scenario: User can use prebuilt rules after upgrading Kibana from version A to B** - -**Automation**: not automated, manual testing required. - -```Gherkin -Given user is upgrading Kibana from version to version -And the instance contains already installed prebuilt rules -When the upgrade is complete -Then user should be able to install new prebuilt rules -And delete installed prebuilt rules -And upgrade installed prebuilt rules that have newer versions in - -Examples: - | A | B | - | 8.7 | 8.9.0 | - | 7.17.x | 8.9.0 | -``` +``` \ No newline at end of file From 5247b52a3c64f570b68165afda0b8204e6edd090 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Mon, 9 Dec 2024 14:46:28 -0300 Subject: [PATCH 02/18] Split test install+upgrade test plans into 3 --- .../prebuilt_rules/installation.md | 102 +++++++++++++ .../package_installation_and_upgrade.md | 42 ++++++ .../prebuilt_rules/upgrade.md | 138 ++++++++++++++++++ 3 files changed, 282 insertions(+) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md index dac2a0e8f1ac6..37b61f3c92740 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md @@ -2,7 +2,11 @@ This is a test plan for the workflows of installing prebuilt rules. +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). +======= +Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ## Table of Contents @@ -46,7 +50,11 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz ### Tickets +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md - [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) +======= +- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md **Milestone 3 - Prebuilt Rules Customization:** - [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) @@ -176,8 +184,13 @@ Examples: **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given no prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +Given no prebuilt rules are installed in Kibana +And no prebuilt rule assets exist +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Rule Management page Then user should NOT see a CTA to install prebuilt rules And user should NOT see a number of rules available to install @@ -191,8 +204,12 @@ And user should NOT see the Rule Updates table **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given the latest prebuilt rule assets exist in Kibana And all the latest prebuilt rules from those rule assets are installed +======= +Given all the latest prebuilt rules are installed in Kibana +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Rule Management page Then user should NOT see a CTA to install prebuilt rules And user should NOT see a number of rules available to install @@ -206,8 +223,12 @@ And user should NOT see the Rule Updates table **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +Given no prebuilt rules are installed in Kibana +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And there are X prebuilt rules available to install When user opens the Rule Management page Then user should see a CTA to install prebuilt rules @@ -222,8 +243,12 @@ And user should NOT see the Rule Updates table **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given Y prebuilt rule assets exist in Kibana And X (where X < Y) prebuilt rules are installed +======= +Given X prebuilt rules are installed in Kibana +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And there are Y more prebuilt rules available to install And for all X installed rules there are no new versions available When user opens the Rule Management page @@ -239,9 +264,14 @@ And user should NOT see the Rule Updates table **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given Y prebuilt rule assets exist in Kibana And X (where X < Y) prebuilt rules are installed And Z (where Z < X) installed rules have matching prebuilt rule assets with higher version available +======= +Given X prebuilt rules are installed in Kibana +And there are Y more prebuilt rules available to install +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And for Z of the installed rules there are new versions available When user opens the Rule Management page Then user should see a CTA to install prebuilt rules @@ -260,7 +290,11 @@ And there are no more prebuilt rules available to install When user opens the Rule Management page And user deletes Y prebuilt rules Then user should see a CTA to install prebuilt rules +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And user should see rules available to install +======= +And user should see the number of rules available to install (Y) +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ``` ### Rule installation workflow: base cases @@ -270,8 +304,13 @@ And user should see rules available to install **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +Given no prebuilt rules are installed in Kibana +And there are X prebuilt rules available to install +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then prebuilt rules available for installation should be displayed in the table When user installs one individual rule without previewing it @@ -287,8 +326,13 @@ And user should see the number of rules available to install decreased by 1 **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +Given no prebuilt rules are installed in Kibana +And there are X prebuilt rules available to install +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then prebuilt rules available for installation should be displayed in the table When user selects rules @@ -311,8 +355,13 @@ Examples: **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +Given no prebuilt rules are installed in Kibana +And there are X prebuilt rules available to install +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then prebuilt rules available for installation should be displayed in the table When user installs all rules @@ -342,8 +391,13 @@ And user should see a CTA that leads to the Rule Management page **Automation**: 1 e2e test ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given 2 prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +Given no prebuilt rules are installed in Kibana +And there are 2 rules available to install +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then all rules available for installation should be displayed in the table When user opens the rule preview for the 1st rule @@ -357,8 +411,13 @@ Then it should disappear **Automation**: 1 e2e test ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given 2 prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +Given no prebuilt rules are installed in Kibana +And there are 2 rules available to install +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then all rules available for installation should be displayed in the table When user opens the rule preview for the rule @@ -377,8 +436,13 @@ And user should see the number of rules available to install as initial number m **Automation**: 1 e2e test ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +Given no prebuilt rules are installed in Kibana +And there are X prebuilt rules of all types available to install +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then all X rules available for installation should be displayed in the table When user opens a rule preview for any rule @@ -391,8 +455,13 @@ And all properties of a rule should be displayed in the correct tab and section **Automation**: 1 e2e test ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given 1 prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +Given no prebuilt rules are installed in Kibana +And there is at least 1 rule available to install +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And this rule has neither Setup guide nor Investigation guide When user opens the Add Rules page Then all rules available for installation should be displayed in the table @@ -440,7 +509,11 @@ There's a legacy prebuilt rules API and a new one. Both should be tested against Given the package is installed And the package contains N rules When user installs all rules via install +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Then the endpoint should return success response (HTTP 200 code) with +======= +Then the endpoint should return 200 with +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And N rule objects should be created And each rule object should have correct id and version @@ -469,9 +542,15 @@ And the package contains N rules When user installs all rules via install And deletes one of the installed rules And gets prebuilt rules status via status +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Then the endpoint should return successful response (HTTP 200 code) with When user installs all rules via install again Then the endpoint should return successful response (HTTP 200 code) with +======= +Then the endpoint should return 200 with +When user installs all rules via install again +Then the endpoint should return 200 with +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Examples: | package_type | api | status_response | install_response | @@ -500,11 +579,19 @@ Given the package is installed And the package contains N rules When user installs all rules via install And user gets prebuilt rules status via status +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Then the endpoint should return successful response (HTTP 200 code) with When user calls install Then the endpoint should return successful response (HTTP 200 code) with When user calls upgrade Then the endpoint should return successful response (HTTP 200 code) with +======= +Then the endpoint should return 200 with +When user calls install +Then the endpoint should return 200 with +When user calls upgrade +Then the endpoint should return 200 with +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Examples: | package_type | api | status_response | install_response | upgrade_response | @@ -527,7 +614,11 @@ Notes: ### Error handling +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md #### **Scenario: Error is handled when any installation operation on prebuilt rules fails** +======= +#### **Scenario: Error is handled when any operation on prebuilt rules fails** +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md **Automation**: e2e test with mock rules @@ -541,6 +632,12 @@ Examples: | installing all | | installing selected | | installing individual | +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= + | upgrading all | + | upgrading selected | + | upgrading individual | +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ``` ### Authorization / RBAC @@ -551,8 +648,13 @@ Examples: ```Gherkin Given user with "Security: read" privileges on Security Solution +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And prebuilt rule assets exist in Kibana And no prebuilt rules are installed +======= +And no prebuilt rules are installed in Kibana +And there are prebuilt rules available to install +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then user should see prebuilt rules available to install But user should not be able to install them diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md index 57e16684facca..b39946c9d9288 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md @@ -4,7 +4,11 @@ This is a test plan for the workflows of installing and updating the Prebuilt Ru > For the test plans for installing and upgrading prebuilt rules, see [Installation of Prebuilt Rules](./installation.md) and [Upgrade of Prebuilt Rules](./upgrade.md). +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). +======= +Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md ## Table of Contents @@ -28,7 +32,11 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz ### Tickets +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md - [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) +======= +- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md **Milestone 3 - Prebuilt Rules Customization:** - [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) @@ -84,8 +92,13 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz **Automation**: 2 e2e tests that install the real package. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Given the prebuilt rules package is not installed When user opens any Security Solution page +======= +Given the package is not installed +When user opens the Rule Management page +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then the package gets installed in the background from EPR ``` @@ -96,11 +109,19 @@ Then the package gets installed in the background from EPR ```Gherkin Given the package is not installed And user is in an air-gapped environment +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user opens any Security Solution page Then the package gets installed in the background from packages bundled into Kibana ``` #### **Scenario: Large package can be installed on a memory restricted Kibana instance** +======= +When user opens the Rule Management page +Then the package gets installed in the background from packages bundled into Kibana +``` + +#### **Scenario: Large package can be installed on a small Kibana instance** +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md **Automation**: 1 integration test. @@ -108,7 +129,11 @@ Then the package gets installed in the background from packages bundled into Kib Given the package is not installed And the package contains the largest amount of historical rule versions (15000) And the Kibana instance has a memory heap size of 700 Mb (see note below) +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user opens any Security Solution page +======= +When user opens the Rule Management page +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then the package is installed without Kibana crashing with an Out Of Memory error ``` @@ -121,8 +146,13 @@ Then the package is installed without Kibana crashing with an Out Of Memory erro **Automation**: 1 integration test with real packages. ```Gherkin +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Given there are two package versions: A and B where A < B And the package of A version is installed +======= +Given there are two package versions: N-1 and N +And the package of N-1 version is installed +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user calls the status endpoint Then it should return a 200 response with some number of rules to install and 0 rules to upgrade When user calls the installation/_review endpoint @@ -130,7 +160,11 @@ Then it should return a 200 response matching the response of the status endpoin When user calls the installation/_perform_ endpoint Then it should return a 200 response matching the response of the status endpoint And rules returned in this response should exist as alert saved objects +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user installs version B of the package +======= +When user installs the package of N version +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then it should be installed successfully When user calls the status endpoint Then it should return a 200 response with some number of new rules to install and some number of rules to upgrade @@ -152,11 +186,19 @@ Then rules returned in this response should exist as alert saved objects ```Gherkin Given user is upgrading Kibana from version to version +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md And the version Kibana instance has already installed prebuilt rules When the Kibana upgrade is complete Then user should be able to install new prebuilt rules And delete installed prebuilt rules And upgrade installed prebuilt rules that have newer versions in Kibana version +======= +And the instance contains already installed prebuilt rules +When the upgrade is complete +Then user should be able to install new prebuilt rules +And delete installed prebuilt rules +And upgrade installed prebuilt rules that have newer versions in +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Examples: | A | B | diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index 20520ae1cab0d..4bbe70355bed0 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -2,7 +2,11 @@ This is a test plan for the workflow of upgrading prebuilt rules. +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). +======= +Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ## Table of Contents @@ -12,6 +16,8 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz - [Assumptions](#assumptions) - [Non-functional requirements](#non-functional-requirements) - [Functional requirements](#functional-requirements) +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +<<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md - [Scenarios](#scenarios) - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) - [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date) @@ -65,6 +71,89 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz - [Authorization / RBAC](#authorization-rbac) - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) +======== +- [Scenarios](#scenarios) + - [Package installation](#package-installation) + - [**Scenario: Package is installed via Fleet**](#scenario-package-is-installed-via-fleet) + - [**Scenario: Package is installed via bundled Fleet package in Kibana**](#scenario-package-is-installed-via-bundled-fleet-package-in-kibana) + - [**Scenario: Large package can be installed on a small Kibana instance**](#scenario-large-package-can-be-installed-on-a-small-kibana-instance) + - [Rule installation and upgrade via the Prebuilt rules API](#rule-installation-and-upgrade-via-the-prebuilt-rules-api) + - [**Scenario: API can install all prebuilt rules**](#scenario-api-can-install-all-prebuilt-rules) + - [**Scenario: API can install prebuilt rules that are not yet installed**](#scenario-api-can-install-prebuilt-rules-that-are-not-yet-installed) + - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) + - [**Scenario: API does not install or upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-install-or-upgrade-prebuilt-rules-if-they-are-up-to-date) + - [Scenarios for the real package](#scenarios-for-the-real-package) + - [**Scenario: User can install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package**](#scenario-user-can-install-prebuilt-rules-from-scratch-then-install-new-rules-and-upgrade-existing-rules-from-the-new-package) + - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) + - [**Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets**](#scenario-user-is-not-notified-when-no-prebuilt-rules-are-installed-and-there-are-no-prebuilt-rules-assets) + - [**Scenario: User is NOT notified when all prebuilt rules are installed and up to date**](#scenario-user-is-not-notified-when-all-prebuilt-rules-are-installed-and-up-to-date) + - [**Scenario: User is notified when no prebuilt rules are installed and there are rules available to install**](#scenario-user-is-notified-when-no-prebuilt-rules-are-installed-and-there-are-rules-available-to-install) + - [**Scenario: User is notified when some prebuilt rules can be installed**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-installed) + - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) + - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) + - [**Scenario: User is notified after a prebuilt rule gets deleted**](#scenario-user-is-notified-after-a-prebuilt-rule-gets-deleted) + - [Rule installation workflow: base cases](#rule-installation-workflow-base-cases) + - [**Scenario: User can install prebuilt rules one by one**](#scenario-user-can-install-prebuilt-rules-one-by-one) + - [**Scenario: User can install multiple prebuilt rules selected on the page**](#scenario-user-can-install-multiple-prebuilt-rules-selected-on-the-page) + - [**Scenario: User can install all available prebuilt rules at once**](#scenario-user-can-install-all-available-prebuilt-rules-at-once) + - [**Scenario: Empty screen is shown when all prebuilt rules are installed**](#scenario-empty-screen-is-shown-when-all-prebuilt-rules-are-installed) + - [**Scenario: User can preview rules available for installation**](#scenario-user-can-preview-rules-available-for-installation) + - [**Scenario: User can install a rule using the rule preview**](#scenario-user-can-install-a-rule-using-the-rule-preview) + - [**Scenario: User can see correct rule information in preview before installing**](#scenario-user-can-see-correct-rule-information-in-preview-before-installing) + - [**Scenario: Tabs and sections without content should be hidden in preview before installing**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-installing) + - [Rule installation workflow: filtering, sorting, pagination](#rule-installation-workflow-filtering-sorting-pagination) + - [Rule installation workflow: misc cases](#rule-installation-workflow-misc-cases) + - [**Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed**](#scenario-user-opening-the-add-rules-page-sees-a-loading-skeleton-until-the-package-installation-is-completed) + - [**Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs**](#scenario-user-can-navigate-from-the-add-rules-page-to-the-rule-management-page-via-breadcrumbs) +======= +- [Scenarios](#scenarios) +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md + - [Rule upgrade workflow: base cases](#rule-upgrade-workflow-base-cases) + - [**Scenario: User can upgrade prebuilt rules one by one**](#scenario-user-can-upgrade-prebuilt-rules-one-by-one) + - [**Scenario: User can upgrade multiple prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-prebuilt-rules-selected-on-the-page) + - [**Scenario: User can upgrade all available prebuilt rules at once**](#scenario-user-can-upgrade-all-available-prebuilt-rules-at-once) + - [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade) + - [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview) + - [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading) + - [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading) + - [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination) + - [Rule upgrade workflow: viewing rule changes in JSON diff view](#rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view) + - [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view) + - [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade) + - [**Scenario: Rule actions and exception lists should not be shown as modified**](#scenario-rule-actions-and-exception-lists-should-not-be-shown-as-modified) + - [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview) + - [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview) + - [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified) + - [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default) + - [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically) + - [Rule upgrade workflow: viewing rule changes in per-field diff view](#rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view) + - [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view) + - [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel) + - [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs) + - [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview) + - [Rule upgrade workflow: preserving rule bound data](#rule-upgrade-workflow-preserving-rule-bound-data) + - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-the-same-rule-type) + - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-a-different-rule-type) + - [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases) + - [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed) +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md + - [Error handling](#error-handling) + - [**Scenario: Error is handled when any operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-operation-on-prebuilt-rules-fails) + - [Authorization / RBAC](#authorization--rbac) + - [**Scenario: User with read privileges on Security Solution cannot install prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-install-prebuilt-rules) + - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) + - [Kibana upgrade](#kibana-upgrade) + - [**Scenario: User can use prebuilt rules after upgrading Kibana from version A to B**](#scenario-user-can-use-prebuilt-rules-after-upgrading-kibana-from-version-a-to-b) +>>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +======= + - [Rule upgrade via the Prebuilt rules API](#rule-installation-and-upgrade-via-the-prebuilt-rules-api) + - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) + - [**Scenario: API does not upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-install-or-upgrade-prebuilt-rules-if-they-are-up-to-date) + - [Error handling](#error-handling) + - [**Scenario: Error is handled when any operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-operation-on-prebuilt-rules-fails) + - [Authorization / RBAC](#authorization--rbac) + - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ## Useful information @@ -238,9 +327,15 @@ And user should see a CTA to upgrade prebuilt rules And user should see the number of rules available to upgrade (Z) ``` +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ### Rule upgrade workflow: individual updates from Rule Updates table #### **Scenario: User can upgrade conflict-free prebuilt rules one by one** +======= +### Rule upgrade workflow: base cases + +#### **Scenario: User can upgrade prebuilt rules one by one** +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. @@ -256,6 +351,7 @@ And the upgraded rule should be removed from the table And user should see the number of rules available to upgrade decreased by 1 ``` +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts** **Automation**: 1 e2e test with mock rules @@ -274,16 +370,27 @@ And there should be a message/tooltip indicating why the rule cannot be upgraded ### Rule upgrade workflow: bulk updates from Rule Updates table #### **Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page** +======= +#### **Scenario: User can upgrade multiple prebuilt rules selected on the page** +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. ```Gherkin Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md And user opens the Rule Updates table Then Y rules available for upgrade should be displayed in the table When user selects Z (where Z < Y) rules, which have no upgrade conflicts Then user should see a CTA to upgrade rules +======= +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user selects rules +Then user should see a CTA to upgrade number of rules +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md When user clicks the CTA Then success message should be displayed after upgrade And all the upgraded rules should be removed from the table @@ -295,6 +402,7 @@ Examples: | all rules on the page, e.g. 12 | ``` +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User cannot upgrade selected prebuilt rules with conflicts** **Automation**: 1 e2e test with mock rules @@ -318,13 +426,19 @@ Examples: ``` #### **Scenario: User can upgrade all available conflict-free prebuilt rules at once** +======= +#### **Scenario: User can upgrade all available prebuilt rules at once** +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. ```Gherkin Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md And those Y new versions don't have conflicts with the current versions +======= +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md And user is on the Rule Management page When user opens the Rule Updates table Then Y rules available for upgrade should be displayed in the table @@ -332,6 +446,7 @@ When user upgrades all rules Then success message should be displayed after upgrade And user should NOT see a CTA to upgrade prebuilt rules And user should NOT see a number of rules available to upgrade +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ``` #### **Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts** @@ -451,6 +566,11 @@ And all X upgraded rules should be removed from the table ### Rule upgrade workflow: rule previews +======= +And user should NOT see the Rule Updates table +``` + +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can preview rules available for upgrade** ```Gherkin @@ -522,9 +642,13 @@ And the Investigation Guide tab should NOT be displayed TODO: add scenarios https://github.com/elastic/kibana/issues/166215 +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in JSON diff view > These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. +======= +### Rule upgrade workflow: viewing rule changes in JSON diff view +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can see changes in a side-by-side JSON diff view** @@ -670,9 +794,13 @@ When a user expands all hidden sections Then all properties of the rule should be sorted alphabetically ``` +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in per-field diff view > These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. +======= +### Rule upgrade workflow: viewing rule changes in per-field diff view +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can see changes in a side-by-side per-field diff view** @@ -810,7 +938,11 @@ Then user should NOT see the Rule Updates tab until the package installation is ### Error handling +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: Error is handled when any upgrade operation on prebuilt rules fails** +======= +#### **Scenario: Error is handled when any operation on prebuilt rules fails** +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md **Automation**: e2e test with mock rules @@ -821,6 +953,12 @@ Then user should see an error message Examples: | operation | +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +======= + | installing all | + | installing selected | + | installing individual | +>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md | upgrading all | | upgrading selected | | upgrading individual | From 898cabfee80f2879c9f847232a401d9d2a6a7828 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Mon, 9 Dec 2024 22:27:13 -0300 Subject: [PATCH 03/18] Edited test plans --- .../prebuilt_rules/installation.md | 7 ++ .../prebuilt_rules/upgrade.md | 72 +++++++++++++++++++ 2 files changed, 79 insertions(+) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md index 37b61f3c92740..7b719ba2956b4 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md @@ -614,11 +614,15 @@ Notes: ### Error handling +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md #### **Scenario: Error is handled when any installation operation on prebuilt rules fails** ======= #### **Scenario: Error is handled when any operation on prebuilt rules fails** >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +#### **Scenario: Error is handled when any installation operation on prebuilt rules fails** +>>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md **Automation**: e2e test with mock rules @@ -633,11 +637,14 @@ Examples: | installing selected | | installing individual | <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ======= | upgrading all | | upgrading selected | | upgrading individual | >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +>>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ``` ### Authorization / RBAC diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index 4bbe70355bed0..61b6462673845 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -73,6 +73,7 @@ Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule ======== - [Scenarios](#scenarios) +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md - [Package installation](#package-installation) - [**Scenario: Package is installed via Fleet**](#scenario-package-is-installed-via-fleet) - [**Scenario: Package is installed via bundled Fleet package in Kibana**](#scenario-package-is-installed-via-bundled-fleet-package-in-kibana) @@ -154,6 +155,49 @@ Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule - [Authorization / RBAC](#authorization--rbac) - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +======= + - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) + - [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date) + - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) + - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) + - [Rule upgrade workflow: individual and bulk updates from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) + - [**Scenario: User can upgrade prebuilt rules one by one**](#scenario-user-can-upgrade-prebuilt-rules-one-by-one) + - [**Scenario: User can upgrade multiple prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-prebuilt-rules-selected-on-the-page) + - [**Scenario: User can upgrade all available prebuilt rules at once**](#scenario-user-can-upgrade-all-available-prebuilt-rules-at-once) + - [Rule upgrade workflow: rule previews](#rule-upgrade-workflow-rule-previews) + - [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade) + - [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview) + - [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading) + - [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading) + - [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination) + - [Rule upgrade workflow: preserving rule bound data](#rule-upgrade-workflow-preserving-rule-bound-data) + - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-the-same-rule-type) + - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-a-different-rule-type) + - [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases) + - [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed) + - [Error handling](#error-handling) + - [**Scenario: Error is handled when any upgrade operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-upgrade-operation-on-prebuilt-rules-fails) + - [Rule upgrade via the Prebuilt rules API](#rule-upgrade-via-the-prebuilt-rules-api) + - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) + - [**Scenario: API does not upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-upgrade-prebuilt-rules-if-they-are-up-to-date) + - [Authorization / RBAC](#authorization-rbac) + - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) + - [MILESTONE 2 - Rule upgrade workflow: viewing rule changes in JSON diff view](#rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view) + - [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view) + - [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade) + - [**Scenario: Rule actions and exception lists should not be shown as modified**](#scenario-rule-actions-and-exception-lists-should-not-be-shown-as-modified) + - [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview) + - [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview) + - [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified) + - [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default) + - [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically) + - [MILESTONE 2 - Rule upgrade workflow: viewing rule changes in per-field diff view](#rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view) + - [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view) + - [**Scenario: User can see changes when updated rule is a different rule type**](#scenario-user-can-see-changes-when-updated-rule-is-a-different-rule-type) + - [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel) + - [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs) + - [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview) +>>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ## Useful information @@ -327,12 +371,16 @@ And user should see a CTA to upgrade prebuilt rules And user should see the number of rules available to upgrade (Z) ``` +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ### Rule upgrade workflow: individual updates from Rule Updates table #### **Scenario: User can upgrade conflict-free prebuilt rules one by one** ======= ### Rule upgrade workflow: base cases +======= +### Rule upgrade workflow: individual and bulk updates from Rule Updates table +>>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can upgrade prebuilt rules one by one** >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -570,7 +618,12 @@ And all X upgraded rules should be removed from the table And user should NOT see the Rule Updates table ``` +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +======= +### Rule upgrade workflow: rule previews + +>>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can preview rules available for upgrade** ```Gherkin @@ -642,6 +695,7 @@ And the Investigation Guide tab should NOT be displayed TODO: add scenarios https://github.com/elastic/kibana/issues/166215 +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in JSON diff view @@ -649,6 +703,11 @@ TODO: add scenarios https://github.com/elastic/kibana/issues/166215 ======= ### Rule upgrade workflow: viewing rule changes in JSON diff view >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +======= +### MILESTONE 2 - Rule upgrade workflow: viewing rule changes in JSON diff view + +> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. +>>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can see changes in a side-by-side JSON diff view** @@ -794,6 +853,7 @@ When a user expands all hidden sections Then all properties of the rule should be sorted alphabetically ``` +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in per-field diff view @@ -801,6 +861,11 @@ Then all properties of the rule should be sorted alphabetically ======= ### Rule upgrade workflow: viewing rule changes in per-field diff view >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +======= +### MILESTONE 2 - Rule upgrade workflow: viewing rule changes in per-field diff view + +> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. +>>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can see changes in a side-by-side per-field diff view** @@ -938,11 +1003,15 @@ Then user should NOT see the Rule Updates tab until the package installation is ### Error handling +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: Error is handled when any upgrade operation on prebuilt rules fails** ======= #### **Scenario: Error is handled when any operation on prebuilt rules fails** >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +======= +#### **Scenario: Error is handled when any upgrade operation on prebuilt rules fails** +>>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md **Automation**: e2e test with mock rules @@ -954,11 +1023,14 @@ Then user should see an error message Examples: | operation | <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ======= | installing all | | installing selected | | installing individual | >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +======= +>>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md | upgrading all | | upgrading selected | | upgrading individual | From 4e29bda8389d4ab22c52a6013ea15de0792dce15 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 10 Dec 2024 00:00:23 -0300 Subject: [PATCH 04/18] Refactor test plans --- .../prebuilt_rules/upgrade.md | 865 ++++++++++++++++++ 1 file changed, 865 insertions(+) create mode 100644 x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md new file mode 100644 index 0000000000000..ab88828edb5e0 --- /dev/null +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -0,0 +1,865 @@ +# Upgrade of Prebuilt Rules + +This is a test plan for the workflow of upgrading prebuilt rules. + +Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. + +## Table of Contents + +- [Useful information](#useful-information) + - [Tickets](#tickets) + - [Terminology](#terminology) + - [Assumptions](#assumptions) + - [Non-functional requirements](#non-functional-requirements) + - [Functional requirements](#functional-requirements) + - [Scenarios](#scenarios) + - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) + - [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date) + - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) + - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) + - [Rule upgrade workflow: individual and bulk updates from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) + - [**Scenario: User can upgrade conflict-free prebuilt rules one by one**](#scenario-user-can-upgrade-conflict-free-prebuilt-rules-one-by-one) + - [**Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts**](#scenario-user-cannot-upgrade-prebuilt-rules-one-by-one-from-rules-update-table-if-they-have-conflicts) + - [**Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-conflict-free-prebuilt-rules-selected-on-the-page) + - [**Scenario: User cannot upgrade multiple prebuilt rules selected on the page when they have upgrade conflicts**](#scenario-user-cannot-upgrade-multiple-prebuilt-rules-selected-on-the-page-when-they-have-upgrade-conflicts) + - [**Scenario: User can upgrade all available conflict-free prebuilt rules at once**](#scenario-user-can-upgrade-all-available-conflict-free-prebuilt-rules-at-once) + - [**Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts**](#scenario-user-cannot-upgrade-all-prebuilt-rules-at-once-if-they-have-upgrade-conflicts) + - [**Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade in the Rules Table**](#scenario-user-can-upgrade-only-conflict-free-rules-when-a-mix-of-rules-with-and-without-conflicts-are-selected-for-upgrade-in-the-rules-table) + - [**Scenario: User can upgrade only conflict-free rules when user attempts to upgrade all rules and only a subset contains upgrade conflicts**](#scenario-user-can-upgrade-only-conflict-free-rules-when-user-attempts-to-upgrade-all-rules-and-only-a-subset-contains-upgrade-conflicts) + - [Rule upgrade workflow: rule previews](#rule-upgrade-workflow-rule-previews) + - [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade) + - [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview) + - [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading) + - [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading) + - [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination) + - [MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in JSON diff view](#milestone-2-legacy---rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view) + - [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view) + - [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade) + - [**Scenario: Rule actions and exception lists should not be shown as modified**](#scenario-rule-actions-and-exception-lists-should-not-be-shown-as-modified) + - [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview) + - [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview) + - [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified) + - [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default) + - [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically) + - [MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in per-field diff view](#milestone-2-legacy---rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view) + - [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view) + - [**Scenario: User can see changes when updated rule is a different rule type**](#scenario-user-can-see-changes-when-updated-rule-is-a-different-rule-type) + - [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel) + - [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs) + - [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview) + - [Rule upgrade workflow: preserving rule bound data](#rule-upgrade-workflow-preserving-rule-bound-data) + - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-the-same-rule-type) + - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-a-different-rule-type) + - [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases) + - [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed) + - [Error handling](#error-handling) + - [**Scenario: Error is handled when any upgrade operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-upgrade-operation-on-prebuilt-rules-fails) + - [Rule upgrade via the Prebuilt rules API](#rule-upgrade-via-the-prebuilt-rules-api) + - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) + - [**Scenario: API does not upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-upgrade-prebuilt-rules-if-they-are-up-to-date) + - [Authorization / RBAC](#authorization-rbac) + - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) + + +## Useful information + +### Tickets + +- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic + +**Milestone 3 - Prebuilt Rules Customization:** +- [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) +- [Tests for prebuilt rule upgrade workflow #202078](https://github.com/elastic/kibana/issues/202078) + +**Milestone 2:** +- [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) +- [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) + +### Terminology + +- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**. + +- **Package**: `security_detection_engine` Fleet package that we use to distribute prebuilt detection rules in the form of `security-rule` assets (saved objects). + +- **Real package**: actual latest stable package distributed and pulled from EPR via Fleet. + +- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly in the test setup, either by using the ES client _in integration tests_ or by an API request _in Cypress tests_. + +- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. + +- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. + - CTA to install prebuilt rules - at this moment, it's a link button with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. + - CTA to upgrade prebuilt rules - at this moment, it's a tab with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. + +### Assumptions + +- Below scenarios only apply to prebuilt detection rules. +- Users should be able to install and upgrade prebuilt rules on the `Basic` license and higher. +- EPR is available for fetching the package unless explicitly indicated otherwise. +- Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored. + +### Non-functional requirements + +- Notifications, rule installation and rule upgrade workflows should work: + - regardless of the package type: with historical rule versions or without; + - regardless of the package registry availability: i.e., they should also work in air-gapped environments. +- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet. +- Kibana should not crash with Out Of Memory exception during package installation. +- For test purposes, it should be possible to use detection rules package versions lower than the latest. + +### Functional requirements + +- User should be able to install prebuilt rules with and without previewing what exactly they would install (rule properties). +- User should be able to upgrade prebuilt rules with and without previewing what updates they would apply (rule properties of target rule versions). +- If user chooses to preview a prebuilt rule to be installed/upgraded, we currently show this preview in a flyout. +- In the prebuilt rule preview a tab that doesn't have any sections should not be displayed and a section that doesn't have any properties also should not be displayed. + +Examples of rule properties we show in the prebuilt rule preview flyout: + +```Gherkin +Examples: +| rule_type | property | tab | section | +│ All rule types │ Author │ Overview │ About │ +│ All rule types │ Building block │ Overview │ About │ +│ All rule types │ Severity │ Overview │ About │ +│ All rule types │ Severity override │ Overview │ About │ +│ All rule types │ Risk score │ Overview │ About │ +│ All rule types │ Risk score override │ Overview │ About │ +│ All rule types │ Reference URLs │ Overview │ About │ +│ All rule types │ False positive examples │ Overview │ About │ +│ All rule types │ Custom highlighted fields │ Overview │ About │ +│ All rule types │ License │ Overview │ About │ +│ All rule types │ Rule name override │ Overview │ About │ +│ All rule types │ MITRE ATT&CK™ │ Overview │ About │ +│ All rule types │ Timestamp override │ Overview │ About │ +│ All rule types │ Tags │ Overview │ About │ +│ All rule types │ Type │ Overview │ Definition │ +│ All rule types │ Related integrations │ Overview │ Definition │ +│ All rule types │ Required fields │ Overview │ Definition │ +│ All rule types │ Timeline template │ Overview │ Definition │ +│ All rule types │ Runs every │ Overview │ Schedule │ +│ All rule types │ Additional look-back time │ Overview │ Schedule │ +│ All rule types │ Setup guide │ Overview │ Setup guide │ +│ All rule types │ Investigation guide │ Investigation guide │ Investigation guide │ +│ Custom Query │ Index patterns │ Overview │ Definition │ +│ Custom Query │ Data view ID │ Overview │ Definition │ +│ Custom Query │ Data view index pattern │ Overview │ Definition │ +│ Custom Query │ Custom query │ Overview │ Definition │ +│ Custom Query │ Filters │ Overview │ Definition │ +│ Custom Query │ Saved query name │ Overview │ Definition │ +│ Custom Query │ Saved query filters │ Overview │ Definition │ +│ Custom Query │ Saved query │ Overview │ Definition │ +│ Custom Query │ Suppress alerts by │ Overview │ Definition │ +│ Custom Query │ Suppress alerts for │ Overview │ Definition │ +│ Custom Query │ If a suppression field is missing │ Overview │ Definition │ +│ Machine Learning │ Anomaly score threshold │ Overview │ Definition │ +│ Machine Learning │ Machine Learning job │ Overview │ Definition │ +│ Threshold │ Threshold │ Overview │ Definition │ +│ Threshold │ Index patterns │ Overview │ Definition │ +│ Threshold │ Data view ID │ Overview │ Definition │ +│ Threshold │ Data view index pattern │ Overview │ Definition │ +│ Threshold │ Custom query │ Overview │ Definition │ +│ Threshold │ Filters │ Overview │ Definition │ +│ Event Correlation │ EQL query │ Overview │ Definition │ +│ Event Correlation │ Filters │ Overview │ Definition │ +│ Event Correlation │ Index patterns │ Overview │ Definition │ +│ Event Correlation │ Data view ID │ Overview │ Definition │ +│ Event Correlation │ Data view index pattern │ Overview │ Definition │ +│ Indicator Match │ Indicator index patterns │ Overview │ Definition │ +│ Indicator Match │ Indicator mapping │ Overview │ Definition │ +│ Indicator Match │ Indicator filters │ Overview │ Definition │ +│ Indicator Match │ Indicator index query │ Overview │ Definition │ +│ Indicator Match │ Index patterns │ Overview │ Definition │ +│ Indicator Match │ Data view ID │ Overview │ Definition │ +│ Indicator Match │ Data view index pattern │ Overview │ Definition │ +│ Indicator Match │ Custom query │ Overview │ Definition │ +│ Indicator Match │ Filters │ Overview │ Definition │ +│ New Terms │ Fields │ Overview │ Definition │ +│ New Terms │ History Window Size │ Overview │ Definition │ +│ New Terms │ Index patterns │ Overview │ Definition │ +│ New Terms │ Data view ID │ Overview │ Definition │ +│ New Terms │ Data view index pattern │ Overview │ Definition │ +│ New Terms │ Custom query │ Overview │ Definition │ +│ New Terms │ Filters │ Overview │ Definition │ +│ ESQL │ ESQL query │ Overview │ Definition │ +│ ESQL │ Suppress alerts by │ Overview │ Definition │ +│ ESQL │ Suppress alerts for │ Overview │ Definition │ +│ ESQL │ If a suppression field is missing │ Overview │ Definition │ +``` + +## Scenarios + +### Rule installation and upgrade notifications on the Rule Management page + +#### **Scenario: User is NOT notified when all installed prebuilt rules are up to date** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given all the latest prebuilt rules are installed in Kibana +When user opens the Rule Management page +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table +``` + +#### **Scenario: User is notified when some prebuilt rules can be upgraded** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And there are no more prebuilt rules available to install +And for Z of the installed rules there are new versions available +When user opens the Rule Management page +Then user should NOT see a CTA to install prebuilt rules +And user should NOT see a number of rules available to install +And user should see a CTA to upgrade prebuilt rules +And user should see the number of rules available to upgrade (Z) +``` + +#### **Scenario: User is notified when both rules to install and upgrade are available** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And there are Y more prebuilt rules available to install +And for Z of the installed rules there are new versions available +When user opens the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install (Y) +And user should see a CTA to upgrade prebuilt rules +And user should see the number of rules available to upgrade (Z) +``` + +### Rule upgrade workflow: individual and bulk updates from Rule Updates table + +#### **Scenario: User can upgrade conflict-free prebuilt rules one by one** + +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user upgrades one individual rule without previewing it +Then success message should be displayed after upgrade +And the upgraded rule should be removed from the table +And user should see the number of rules available to upgrade decreased by 1 +``` + +#### **Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts** + +**Automation**: 1 e2e test with mock rules + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +And for Z of the rules with upgrades there are upgrade conflicts +Then for those Z rules the Upgrade Rule button should be disabled +And the user should not be able to upgrade them directly from the table +And there should be a message/tooltip indicating why the rule cannot be updated directly +``` + +#### **Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page** + +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user selects rules, which have no upgrade conflicts +Then user should see a CTA to upgrade number of rules +When user clicks the CTA +Then success message should be displayed after upgrade +And all the upgraded rules should be removed from the table +And user should see the number of rules available to upgrade decreased by number of upgraded rules + +Examples: + | Z | + | a few rules on the page, e.g. 2 | + | all rules on the page, e.g. 12 | +``` + +#### **Scenario: User cannot upgrade multiple prebuilt rules selected on the page when they have upgrade conflicts** + +**Automation**: 1 e2e test with mock rules + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And all of those Y new versions have conflicts with the current versions +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user selects rules, all of which have upgrade conflicts +Then user should see a CTA to upgrade number of rules, which should be disabled +When user hovers on the CTA to upgrade multiple rules +Then a message should be displayed that informs the user why the rules cannot be updated + +Examples: + | Z | + | a few rules on the page, e.g. 2 | + | all rules on the page, e.g. 12 | +``` + +#### **Scenario: User can upgrade all available conflict-free prebuilt rules at once** + +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And those Y new versions don't have conflicts with the current versions +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user upgrades all rules +Then success message should be displayed after upgrade +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +``` + +#### **Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts** + +**Automation**: 1 e2e test with mock rules + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And all Y new versions have conflicts with the current versions +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +Then user should see a CTA to upgrade all rules +And the CTA to upgrade all rules should be disabled +When user hovers on the CTA to upgrade all rules +Then a message should be displayed that informs the user why the rules cannot be updated +``` + +#### **Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade in the Rules Table** + +**Automation**: 1 e2e test with mock rules + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And a subset Z of the rules have conflicts with the current versions +And user is on the Rule Management page +Then Y rules available for upgrade should be displayed in the table +And user selects rules, which is a mixture of rules with and without upgrade conflicts +Then user should see a CTA to upgrade number of rules, which is enabled +When user clicks the CTA +A modal window should inform the user that only W rules without conflicts will be upgraded +When user confirms the action in the modal +Then success message should be displayed after upgrade informing that W rules were updated +And the W upgraded rules should be removed from the table +And the remaining Z - W rules should still be present in the table +And user should see the number of rules available to upgrade decreased by W number of upgraded rules + +Examples: + | Z | + | a few rules on the page, e.g. 2 | + | all rules on the page, e.g. 12 | +``` + +#### **Scenario: User can upgrade only conflict-free rules when user attempts to upgrade all rules and only a subset contains upgrade conflicts** + +**Automation**: 1 e2e test with mock rules + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And a subset Z of the rules have conflicts with the current versions +And user is on the Rule Management page +Then Y rules available for upgrade should be displayed in the table +Then user should see an enabled CTA to upgrade all rules +When user clicks the CTA +A modal window should inform the user that only Z rules without conflicts will be upgraded +When user confirms the action in the modal +Then success message should be displayed after upgrade informing that Z rules were updated +And the Z upgraded rules should be removed from the table +And the remaining Y - Z rules should still be present in the table +And user should see the number of rules available to upgrade decreased by Z number of upgraded rules +``` + +### Rule upgrade workflow: rule previews + +#### **Scenario: User can preview rules available for upgrade** + +```Gherkin +Given there is at least one prebuilt rule installed in Kibana +And for this rule there is a new version available +And user is on the Rule Management page +When user opens the Rule Updates table +Then this rule should be displayed in the table +When user opens the rule preview for this rule +Then the preview should open +When user closes the preview +Then it should disappear +``` + +#### **Scenario: User can upgrade a rule using the rule preview** + +**Automation**: 1 e2e test + +```Gherkin +Given there is at least one prebuilt rule installed in Kibana +And for this rule there is a new version available +And user is on the Rule Management page +When user opens the Rule Updates table +Then this rule should be displayed in the table +When user opens the rule preview for this rule +Then the preview should open +When user upgrades the rule using a CTA in the rule preview +Then the rule should be upgraded to the latest version +And a success message should be displayed after upgrade +And the rule should be removed from the Rule Updates table +And user should see the number of rules available to upgrade as initial number minus 1 +``` + +#### **Scenario: User can see correct rule information in preview before upgrading** + +**Automation**: 1 e2e test + +```Gherkin +Given X prebuilt rules of all types are installed in Kibana +And for all of the installed rules there are new versions available +And user is on the Rule Management page +When user opens the Rule Updates table +Then all X rules available for upgrade should be displayed in the table +When user opens a rule preview for any rule +Then the preview should appear +And the "Updates" tab should be active +When user selects the "Overview" tab +Then all properties of the new version of a rule should be displayed in the correct tab and section of the preview (see examples of rule properties above) +``` + +#### **Scenario: Tabs and sections without content should be hidden in preview before upgrading** + +**Automation**: 1 e2e test + +```Gherkin +Given at least 1 prebuilt rule is installed in Kibana +And for this rule there is a new version available +And the updated version of a rule has neither Setup guide nor Investigation guide +And user is on the Rule Management page +When user opens the Rule Updates table +Then all rules available for upgrade should be displayed in the table +When user opens the rule preview for a rule without Setup guide and Investigation guide +Then the preview should open +And the Setup Guide section should NOT be displayed in the Overview tab +And the Investigation Guide tab should NOT be displayed +``` + +### Rule upgrade workflow: filtering, sorting, pagination + +TODO: add scenarios https://github.com/elastic/kibana/issues/166215 + +### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in JSON diff view + +> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. + +#### **Scenario: User can see changes in a side-by-side JSON diff view** + +**Automation**: 1 e2e test + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of these rules new versions are available +When user opens the Rule Updates table and selects a rule +Then the upgrade preview should open +And rule changes should be displayed in a two-column JSON diff view +And correct rule version numbers should be displayed in their respective columns +When the user selects another rule without closing the preview +Then the preview should display the changes for the newly selected rule +``` + +#### **Scenario: User can see precisely how property values would change after upgrade** + +**Automation**: 1 UI integration test + +```Gherkin +Given a rule preview with rule changes is open +Then each line of that was should have background +And marked with badge +And each changed word in should be highlighted with + +Examples: +| change_type | column | bg_color | accent_color | line_badge | +| updated | Current rule | removed_bg_color | removed_accent_color | - | +| updated | Elastic update | added_bg_color | added_accent_color | + | +| removed | Current rule | removed_bg_color | none | - | +| removed | Elastic update | none | none | none | +| added | Current rule | none | none | none | +| added | Elastic update | added_bg_color | none | + | +``` + +#### **Scenario: Rule actions and exception lists should not be shown as modified** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And the currently installed version of this rule doesn't have any actions or an exception list +And a user has set up actions and an exception list for this rule +And this rule has an update available +And the update doesn't define any actions or an exception list +When a user opens the upgrade preview for this rule +Then the preview should open +And the JSON diff shouldn't show any modifications to rule's actions or exception list +``` + +#### **Scenario: Dynamic properties should not be included in preview** + +**Automation**: 1 e2e test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule is disabled by default +And a user has enabled this rule +And this rule executed at least once +And this rule has an update available +When user opens the upgrade preview +Then the preview should open +And the JSON diff shouldn't show any properties on both sides + +Examples: +| property | +| execution_summary | +| enabled | +``` + +#### **Scenario: Technical properties should not be included in preview** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has an update available +When a user opens the upgrade preview +Then the preview should open +And the JSON diff shouldn't show any properties on both sides + +Examples: +| technical_property | +| revision | +| updated_at | +| updated_by | +| created_at | +| created_by | +``` + +#### **Scenario: Properties with semantically equal values should not be shown as modified** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has an update available +And the update has properties with different, but semantically equal values +When a user opens the upgrade preview +Then the preview should open +And the JSON diff shouldn't show any changes to properties with semantically equal values + +Duration examples: +| 1h | +| 60m | +| 3600s | + +Empty value examples: +| no value | +| '' | +| [] | +| undefined | +| null | +``` + +#### **Scenario: Unchanged sections of a rule should be hidden by default** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has an update available +When a user opens the upgrade preview +Then the preview should open +And only the sections of the diff that have changes should be visible +And unchanged sections should be hidden behind a button with a number of unchanged lines +When a user clicks on the hidden section button +Then the section should expand and show the unchanged properties +``` + +#### **Scenario: Properties should be sorted alphabetically** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has an update available +When a user opens the upgrade preview +Then the preview should open +And visible properties should be sorted alphabetically +When a user expands all hidden sections +Then all properties of the rule should be sorted alphabetically +``` + +### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in per-field diff view + +> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. + +#### **Scenario: User can see changes in a side-by-side per-field diff view** + +**Automation**: 1 e2e test + +```Gherkin +Given X prebuilt rules are installed in Kibana +And for Y of these rules new versions are available +When user opens the Rule Updates table and selects a rule +Then the per-field upgrade preview should open +And rule changes should be displayed in a two-column diff view with each field in its own accordion component +And all field diff accordions should be open by default +And correct rule version numbers should be displayed in their respective columns +When the user selects another rule without closing the preview +Then the preview should display the changes for the newly selected rule +``` + +#### **Scenario: User can see changes when updated rule is a different rule type** + +**Automation**: 1 e2e test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has an update available that changes the rule type +When user opens the upgrade preview +Then the rule type changes should be displayed in grouped field diffs with corresponding query fields +# When tooltip enhancement is added, this step needs to be added to the corresponding test scenario +And a tooltip is displayed with information about changing rule types +``` + +#### **Scenario: Field groupings should be rendered together in the same accordion panel** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule contains one or more values +When user opens the upgrade preview +The diff accordion panel should display its grouped rule properties +And each property should have its name displayed inside the panel above its value + +Examples: +| field | +| data_source | +| kql_query | +| eql_query | +| esql_query | +| threat_query | +| rule_schedule | +| rule_name_override | +| timestamp_override | +| timeline_template | +| building_block | +| threshold | +``` + +#### **Scenario: Undefined values are displayed with empty diffs** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has field in the version that didn't exist in the version +When a user opens the upgrade preview +Then the preview should open +And the old/new field should render an empty panel + +Examples: +| version_one | version_two | +| target | current | +| current | target | +``` + +#### **Scenario: Field diff components have the same grouping and order as in rule details overview** + +**Automation**: 1 UI integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has multiple fields that are different between the current and target version +When a user opens the upgrade preview +Then the multiple field diff accordions should be sorted in the same order as on the rule details overview tab +And the field diff accordions should be grouped inside its corresponding
accordion +And any
accordion that doesn't have fields inside it shouldn't be displayed + +Examples: +| section | +| About | +| Definition | +| Schedule | +| Setup Guide | +``` + +### Rule upgrade workflow: preserving rule bound data + +#### **Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type** + +**Automation**: 1 unit test per case, 1 integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has an update available +And the update has the same rule type +When a user upgrades the rule +Then the rule bound data should be preserved +``` + +Examples: generated alerts, exception lists (rule exception list, shared exception list, endpoint exception list), timeline reference, actions, enabled state, execution results and execution events. + +#### **Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type** + +**Automation**: 1 unit test per case, 1 integration test + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has an update available +And the update has a different rule type +When a user upgrades the rule +Then the rule bound data should be preserved +``` + +Examples: generated alerts, exception lists (rule exception list, shared exception list, endpoint exception list), timeline reference, actions, enabled state, execution results and execution events. + +### Rule upgrade workflow: misc cases + +#### **Scenario: User doesn't see the Rule Updates tab until the package installation is completed** + +**Automation**: unit tests. + +```Gherkin +Given prebuilt rules package is not installed +When user opens the Rule Management page +Then user should NOT see the Rule Updates tab until the package installation is completed and there are rules available for upgrade +``` + +### Error handling + +#### **Scenario: Error is handled when any upgrade operation on prebuilt rules fails** + +**Automation**: e2e test with mock rules + +```Gherkin +When user is prebuilt rules +And this operation fails +Then user should see an error message + +Examples: + | operation | + | upgrading all | + | upgrading selected | + | upgrading individual | +``` + + +### Rule upgrade via the Prebuilt rules API + +There's a legacy prebuilt rules API and a new one. Both should be tested against two types of the package: with and without historical rule versions. + +#### **Scenario: API can upgrade prebuilt rules that are outdated** + +**Automation**: 4 integration tests with mock rules. + +```Gherkin +Given the package is installed +And the package contains N rules +When user installs all rules via install +And new X+1 version of a rule asset +And user gets prebuilt rules status via status +Then the endpoint should return 200 with +When user upgrades all rules via upgrade +Then the endpoint should return 200 with + +Examples: + | package_type | api | assets_update | status_response | upgrade_response | + | with historical versions | legacy | gets added | not_updated: 1 | installed: 0, updated: 1 | + | w/o historical versions | legacy | replaces X | not_updated: 1 | installed: 0, updated: 1 | + | with historical versions | new | gets added | to_upgrade: 1 | total: 1, succeeded: 1 | + | w/o historical versions | new | replaces X | to_upgrade: 1 | total: 1, succeeded: 1 | +``` + +TODO: Check why for the legacy API Dmitrii has added 2 integration tests for `rule package with historical versions` instead of 1: + +- `should update outdated prebuilt rules when previous historical versions available` +- `should update outdated prebuilt rules when previous historical versions unavailable` + +(NOTE: the second scenario tests that, if a new version of a rule is released, it can upgrade the current instance of that rule even if the historical versions of that rule are no longer in the package) + +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` + - upgrade: `PUT /api/detection_engine/rules/prepackaged` + - status: `GET /api/detection_engine/rules/prepackaged/_status` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` + - status: `GET /internal/detection_engine/prebuilt_rules/status` + +#### **Scenario: API does not upgrade prebuilt rules if they are up to date** + +**Automation**: 4 integration tests with mock rules. + +```Gherkin +Given the package is installed +And the package contains N rules +When user installs all rules via install +And user gets prebuilt rules status via status +Then the endpoint should return 200 with +When user calls install +Then the endpoint should return 200 with +When user calls upgrade +Then the endpoint should return 200 with + +Examples: + | package_type | api | status_response | install_response | upgrade_response | + | with historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | + | w/o historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | + | with historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | + | w/o historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | +``` + +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` + - upgrade: `PUT /api/detection_engine/rules/prepackaged` + - status: `GET /api/detection_engine/rules/prepackaged/_status` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` + - status: `GET /internal/detection_engine/prebuilt_rules/status` + +### Authorization / RBAC + +#### **Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules** + +**Automation**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and upgrade endpoints. + +```Gherkin +Given user with "Security: read" privileges on Security Solution +And X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +When user opens the Rule Management page +And user opens the Rule Updates table +Then user should see prebuilt rules available to upgrade +But user should not be able to upgrade them +``` \ No newline at end of file From ab4931587df11778f603259d4b8b9338e1e30158 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 17 Dec 2024 15:35:44 -0300 Subject: [PATCH 05/18] Apply suggestions --- .../prebuilt_rules/installation.md | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md index 7b719ba2956b4..1706e34178811 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md @@ -2,11 +2,15 @@ This is a test plan for the workflows of installing prebuilt rules. +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). ======= Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). +>>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ## Table of Contents @@ -50,11 +54,15 @@ Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule ### Tickets +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md - [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) ======= - [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +- [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) +>>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md **Milestone 3 - Prebuilt Rules Customization:** - [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) @@ -185,12 +193,17 @@ Examples: ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given no prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= Given no prebuilt rules are installed in Kibana And no prebuilt rule assets exist >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given no prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Rule Management page Then user should NOT see a CTA to install prebuilt rules And user should NOT see a number of rules available to install @@ -205,11 +218,16 @@ And user should NOT see the Rule Updates table ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given the latest prebuilt rule assets exist in Kibana And all the latest prebuilt rules from those rule assets are installed ======= Given all the latest prebuilt rules are installed in Kibana >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given the latest prebuilt rule assets exist in Kibana +And all the latest prebuilt rules from those rule assets are installed +>>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Rule Management page Then user should NOT see a CTA to install prebuilt rules And user should NOT see a number of rules available to install @@ -224,11 +242,16 @@ And user should NOT see the Rule Updates table ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= Given no prebuilt rules are installed in Kibana >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And there are X prebuilt rules available to install When user opens the Rule Management page Then user should see a CTA to install prebuilt rules @@ -244,11 +267,16 @@ And user should NOT see the Rule Updates table ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given Y prebuilt rule assets exist in Kibana And X (where X < Y) prebuilt rules are installed ======= Given X prebuilt rules are installed in Kibana >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given Y prebuilt rule assets exist in Kibana +And X (where X < Y) prebuilt rules are installed +>>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And there are Y more prebuilt rules available to install And for all X installed rules there are no new versions available When user opens the Rule Management page @@ -265,6 +293,7 @@ And user should NOT see the Rule Updates table ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given Y prebuilt rule assets exist in Kibana And X (where X < Y) prebuilt rules are installed And Z (where Z < X) installed rules have matching prebuilt rule assets with higher version available @@ -272,6 +301,11 @@ And Z (where Z < X) installed rules have matching prebuilt rule assets with hig Given X prebuilt rules are installed in Kibana And there are Y more prebuilt rules available to install >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given Y prebuilt rule assets exist in Kibana +And X (where X < Y) prebuilt rules are installed +And Z (where Z < X) installed rules have matching prebuilt rule assets with higher version available +>>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And for Z of the installed rules there are new versions available When user opens the Rule Management page Then user should see a CTA to install prebuilt rules @@ -291,10 +325,14 @@ When user opens the Rule Management page And user deletes Y prebuilt rules Then user should see a CTA to install prebuilt rules <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And user should see rules available to install ======= And user should see the number of rules available to install (Y) >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +And user should see rules available to install +>>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ``` ### Rule installation workflow: base cases From 66746dd8244ea6c517404b2aeaef070f6bb76579 Mon Sep 17 00:00:00 2001 From: Juan Pablo Djeredjian Date: Tue, 17 Dec 2024 15:41:38 -0300 Subject: [PATCH 06/18] Apply suggestion for installation.md Co-authored-by: Maxim Palenov --- .../prebuilt_rules/installation.md | 60 ++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md index 1706e34178811..a03b4f3cd88fc 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md @@ -343,12 +343,17 @@ And user should see rules available to install ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= Given no prebuilt rules are installed in Kibana And there are X prebuilt rules available to install >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then prebuilt rules available for installation should be displayed in the table When user installs one individual rule without previewing it @@ -365,12 +370,17 @@ And user should see the number of rules available to install decreased by 1 ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= Given no prebuilt rules are installed in Kibana And there are X prebuilt rules available to install >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then prebuilt rules available for installation should be displayed in the table When user selects rules @@ -394,12 +404,17 @@ Examples: ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= Given no prebuilt rules are installed in Kibana And there are X prebuilt rules available to install >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then prebuilt rules available for installation should be displayed in the table When user installs all rules @@ -430,12 +445,17 @@ And user should see a CTA that leads to the Rule Management page ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given 2 prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= Given no prebuilt rules are installed in Kibana And there are 2 rules available to install >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given 2 prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then all rules available for installation should be displayed in the table When user opens the rule preview for the 1st rule @@ -450,12 +470,17 @@ Then it should disappear ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given 2 prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= Given no prebuilt rules are installed in Kibana And there are 2 rules available to install >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given 2 prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then all rules available for installation should be displayed in the table When user opens the rule preview for the rule @@ -475,12 +500,17 @@ And user should see the number of rules available to install as initial number m ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= Given no prebuilt rules are installed in Kibana And there are X prebuilt rules of all types available to install >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given X prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then all X rules available for installation should be displayed in the table When user opens a rule preview for any rule @@ -494,12 +524,17 @@ And all properties of a rule should be displayed in the correct tab and section ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given 1 prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= Given no prebuilt rules are installed in Kibana And there is at least 1 rule available to install >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Given 1 prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And this rule has neither Setup guide nor Investigation guide When user opens the Add Rules page Then all rules available for installation should be displayed in the table @@ -548,10 +583,14 @@ Given the package is installed And the package contains N rules When user installs all rules via install <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Then the endpoint should return success response (HTTP 200 code) with ======= Then the endpoint should return 200 with >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Then the endpoint should return success response (HTTP 200 code) with +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And N rule objects should be created And each rule object should have correct id and version @@ -581,6 +620,7 @@ When user installs all rules via install And deletes one of the installed rules And gets prebuilt rules status via status <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Then the endpoint should return successful response (HTTP 200 code) with When user installs all rules via install again Then the endpoint should return successful response (HTTP 200 code) with @@ -589,6 +629,11 @@ Then the endpoint should return 200 with When user installs all rules via install again Then the endpoint should return 200 with >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Then the endpoint should return successful response (HTTP 200 code) with +When user installs all rules via install again +Then the endpoint should return successful response (HTTP 200 code) with +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Examples: | package_type | api | status_response | install_response | @@ -618,6 +663,7 @@ And the package contains N rules When user installs all rules via install And user gets prebuilt rules status via status <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Then the endpoint should return successful response (HTTP 200 code) with When user calls install Then the endpoint should return successful response (HTTP 200 code) with @@ -625,11 +671,18 @@ When user calls upgrade Then the endpoint should return successful response (HTTP 200 code) with ======= Then the endpoint should return 200 with +======= +Then the endpoint should return successful response (HTTP 200 code) with +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user calls install -Then the endpoint should return 200 with +Then the endpoint should return successful response (HTTP 200 code) with When user calls upgrade +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Then the endpoint should return 200 with >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +Then the endpoint should return successful response (HTTP 200 code) with +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Examples: | package_type | api | status_response | install_response | upgrade_response | @@ -694,12 +747,17 @@ Examples: ```Gherkin Given user with "Security: read" privileges on Security Solution <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And prebuilt rule assets exist in Kibana And no prebuilt rules are installed ======= And no prebuilt rules are installed in Kibana And there are prebuilt rules available to install >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +======= +And prebuilt rule assets exist in Kibana +And no prebuilt rules are installed +>>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then user should see prebuilt rules available to install But user should not be able to install them From 2df76b124602c62659d5305579f698f4d1ab4d78 Mon Sep 17 00:00:00 2001 From: Juan Pablo Djeredjian Date: Tue, 17 Dec 2024 15:43:38 -0300 Subject: [PATCH 07/18] Apply partial suggestions for `package_installation_and_upgrade.md` Co-authored-by: Maxim Palenov --- .../package_installation_and_upgrade.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md index b39946c9d9288..f7a34d96a098a 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md @@ -4,11 +4,15 @@ This is a test plan for the workflows of installing and updating the Prebuilt Ru > For the test plans for installing and upgrading prebuilt rules, see [Installation of Prebuilt Rules](./installation.md) and [Upgrade of Prebuilt Rules](./upgrade.md). +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). ======= Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +======= +Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). +>>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md ## Table of Contents @@ -32,11 +36,15 @@ Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule ### Tickets +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md - [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) ======= - [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +======= +- [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) +>>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md **Milestone 3 - Prebuilt Rules Customization:** - [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) @@ -93,12 +101,17 @@ Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Given the prebuilt rules package is not installed When user opens any Security Solution page ======= Given the package is not installed When user opens the Rule Management page >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +======= +Given the prebuilt rules package is not installed +When user opens any Security Solution page +>>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then the package gets installed in the background from EPR ``` @@ -110,6 +123,7 @@ Then the package gets installed in the background from EPR Given the package is not installed And user is in an air-gapped environment <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user opens any Security Solution page Then the package gets installed in the background from packages bundled into Kibana ``` @@ -122,6 +136,13 @@ Then the package gets installed in the background from packages bundled into Kib #### **Scenario: Large package can be installed on a small Kibana instance** >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +======= +When user opens any Security Solution page +Then the package gets installed in the background from packages bundled into Kibana +``` + +#### **Scenario: Large package can be installed on a memory restricted Kibana instance** +>>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md **Automation**: 1 integration test. @@ -130,10 +151,14 @@ Given the package is not installed And the package contains the largest amount of historical rule versions (15000) And the Kibana instance has a memory heap size of 700 Mb (see note below) <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user opens any Security Solution page ======= When user opens the Rule Management page >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +======= +When user opens any Security Solution page +>>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then the package is installed without Kibana crashing with an Out Of Memory error ``` From d7ffa1b57ff2c21b0c8b77590e7c5aac7348a8ee Mon Sep 17 00:00:00 2001 From: Juan Pablo Djeredjian Date: Tue, 17 Dec 2024 15:53:47 -0300 Subject: [PATCH 08/18] Finish applying suggestions in package_installation_and_upgrade.md Co-authored-by: Maxim Palenov --- .../package_installation_and_upgrade.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md index f7a34d96a098a..56f00e93ef52f 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md @@ -172,12 +172,17 @@ Then the package is installed without Kibana crashing with an Out Of Memory erro ```Gherkin <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Given there are two package versions: A and B where A < B And the package of A version is installed ======= Given there are two package versions: N-1 and N And the package of N-1 version is installed >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +======= +Given there are two package versions: A and B where A < B +And the package of A version is installed +>>>>>>> 5894739cfd3 (Finish applying suggestions in package_installation_and_upgrade.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user calls the status endpoint Then it should return a 200 response with some number of rules to install and 0 rules to upgrade When user calls the installation/_review endpoint @@ -186,10 +191,14 @@ When user calls the installation/_perform_ endpoint Then it should return a 200 response matching the response of the status endpoint And rules returned in this response should exist as alert saved objects <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user installs version B of the package ======= When user installs the package of N version >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +======= +When user installs version B of the package +>>>>>>> 5894739cfd3 (Finish applying suggestions in package_installation_and_upgrade.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then it should be installed successfully When user calls the status endpoint Then it should return a 200 response with some number of new rules to install and some number of rules to upgrade @@ -212,6 +221,7 @@ Then rules returned in this response should exist as alert saved objects ```Gherkin Given user is upgrading Kibana from version to version <<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md And the version Kibana instance has already installed prebuilt rules When the Kibana upgrade is complete Then user should be able to install new prebuilt rules @@ -220,6 +230,10 @@ And upgrade installed prebuilt rules that have newer versions in Kibana version ======= And the instance contains already installed prebuilt rules When the upgrade is complete +======= +And the version Kibana instance has already installed prebuilt rules +When the Kibana upgrade is complete +>>>>>>> 5894739cfd3 (Finish applying suggestions in package_installation_and_upgrade.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then user should be able to install new prebuilt rules And delete installed prebuilt rules And upgrade installed prebuilt rules that have newer versions in From 45bdd7ea18dca589bd917df62732daa4aa707457 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 17 Dec 2024 15:54:40 -0300 Subject: [PATCH 09/18] fix --- .../prebuilt_rules/package_installation_and_upgrade.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md index 56f00e93ef52f..bd76975dd0ad4 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md @@ -227,6 +227,7 @@ When the Kibana upgrade is complete Then user should be able to install new prebuilt rules And delete installed prebuilt rules And upgrade installed prebuilt rules that have newer versions in Kibana version +<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md ======= And the instance contains already installed prebuilt rules When the upgrade is complete @@ -238,6 +239,8 @@ Then user should be able to install new prebuilt rules And delete installed prebuilt rules And upgrade installed prebuilt rules that have newer versions in >>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +======= +>>>>>>> b19262ad8ea (fix):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Examples: | A | B | From 28582c317d55aa2667863d9077a35cb4c4199bff Mon Sep 17 00:00:00 2001 From: Juan Pablo Djeredjian Date: Tue, 17 Dec 2024 22:15:53 -0300 Subject: [PATCH 10/18] Apply suggestionsin upgrade.md - part 1 Co-authored-by: Maxim Palenov --- .../detection_response/prebuilt_rules/upgrade.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index ab88828edb5e0..81d3370d1625c 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -2,7 +2,7 @@ This is a test plan for the workflow of upgrading prebuilt rules. -Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. +Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). ## Table of Contents @@ -261,7 +261,7 @@ And for Y of the installed rules there are new versions available And user is on the Rule Management page When user opens the Rule Updates table Then Y rules available for upgrade should be displayed in the table -And for Z of the rules with upgrades there are upgrade conflicts +And for Z (where Z < Y) of the rules with upgrades there are upgrade conflicts Then for those Z rules the Upgrade Rule button should be disabled And the user should not be able to upgrade them directly from the table And there should be a message/tooltip indicating why the rule cannot be updated directly @@ -274,11 +274,10 @@ And there should be a message/tooltip indicating why the rule cannot be updated ```Gherkin Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available -And user is on the Rule Management page -When user opens the Rule Updates table +And user opens the Rule Updates table Then Y rules available for upgrade should be displayed in the table -When user selects rules, which have no upgrade conflicts -Then user should see a CTA to upgrade number of rules +When user selects Z (where Z < Y) rules, which have no upgrade conflicts +Then user should see a CTA to upgrade rules When user clicks the CTA Then success message should be displayed after upgrade And all the upgraded rules should be removed from the table From 32c1d17a822400a0b2f809eaad84eec3a4c51a07 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 17 Dec 2024 22:16:58 -0300 Subject: [PATCH 11/18] fixes --- .../detection_response/prebuilt_rules/upgrade.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index 81d3370d1625c..6345dc98af9f9 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -17,9 +17,10 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz - [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date) - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) - - [Rule upgrade workflow: individual and bulk updates from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) + - [Rule upgrade workflow: individual updates from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) - [**Scenario: User can upgrade conflict-free prebuilt rules one by one**](#scenario-user-can-upgrade-conflict-free-prebuilt-rules-one-by-one) - [**Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts**](#scenario-user-cannot-upgrade-prebuilt-rules-one-by-one-from-rules-update-table-if-they-have-conflicts) + - [Rule upgrade workflow: bulk updates from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) - [**Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-conflict-free-prebuilt-rules-selected-on-the-page) - [**Scenario: User cannot upgrade multiple prebuilt rules selected on the page when they have upgrade conflicts**](#scenario-user-cannot-upgrade-multiple-prebuilt-rules-selected-on-the-page-when-they-have-upgrade-conflicts) - [**Scenario: User can upgrade all available conflict-free prebuilt rules at once**](#scenario-user-can-upgrade-all-available-conflict-free-prebuilt-rules-at-once) @@ -233,7 +234,7 @@ And user should see a CTA to upgrade prebuilt rules And user should see the number of rules available to upgrade (Z) ``` -### Rule upgrade workflow: individual and bulk updates from Rule Updates table +### Rule upgrade workflow: individual updates from Rule Updates table #### **Scenario: User can upgrade conflict-free prebuilt rules one by one** @@ -258,15 +259,16 @@ And user should see the number of rules available to upgrade decreased by 1 ```Gherkin Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available -And user is on the Rule Management page -When user opens the Rule Updates table +And user is on the Rule Updates table Then Y rules available for upgrade should be displayed in the table And for Z (where Z < Y) of the rules with upgrades there are upgrade conflicts Then for those Z rules the Upgrade Rule button should be disabled And the user should not be able to upgrade them directly from the table -And there should be a message/tooltip indicating why the rule cannot be updated directly +And there should be a message/tooltip indicating why the rule cannot be upgraded directly ``` +### Rule upgrade workflow: bulk updates from Rule Updates table + #### **Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page** **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. From a992629113fcd918835811a1c0d69da20cb7c134 Mon Sep 17 00:00:00 2001 From: Juan Pablo Djeredjian Date: Tue, 17 Dec 2024 22:21:26 -0300 Subject: [PATCH 12/18] part2 Co-authored-by: Maxim Palenov --- .../detection_response/prebuilt_rules/upgrade.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index 6345dc98af9f9..de16552b1b4bf 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -291,7 +291,7 @@ Examples: | all rules on the page, e.g. 12 | ``` -#### **Scenario: User cannot upgrade multiple prebuilt rules selected on the page when they have upgrade conflicts** +#### **Scenario: User cannot upgrade selected prebuilt rules with conflicts** **Automation**: 1 e2e test with mock rules @@ -373,19 +373,19 @@ Examples: | all rules on the page, e.g. 12 | ``` -#### **Scenario: User can upgrade only conflict-free rules when user attempts to upgrade all rules and only a subset contains upgrade conflicts** +#### **Scenario: User can upgrade only conflict-free rules when attempting to upgrade all rules** **Automation**: 1 e2e test with mock rules ```Gherkin Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available -And a subset Z of the rules have conflicts with the current versions +And Z (where Z < Y) rules have conflicts with the current versions And user is on the Rule Management page Then Y rules available for upgrade should be displayed in the table Then user should see an enabled CTA to upgrade all rules When user clicks the CTA -A modal window should inform the user that only Z rules without conflicts will be upgraded +A modal window should inform the user that only K (where K < Z) rules without conflicts will be upgraded When user confirms the action in the modal Then success message should be displayed after upgrade informing that Z rules were updated And the Z upgraded rules should be removed from the table From 3c9c5ef95fa8af6c936aae2be472dc391f60ded0 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 17 Dec 2024 22:25:28 -0300 Subject: [PATCH 13/18] math fix --- .../detection_response/prebuilt_rules/upgrade.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index de16552b1b4bf..24d40b3828255 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -355,7 +355,7 @@ Then a message should be displayed that informs the user why the rules cannot be Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available And a subset Z of the rules have conflicts with the current versions -And user is on the Rule Management page +And user is on the Rule Updates table Then Y rules available for upgrade should be displayed in the table And user selects rules, which is a mixture of rules with and without upgrade conflicts Then user should see a CTA to upgrade number of rules, which is enabled @@ -381,16 +381,16 @@ Examples: Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available And Z (where Z < Y) rules have conflicts with the current versions -And user is on the Rule Management page +And user is on the Rule Updates table Then Y rules available for upgrade should be displayed in the table Then user should see an enabled CTA to upgrade all rules When user clicks the CTA -A modal window should inform the user that only K (where K < Z) rules without conflicts will be upgraded +A modal window should inform the user that only K (where K < Y) rules without conflicts will be upgraded When user confirms the action in the modal -Then success message should be displayed after upgrade informing that Z rules were updated -And the Z upgraded rules should be removed from the table -And the remaining Y - Z rules should still be present in the table -And user should see the number of rules available to upgrade decreased by Z number of upgraded rules +Then success message should be displayed after upgrade informing that K rules were updated +And the K upgraded rules should be removed from the table +And the remaining Y - K rules should still be present in the table +And user should see the number of rules available to upgrade decreased by K number of upgraded rules ``` ### Rule upgrade workflow: rule previews From 1ffb1700ba190beb0bf307d4ad6c5b31b1c5cda3 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 17 Dec 2024 22:28:40 -0300 Subject: [PATCH 14/18] fix --- .../test_plans/detection_response/prebuilt_rules/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index 24d40b3828255..04a5bebf351cb 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -389,7 +389,7 @@ A modal window should inform the user that only K (where K < Y) rules without co When user confirms the action in the modal Then success message should be displayed after upgrade informing that K rules were updated And the K upgraded rules should be removed from the table -And the remaining Y - K rules should still be present in the table +And the remaining M = Y - K rules should still be present in the table And user should see the number of rules available to upgrade decreased by K number of upgraded rules ``` From d5017d62c3616c1cb88a24a0e729e4a4cbf3ac03 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 17 Dec 2024 22:43:20 -0300 Subject: [PATCH 15/18] Added rule type changes tests --- .../prebuilt_rules/upgrade.md | 60 ++++++++++++++++++- 1 file changed, 58 insertions(+), 2 deletions(-) diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index 04a5bebf351cb..20520ae1cab0d 100644 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -17,16 +17,20 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz - [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date) - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) - - [Rule upgrade workflow: individual updates from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) + - [Rule upgrade workflow: individual upgrade from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) - [**Scenario: User can upgrade conflict-free prebuilt rules one by one**](#scenario-user-can-upgrade-conflict-free-prebuilt-rules-one-by-one) - [**Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts**](#scenario-user-cannot-upgrade-prebuilt-rules-one-by-one-from-rules-update-table-if-they-have-conflicts) - - [Rule upgrade workflow: bulk updates from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) + - [Rule upgrade workflow: bulk upgrade from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) - [**Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-conflict-free-prebuilt-rules-selected-on-the-page) - [**Scenario: User cannot upgrade multiple prebuilt rules selected on the page when they have upgrade conflicts**](#scenario-user-cannot-upgrade-multiple-prebuilt-rules-selected-on-the-page-when-they-have-upgrade-conflicts) - [**Scenario: User can upgrade all available conflict-free prebuilt rules at once**](#scenario-user-can-upgrade-all-available-conflict-free-prebuilt-rules-at-once) - [**Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts**](#scenario-user-cannot-upgrade-all-prebuilt-rules-at-once-if-they-have-upgrade-conflicts) - [**Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade in the Rules Table**](#scenario-user-can-upgrade-only-conflict-free-rules-when-a-mix-of-rules-with-and-without-conflicts-are-selected-for-upgrade-in-the-rules-table) - [**Scenario: User can upgrade only conflict-free rules when user attempts to upgrade all rules and only a subset contains upgrade conflicts**](#scenario-user-can-upgrade-only-conflict-free-rules-when-user-attempts-to-upgrade-all-rules-and-only-a-subset-contains-upgrade-conflicts) + - [Rule upgrade workflow: upgrading rules with rule type change](#rule-upgrade-workflow-upgrading-rules-with-rule-type-change) + - [**Scenario: User can upgrade rule with rule type change individually**](#scenario-user-can-upgrade-rule-with-rule-type-change-individually) + - [**Scenario: User can bulk upgrade selected rules with rule type changes**](#scenario-user-can-bulk-upgrade-selected-rules-with-rule-type-changes) + - [**Scenario: User can bulk upgrade all rules with rule type changes**](#scenario-user-can-bulk-upgrade-all-rules-with-rule-type-changes) - [Rule upgrade workflow: rule previews](#rule-upgrade-workflow-rule-previews) - [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade) - [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview) @@ -393,6 +397,58 @@ And the remaining M = Y - K rules should still be present in the table And user should see the number of rules available to upgrade decreased by K number of upgraded rules ``` + +### Rule upgrade workflow: upgrading rules with rule type change + +#### **Scenario: User can upgrade rule with rule type change individually** + +**Automation**: 1 e2e test with mock rules + +```Gherkin +Given a prebuilt rule is installed in Kibana +And this rule has an update available that changes its rule type +When user opens the Rule Updates table +Then this rule should be displayed in the table +And the Upgrade Rule button should be enabled +When user upgrades the rule +Then success message should be displayed after upgrade +And the upgraded rule should be removed from the table +And user should see the number of rules available to upgrade decreased by 1 +``` + +#### **Scenario: User can bulk upgrade selected rules with rule type changes** + + +**Automation**: 1 e2e test with mock rules + +```Gherkin +Given X prebuilt rules are installed in Kibana +And Y of these rules have updates available that change their rule types +When user opens the Rule Updates table +Then Y rules should be displayed in the table +When user selects Z rules (where Z < Y) with rule type changes +Then user should see an enabled CTA to upgrade Z rules +When user clicks the CTA +Then success message should be displayed after upgrade +And all Z upgraded rules should be removed from the table +And user should see the number of rules available to upgrade decreased by Z +``` + +#### **Scenario: User can bulk upgrade all rules with rule type changes** + +**Automation**: 1 e2e test with mock rules + +```Gherkin +Given X prebuilt rules are installed in Kibana +And all X rules have updates available that change their rule types +When user opens the Rule Updates table +Then X rules should be displayed in the table +And user should see an enabled CTA to upgrade all rules +When user clicks the CTA to upgrade all rules +Then success message should be displayed after upgrade +And all X upgraded rules should be removed from the table +``` + ### Rule upgrade workflow: rule previews #### **Scenario: User can preview rules available for upgrade** From aef6ff3e2e57475bc54a87a619333cb41c88548f Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 17 Dec 2024 23:02:44 -0300 Subject: [PATCH 16/18] Updated test plans --- .../prebuilt_rules/installation.md | 205 ----------------- .../package_installation_and_upgrade.md | 84 ------- .../prebuilt_rules/upgrade.md | 210 ------------------ 3 files changed, 499 deletions(-) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md index a03b4f3cd88fc..dac2a0e8f1ac6 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md @@ -2,15 +2,7 @@ This is a test plan for the workflows of installing prebuilt rules. -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). -======= -Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= -Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). ->>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ## Table of Contents @@ -54,15 +46,7 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz ### Tickets -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -- [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) -======= -- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= - [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) ->>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md **Milestone 3 - Prebuilt Rules Customization:** - [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) @@ -192,18 +176,8 @@ Examples: **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Given no prebuilt rule assets exist in Kibana -And no prebuilt rules are installed -======= -Given no prebuilt rules are installed in Kibana -And no prebuilt rule assets exist ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= Given no prebuilt rule assets exist in Kibana And no prebuilt rules are installed ->>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Rule Management page Then user should NOT see a CTA to install prebuilt rules And user should NOT see a number of rules available to install @@ -217,17 +191,8 @@ And user should NOT see the Rule Updates table **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Given the latest prebuilt rule assets exist in Kibana -And all the latest prebuilt rules from those rule assets are installed -======= -Given all the latest prebuilt rules are installed in Kibana ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= Given the latest prebuilt rule assets exist in Kibana And all the latest prebuilt rules from those rule assets are installed ->>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Rule Management page Then user should NOT see a CTA to install prebuilt rules And user should NOT see a number of rules available to install @@ -241,17 +206,8 @@ And user should NOT see the Rule Updates table **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed -======= -Given no prebuilt rules are installed in Kibana ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= -Given X prebuilt rule assets exist in Kibana -And no prebuilt rules are installed ->>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And there are X prebuilt rules available to install When user opens the Rule Management page Then user should see a CTA to install prebuilt rules @@ -266,17 +222,8 @@ And user should NOT see the Rule Updates table **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Given Y prebuilt rule assets exist in Kibana -And X (where X < Y) prebuilt rules are installed -======= -Given X prebuilt rules are installed in Kibana ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= Given Y prebuilt rule assets exist in Kibana And X (where X < Y) prebuilt rules are installed ->>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And there are Y more prebuilt rules available to install And for all X installed rules there are no new versions available When user opens the Rule Management page @@ -292,20 +239,9 @@ And user should NOT see the Rule Updates table **Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Given Y prebuilt rule assets exist in Kibana -And X (where X < Y) prebuilt rules are installed -And Z (where Z < X) installed rules have matching prebuilt rule assets with higher version available -======= -Given X prebuilt rules are installed in Kibana -And there are Y more prebuilt rules available to install ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= Given Y prebuilt rule assets exist in Kibana And X (where X < Y) prebuilt rules are installed And Z (where Z < X) installed rules have matching prebuilt rule assets with higher version available ->>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And for Z of the installed rules there are new versions available When user opens the Rule Management page Then user should see a CTA to install prebuilt rules @@ -324,15 +260,7 @@ And there are no more prebuilt rules available to install When user opens the Rule Management page And user deletes Y prebuilt rules Then user should see a CTA to install prebuilt rules -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -And user should see rules available to install -======= -And user should see the number of rules available to install (Y) ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= And user should see rules available to install ->>>>>>> 45f3f0068f7 (Apply suggestions):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ``` ### Rule installation workflow: base cases @@ -342,18 +270,8 @@ And user should see rules available to install **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed -======= -Given no prebuilt rules are installed in Kibana -And there are X prebuilt rules available to install ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= -Given X prebuilt rule assets exist in Kibana -And no prebuilt rules are installed ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then prebuilt rules available for installation should be displayed in the table When user installs one individual rule without previewing it @@ -369,18 +287,8 @@ And user should see the number of rules available to install decreased by 1 **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Given X prebuilt rule assets exist in Kibana -And no prebuilt rules are installed -======= -Given no prebuilt rules are installed in Kibana -And there are X prebuilt rules available to install ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then prebuilt rules available for installation should be displayed in the table When user selects rules @@ -403,18 +311,8 @@ Examples: **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/\* endpoints in integration. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed -======= -Given no prebuilt rules are installed in Kibana -And there are X prebuilt rules available to install ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= -Given X prebuilt rule assets exist in Kibana -And no prebuilt rules are installed ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then prebuilt rules available for installation should be displayed in the table When user installs all rules @@ -444,18 +342,8 @@ And user should see a CTA that leads to the Rule Management page **Automation**: 1 e2e test ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given 2 prebuilt rule assets exist in Kibana And no prebuilt rules are installed -======= -Given no prebuilt rules are installed in Kibana -And there are 2 rules available to install ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= -Given 2 prebuilt rule assets exist in Kibana -And no prebuilt rules are installed ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then all rules available for installation should be displayed in the table When user opens the rule preview for the 1st rule @@ -469,18 +357,8 @@ Then it should disappear **Automation**: 1 e2e test ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Given 2 prebuilt rule assets exist in Kibana -And no prebuilt rules are installed -======= -Given no prebuilt rules are installed in Kibana -And there are 2 rules available to install ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= Given 2 prebuilt rule assets exist in Kibana And no prebuilt rules are installed ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then all rules available for installation should be displayed in the table When user opens the rule preview for the rule @@ -499,18 +377,8 @@ And user should see the number of rules available to install as initial number m **Automation**: 1 e2e test ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Given X prebuilt rule assets exist in Kibana -And no prebuilt rules are installed -======= -Given no prebuilt rules are installed in Kibana -And there are X prebuilt rules of all types available to install ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= Given X prebuilt rule assets exist in Kibana And no prebuilt rules are installed ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then all X rules available for installation should be displayed in the table When user opens a rule preview for any rule @@ -523,18 +391,8 @@ And all properties of a rule should be displayed in the correct tab and section **Automation**: 1 e2e test ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Given 1 prebuilt rule assets exist in Kibana And no prebuilt rules are installed -======= -Given no prebuilt rules are installed in Kibana -And there is at least 1 rule available to install ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= -Given 1 prebuilt rule assets exist in Kibana -And no prebuilt rules are installed ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And this rule has neither Setup guide nor Investigation guide When user opens the Add Rules page Then all rules available for installation should be displayed in the table @@ -582,15 +440,7 @@ There's a legacy prebuilt rules API and a new one. Both should be tested against Given the package is installed And the package contains N rules When user installs all rules via install -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Then the endpoint should return success response (HTTP 200 code) with -======= -Then the endpoint should return 200 with ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= Then the endpoint should return success response (HTTP 200 code) with ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md And N rule objects should be created And each rule object should have correct id and version @@ -619,21 +469,9 @@ And the package contains N rules When user installs all rules via install And deletes one of the installed rules And gets prebuilt rules status via status -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Then the endpoint should return successful response (HTTP 200 code) with When user installs all rules via install again Then the endpoint should return successful response (HTTP 200 code) with -======= -Then the endpoint should return 200 with -When user installs all rules via install again -Then the endpoint should return 200 with ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= -Then the endpoint should return successful response (HTTP 200 code) with -When user installs all rules via install again -Then the endpoint should return successful response (HTTP 200 code) with ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Examples: | package_type | api | status_response | install_response | @@ -662,27 +500,11 @@ Given the package is installed And the package contains N rules When user installs all rules via install And user gets prebuilt rules status via status -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Then the endpoint should return successful response (HTTP 200 code) with -When user calls install -Then the endpoint should return successful response (HTTP 200 code) with -When user calls upgrade -Then the endpoint should return successful response (HTTP 200 code) with -======= -Then the endpoint should return 200 with -======= Then the endpoint should return successful response (HTTP 200 code) with ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user calls install Then the endpoint should return successful response (HTTP 200 code) with When user calls upgrade -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -Then the endpoint should return 200 with ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= Then the endpoint should return successful response (HTTP 200 code) with ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md Examples: | package_type | api | status_response | install_response | upgrade_response | @@ -705,15 +527,7 @@ Notes: ### Error handling -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md #### **Scenario: Error is handled when any installation operation on prebuilt rules fails** -======= -#### **Scenario: Error is handled when any operation on prebuilt rules fails** ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= -#### **Scenario: Error is handled when any installation operation on prebuilt rules fails** ->>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md **Automation**: e2e test with mock rules @@ -727,15 +541,6 @@ Examples: | installing all | | installing selected | | installing individual | -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= - | upgrading all | - | upgrading selected | - | upgrading individual | ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= ->>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md ``` ### Authorization / RBAC @@ -746,18 +551,8 @@ Examples: ```Gherkin Given user with "Security: read" privileges on Security Solution -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -And prebuilt rule assets exist in Kibana -And no prebuilt rules are installed -======= -And no prebuilt rules are installed in Kibana -And there are prebuilt rules available to install ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md -======= And prebuilt rule assets exist in Kibana And no prebuilt rules are installed ->>>>>>> 142b4da45f1 (Apply suggestion for installation.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md When user opens the Add Rules page Then user should see prebuilt rules available to install But user should not be able to install them diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md index bd76975dd0ad4..57e16684facca 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md @@ -4,15 +4,7 @@ This is a test plan for the workflows of installing and updating the Prebuilt Ru > For the test plans for installing and upgrading prebuilt rules, see [Installation of Prebuilt Rules](./installation.md) and [Upgrade of Prebuilt Rules](./upgrade.md). -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). -======= -Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -======= -Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). ->>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md ## Table of Contents @@ -36,15 +28,7 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz ### Tickets -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -- [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) -======= -- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -======= - [Rule Immutability/Customization epic](https://github.com/elastic/security-team/issues/1974)(internal) ->>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md **Milestone 3 - Prebuilt Rules Customization:** - [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) @@ -100,18 +84,8 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz **Automation**: 2 e2e tests that install the real package. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -Given the prebuilt rules package is not installed -When user opens any Security Solution page -======= -Given the package is not installed -When user opens the Rule Management page ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -======= Given the prebuilt rules package is not installed When user opens any Security Solution page ->>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then the package gets installed in the background from EPR ``` @@ -122,27 +96,11 @@ Then the package gets installed in the background from EPR ```Gherkin Given the package is not installed And user is in an air-gapped environment -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user opens any Security Solution page Then the package gets installed in the background from packages bundled into Kibana ``` #### **Scenario: Large package can be installed on a memory restricted Kibana instance** -======= -When user opens the Rule Management page -Then the package gets installed in the background from packages bundled into Kibana -``` - -#### **Scenario: Large package can be installed on a small Kibana instance** ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -======= -When user opens any Security Solution page -Then the package gets installed in the background from packages bundled into Kibana -``` - -#### **Scenario: Large package can be installed on a memory restricted Kibana instance** ->>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md **Automation**: 1 integration test. @@ -150,15 +108,7 @@ Then the package gets installed in the background from packages bundled into Kib Given the package is not installed And the package contains the largest amount of historical rule versions (15000) And the Kibana instance has a memory heap size of 700 Mb (see note below) -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -When user opens any Security Solution page -======= -When user opens the Rule Management page ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -======= When user opens any Security Solution page ->>>>>>> a71de796f01 (Apply partial suggestions for `package_installation_and_upgrade.md`):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then the package is installed without Kibana crashing with an Out Of Memory error ``` @@ -171,18 +121,8 @@ Then the package is installed without Kibana crashing with an Out Of Memory erro **Automation**: 1 integration test with real packages. ```Gherkin -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -Given there are two package versions: A and B where A < B -And the package of A version is installed -======= -Given there are two package versions: N-1 and N -And the package of N-1 version is installed ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -======= Given there are two package versions: A and B where A < B And the package of A version is installed ->>>>>>> 5894739cfd3 (Finish applying suggestions in package_installation_and_upgrade.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user calls the status endpoint Then it should return a 200 response with some number of rules to install and 0 rules to upgrade When user calls the installation/_review endpoint @@ -190,15 +130,7 @@ Then it should return a 200 response matching the response of the status endpoin When user calls the installation/_perform_ endpoint Then it should return a 200 response matching the response of the status endpoint And rules returned in this response should exist as alert saved objects -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md When user installs version B of the package -======= -When user installs the package of N version ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -======= -When user installs version B of the package ->>>>>>> 5894739cfd3 (Finish applying suggestions in package_installation_and_upgrade.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Then it should be installed successfully When user calls the status endpoint Then it should return a 200 response with some number of new rules to install and some number of rules to upgrade @@ -220,27 +152,11 @@ Then rules returned in this response should exist as alert saved objects ```Gherkin Given user is upgrading Kibana from version to version -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md And the version Kibana instance has already installed prebuilt rules When the Kibana upgrade is complete Then user should be able to install new prebuilt rules And delete installed prebuilt rules And upgrade installed prebuilt rules that have newer versions in Kibana version -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -======= -And the instance contains already installed prebuilt rules -When the upgrade is complete -======= -And the version Kibana instance has already installed prebuilt rules -When the Kibana upgrade is complete ->>>>>>> 5894739cfd3 (Finish applying suggestions in package_installation_and_upgrade.md):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -Then user should be able to install new prebuilt rules -And delete installed prebuilt rules -And upgrade installed prebuilt rules that have newer versions in ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md -======= ->>>>>>> b19262ad8ea (fix):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/package_installation_and_upgrade.md Examples: | A | B | diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index 61b6462673845..20520ae1cab0d 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -2,11 +2,7 @@ This is a test plan for the workflow of upgrading prebuilt rules. -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). -======= -Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/kibana/issues/174168) epic. ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ## Table of Contents @@ -16,8 +12,6 @@ Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule - [Assumptions](#assumptions) - [Non-functional requirements](#non-functional-requirements) - [Functional requirements](#functional-requirements) -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -<<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md - [Scenarios](#scenarios) - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) - [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date) @@ -71,133 +65,6 @@ Status: `in progress`. The current test plan matches `Milestone 3` of the [Rule - [Authorization / RBAC](#authorization-rbac) - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) -======== -- [Scenarios](#scenarios) -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md - - [Package installation](#package-installation) - - [**Scenario: Package is installed via Fleet**](#scenario-package-is-installed-via-fleet) - - [**Scenario: Package is installed via bundled Fleet package in Kibana**](#scenario-package-is-installed-via-bundled-fleet-package-in-kibana) - - [**Scenario: Large package can be installed on a small Kibana instance**](#scenario-large-package-can-be-installed-on-a-small-kibana-instance) - - [Rule installation and upgrade via the Prebuilt rules API](#rule-installation-and-upgrade-via-the-prebuilt-rules-api) - - [**Scenario: API can install all prebuilt rules**](#scenario-api-can-install-all-prebuilt-rules) - - [**Scenario: API can install prebuilt rules that are not yet installed**](#scenario-api-can-install-prebuilt-rules-that-are-not-yet-installed) - - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) - - [**Scenario: API does not install or upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-install-or-upgrade-prebuilt-rules-if-they-are-up-to-date) - - [Scenarios for the real package](#scenarios-for-the-real-package) - - [**Scenario: User can install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package**](#scenario-user-can-install-prebuilt-rules-from-scratch-then-install-new-rules-and-upgrade-existing-rules-from-the-new-package) - - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) - - [**Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets**](#scenario-user-is-not-notified-when-no-prebuilt-rules-are-installed-and-there-are-no-prebuilt-rules-assets) - - [**Scenario: User is NOT notified when all prebuilt rules are installed and up to date**](#scenario-user-is-not-notified-when-all-prebuilt-rules-are-installed-and-up-to-date) - - [**Scenario: User is notified when no prebuilt rules are installed and there are rules available to install**](#scenario-user-is-notified-when-no-prebuilt-rules-are-installed-and-there-are-rules-available-to-install) - - [**Scenario: User is notified when some prebuilt rules can be installed**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-installed) - - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) - - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) - - [**Scenario: User is notified after a prebuilt rule gets deleted**](#scenario-user-is-notified-after-a-prebuilt-rule-gets-deleted) - - [Rule installation workflow: base cases](#rule-installation-workflow-base-cases) - - [**Scenario: User can install prebuilt rules one by one**](#scenario-user-can-install-prebuilt-rules-one-by-one) - - [**Scenario: User can install multiple prebuilt rules selected on the page**](#scenario-user-can-install-multiple-prebuilt-rules-selected-on-the-page) - - [**Scenario: User can install all available prebuilt rules at once**](#scenario-user-can-install-all-available-prebuilt-rules-at-once) - - [**Scenario: Empty screen is shown when all prebuilt rules are installed**](#scenario-empty-screen-is-shown-when-all-prebuilt-rules-are-installed) - - [**Scenario: User can preview rules available for installation**](#scenario-user-can-preview-rules-available-for-installation) - - [**Scenario: User can install a rule using the rule preview**](#scenario-user-can-install-a-rule-using-the-rule-preview) - - [**Scenario: User can see correct rule information in preview before installing**](#scenario-user-can-see-correct-rule-information-in-preview-before-installing) - - [**Scenario: Tabs and sections without content should be hidden in preview before installing**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-installing) - - [Rule installation workflow: filtering, sorting, pagination](#rule-installation-workflow-filtering-sorting-pagination) - - [Rule installation workflow: misc cases](#rule-installation-workflow-misc-cases) - - [**Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed**](#scenario-user-opening-the-add-rules-page-sees-a-loading-skeleton-until-the-package-installation-is-completed) - - [**Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs**](#scenario-user-can-navigate-from-the-add-rules-page-to-the-rule-management-page-via-breadcrumbs) -======= -- [Scenarios](#scenarios) ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md - - [Rule upgrade workflow: base cases](#rule-upgrade-workflow-base-cases) - - [**Scenario: User can upgrade prebuilt rules one by one**](#scenario-user-can-upgrade-prebuilt-rules-one-by-one) - - [**Scenario: User can upgrade multiple prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-prebuilt-rules-selected-on-the-page) - - [**Scenario: User can upgrade all available prebuilt rules at once**](#scenario-user-can-upgrade-all-available-prebuilt-rules-at-once) - - [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade) - - [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview) - - [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading) - - [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading) - - [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination) - - [Rule upgrade workflow: viewing rule changes in JSON diff view](#rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view) - - [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view) - - [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade) - - [**Scenario: Rule actions and exception lists should not be shown as modified**](#scenario-rule-actions-and-exception-lists-should-not-be-shown-as-modified) - - [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview) - - [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview) - - [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified) - - [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default) - - [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically) - - [Rule upgrade workflow: viewing rule changes in per-field diff view](#rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view) - - [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view) - - [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel) - - [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs) - - [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview) - - [Rule upgrade workflow: preserving rule bound data](#rule-upgrade-workflow-preserving-rule-bound-data) - - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-the-same-rule-type) - - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-a-different-rule-type) - - [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases) - - [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed) -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md - - [Error handling](#error-handling) - - [**Scenario: Error is handled when any operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-operation-on-prebuilt-rules-fails) - - [Authorization / RBAC](#authorization--rbac) - - [**Scenario: User with read privileges on Security Solution cannot install prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-install-prebuilt-rules) - - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) - - [Kibana upgrade](#kibana-upgrade) - - [**Scenario: User can use prebuilt rules after upgrading Kibana from version A to B**](#scenario-user-can-use-prebuilt-rules-after-upgrading-kibana-from-version-a-to-b) ->>>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md -======= - - [Rule upgrade via the Prebuilt rules API](#rule-installation-and-upgrade-via-the-prebuilt-rules-api) - - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) - - [**Scenario: API does not upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-install-or-upgrade-prebuilt-rules-if-they-are-up-to-date) - - [Error handling](#error-handling) - - [**Scenario: Error is handled when any operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-operation-on-prebuilt-rules-fails) - - [Authorization / RBAC](#authorization--rbac) - - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -======= - - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) - - [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date) - - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) - - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) - - [Rule upgrade workflow: individual and bulk updates from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) - - [**Scenario: User can upgrade prebuilt rules one by one**](#scenario-user-can-upgrade-prebuilt-rules-one-by-one) - - [**Scenario: User can upgrade multiple prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-prebuilt-rules-selected-on-the-page) - - [**Scenario: User can upgrade all available prebuilt rules at once**](#scenario-user-can-upgrade-all-available-prebuilt-rules-at-once) - - [Rule upgrade workflow: rule previews](#rule-upgrade-workflow-rule-previews) - - [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade) - - [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview) - - [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading) - - [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading) - - [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination) - - [Rule upgrade workflow: preserving rule bound data](#rule-upgrade-workflow-preserving-rule-bound-data) - - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-the-same-rule-type) - - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-a-different-rule-type) - - [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases) - - [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed) - - [Error handling](#error-handling) - - [**Scenario: Error is handled when any upgrade operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-upgrade-operation-on-prebuilt-rules-fails) - - [Rule upgrade via the Prebuilt rules API](#rule-upgrade-via-the-prebuilt-rules-api) - - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) - - [**Scenario: API does not upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-upgrade-prebuilt-rules-if-they-are-up-to-date) - - [Authorization / RBAC](#authorization-rbac) - - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) - - [MILESTONE 2 - Rule upgrade workflow: viewing rule changes in JSON diff view](#rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view) - - [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view) - - [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade) - - [**Scenario: Rule actions and exception lists should not be shown as modified**](#scenario-rule-actions-and-exception-lists-should-not-be-shown-as-modified) - - [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview) - - [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview) - - [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified) - - [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default) - - [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically) - - [MILESTONE 2 - Rule upgrade workflow: viewing rule changes in per-field diff view](#rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view) - - [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view) - - [**Scenario: User can see changes when updated rule is a different rule type**](#scenario-user-can-see-changes-when-updated-rule-is-a-different-rule-type) - - [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel) - - [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs) - - [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview) ->>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ## Useful information @@ -371,19 +238,9 @@ And user should see a CTA to upgrade prebuilt rules And user should see the number of rules available to upgrade (Z) ``` -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ### Rule upgrade workflow: individual updates from Rule Updates table #### **Scenario: User can upgrade conflict-free prebuilt rules one by one** -======= -### Rule upgrade workflow: base cases -======= -### Rule upgrade workflow: individual and bulk updates from Rule Updates table ->>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md - -#### **Scenario: User can upgrade prebuilt rules one by one** ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. @@ -399,7 +256,6 @@ And the upgraded rule should be removed from the table And user should see the number of rules available to upgrade decreased by 1 ``` -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts** **Automation**: 1 e2e test with mock rules @@ -418,27 +274,16 @@ And there should be a message/tooltip indicating why the rule cannot be upgraded ### Rule upgrade workflow: bulk updates from Rule Updates table #### **Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page** -======= -#### **Scenario: User can upgrade multiple prebuilt rules selected on the page** ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. ```Gherkin Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md And user opens the Rule Updates table Then Y rules available for upgrade should be displayed in the table When user selects Z (where Z < Y) rules, which have no upgrade conflicts Then user should see a CTA to upgrade rules -======= -And user is on the Rule Management page -When user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -When user selects rules -Then user should see a CTA to upgrade number of rules ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md When user clicks the CTA Then success message should be displayed after upgrade And all the upgraded rules should be removed from the table @@ -450,7 +295,6 @@ Examples: | all rules on the page, e.g. 12 | ``` -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User cannot upgrade selected prebuilt rules with conflicts** **Automation**: 1 e2e test with mock rules @@ -474,19 +318,13 @@ Examples: ``` #### **Scenario: User can upgrade all available conflict-free prebuilt rules at once** -======= -#### **Scenario: User can upgrade all available prebuilt rules at once** ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md **Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. ```Gherkin Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md And those Y new versions don't have conflicts with the current versions -======= ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md And user is on the Rule Management page When user opens the Rule Updates table Then Y rules available for upgrade should be displayed in the table @@ -494,7 +332,6 @@ When user upgrades all rules Then success message should be displayed after upgrade And user should NOT see a CTA to upgrade prebuilt rules And user should NOT see a number of rules available to upgrade -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ``` #### **Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts** @@ -614,16 +451,6 @@ And all X upgraded rules should be removed from the table ### Rule upgrade workflow: rule previews -======= -And user should NOT see the Rule Updates table -``` - -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -======= -### Rule upgrade workflow: rule previews - ->>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can preview rules available for upgrade** ```Gherkin @@ -695,19 +522,9 @@ And the Investigation Guide tab should NOT be displayed TODO: add scenarios https://github.com/elastic/kibana/issues/166215 -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in JSON diff view > These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. -======= -### Rule upgrade workflow: viewing rule changes in JSON diff view ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -======= -### MILESTONE 2 - Rule upgrade workflow: viewing rule changes in JSON diff view - -> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. ->>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can see changes in a side-by-side JSON diff view** @@ -853,19 +670,9 @@ When a user expands all hidden sections Then all properties of the rule should be sorted alphabetically ``` -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md ### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in per-field diff view > These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. -======= -### Rule upgrade workflow: viewing rule changes in per-field diff view ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -======= -### MILESTONE 2 - Rule upgrade workflow: viewing rule changes in per-field diff view - -> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. ->>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md #### **Scenario: User can see changes in a side-by-side per-field diff view** @@ -1003,15 +810,7 @@ Then user should NOT see the Rule Updates tab until the package installation is ### Error handling -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -#### **Scenario: Error is handled when any upgrade operation on prebuilt rules fails** -======= -#### **Scenario: Error is handled when any operation on prebuilt rules fails** ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -======= #### **Scenario: Error is handled when any upgrade operation on prebuilt rules fails** ->>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md **Automation**: e2e test with mock rules @@ -1022,15 +821,6 @@ Then user should see an error message Examples: | operation | -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -<<<<<<< HEAD:x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -======= - | installing all | - | installing selected | - | installing individual | ->>>>>>> d44834d79cd (Split test install+upgrade test plans into 3):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md -======= ->>>>>>> 3815f2b555c (Edited test plans):x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md | upgrading all | | upgrading selected | | upgrading individual | From 784da509c3603bd504add3516e62e56d89c77b8e Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 17 Dec 2024 23:06:04 -0300 Subject: [PATCH 17/18] delete file --- .../prebuilt_rules/upgrade.md | 922 ------------------ 1 file changed, 922 deletions(-) delete mode 100644 x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md diff --git a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md deleted file mode 100644 index 20520ae1cab0d..0000000000000 --- a/x-pack/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ /dev/null @@ -1,922 +0,0 @@ -# Upgrade of Prebuilt Rules - -This is a test plan for the workflow of upgrading prebuilt rules. - -Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168). - -## Table of Contents - -- [Useful information](#useful-information) - - [Tickets](#tickets) - - [Terminology](#terminology) - - [Assumptions](#assumptions) - - [Non-functional requirements](#non-functional-requirements) - - [Functional requirements](#functional-requirements) - - [Scenarios](#scenarios) - - [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page) - - [**Scenario: User is NOT notified when all installed prebuilt rules are up to date**](#scenario-user-is-not-notified-when-all-installed-prebuilt-rules-are-up-to-date) - - [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded) - - [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available) - - [Rule upgrade workflow: individual upgrade from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) - - [**Scenario: User can upgrade conflict-free prebuilt rules one by one**](#scenario-user-can-upgrade-conflict-free-prebuilt-rules-one-by-one) - - [**Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts**](#scenario-user-cannot-upgrade-prebuilt-rules-one-by-one-from-rules-update-table-if-they-have-conflicts) - - [Rule upgrade workflow: bulk upgrade from Rule Updates table](#rule-upgrade-workflow-individual-and-bulk-updates-from-rule-updates-table) - - [**Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-conflict-free-prebuilt-rules-selected-on-the-page) - - [**Scenario: User cannot upgrade multiple prebuilt rules selected on the page when they have upgrade conflicts**](#scenario-user-cannot-upgrade-multiple-prebuilt-rules-selected-on-the-page-when-they-have-upgrade-conflicts) - - [**Scenario: User can upgrade all available conflict-free prebuilt rules at once**](#scenario-user-can-upgrade-all-available-conflict-free-prebuilt-rules-at-once) - - [**Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts**](#scenario-user-cannot-upgrade-all-prebuilt-rules-at-once-if-they-have-upgrade-conflicts) - - [**Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade in the Rules Table**](#scenario-user-can-upgrade-only-conflict-free-rules-when-a-mix-of-rules-with-and-without-conflicts-are-selected-for-upgrade-in-the-rules-table) - - [**Scenario: User can upgrade only conflict-free rules when user attempts to upgrade all rules and only a subset contains upgrade conflicts**](#scenario-user-can-upgrade-only-conflict-free-rules-when-user-attempts-to-upgrade-all-rules-and-only-a-subset-contains-upgrade-conflicts) - - [Rule upgrade workflow: upgrading rules with rule type change](#rule-upgrade-workflow-upgrading-rules-with-rule-type-change) - - [**Scenario: User can upgrade rule with rule type change individually**](#scenario-user-can-upgrade-rule-with-rule-type-change-individually) - - [**Scenario: User can bulk upgrade selected rules with rule type changes**](#scenario-user-can-bulk-upgrade-selected-rules-with-rule-type-changes) - - [**Scenario: User can bulk upgrade all rules with rule type changes**](#scenario-user-can-bulk-upgrade-all-rules-with-rule-type-changes) - - [Rule upgrade workflow: rule previews](#rule-upgrade-workflow-rule-previews) - - [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade) - - [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview) - - [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading) - - [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading) - - [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination) - - [MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in JSON diff view](#milestone-2-legacy---rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view) - - [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view) - - [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade) - - [**Scenario: Rule actions and exception lists should not be shown as modified**](#scenario-rule-actions-and-exception-lists-should-not-be-shown-as-modified) - - [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview) - - [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview) - - [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified) - - [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default) - - [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically) - - [MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in per-field diff view](#milestone-2-legacy---rule-upgrade-workflow-viewing-rule-changes-in-per-field-diff-view) - - [**Scenario: User can see changes in a side-by-side per-field diff view**](#scenario-user-can-see-changes-in-a-side-by-side-per-field-diff-view) - - [**Scenario: User can see changes when updated rule is a different rule type**](#scenario-user-can-see-changes-when-updated-rule-is-a-different-rule-type) - - [**Scenario: Field groupings should be rendered together in the same accordion panel**](#scenario-field-groupings-should-be-rendered-together-in-the-same-accordion-panel) - - [**Scenario: Undefined values are displayed with empty diffs**](#scenario-undefined-values-are-displayed-with-empty-diffs) - - [**Scenario: Field diff components have the same grouping and order as in rule details overview**](#scenario-field-diff-components-have-the-same-grouping-and-order-as-in-rule-details-overview) - - [Rule upgrade workflow: preserving rule bound data](#rule-upgrade-workflow-preserving-rule-bound-data) - - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-the-same-rule-type) - - [**Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type**](#scenario-rule-bound-data-is-preserved-after-upgrading-a-rule-to-a-newer-version-with-a-different-rule-type) - - [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases) - - [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed) - - [Error handling](#error-handling) - - [**Scenario: Error is handled when any upgrade operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-upgrade-operation-on-prebuilt-rules-fails) - - [Rule upgrade via the Prebuilt rules API](#rule-upgrade-via-the-prebuilt-rules-api) - - [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated) - - [**Scenario: API does not upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-upgrade-prebuilt-rules-if-they-are-up-to-date) - - [Authorization / RBAC](#authorization-rbac) - - [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules) - - -## Useful information - -### Tickets - -- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic - -**Milestone 3 - Prebuilt Rules Customization:** -- [Milestone 3 epic ticket](https://github.com/elastic/kibana/issues/174168) -- [Tests for prebuilt rule upgrade workflow #202078](https://github.com/elastic/kibana/issues/202078) - -**Milestone 2:** -- [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) -- [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) - -### Terminology - -- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**. - -- **Package**: `security_detection_engine` Fleet package that we use to distribute prebuilt detection rules in the form of `security-rule` assets (saved objects). - -- **Real package**: actual latest stable package distributed and pulled from EPR via Fleet. - -- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly in the test setup, either by using the ES client _in integration tests_ or by an API request _in Cypress tests_. - -- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. - -- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. - - CTA to install prebuilt rules - at this moment, it's a link button with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. - - CTA to upgrade prebuilt rules - at this moment, it's a tab with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. - -### Assumptions - -- Below scenarios only apply to prebuilt detection rules. -- Users should be able to install and upgrade prebuilt rules on the `Basic` license and higher. -- EPR is available for fetching the package unless explicitly indicated otherwise. -- Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored. - -### Non-functional requirements - -- Notifications, rule installation and rule upgrade workflows should work: - - regardless of the package type: with historical rule versions or without; - - regardless of the package registry availability: i.e., they should also work in air-gapped environments. -- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet. -- Kibana should not crash with Out Of Memory exception during package installation. -- For test purposes, it should be possible to use detection rules package versions lower than the latest. - -### Functional requirements - -- User should be able to install prebuilt rules with and without previewing what exactly they would install (rule properties). -- User should be able to upgrade prebuilt rules with and without previewing what updates they would apply (rule properties of target rule versions). -- If user chooses to preview a prebuilt rule to be installed/upgraded, we currently show this preview in a flyout. -- In the prebuilt rule preview a tab that doesn't have any sections should not be displayed and a section that doesn't have any properties also should not be displayed. - -Examples of rule properties we show in the prebuilt rule preview flyout: - -```Gherkin -Examples: -| rule_type | property | tab | section | -│ All rule types │ Author │ Overview │ About │ -│ All rule types │ Building block │ Overview │ About │ -│ All rule types │ Severity │ Overview │ About │ -│ All rule types │ Severity override │ Overview │ About │ -│ All rule types │ Risk score │ Overview │ About │ -│ All rule types │ Risk score override │ Overview │ About │ -│ All rule types │ Reference URLs │ Overview │ About │ -│ All rule types │ False positive examples │ Overview │ About │ -│ All rule types │ Custom highlighted fields │ Overview │ About │ -│ All rule types │ License │ Overview │ About │ -│ All rule types │ Rule name override │ Overview │ About │ -│ All rule types │ MITRE ATT&CK™ │ Overview │ About │ -│ All rule types │ Timestamp override │ Overview │ About │ -│ All rule types │ Tags │ Overview │ About │ -│ All rule types │ Type │ Overview │ Definition │ -│ All rule types │ Related integrations │ Overview │ Definition │ -│ All rule types │ Required fields │ Overview │ Definition │ -│ All rule types │ Timeline template │ Overview │ Definition │ -│ All rule types │ Runs every │ Overview │ Schedule │ -│ All rule types │ Additional look-back time │ Overview │ Schedule │ -│ All rule types │ Setup guide │ Overview │ Setup guide │ -│ All rule types │ Investigation guide │ Investigation guide │ Investigation guide │ -│ Custom Query │ Index patterns │ Overview │ Definition │ -│ Custom Query │ Data view ID │ Overview │ Definition │ -│ Custom Query │ Data view index pattern │ Overview │ Definition │ -│ Custom Query │ Custom query │ Overview │ Definition │ -│ Custom Query │ Filters │ Overview │ Definition │ -│ Custom Query │ Saved query name │ Overview │ Definition │ -│ Custom Query │ Saved query filters │ Overview │ Definition │ -│ Custom Query │ Saved query │ Overview │ Definition │ -│ Custom Query │ Suppress alerts by │ Overview │ Definition │ -│ Custom Query │ Suppress alerts for │ Overview │ Definition │ -│ Custom Query │ If a suppression field is missing │ Overview │ Definition │ -│ Machine Learning │ Anomaly score threshold │ Overview │ Definition │ -│ Machine Learning │ Machine Learning job │ Overview │ Definition │ -│ Threshold │ Threshold │ Overview │ Definition │ -│ Threshold │ Index patterns │ Overview │ Definition │ -│ Threshold │ Data view ID │ Overview │ Definition │ -│ Threshold │ Data view index pattern │ Overview │ Definition │ -│ Threshold │ Custom query │ Overview │ Definition │ -│ Threshold │ Filters │ Overview │ Definition │ -│ Event Correlation │ EQL query │ Overview │ Definition │ -│ Event Correlation │ Filters │ Overview │ Definition │ -│ Event Correlation │ Index patterns │ Overview │ Definition │ -│ Event Correlation │ Data view ID │ Overview │ Definition │ -│ Event Correlation │ Data view index pattern │ Overview │ Definition │ -│ Indicator Match │ Indicator index patterns │ Overview │ Definition │ -│ Indicator Match │ Indicator mapping │ Overview │ Definition │ -│ Indicator Match │ Indicator filters │ Overview │ Definition │ -│ Indicator Match │ Indicator index query │ Overview │ Definition │ -│ Indicator Match │ Index patterns │ Overview │ Definition │ -│ Indicator Match │ Data view ID │ Overview │ Definition │ -│ Indicator Match │ Data view index pattern │ Overview │ Definition │ -│ Indicator Match │ Custom query │ Overview │ Definition │ -│ Indicator Match │ Filters │ Overview │ Definition │ -│ New Terms │ Fields │ Overview │ Definition │ -│ New Terms │ History Window Size │ Overview │ Definition │ -│ New Terms │ Index patterns │ Overview │ Definition │ -│ New Terms │ Data view ID │ Overview │ Definition │ -│ New Terms │ Data view index pattern │ Overview │ Definition │ -│ New Terms │ Custom query │ Overview │ Definition │ -│ New Terms │ Filters │ Overview │ Definition │ -│ ESQL │ ESQL query │ Overview │ Definition │ -│ ESQL │ Suppress alerts by │ Overview │ Definition │ -│ ESQL │ Suppress alerts for │ Overview │ Definition │ -│ ESQL │ If a suppression field is missing │ Overview │ Definition │ -``` - -## Scenarios - -### Rule installation and upgrade notifications on the Rule Management page - -#### **Scenario: User is NOT notified when all installed prebuilt rules are up to date** - -**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. - -```Gherkin -Given all the latest prebuilt rules are installed in Kibana -When user opens the Rule Management page -And user should NOT see a CTA to upgrade prebuilt rules -And user should NOT see a number of rules available to upgrade -And user should NOT see the Rule Updates table -``` - -#### **Scenario: User is notified when some prebuilt rules can be upgraded** - -**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. - -```Gherkin -Given X prebuilt rules are installed in Kibana -And there are no more prebuilt rules available to install -And for Z of the installed rules there are new versions available -When user opens the Rule Management page -Then user should NOT see a CTA to install prebuilt rules -And user should NOT see a number of rules available to install -And user should see a CTA to upgrade prebuilt rules -And user should see the number of rules available to upgrade (Z) -``` - -#### **Scenario: User is notified when both rules to install and upgrade are available** - -**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. - -```Gherkin -Given X prebuilt rules are installed in Kibana -And there are Y more prebuilt rules available to install -And for Z of the installed rules there are new versions available -When user opens the Rule Management page -Then user should see a CTA to install prebuilt rules -And user should see the number of rules available to install (Y) -And user should see a CTA to upgrade prebuilt rules -And user should see the number of rules available to upgrade (Z) -``` - -### Rule upgrade workflow: individual updates from Rule Updates table - -#### **Scenario: User can upgrade conflict-free prebuilt rules one by one** - -**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And user is on the Rule Management page -When user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -When user upgrades one individual rule without previewing it -Then success message should be displayed after upgrade -And the upgraded rule should be removed from the table -And user should see the number of rules available to upgrade decreased by 1 -``` - -#### **Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts** - -**Automation**: 1 e2e test with mock rules - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And user is on the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -And for Z (where Z < Y) of the rules with upgrades there are upgrade conflicts -Then for those Z rules the Upgrade Rule button should be disabled -And the user should not be able to upgrade them directly from the table -And there should be a message/tooltip indicating why the rule cannot be upgraded directly -``` - -### Rule upgrade workflow: bulk updates from Rule Updates table - -#### **Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page** - -**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -When user selects Z (where Z < Y) rules, which have no upgrade conflicts -Then user should see a CTA to upgrade rules -When user clicks the CTA -Then success message should be displayed after upgrade -And all the upgraded rules should be removed from the table -And user should see the number of rules available to upgrade decreased by number of upgraded rules - -Examples: - | Z | - | a few rules on the page, e.g. 2 | - | all rules on the page, e.g. 12 | -``` - -#### **Scenario: User cannot upgrade selected prebuilt rules with conflicts** - -**Automation**: 1 e2e test with mock rules - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And all of those Y new versions have conflicts with the current versions -And user is on the Rule Management page -When user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -When user selects rules, all of which have upgrade conflicts -Then user should see a CTA to upgrade number of rules, which should be disabled -When user hovers on the CTA to upgrade multiple rules -Then a message should be displayed that informs the user why the rules cannot be updated - -Examples: - | Z | - | a few rules on the page, e.g. 2 | - | all rules on the page, e.g. 12 | -``` - -#### **Scenario: User can upgrade all available conflict-free prebuilt rules at once** - -**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/\* endpoints in integration. - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And those Y new versions don't have conflicts with the current versions -And user is on the Rule Management page -When user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -When user upgrades all rules -Then success message should be displayed after upgrade -And user should NOT see a CTA to upgrade prebuilt rules -And user should NOT see a number of rules available to upgrade -``` - -#### **Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts** - -**Automation**: 1 e2e test with mock rules - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And all Y new versions have conflicts with the current versions -And user is on the Rule Management page -When user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -Then user should see a CTA to upgrade all rules -And the CTA to upgrade all rules should be disabled -When user hovers on the CTA to upgrade all rules -Then a message should be displayed that informs the user why the rules cannot be updated -``` - -#### **Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade in the Rules Table** - -**Automation**: 1 e2e test with mock rules - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And a subset Z of the rules have conflicts with the current versions -And user is on the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -And user selects rules, which is a mixture of rules with and without upgrade conflicts -Then user should see a CTA to upgrade number of rules, which is enabled -When user clicks the CTA -A modal window should inform the user that only W rules without conflicts will be upgraded -When user confirms the action in the modal -Then success message should be displayed after upgrade informing that W rules were updated -And the W upgraded rules should be removed from the table -And the remaining Z - W rules should still be present in the table -And user should see the number of rules available to upgrade decreased by W number of upgraded rules - -Examples: - | Z | - | a few rules on the page, e.g. 2 | - | all rules on the page, e.g. 12 | -``` - -#### **Scenario: User can upgrade only conflict-free rules when attempting to upgrade all rules** - -**Automation**: 1 e2e test with mock rules - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -And Z (where Z < Y) rules have conflicts with the current versions -And user is on the Rule Updates table -Then Y rules available for upgrade should be displayed in the table -Then user should see an enabled CTA to upgrade all rules -When user clicks the CTA -A modal window should inform the user that only K (where K < Y) rules without conflicts will be upgraded -When user confirms the action in the modal -Then success message should be displayed after upgrade informing that K rules were updated -And the K upgraded rules should be removed from the table -And the remaining M = Y - K rules should still be present in the table -And user should see the number of rules available to upgrade decreased by K number of upgraded rules -``` - - -### Rule upgrade workflow: upgrading rules with rule type change - -#### **Scenario: User can upgrade rule with rule type change individually** - -**Automation**: 1 e2e test with mock rules - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has an update available that changes its rule type -When user opens the Rule Updates table -Then this rule should be displayed in the table -And the Upgrade Rule button should be enabled -When user upgrades the rule -Then success message should be displayed after upgrade -And the upgraded rule should be removed from the table -And user should see the number of rules available to upgrade decreased by 1 -``` - -#### **Scenario: User can bulk upgrade selected rules with rule type changes** - - -**Automation**: 1 e2e test with mock rules - -```Gherkin -Given X prebuilt rules are installed in Kibana -And Y of these rules have updates available that change their rule types -When user opens the Rule Updates table -Then Y rules should be displayed in the table -When user selects Z rules (where Z < Y) with rule type changes -Then user should see an enabled CTA to upgrade Z rules -When user clicks the CTA -Then success message should be displayed after upgrade -And all Z upgraded rules should be removed from the table -And user should see the number of rules available to upgrade decreased by Z -``` - -#### **Scenario: User can bulk upgrade all rules with rule type changes** - -**Automation**: 1 e2e test with mock rules - -```Gherkin -Given X prebuilt rules are installed in Kibana -And all X rules have updates available that change their rule types -When user opens the Rule Updates table -Then X rules should be displayed in the table -And user should see an enabled CTA to upgrade all rules -When user clicks the CTA to upgrade all rules -Then success message should be displayed after upgrade -And all X upgraded rules should be removed from the table -``` - -### Rule upgrade workflow: rule previews - -#### **Scenario: User can preview rules available for upgrade** - -```Gherkin -Given there is at least one prebuilt rule installed in Kibana -And for this rule there is a new version available -And user is on the Rule Management page -When user opens the Rule Updates table -Then this rule should be displayed in the table -When user opens the rule preview for this rule -Then the preview should open -When user closes the preview -Then it should disappear -``` - -#### **Scenario: User can upgrade a rule using the rule preview** - -**Automation**: 1 e2e test - -```Gherkin -Given there is at least one prebuilt rule installed in Kibana -And for this rule there is a new version available -And user is on the Rule Management page -When user opens the Rule Updates table -Then this rule should be displayed in the table -When user opens the rule preview for this rule -Then the preview should open -When user upgrades the rule using a CTA in the rule preview -Then the rule should be upgraded to the latest version -And a success message should be displayed after upgrade -And the rule should be removed from the Rule Updates table -And user should see the number of rules available to upgrade as initial number minus 1 -``` - -#### **Scenario: User can see correct rule information in preview before upgrading** - -**Automation**: 1 e2e test - -```Gherkin -Given X prebuilt rules of all types are installed in Kibana -And for all of the installed rules there are new versions available -And user is on the Rule Management page -When user opens the Rule Updates table -Then all X rules available for upgrade should be displayed in the table -When user opens a rule preview for any rule -Then the preview should appear -And the "Updates" tab should be active -When user selects the "Overview" tab -Then all properties of the new version of a rule should be displayed in the correct tab and section of the preview (see examples of rule properties above) -``` - -#### **Scenario: Tabs and sections without content should be hidden in preview before upgrading** - -**Automation**: 1 e2e test - -```Gherkin -Given at least 1 prebuilt rule is installed in Kibana -And for this rule there is a new version available -And the updated version of a rule has neither Setup guide nor Investigation guide -And user is on the Rule Management page -When user opens the Rule Updates table -Then all rules available for upgrade should be displayed in the table -When user opens the rule preview for a rule without Setup guide and Investigation guide -Then the preview should open -And the Setup Guide section should NOT be displayed in the Overview tab -And the Investigation Guide tab should NOT be displayed -``` - -### Rule upgrade workflow: filtering, sorting, pagination - -TODO: add scenarios https://github.com/elastic/kibana/issues/166215 - -### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in JSON diff view - -> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. - -#### **Scenario: User can see changes in a side-by-side JSON diff view** - -**Automation**: 1 e2e test - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of these rules new versions are available -When user opens the Rule Updates table and selects a rule -Then the upgrade preview should open -And rule changes should be displayed in a two-column JSON diff view -And correct rule version numbers should be displayed in their respective columns -When the user selects another rule without closing the preview -Then the preview should display the changes for the newly selected rule -``` - -#### **Scenario: User can see precisely how property values would change after upgrade** - -**Automation**: 1 UI integration test - -```Gherkin -Given a rule preview with rule changes is open -Then each line of that was should have background -And marked with badge -And each changed word in should be highlighted with - -Examples: -| change_type | column | bg_color | accent_color | line_badge | -| updated | Current rule | removed_bg_color | removed_accent_color | - | -| updated | Elastic update | added_bg_color | added_accent_color | + | -| removed | Current rule | removed_bg_color | none | - | -| removed | Elastic update | none | none | none | -| added | Current rule | none | none | none | -| added | Elastic update | added_bg_color | none | + | -``` - -#### **Scenario: Rule actions and exception lists should not be shown as modified** - -**Automation**: 1 UI integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And the currently installed version of this rule doesn't have any actions or an exception list -And a user has set up actions and an exception list for this rule -And this rule has an update available -And the update doesn't define any actions or an exception list -When a user opens the upgrade preview for this rule -Then the preview should open -And the JSON diff shouldn't show any modifications to rule's actions or exception list -``` - -#### **Scenario: Dynamic properties should not be included in preview** - -**Automation**: 1 e2e test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule is disabled by default -And a user has enabled this rule -And this rule executed at least once -And this rule has an update available -When user opens the upgrade preview -Then the preview should open -And the JSON diff shouldn't show any properties on both sides - -Examples: -| property | -| execution_summary | -| enabled | -``` - -#### **Scenario: Technical properties should not be included in preview** - -**Automation**: 1 UI integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has an update available -When a user opens the upgrade preview -Then the preview should open -And the JSON diff shouldn't show any properties on both sides - -Examples: -| technical_property | -| revision | -| updated_at | -| updated_by | -| created_at | -| created_by | -``` - -#### **Scenario: Properties with semantically equal values should not be shown as modified** - -**Automation**: 1 UI integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has an update available -And the update has properties with different, but semantically equal values -When a user opens the upgrade preview -Then the preview should open -And the JSON diff shouldn't show any changes to properties with semantically equal values - -Duration examples: -| 1h | -| 60m | -| 3600s | - -Empty value examples: -| no value | -| '' | -| [] | -| undefined | -| null | -``` - -#### **Scenario: Unchanged sections of a rule should be hidden by default** - -**Automation**: 1 UI integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has an update available -When a user opens the upgrade preview -Then the preview should open -And only the sections of the diff that have changes should be visible -And unchanged sections should be hidden behind a button with a number of unchanged lines -When a user clicks on the hidden section button -Then the section should expand and show the unchanged properties -``` - -#### **Scenario: Properties should be sorted alphabetically** - -**Automation**: 1 UI integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has an update available -When a user opens the upgrade preview -Then the preview should open -And visible properties should be sorted alphabetically -When a user expands all hidden sections -Then all properties of the rule should be sorted alphabetically -``` - -### MILESTONE 2 (Legacy) - Rule upgrade workflow: viewing rule changes in per-field diff view - -> These flow were created for Milestone 2 of the Prebuilt Rules Customization epic, before users could customize prebuilt rules. This section should be deleted once Milestone 3 goes live. - -#### **Scenario: User can see changes in a side-by-side per-field diff view** - -**Automation**: 1 e2e test - -```Gherkin -Given X prebuilt rules are installed in Kibana -And for Y of these rules new versions are available -When user opens the Rule Updates table and selects a rule -Then the per-field upgrade preview should open -And rule changes should be displayed in a two-column diff view with each field in its own accordion component -And all field diff accordions should be open by default -And correct rule version numbers should be displayed in their respective columns -When the user selects another rule without closing the preview -Then the preview should display the changes for the newly selected rule -``` - -#### **Scenario: User can see changes when updated rule is a different rule type** - -**Automation**: 1 e2e test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has an update available that changes the rule type -When user opens the upgrade preview -Then the rule type changes should be displayed in grouped field diffs with corresponding query fields -# When tooltip enhancement is added, this step needs to be added to the corresponding test scenario -And a tooltip is displayed with information about changing rule types -``` - -#### **Scenario: Field groupings should be rendered together in the same accordion panel** - -**Automation**: 1 UI integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule contains one or more values -When user opens the upgrade preview -The diff accordion panel should display its grouped rule properties -And each property should have its name displayed inside the panel above its value - -Examples: -| field | -| data_source | -| kql_query | -| eql_query | -| esql_query | -| threat_query | -| rule_schedule | -| rule_name_override | -| timestamp_override | -| timeline_template | -| building_block | -| threshold | -``` - -#### **Scenario: Undefined values are displayed with empty diffs** - -**Automation**: 1 UI integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has field in the version that didn't exist in the version -When a user opens the upgrade preview -Then the preview should open -And the old/new field should render an empty panel - -Examples: -| version_one | version_two | -| target | current | -| current | target | -``` - -#### **Scenario: Field diff components have the same grouping and order as in rule details overview** - -**Automation**: 1 UI integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has multiple fields that are different between the current and target version -When a user opens the upgrade preview -Then the multiple field diff accordions should be sorted in the same order as on the rule details overview tab -And the field diff accordions should be grouped inside its corresponding
accordion -And any
accordion that doesn't have fields inside it shouldn't be displayed - -Examples: -| section | -| About | -| Definition | -| Schedule | -| Setup Guide | -``` - -### Rule upgrade workflow: preserving rule bound data - -#### **Scenario: Rule bound data is preserved after upgrading a rule to a newer version with the same rule type** - -**Automation**: 1 unit test per case, 1 integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has an update available -And the update has the same rule type -When a user upgrades the rule -Then the rule bound data should be preserved -``` - -Examples: generated alerts, exception lists (rule exception list, shared exception list, endpoint exception list), timeline reference, actions, enabled state, execution results and execution events. - -#### **Scenario: Rule bound data is preserved after upgrading a rule to a newer version with a different rule type** - -**Automation**: 1 unit test per case, 1 integration test - -```Gherkin -Given a prebuilt rule is installed in Kibana -And this rule has an update available -And the update has a different rule type -When a user upgrades the rule -Then the rule bound data should be preserved -``` - -Examples: generated alerts, exception lists (rule exception list, shared exception list, endpoint exception list), timeline reference, actions, enabled state, execution results and execution events. - -### Rule upgrade workflow: misc cases - -#### **Scenario: User doesn't see the Rule Updates tab until the package installation is completed** - -**Automation**: unit tests. - -```Gherkin -Given prebuilt rules package is not installed -When user opens the Rule Management page -Then user should NOT see the Rule Updates tab until the package installation is completed and there are rules available for upgrade -``` - -### Error handling - -#### **Scenario: Error is handled when any upgrade operation on prebuilt rules fails** - -**Automation**: e2e test with mock rules - -```Gherkin -When user is prebuilt rules -And this operation fails -Then user should see an error message - -Examples: - | operation | - | upgrading all | - | upgrading selected | - | upgrading individual | -``` - - -### Rule upgrade via the Prebuilt rules API - -There's a legacy prebuilt rules API and a new one. Both should be tested against two types of the package: with and without historical rule versions. - -#### **Scenario: API can upgrade prebuilt rules that are outdated** - -**Automation**: 4 integration tests with mock rules. - -```Gherkin -Given the package is installed -And the package contains N rules -When user installs all rules via install -And new X+1 version of a rule asset -And user gets prebuilt rules status via status -Then the endpoint should return 200 with -When user upgrades all rules via upgrade -Then the endpoint should return 200 with - -Examples: - | package_type | api | assets_update | status_response | upgrade_response | - | with historical versions | legacy | gets added | not_updated: 1 | installed: 0, updated: 1 | - | w/o historical versions | legacy | replaces X | not_updated: 1 | installed: 0, updated: 1 | - | with historical versions | new | gets added | to_upgrade: 1 | total: 1, succeeded: 1 | - | w/o historical versions | new | replaces X | to_upgrade: 1 | total: 1, succeeded: 1 | -``` - -TODO: Check why for the legacy API Dmitrii has added 2 integration tests for `rule package with historical versions` instead of 1: - -- `should update outdated prebuilt rules when previous historical versions available` -- `should update outdated prebuilt rules when previous historical versions unavailable` - -(NOTE: the second scenario tests that, if a new version of a rule is released, it can upgrade the current instance of that rule even if the historical versions of that rule are no longer in the package) - -Notes: - -- Legacy API: - - install: `PUT /api/detection_engine/rules/prepackaged` - - upgrade: `PUT /api/detection_engine/rules/prepackaged` - - status: `GET /api/detection_engine/rules/prepackaged/_status` -- New API: - - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` - - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` - - status: `GET /internal/detection_engine/prebuilt_rules/status` - -#### **Scenario: API does not upgrade prebuilt rules if they are up to date** - -**Automation**: 4 integration tests with mock rules. - -```Gherkin -Given the package is installed -And the package contains N rules -When user installs all rules via install -And user gets prebuilt rules status via status -Then the endpoint should return 200 with -When user calls install -Then the endpoint should return 200 with -When user calls upgrade -Then the endpoint should return 200 with - -Examples: - | package_type | api | status_response | install_response | upgrade_response | - | with historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | - | w/o historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | - | with historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | - | w/o historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | -``` - -Notes: - -- Legacy API: - - install: `PUT /api/detection_engine/rules/prepackaged` - - upgrade: `PUT /api/detection_engine/rules/prepackaged` - - status: `GET /api/detection_engine/rules/prepackaged/_status` -- New API: - - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` - - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` - - status: `GET /internal/detection_engine/prebuilt_rules/status` - -### Authorization / RBAC - -#### **Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules** - -**Automation**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and upgrade endpoints. - -```Gherkin -Given user with "Security: read" privileges on Security Solution -And X prebuilt rules are installed in Kibana -And for Y of the installed rules there are new versions available -When user opens the Rule Management page -And user opens the Rule Updates table -Then user should see prebuilt rules available to upgrade -But user should not be able to upgrade them -``` \ No newline at end of file From f068731ed2c23c7023914f28b6ceb514b570aeca Mon Sep 17 00:00:00 2001 From: jpdjere Date: Wed, 18 Dec 2024 14:13:56 -0300 Subject: [PATCH 18/18] Fixes --- .../prebuilt_rules/installation.md | 2 +- .../prebuilt_rules/upgrade.md | 39 +++++++------------ 2 files changed, 16 insertions(+), 25 deletions(-) diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md index dac2a0e8f1ac6..6f91d4958650c 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/installation.md @@ -386,7 +386,7 @@ Then the preview should appear And all properties of a rule should be displayed in the correct tab and section of the preview (see examples of rule properties above) ``` -#### **Scenario: Tabs and sections without content should be hidden in preview before installing** +#### **Scenario: Optional tabs and sections without content should be hidden in preview before installing** **Automation**: 1 e2e test diff --git a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md index 20520ae1cab0d..4beb517f9598a 100644 --- a/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md +++ b/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade.md @@ -194,7 +194,7 @@ Examples: ## Scenarios -### Rule installation and upgrade notifications on the Rule Management page +### Rule upgrade notifications on the Rule Management page #### **Scenario: User is NOT notified when all installed prebuilt rules are up to date** @@ -247,8 +247,7 @@ And user should see the number of rules available to upgrade (Z) ```Gherkin Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available -And user is on the Rule Management page -When user opens the Rule Updates table +When user is on the Rule Updates table Then Y rules available for upgrade should be displayed in the table When user upgrades one individual rule without previewing it Then success message should be displayed after upgrade @@ -304,8 +303,7 @@ Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available And all of those Y new versions have conflicts with the current versions And user is on the Rule Management page -When user opens the Rule Updates table -Then Y rules available for upgrade should be displayed in the table +When user is on the Rule Updates table When user selects rules, all of which have upgrade conflicts Then user should see a CTA to upgrade number of rules, which should be disabled When user hovers on the CTA to upgrade multiple rules @@ -325,8 +323,7 @@ Examples: Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available And those Y new versions don't have conflicts with the current versions -And user is on the Rule Management page -When user opens the Rule Updates table +When user is on the Rule Updates table Then Y rules available for upgrade should be displayed in the table When user upgrades all rules Then success message should be displayed after upgrade @@ -342,8 +339,7 @@ And user should NOT see a number of rules available to upgrade Given X prebuilt rules are installed in Kibana And for Y of the installed rules there are new versions available And all Y new versions have conflicts with the current versions -And user is on the Rule Management page -When user opens the Rule Updates table +When user opens the Rule Updates table Then Y rules available for upgrade should be displayed in the table Then user should see a CTA to upgrade all rules And the CTA to upgrade all rules should be disabled @@ -351,7 +347,7 @@ When user hovers on the CTA to upgrade all rules Then a message should be displayed that informs the user why the rules cannot be updated ``` -#### **Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade in the Rules Table** +#### **Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade** **Automation**: 1 e2e test with mock rules @@ -409,11 +405,9 @@ Given a prebuilt rule is installed in Kibana And this rule has an update available that changes its rule type When user opens the Rule Updates table Then this rule should be displayed in the table -And the Upgrade Rule button should be enabled -When user upgrades the rule -Then success message should be displayed after upgrade -And the upgraded rule should be removed from the table -And user should see the number of rules available to upgrade decreased by 1 +And the Upgrade Rule button should be disabled +And the user should not be able to upgrade them directly from the table +And there should be a message/tooltip indicating why the rule cannot be upgraded directly ``` #### **Scenario: User can bulk upgrade selected rules with rule type changes** @@ -427,11 +421,9 @@ And Y of these rules have updates available that change their rule types When user opens the Rule Updates table Then Y rules should be displayed in the table When user selects Z rules (where Z < Y) with rule type changes -Then user should see an enabled CTA to upgrade Z rules -When user clicks the CTA -Then success message should be displayed after upgrade -And all Z upgraded rules should be removed from the table -And user should see the number of rules available to upgrade decreased by Z +The button to upgrade the Z selected rules should be disabled +And the user should not be able to upgrade them directly from the table +And there should be a message/tooltip indicating why the rule cannot be upgraded directly ``` #### **Scenario: User can bulk upgrade all rules with rule type changes** @@ -443,10 +435,9 @@ Given X prebuilt rules are installed in Kibana And all X rules have updates available that change their rule types When user opens the Rule Updates table Then X rules should be displayed in the table -And user should see an enabled CTA to upgrade all rules -When user clicks the CTA to upgrade all rules -Then success message should be displayed after upgrade -And all X upgraded rules should be removed from the table +The button to upgrade all rules with should be disabled +And the user should not be able to upgrade them directly from the table +And there should be a message/tooltip indicating why the rule cannot be upgraded directly ``` ### Rule upgrade workflow: rule previews