[Security Solution] Refactor prebuilt rules integration tests by maximpn · Pull Request #219831 · elastic/kibana

maximpn · 2025-05-01T11:32:55Z

Related Epic: #179907

Summary

This PR refactors prebuilt rules integration tests structure to streamline the implementation of test plans targeted Prebuilt Rules Customization Milestone 4.

Details

Existing integration tests structure have prebuilt rules related integration tests scattered around rules management area. Due to historical reasons and pace of the prebuilt rules customization development some old tests were updated, some new were added as random spots as well as some new tests structure was suggested.

This PR moves files and some tests around to the following structure

test_suites/detection_response/rules_management/prebuilt_rules is the root folder for prebuilt rules related integration tests
customization_disabled subfolder contains prebuilt rules integration tests covering scenarios when users have insufficient for customization license level (basic/essentials)
customization_enabled subfolder contains prebuilt rules integration tests covering scenarios when users have sufficient for customization license level
customization_disabled and customization_enabled subfoldera have test suites grouped by sub domains
- prebuilt_rules_package - contains integration tests related to detection rules Fleet package installation and updating, bootstrap is also belong to here
- install_prebuilt_rules - contains tests related to prebuilt rules installation from prebuilt rule assets
- upgrade_prebuilt_rules - contains tests related to prebuilt rules upgrade workflow
- customization - contains tests related to prebuilt rules customization including is_customized calculation
- import_export - contains tests related to exporting and importing customized and non-customized prebuilt rules
- status - contain status endpoints related tests

elasticmachine · 2025-05-01T11:32:58Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2025-05-01T11:32:58Z

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

elasticmachine · 2025-05-01T11:32:58Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

prodsecmachine · 2025-05-01T11:34:12Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

dplumlee

LGTM @maximpn I think this structure is much cleaner and readable than what we currently have. Left a few comments and questions but approving in advance

dplumlee · 2025-05-27T19:27:52Z

...lt_rules/customization_enabled/install_prebuilt_rules/install_mocked_prebuilt_rule_assets.ts

+      it('installs prebuilt rules', async () => {
+        await createPrebuiltRuleAssetSavedObjects(es, getRuleAssetSavedObjects());
+        const body = await installPrebuiltRules(es, supertest);
+
+        expect(body.summary.succeeded).toBe(RULES_COUNT);
+        expect(body.summary.failed).toBe(0);
+        expect(body.summary.skipped).toBe(0);
+      });
+
+      it('installs correct prebuilt rule versions', async () => {
+        await createPrebuiltRuleAssetSavedObjects(es, getRuleAssetSavedObjects());
+        const body = await installPrebuiltRules(es, supertest);
+
+        // Check that all prebuilt rules were actually installed and their versions match the latest
+        expect(body.results.created).toEqual(
+          expect.arrayContaining([
+            expect.objectContaining({ rule_id: 'rule-1', version: 1 }),
+            expect.objectContaining({ rule_id: 'rule-2', version: 2 }),
+            expect.objectContaining({ rule_id: 'rule-3', version: 3 }),
+            expect.objectContaining({ rule_id: 'rule-4', version: 4 }),
+          ])
+        );
+      });


Are these checks worth two separate tests?

Could you elaborate on the question? Do you mean separate test suites?

Meaning these two tests (installs rules/installs correct rules) do the same operation and then just check different items on the same response body. Wouldn't one test checking all of these lead to the same results?

Indeed. But I didn't have a goal to rectify all tests as part of this PR. It just merged existing and scattered tests together. In the next PRs test suites will be rectified. And it will be addressed by sub domains.

dplumlee · 2025-05-27T19:44:22Z

...ules/customization_enabled/prebuilt_rules_package/air_gapped/fixtures/packages/.eslintignore

Why is this file needed? Can we not do this from the main .eslintignore file?

It was a leftover after tossing files around. It doesn't ignore any files and has been removed.

dplumlee · 2025-05-27T19:47:07Z

...ebuilt_rules/customization_enabled/upgrade_prebuilt_rules/bulk_upgrade_all_prebuilt_rules.ts

+      await deleteAllPrebuiltRuleAssets(es, log);
+    });
+
+    describe('w/ historical versions', () => {


I think it's better to leave these as with/without for people who don't know the shorthand

dplumlee · 2025-05-27T19:47:21Z

...ebuilt_rules/customization_enabled/upgrade_prebuilt_rules/bulk_upgrade_all_prebuilt_rules.ts

    });

-    describe('without historical versions', () => {
+    describe('w/o historical versions', () => {


Same thing as above

dplumlee · 2025-05-27T19:48:15Z

...t_rules/customization_enabled/upgrade_prebuilt_rules/bulk_upgrade_selected_prebuilt_rules.ts

+      await deleteAllPrebuiltRuleAssets(es, log);
+    });
+
+    describe('w/ historical versions', () => {


Same as above, prefer with/without

dplumlee · 2025-05-27T19:49:56Z

...rebuilt_rules/customization_enabled/upgrade_prebuilt_rules/preview_prebuilt_rules_upgrade.ts

  };

-  describe('@ess @serverless @skipInServerlessMKI preview prebuilt rules upgrade', () => {
+  describe('@ess @serverless @skipInServerlessMKI review prebuilt rules upgrade', () => {


If we're renaming this to review prebuilt rules upgrade, do you also want to change this file name?

Yes, it makes sense

…n folder (#225495) **Relates to:** #219831 ## Summary This PR moves prebuilt rules license independent integration tests to the `prebuilt_rules/common` folder.

…n folder (elastic#225495) **Relates to:** elastic#219831 ## Summary This PR moves prebuilt rules license independent integration tests to the `prebuilt_rules/common` folder. (cherry picked from commit 55e3292)

…n folder (elastic#225495) **Relates to:** elastic#219831 ## Summary This PR moves prebuilt rules license independent integration tests to the `prebuilt_rules/common` folder. (cherry picked from commit 55e3292) # Conflicts: # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/install_prebuilt_rules/install_mocked_prebuilt_rule_assets.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/revert_prebuilt_rules/revert_prebuilt_rules.ts

…c#219831) **Related Epic:** elastic#179907 ## Summary This PR refactors prebuilt rules integration tests structure to streamline the implementation of test plans targeted [Prebuilt Rules Customization Milestone 4](elastic#179907). ## Details Existing integration tests structure have prebuilt rules related integration tests scattered around rules management area. Due to historical reasons and pace of the prebuilt rules customization development some old tests were updated, some new were added as random spots as well as some new tests structure was suggested. This PR moves files and some tests around to the following structure - `test_suites/detection_response/rules_management/prebuilt_rules` is the root folder for prebuilt rules related integration tests - `customization_disabled` subfolder contains prebuilt rules integration tests covering scenarios when users have **insufficient** for customization license level (basic/essentials) - `customization_enabled` subfolder contains prebuilt rules integration tests covering scenarios when users have **sufficient** for customization license level - `customization_disabled` and `customization_enabled` subfoldera have test suites grouped by sub domains - `prebuilt_rules_package` - contains integration tests related to detection rules Fleet package installation and updating, bootstrap is also belong to here - `install_prebuilt_rules` - contains tests related to prebuilt rules installation from prebuilt rule assets - `upgrade_prebuilt_rules` - contains tests related to prebuilt rules upgrade workflow - `customization` - contains tests related to prebuilt rules customization including `is_customized` calculation - `import_export` - contains tests related to exporting and importing customized and non-customized prebuilt rules - `status` - contain status endpoints related tests (cherry picked from commit 3ea69af) # Conflicts: # .buildkite/ftr_security_serverless_configs.yml # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/prebuilt_rules_package/index.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/prebuilt_rule_customization/customization_enabled/is_customized_calculation.ts

maximpn · 2025-07-03T10:44:48Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0
✅	8.18

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…219831) (#226383) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Refactor prebuilt rules integration tests (#219831)](#219831)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…219831) (#226385) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Refactor prebuilt rules integration tests (#219831)](#219831)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

kibanamachine · 2025-07-03T14:15:59Z

Starting backport for target branches: 8.19, 9.0, 9.1

https://github.com/elastic/kibana/actions/runs/16052870896

kibanamachine · 2025-07-03T14:16:13Z

Starting backport for target branches: 8.19, 9.0, 9.1

https://github.com/elastic/kibana/actions/runs/16052877320

kibanamachine · 2025-07-03T14:16:16Z

Starting backport for target branches: 8.18, 8.19, 9.0, 9.1

https://github.com/elastic/kibana/actions/runs/16052878744

kibanamachine · 2025-07-03T14:23:34Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts
❌	9.1	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 219831

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-07-03T14:23:39Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts
❌	9.1	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 219831

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-07-03T14:24:05Z

💔 All backports failed

Status	Branch	Result
❌	8.18	Backport failed because of merge conflicts
❌	8.19	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts
❌	9.1	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 219831

Questions ?

Please refer to the Backport tool documentation

**Relates to:** elastic#219831 ## Summary This PR fixes MKI tests configuration for prebuilt rules after performing refactoring in elastic#219831. Since shorthand script in `x-pack/test/security_solution_api_integration/package.json` are used only for running MKI tests and local development it didn't cause PR pipeline failures and became prominent after running MKI tests. ## Pipelines [Kibana / Serverless / Security Solution Quality Gate / Rule Management](https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management) pipeline was manually ran against this PR. - ✅ [Periodic pipeline](https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/2786) - ✅ [Quality gate pipeline](https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/2786) (cherry picked from commit d853886)

…226501) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Fix MKI tests for prebuilt rules (#222170)](#222170)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…#226502) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Fix MKI tests for prebuilt rules (#222170)](#222170)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…n folder (elastic#225495) **Relates to:** elastic#219831 ## Summary This PR moves prebuilt rules license independent integration tests to the `prebuilt_rules/common` folder. (cherry picked from commit 55e3292) # Conflicts: # .buildkite/ftr_security_stateful_configs.yml # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/revert_prebuilt_rules/get_prebuilt_rule_base_version.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/revert_prebuilt_rules/index.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/revert_prebuilt_rules/revert_prebuilt_rules.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/revert_prebuilt_rules/index.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/ml_disabled/perform_installation/index.ts

…n folder (elastic#225495) **Relates to:** elastic#219831 ## Summary This PR moves prebuilt rules license independent integration tests to the `prebuilt_rules/common` folder. (cherry picked from commit 55e3292) # Conflicts: # .buildkite/ftr_security_stateful_configs.yml # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/install_prebuilt_rules/install_mocked_prebuilt_rule_assets.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/revert_prebuilt_rules/get_prebuilt_rule_base_version.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/revert_prebuilt_rules/index.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/revert_prebuilt_rules/revert_prebuilt_rules.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/revert_prebuilt_rules/index.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/ml_disabled/perform_installation/index.ts

…n folder (elastic#225495) **Relates to:** elastic#219831 This PR moves prebuilt rules license independent integration tests to the `prebuilt_rules/common` folder. (cherry picked from commit 55e3292)

…n folder (elastic#225495) **Relates to:** elastic#219831 ## Summary This PR moves prebuilt rules license independent integration tests to the `prebuilt_rules/common` folder.

maximpn self-assigned this May 1, 2025

maximpn requested review from a team as code owners May 1, 2025 11:32

maximpn requested a review from dplumlee May 1, 2025 11:32

maximpn marked this pull request as draft May 1, 2025 19:08

maximpn force-pushed the refactor-functional-tests branch from cb2e177 to c16de20 Compare May 2, 2025 11:35

maximpn force-pushed the refactor-functional-tests branch 2 times, most recently from bd68e05 to 7e17aad Compare May 22, 2025 08:25

maximpn marked this pull request as ready for review May 22, 2025 10:51

maximpn requested a review from a team as a code owner May 22, 2025 10:51

Ikuni17 approved these changes May 23, 2025

View reviewed changes

dplumlee approved these changes May 27, 2025

View reviewed changes

maximpn added 3 commits May 28, 2025 14:32

reorganize prebuilt rule customizarion integration tests

203befb

toss prebuilt rules relates tests to appropriate folders

fa97d5c

fix broken tests

cbe7ff6

maximpn mentioned this pull request Jul 3, 2025

[9.0] [Security Solution] Refactor prebuilt rules integration tests (#219831) #226383

Merged

maximpn mentioned this pull request Jul 3, 2025

[8.18] [Security Solution] Refactor prebuilt rules integration tests (#219831) #226385

Merged

kibanamachine added the v9.0.4 label Jul 3, 2025

kibanamachine added the v9.2.0 label Jul 3, 2025

kibanamachine added the v8.18.4 label Jul 3, 2025

banderror mentioned this pull request Jul 7, 2025

[Security Solution] Tests for prebuilt rule customization workflow #202068

Closed

21 tasks

Comments

Conversation

maximpn commented May 1, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Uh oh!

elasticmachine commented May 1, 2025

Uh oh!

elasticmachine commented May 1, 2025

Uh oh!

elasticmachine commented May 1, 2025

Uh oh!

prodsecmachine commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎉 Snyk checks have passed. No issues have been found so far.

Uh oh!

dplumlee left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dplumlee May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maximpn commented Jul 3, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Jul 3, 2025

Uh oh!

kibanamachine commented Jul 3, 2025

Uh oh!

kibanamachine commented Jul 3, 2025

Uh oh!

kibanamachine commented Jul 3, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

kibanamachine commented Jul 3, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

kibanamachine commented Jul 3, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

maximpn commented May 1, 2025 •

edited by kibanamachine

Loading

prodsecmachine commented May 1, 2025 •

edited

Loading

dplumlee May 27, 2025 •

edited

Loading