[Security Solution] Implement Prebuilt Rules Customization test plan by maximpn · Pull Request #232776 · elastic/kibana

maximpn · 2025-08-25T12:11:20Z

Resolves: #202068

Summary

This PR implements prebuilt rules customization test plan.

Details

The major part of the prebuilt rules customization test plan had been implemented already so this PR adds missing test scenario implementations and refactors the existing tests. In particular the following has been done

Prebuilt rules reverting customization test plan has been extracted
Existing prebuilt rule customization tests have been refactored to improve naming and remove duplicates
Test scenario implementations for missing base version have been added
Test scenario implementations for insufficient license have been added
Per field customization integration tests now also verify the customization can be reverted by saving original field value

Extended test coverage is crucial for making sure changes suggested in #230856 don't bring in new issues.

Flaky test runner

Prebuilt rule customization integration tests (200 runs)
Prebuilt rule customization e2e tests (100 runs)

kibanamachine · 2025-08-29T03:11:28Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#9251

[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/configs/ess.config.ts: 200/200 tests passed.

see run history

kibanamachine · 2025-08-29T08:43:39Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#9254

[✅] Security Solution Rule Management - Prebuilt Rules Customization - Cypress: 100/100 tests passed.

see run history

elasticmachine · 2025-08-29T11:12:35Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2025-08-29T11:12:37Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2025-08-29T11:12:38Z

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

dplumlee

Left a few comments but overall the improved tests look good @maximpn. I'm approving in advance as the comments should be pretty trivial to address and I don't want to block 👍

dplumlee · 2025-09-02T19:17:01Z

...agement/prebuilt_rules/customization_disabled/customization/unable_to_customize_on_import.ts

Github won't let me comment on the specific file line but on line 28, the

describe('@ess @serverless @skipInServerlessMKI Calculate "is_customized"', () => {

should be updated to align with the new file name

Thanks for noticing that.

I overlooked that implementation in customization_disabled/customization/unable_to_customize_on_import.ts doesn't match the title. In fact we allow Prebuilt Rule Customization via Prebuilt Rules Importing under insufficient for Prebuilt Rules Customization license. You many notice that Prebuilt Rules import and export API integration tests in x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/import_export run under the low-tier license.

Taking the above into account I removed customization_disabled/customization/unable_to_customize_on_import.ts as a duplicate of Prebuilt Rules import and export API integration tests.

dplumlee · 2025-09-02T19:23:19Z

..._management/prebuilt_rules/customization_enabled/customization/customize_via_bulk_editing.ts

+      it(`applies "${BulkActionEditTypeEnum.add_index_patterns}" bulk edit action to prebuilt rules`, async () => {
+        const bulkResponse = await performBulkEditOnPrebuiltRules({
+          type: BulkActionEditTypeEnum.add_index_patterns,
+          value: ['test-*'],
+        });
+
+        expect(bulkResponse.attributes.results.updated).toEqual(
+          expect.arrayContaining([
+            expect.objectContaining({
+              rule_id: QUERY_PREBUILT_RULE_ID,
+              index: ['existing-index-pattern-1', 'existing-index-pattern-2', 'test-*'],
+            }),
+            expect.objectContaining({
+              rule_id: SAVED_QUERY_PREBUILT_RULE_ID,
+              index: ['existing-index-pattern-1', 'existing-index-pattern-2', 'test-*'],
+            }),
+            expect.objectContaining({
+              rule_id: EQL_PREBUILT_RULE_ID,
+              index: ['existing-index-pattern-1', 'existing-index-pattern-2', 'test-*'],
+            }),
+          ])
+        );


This looks like a duplication of the test above

dplumlee · 2025-09-02T19:32:42Z

...prebuilt_rules/customization_enabled/customization/detect_customization_with_base_version.ts

+      });
+    });
+
+    describe('custom query rule fields', () => {


Probably also want a "saved query rule fields" block in this section, or at least something testing for saved_id since the fields are mostly the same between saved query and custom query rule types.

dplumlee · 2025-09-02T19:36:45Z

...built_rules/customization_enabled/customization/detect_customization_without_base_version.ts

+      });
+    });
+
+    describe('custom query rule fields', () => {


Same thing with the saved query rule type here

dplumlee · 2025-09-02T19:43:49Z

...e2e/detection_response/rule_management/prebuilt_rules/customization/rule_customization.cy.ts

-    it('user can edit a non-customized prebuilt rule from the rule edit page', function () {
-      const newDescriptionValue = 'New rule description';
-      cy.get(RULE_NAME).contains('Non-customized prebuilt rule').click();
+    describe('navigation to the prebuilt rule editing page', () => {


nit:

Suggested change

describe('navigation to the prebuilt rule editing page', () => {

describe('navigation to the rule editing page', () => {

…lans

kibanamachine · 2025-09-03T16:59:56Z

💔 All backports failed

Status	Branch	Result
❌	8.18	Backport failed because of merge conflicts
❌	8.19	Backport failed because of merge conflicts You might need to backport the following PRs to 8.19: - Upgrading backport to 10.0.2 (#233351)
❌	9.0	Backport failed because of merge conflicts
❌	9.1	Backport failed because of merge conflicts You might need to backport the following PRs to 9.1: - [Security Solution][EDR] Fix import of endpoint exceptions (#233142)

Manual backport

To create the backport manually run:

node scripts/backport --pr 232776

Questions ?

Please refer to the Backport tool documentation

…lastic#232776) **Resolves: elastic#202068 ## Summary This PR implements [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md). ## Details The major part of the [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md) had been implemented already so this PR adds missing test scenario implementations and refactors the existing tests. In particular the following has been done - Prebuilt rules reverting customization test plan has been extracted - Existing prebuilt rule customization tests have been refactored to improve naming and remove duplicates - Test scenario implementations for missing base version have been added - Test scenario implementations for insufficient license have been added - Per field customization integration tests now also verify the customization can be reverted by saving original field value Extended test coverage is crucial for making sure changes suggested in elastic#230856 don't bring in new issues. ## Flaky test runner - [Prebuilt rule customization integration tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9251) (200 runs) - [Prebuilt rule customization e2e tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9254) (100 runs) (cherry picked from commit 57d96cf) # Conflicts: # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/calculate_is_customized.ts # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/calculate_is_customized.ts # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/customize_prebuilt_rules.ts

…lastic#232776) **Resolves: elastic#202068 ## Summary This PR implements [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md). ## Details The major part of the [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md) had been implemented already so this PR adds missing test scenario implementations and refactors the existing tests. In particular the following has been done - Prebuilt rules reverting customization test plan has been extracted - Existing prebuilt rule customization tests have been refactored to improve naming and remove duplicates - Test scenario implementations for missing base version have been added - Test scenario implementations for insufficient license have been added - Per field customization integration tests now also verify the customization can be reverted by saving original field value Extended test coverage is crucial for making sure changes suggested in elastic#230856 don't bring in new issues. ## Flaky test runner - [Prebuilt rule customization integration tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9251) (200 runs) - [Prebuilt rule customization e2e tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9254) (100 runs) (cherry picked from commit 57d96cf) # Conflicts: # x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/customize_via_bulk_editing.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/calculate_is_customized.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/customize_via_bulk_editing.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/unable_to_customize_via_bulk_editing.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/calculate_is_customized.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/customize_prebuilt_rules.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/detect_customization_with_base_version.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/detect_customization_without_base_version.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/unaffected_fields.ts # x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/customization/rule_customization.cy.ts # x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/customization/rule_customization_basic_license.cy.ts

maximpn · 2025-09-04T07:48:14Z

💚 All backports created successfully

Status	Branch	Result
✅	9.1
✅	9.0
✅	8.18

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…lastic#232776) **Resolves: elastic#202068 ## Summary This PR implements [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md). ## Details The major part of the [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md) had been implemented already so this PR adds missing test scenario implementations and refactors the existing tests. In particular the following has been done - Prebuilt rules reverting customization test plan has been extracted - Existing prebuilt rule customization tests have been refactored to improve naming and remove duplicates - Test scenario implementations for missing base version have been added - Test scenario implementations for insufficient license have been added - Per field customization integration tests now also verify the customization can be reverted by saving original field value Extended test coverage is crucial for making sure changes suggested in elastic#230856 don't bring in new issues. ## Flaky test runner - [Prebuilt rule customization integration tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9251) (200 runs) - [Prebuilt rule customization e2e tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9254) (100 runs) (cherry picked from commit 57d96cf) # Conflicts: # x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/customize_via_bulk_editing.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/calculate_is_customized.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/customize_via_bulk_editing.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/unable_to_customize_via_bulk_editing.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/calculate_is_customized.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/customize_prebuilt_rules.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/detect_customization_with_base_version.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/detect_customization_without_base_version.ts # x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/unaffected_fields.ts # x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/customization/rule_customization.cy.ts # x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/customization/rule_customization_basic_license.cy.ts

…lastic#232776) **Resolves: elastic#202068 ## Summary This PR implements [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md). ## Details The major part of the [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md) had been implemented already so this PR adds missing test scenario implementations and refactors the existing tests. In particular the following has been done - Prebuilt rules reverting customization test plan has been extracted - Existing prebuilt rule customization tests have been refactored to improve naming and remove duplicates - Test scenario implementations for missing base version have been added - Test scenario implementations for insufficient license have been added - Per field customization integration tests now also verify the customization can be reverted by saving original field value Extended test coverage is crucial for making sure changes suggested in elastic#230856 don't bring in new issues. ## Flaky test runner - [Prebuilt rule customization integration tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9251) (200 runs) - [Prebuilt rule customization e2e tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9254) (100 runs)

kibanamachine · 2025-09-05T17:48:55Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @maximpn

kibanamachine · 2025-09-08T18:49:12Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @maximpn

maximpn · 2025-09-09T19:16:32Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…lastic#232776) **Resolves: elastic#202068 ## Summary This PR implements [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md). ## Details The major part of the [prebuilt rules customization test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_customization.md) had been implemented already so this PR adds missing test scenario implementations and refactors the existing tests. In particular the following has been done - Prebuilt rules reverting customization test plan has been extracted - Existing prebuilt rule customization tests have been refactored to improve naming and remove duplicates - Test scenario implementations for missing base version have been added - Test scenario implementations for insufficient license have been added - Per field customization integration tests now also verify the customization can be reverted by saving original field value Extended test coverage is crucial for making sure changes suggested in elastic#230856 don't bring in new issues. ## Flaky test runner - [Prebuilt rule customization integration tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9251) (200 runs) - [Prebuilt rule customization e2e tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9254) (100 runs) (cherry picked from commit 57d96cf) # Conflicts: # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_disabled/customization/calculate_is_customized.ts # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/calculate_is_customized.ts # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/customization/customize_prebuilt_rules.ts

kibanamachine · 2025-09-09T19:48:47Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @maximpn

… plan (#232776) (#233905) # Backport This will backport the following commits from `main` to `9.1`: - [[Security Solution] Implement Prebuilt Rules Customization test plan (#232776)](#232776)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

… plan (#232776) (#233952) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Implement Prebuilt Rules Customization test plan (#232776)](#232776)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…t plan (#232776) (#234501) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution] Implement Prebuilt Rules Customization test plan (#232776)](#232776)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…t plan (#232776) (#233954) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Implement Prebuilt Rules Customization test plan (#232776)](#232776)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

maximpn self-assigned this Aug 25, 2025

maximpn changed the title ~~reorganize prebuilt rule customization tests~~ [Security Solution] Implement Prebuilt Rules Customization test plan Aug 25, 2025

maximpn force-pushed the implement-prebuilt-rule-customizations-test-plan branch 3 times, most recently from f405650 to 376bd9a Compare August 28, 2025 08:10

maximpn requested a review from dplumlee August 29, 2025 11:12

maximpn marked this pull request as ready for review August 29, 2025 11:12

maximpn requested a review from a team as a code owner August 29, 2025 11:12

banderror mentioned this pull request Aug 30, 2025

[Security Solution] Tests for prebuilt rule customization workflow #202068

Closed

21 tasks

dplumlee approved these changes Sep 2, 2025

View reviewed changes

maximpn added 3 commits September 3, 2025 10:29

reorganize prebuilt rule customization tests

eacec7d

implement test scenarios for missing base version

0fd1eb2

split prebuilt rule customization and reverting customizations test p…

86976e6

…lans

maximpn deleted the implement-prebuilt-rule-customizations-test-plan branch September 3, 2025 16:56

maximpn mentioned this pull request Sep 3, 2025

[9.1] [Security Solution] Implement Prebuilt Rules Customization test plan (#232776) #233905

Merged

maximpn mentioned this pull request Sep 4, 2025

[9.0] [Security Solution] Implement Prebuilt Rules Customization test plan (#232776) #233952

Merged

maximpn mentioned this pull request Sep 4, 2025

[8.18] [Security Solution] Implement Prebuilt Rules Customization test plan (#232776) #233954

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Sep 5, 2025

maximpn mentioned this pull request Sep 9, 2025

[8.19] [Security Solution] Implement Prebuilt Rules Customization test plan (#232776) #234501

Merged

kibanamachine added the v9.1.4 label Sep 10, 2025

kibanamachine added the v9.0.7 label Sep 10, 2025

kibanamachine added v8.19.4 v8.18.7 and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels Sep 10, 2025

banderror removed v8.19.3 v8.18.6 v9.1.3 v9.0.6 labels Sep 11, 2025

	describe('navigation to the prebuilt rule editing page', () => {
	describe('navigation to the rule editing page', () => {

Conversation

maximpn commented Aug 25, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Flaky test runner

Uh oh!

kibanamachine commented Aug 29, 2025

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#9251

Uh oh!

kibanamachine commented Aug 29, 2025

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#9254

Uh oh!

elasticmachine commented Aug 29, 2025

Uh oh!

elasticmachine commented Aug 29, 2025

Uh oh!

elasticmachine commented Aug 29, 2025

Uh oh!

dplumlee left a comment

Choose a reason for hiding this comment

Uh oh!

dplumlee Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

maximpn Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

dplumlee Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

dplumlee Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

dplumlee Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

dplumlee Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Sep 3, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

maximpn commented Sep 4, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Sep 5, 2025

Uh oh!

kibanamachine commented Sep 8, 2025

Uh oh!

maximpn commented Sep 9, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Sep 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

maximpn commented Aug 25, 2025 •

edited by kibanamachine

Loading