[Security Solution] Unskip prebuilt rules bundled package integration tests by maximpn · Pull Request #227371 · elastic/kibana

maximpn · 2025-07-10T08:29:06Z

Resolves: #180087

Summary

This PR fixes and unskips Prebuilt Rules bundled package integration tests.

Details

Flakiness was caused by improper test setup which failed to remove already installed prebuilt rules package.

Flaky test runner

✅ 200 runs

kibanamachine · 2025-07-10T10:02:02Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#8579

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/common/configs/ess_air_gapped.config.ts: 200/200 tests passed.

see run history

elasticmachine · 2025-07-10T13:08:01Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2025-07-10T13:08:02Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2025-07-10T13:08:03Z

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

nikitaindik · 2025-07-11T09:48:42Z

...anagement/prebuilt_rules/common/prebuilt_rules_package/air_gapped/install_bundled_package.ts

+      await deletePrebuiltRulesFleetPackage({ es, supertest, retryService: retry, log });
    });

    it('should list `security_detection_engine` as a bundled fleet package in the `fleet_package.json` file', async () => {


Noticed that assertion on line 54

expect(securityDetectionEnginePackage?.name).toBe('security_detection_engine');

seems to be unnecessary since it's kind of covered by assertion on line 52

expect(securityDetectionEnginePackage).not.toBeUndefined();

Indeed, this assertion looks redundant. I simply didn't touch them. But it makes sense to simplify the assertions in the scope of this PR.

nikitaindik

Please help me understand. Is this what seems to be happening here?

Some previous test installs the package, but removes all of its assets.
In this test, we attempt to install the package, but the installation is skipped since the package is already installed (though empty).
We expect to have installable rules, but we get zero since the package is empty and this asserting fails:

expect(statusAfterPackageInstallation.stats.num_prebuilt_rules_to_install).toBeGreaterThan(0)

maximpn · 2025-07-11T10:10:27Z

@nikitaindik I've addressed your comment. Could you have a look?

maximpn · 2025-07-11T10:17:11Z

@nikitaindik,

Some previous test installs the package, but removes all of its assets.
In this test, we attempt to install the package, but the installation is skipped since the package is already installed (though empty).
We expect to have installable rules, but we get zero since the package is empty and this asserting fails:

Yes. This is correct. It's dependency in the previous test suits. Each test should prepare the environment for its flawless execution. In that particular case removing all prebuilt rule assets doesn't lead to removing Fleet's SO. We the test tried to install the package Fleet sees there is an SO matching this package and doesn't install prebuilt rule assets for the package.

elasticmachine · 2025-07-11T12:09:30Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: ed64b0c

Failed CI Steps

FTR Configs #99

Test Failures

[job] [logs] FTR Configs #99 / Serverless Common UI - Dev Tools Search Profiler Editor With a test index profiles a simple query

Metrics [docs]

✅ unchanged

History

💚 Build #317571 succeeded 97b0187

cc @maximpn

nikitaindik

Thanks, LGTM now! In the future we might want to add a utility function that removes both the package and its contents: basically calls both deleteAllPrebuiltRuleAssets and deletePrebuiltRulesFleetPackage.

kibanamachine · 2025-07-11T12:26:39Z

Starting backport for target branches: 8.18, 8.19, 9.0, 9.1

https://github.com/elastic/kibana/actions/runs/16219987944

… tests (elastic#227371) **Resolves: elastic#180087 ## Summary This PR fixes and unskips Prebuilt Rules bundled package integration tests. ## Details Flakiness was caused by improper test setup which failed to remove already installed prebuilt rules package. ## Flaky test runner - ✅ [200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8579) (cherry picked from commit c43d8cc)

kibanamachine · 2025-07-11T12:33:49Z

💚 All backports created successfully

Status	Branch	Result
✅	8.18
✅	8.19
✅	9.0
✅	9.1

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ration tests (#227371) (#227646) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Unskip prebuilt rules bundled package integration tests (#227371)](#227371)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>

…gration tests (#227371) (#227645) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution] Unskip prebuilt rules bundled package integration tests (#227371)](#227371)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>

…ration tests (#227371) (#227647) # Backport This will backport the following commits from `main` to `9.1`: - [[Security Solution] Unskip prebuilt rules bundled package integration tests (#227371)](#227371)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>

…gration tests (#227371) (#227644) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Unskip prebuilt rules bundled package integration tests (#227371)](#227371)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>

… tests (elastic#227371) **Resolves: elastic#180087 ## Summary This PR fixes and unskips Prebuilt Rules bundled package integration tests. ## Details Flakiness was caused by improper test setup which failed to remove already installed prebuilt rules package. ## Flaky test runner - ✅ [200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8579)

maximpn self-assigned this Jul 10, 2025

maximpn marked this pull request as ready for review July 10, 2025 13:07

maximpn requested a review from a team as a code owner July 10, 2025 13:07

maximpn requested a review from jkelas July 10, 2025 13:07

nikitaindik requested review from nikitaindik and removed request for jkelas July 10, 2025 13:09

nikitaindik reviewed Jul 11, 2025

View reviewed changes

unskip prebuilt rules bundle package integration tests

ed64b0c

maximpn force-pushed the unskip-prebuilt-rules-bundled-package-integration-tests branch from 97b0187 to ed64b0c Compare July 11, 2025 10:09

maximpn requested a review from nikitaindik July 11, 2025 10:09

nikitaindik approved these changes Jul 11, 2025

View reviewed changes

maximpn merged commit c43d8cc into elastic:main Jul 11, 2025
12 checks passed

maximpn deleted the unskip-prebuilt-rules-bundled-package-integration-tests branch July 11, 2025 12:26

kibanamachine mentioned this pull request Jul 11, 2025

[8.18] [Security Solution] Unskip prebuilt rules bundled package integration tests (#227371) #227644

Merged

kibanamachine mentioned this pull request Jul 11, 2025

[8.19] [Security Solution] Unskip prebuilt rules bundled package integration tests (#227371) #227645

Merged

kibanamachine mentioned this pull request Jul 11, 2025

[9.0] [Security Solution] Unskip prebuilt rules bundled package integration tests (#227371) #227646

Merged

kibanamachine mentioned this pull request Jul 11, 2025

[9.1] [Security Solution] Unskip prebuilt rules bundled package integration tests (#227371) #227647

Merged

kibanamachine added the v9.0.4 label Jul 11, 2025

kibanamachine added the v8.18.4 label Jul 11, 2025

banderror removed v9.0.0 v8.18.0 labels Jul 14, 2025

Conversation

maximpn commented Jul 10, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Flaky test runner

Uh oh!

kibanamachine commented Jul 10, 2025

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#8579

Uh oh!

elasticmachine commented Jul 10, 2025

Uh oh!

elasticmachine commented Jul 10, 2025

Uh oh!

elasticmachine commented Jul 10, 2025

Uh oh!

nikitaindik Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

maximpn Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

nikitaindik left a comment

Choose a reason for hiding this comment

Uh oh!

maximpn commented Jul 11, 2025

Uh oh!

maximpn commented Jul 11, 2025

Uh oh!

elasticmachine commented Jul 11, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

nikitaindik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Jul 11, 2025

Uh oh!

kibanamachine commented Jul 11, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

maximpn commented Jul 10, 2025 •

edited by kibanamachine

Loading