Skip to content

[8.18] [Security Solution] Split prebuilt rule Cypress tests (#231150) #231507

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nikitaindik merged 3 commits into from
Aug 13, 2025
Merged

[8.18] [Security Solution] Split prebuilt rule Cypress tests (#231150) #231507

nikitaindik merged 3 commits into from
Aug 13, 2025

Conversation

@nikitaindik
Copy link
Contributor

@nikitaindik nikitaindik commented Aug 12, 2025

Backport

This will backport the following commits from main to 8.18:

Questions ?

Please refer to the Backport tool documentation

@nikitaindik
[Security Solution] Split prebuilt rule Cypress tests (elastic#231150)
dde7880 
**Part of epic: elastic#229688

## Summary
We have recently observed that some prebuilt detection rules in Cypress
tests run for over 60 minutes in CI pipelines. This leads to pipeline
timeouts. As a temporary measure `parallelism` was increased to 2
[recently](elastic#230969).

This PR divides the Cypress tests for prebuilt detection rules into four
separate groups to avoid hitting the 60m limit.

## Changes
- Divided prebuilt rules directory `prebuilt_rules` into 4 subdirs:
  - `installation`
  - `upgrade`
  - `customization`
- `management` (enabling/disabling, deletion, import and export of
rules)
- Added `yarn` commands and `bash` scripts to run tests from each
subdirectory
- Registered new groups in BuildKite pipelines (with `parallelism: 1`):
  - `pull_request/security_solution/rule_management.yml`
  - `on_merge.yml`
  - `chrome_forward_testing.yml`
  - `pointer_compression.yml`
  - `verify_es_serverless_image.yml` (Serverless tests only)
- Registered the new scripts in the Flaky Test Runner

## Running times in pull request pipeline
Link to this PR's run in pull request pipeline:
https://buildkite.com/elastic/kibana-pull-request/builds/327944

**Non-Serverless groups:**
- 13 min: Rule Management - Prebuilt Rules Customization - Security
Solution Cypress Tests
- 13 min: Rule Management - Prebuilt Rules Installation - Security
Solution Cypress Tests
- 10 min: Rule Management - Prebuilt Rules Management - Security
Solution Cypress Tests
- 31 min: Rule Management - Prebuilt Rules Upgrade - Security Solution
Cypress Tests

**Serverless groups:**
- 17 min: Serverless Rule Management - Prebuilt Rules Customization -
Security Solution Cypress Tests
- 16 min: Serverless Rule Management - Prebuilt Rules Installation -
Security Solution Cypress Tests
- 11 min: Serverless Rule Management - Prebuilt Rules Management -
Security Solution Cypress Tests
- 27 min: Serverless Rule Management - Prebuilt Rules Upgrade - Security
Solution Cypress Tests

Each group's running time is significantly below the 60-minute limit.

(cherry picked from commit 40b7b42)

# Conflicts:
#	.buildkite/pipelines/chrome_forward_testing.yml
#	.buildkite/pipelines/es_serverless/verify_es_serverless_image.yml
#	.buildkite/pipelines/on_merge.yml
#	.buildkite/pipelines/pointer_compression.yml
#	.buildkite/pipelines/pull_request/security_solution/rule_management.yml
#	x-pack/solutions/security/test/security_solution_cypress/cypress/README.md
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/customization/revert_prebuilt_rule.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/customization/rule_customization.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/export_prebuilt_rule.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/import_prebuilt_rule.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_error_handling.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_notifications.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_update_authorization.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_via_fleet.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_with_preview.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_workflow.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/installation/install_error_handling.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/installation/install_notifications.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/installation/install_update_authorization.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/installation/install_via_fleet.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/installation/install_with_preview.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/installation/install_workflow.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/management.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/management/export_prebuilt_rule.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/management/import_prebuilt_rule.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/management/management.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/rule_customization.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade/upgrade_error_handling.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade/upgrade_notifications.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade/upgrade_with_preview.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade/upgrade_with_preview_basic_license.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade/upgrade_without_preview.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade/upgrade_without_preview_basic_license.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_error_handling.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_notifications.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_with_preview.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_with_preview_basic_license.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_without_preview.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_without_preview_basic_license.cy.ts
#	x-pack/solutions/security/test/security_solution_cypress/package.json
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/export_prebuilt_rule.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/import_prebuilt_rule.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_error_handling.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_notifications.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_update_authorization.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_via_fleet.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_with_preview.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/install_workflow.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/management.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/rule_customization.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_error_handling.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_notifications.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_with_preview.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_with_preview_basic_license.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_without_preview.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/upgrade_without_preview_basic_license.cy.ts
@nikitaindik nikitaindik added the backport This PR is a backport of another PR label Aug 12, 2025
@nikitaindik nikitaindik enabled auto-merge (squash) August 12, 2025 19:18
@nikitaindik nikitaindik mentioned this pull request Aug 12, 2025
Merged
banderror
banderror reviewed Aug 12, 2025
View reviewed changes
Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nikitaindik Compared every file change with the base PR, found a few potential issues. Please take a look.

@nikitaindik
Copy link
Contributor Author

nikitaindik commented Aug 12, 2025 •
edited
Loading

Thanks for taking a look, @banderror! I think changes in pointer_compression.yml are fine. There's 4 pipeline steps for prebuilt rules in Serverless and 4 steps in non-Serverless. Did you expect a different diff?

@nikitaindik nikitaindik requested a review from banderror August 12, 2025 21:57
banderror
banderror approved these changes Aug 12, 2025
View reviewed changes
Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @nikitaindik for working late hours to address the remaining issues. The updated backport LGTM 👍

@nikitaindik nikitaindik merged commit b83b0e1 into elastic:8.18 Aug 13, 2025
8 checks passed
@elasticmachine
Copy link
Contributor

elasticmachine commented Aug 13, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #51 / Observability Logs Explorer DataSourceSelector with installed integrations and uncategorized data streams when open on the integrations tab should load more integrations by scrolling to the end of the list

Metrics [docs]

✅ unchanged

History

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@banderror banderror banderror approved these changes

@kibanamachine kibanamachine Awaiting requested review from kibanamachine kibanamachine is a code owner

Assignees

No one assigned

Labels

backport This PR is a backport of another PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nikitaindik @elasticmachine @banderror