[Security Solution] Enable critical Rule Management tests in MKI periodic and 2nd quality gate pipelines#193666
Conversation
maximpn
added
test
release_note:skip
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
There was a problem hiding this comment.
Not sure I understand why these two have been deleted. Have they been moved somewhere else?
There was a problem hiding this comment.
Ah, ok, I see should enable rules has been moved to trial_license_complete_tier/perform_bulk_enable_disable.ts and that Rule Deletion now has their own section.
My only question now is: for bulk rule deletion, the test that was in this file was under Trial License. The new delete_rules_bulk.ts file is under Basic License. Is that intended? Does it matter?
There was a problem hiding this comment.
Bulk enable/disable tests were moved to a separate file and enabled in MKI periodic/2nd quality gate pipelines since rule enabling/disabling functionality is critical.
Rule deletion test was removed by mistake instead of rule disabling test. Thanks for noticing it 👍
|
@maximpn Is there somewhere where the current behaviour of tags in integration tests is described? In this ticket I see how they should be, but not how they currently behave. I want to udnerstand, for example, what tag should be used if you want a test to run in ServerlessPeriodic |
There was a problem hiding this comment.
My understanding is that this endpoint will be deprecated in 9.0.
Do you think that could have any impact on our decision to add it to the QA test gate?
There was a problem hiding this comment.
You're right. This test shouldn't be enabled in MKI.
There was a problem hiding this comment.
I guess we have no test coverage for creating other rule types. But just wondering why you've decided to add this one to ServerlessKibanaQAGate. Do you think that there should be similar tests to this, one for each rule type, and all should be part of ServerlessKibanaQAGate?
There was a problem hiding this comment.
You're correct we have a limited test coverage. The goal of this PR is to enable smoke tests from what's available. Since we have a test for new terms rule type it makes sense to enable it in MKI.
There was a problem hiding this comment.
What's the reason to add import_rules to ServerlessKibanaQAGate, but not import_rules_with_overwrite?
I'm just thinking that the import function is becoming increasingly important, especially since it will be the main way to bulk create rules once the Bulk Create API is fully deprecated in 9.0
There was a problem hiding this comment.
The goal is to enable smoke testing for importing functionality. There is an assumption that a scenario when import_rules_with_overwrite fails only in MKI while passes in PR and on merge pipelines but import_rules passes everywhere is quite rare. That's why the decision is to enable only basic tests.
There was a problem hiding this comment.
Also here I'd like to understand the difference between Basic and Trial Licenses, and why we're adding the Basic tests to QA, but leaving Trial out
There was a problem hiding this comment.
The goal is to have critical functionality smoke tests running in MKI. If one test fails it will block a Serverless release and require our team to do a hotfix. More tests enabled the higher such risk but of course we want to deliver stable features. The decision is to find a balance in that trade off by enabling only basic license tests. It's a critical functionality and always gives an option for a user to roll back to a basic license and wait for the next Serverless release.
Nothing blocks us from enabling more tests later on when it's necessary.
There was a problem hiding this comment.
What about here? Are we skipping it because it's flaky?
Import looks important enough to be in QA
There was a problem hiding this comment.
Cypress tests assert UI functionality. It's not critical since users can overcome it by using our API directly. Rule CRUD, import and rules table are critical features.
|
@maximpn Thanks for the audit and this PR. Overall looks good to me, but I left a bunch of questions that I'd like to understand before approving. |
maximpn
force-pushed
the
enable-specific-tests-in-MKI
branch
from
1750205 to
795e7c9
Compare
|
@jpdjere Thanks for your review 🙏 Aggregating your questions in one we get "Why do we enable only some tests in MKI leaving the others skipped?" The answer: We want to have a bare minimum smoke tests enabled in 2nd quality gate. Any failure at that stage would mean we have absolutely critical problem blocking the release. Critical Rule Management functionality is Rules Table, Rules CRUD, rule enabling/disabling and rule importing (rule importing is able to replace rule bulk operations). Additionally any non basic license tests aren't critical since users are able to roll back to a basic license. |
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @maximpn |
jpdjere
left a comment
jpdjere
left a comment
There was a problem hiding this comment.
Thanks for all the explanations
|
Starting backport for target branches: 8.x https://github.com/elastic/kibana/actions/runs/11064628936 |
…odic and 2nd quality gate pipelines (elastic#193666) **Resolves:** elastic/security-team#10609 ## Summary This PR enables critical Rule Management tests in periodic and 2nd quality gate MKI pipelines. In the other words it enable smoke testing for Rule Management critical features. ## Details A prerequisite Rule Management FTR and Cypress tests audit result are summarized in a [Google Sheet document](https://docs.google.com/spreadsheets/d/1jyNXlMpTLAxt5u_ZTNJVms7UnTmwA2xALq2C3HEOEqQ). Based on feature criticality it was decided to enable critical FTR and Cypress tests in periodic and 2nd quality gate MKI pipelines and disable non critical tests in MKI pipelines. (cherry picked from commit 4ab4c66)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…I periodic and 2nd quality gate pipelines (#193666) (#194249) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Enable critical Rule Management tests in MKI periodic and 2nd quality gate pipelines (#193666)](#193666) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"maxim.palenov@elastic.co"},"sourceCommit":{"committedDate":"2024-09-27T04:41:30Z","message":"[Security Solution] Enable critical Rule Management tests in MKI periodic and 2nd quality gate pipelines (#193666)\n\n**Resolves:** https://github.com/elastic/security-team/issues/10609\r\n\r\n## Summary\r\n\r\nThis PR enables critical Rule Management tests in periodic and 2nd quality gate MKI pipelines. In the other words it enable smoke testing for Rule Management critical features.\r\n\r\n## Details\r\n\r\nA prerequisite Rule Management FTR and Cypress tests audit result are summarized in a [Google Sheet document](https://docs.google.com/spreadsheets/d/1jyNXlMpTLAxt5u_ZTNJVms7UnTmwA2xALq2C3HEOEqQ). Based on feature criticality it was decided to enable critical FTR and Cypress tests in periodic and 2nd quality gate MKI pipelines and disable non critical tests in MKI pipelines.","sha":"4ab4c661272fa2763f027204461c37d0a1620aad","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","backport:prev-minor","Project:Serverless","v8.16.0"],"title":"[Security Solution] Enable critical Rule Management tests in MKI periodic and 2nd quality gate pipelines","number":193666,"url":"https://github.com/elastic/kibana/pull/193666","mergeCommit":{"message":"[Security Solution] Enable critical Rule Management tests in MKI periodic and 2nd quality gate pipelines (#193666)\n\n**Resolves:** https://github.com/elastic/security-team/issues/10609\r\n\r\n## Summary\r\n\r\nThis PR enables critical Rule Management tests in periodic and 2nd quality gate MKI pipelines. In the other words it enable smoke testing for Rule Management critical features.\r\n\r\n## Details\r\n\r\nA prerequisite Rule Management FTR and Cypress tests audit result are summarized in a [Google Sheet document](https://docs.google.com/spreadsheets/d/1jyNXlMpTLAxt5u_ZTNJVms7UnTmwA2xALq2C3HEOEqQ). Based on feature criticality it was decided to enable critical FTR and Cypress tests in periodic and 2nd quality gate MKI pipelines and disable non critical tests in MKI pipelines.","sha":"4ab4c661272fa2763f027204461c37d0a1620aad"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/193666","number":193666,"mergeCommit":{"message":"[Security Solution] Enable critical Rule Management tests in MKI periodic and 2nd quality gate pipelines (#193666)\n\n**Resolves:** https://github.com/elastic/security-team/issues/10609\r\n\r\n## Summary\r\n\r\nThis PR enables critical Rule Management tests in periodic and 2nd quality gate MKI pipelines. In the other words it enable smoke testing for Rule Management critical features.\r\n\r\n## Details\r\n\r\nA prerequisite Rule Management FTR and Cypress tests audit result are summarized in a [Google Sheet document](https://docs.google.com/spreadsheets/d/1jyNXlMpTLAxt5u_ZTNJVms7UnTmwA2xALq2C3HEOEqQ). Based on feature criticality it was decided to enable critical FTR and Cypress tests in periodic and 2nd quality gate MKI pipelines and disable non critical tests in MKI pipelines.","sha":"4ab4c661272fa2763f027204461c37d0a1620aad"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
## Summary This test shouldn't run in periodic pipeline. We expect only FTR and Cypress running in periodic and 2nd quality gate pipelines enabled in #193666.
## Summary This test shouldn't run in periodic pipeline. We expect only FTR and Cypress running in periodic and 2nd quality gate pipelines enabled in elastic#193666. (cherry picked from commit 6f1449b)
5 participants
Resolves: https://github.com/elastic/security-team/issues/10609
Summary
This PR enables critical Rule Management tests in periodic and 2nd quality gate MKI pipelines. In the other words it enable smoke testing for Rule Management critical features.
Details
A prerequisite Rule Management FTR and Cypress tests audit result are summarized in a Google Sheet document. Based on feature criticality it was decided to enable critical FTR and Cypress tests in periodic and 2nd quality gate MKI pipelines and disable non critical tests in MKI pipelines.