[Security Solution][Detection Engine] fix cypress MKI flaky test#194866
Merged
vitaliidm merged 13 commits intoelastic:mainfrom Oct 9, 2024
Merged
[Security Solution][Detection Engine] fix cypress MKI flaky test#194866vitaliidm merged 13 commits intoelastic:mainfrom
vitaliidm merged 13 commits intoelastic:mainfrom
Conversation
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#7078[✅] [Serverless] Security Solution Detection Engine - Cypress: 100/100 tests passed. |
…/github.com/vitaliidm/kibana into de_8_16/threshold_suppression_flaky_cy_test
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#7084[✅] [Serverless] Security Solution Detection Engine - Cypress: 200/200 tests passed. |
vitaliidm
changed the title
vitaliidm
added
release_note:skip
Contributor
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
Contributor
|
Pinging @elastic/security-detection-engine (Team:Detection Engine) |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]
History
To update your PR or re-run it, just comment with: cc @vitaliidm |
rylnd
approved these changes
Oct 9, 2024
Contributor
rylnd
left a comment
rylnd
left a comment
There was a problem hiding this comment.
I like the theory of the suppression fields not being in the proper state when cypress tries to interact with them, and the strategy of performing them at the end seems valid and borne out by the flaky test runner. But since we were unable to reproduce in the first place, this is all just guesswork.
Regardless, this code is an improvement on the hypothesis, and inert otherwise, and we should merge it. LGTM.
Contributor
💚 Build Succeeded
Metrics [docs]
cc @vitaliidm |
Contributor
|
Starting backport for target branches: 8.x https://github.com/elastic/kibana/actions/runs/11251923749 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 9, 2024
…stic#194866) ## Summary This an attempt to fix flaky Cypress test: https://buildkite.com/organizations/elastic/analytics/suites/serverless-mki-cypress-detection-engine/tests/9cd134bd-fa8b-8ff3-858e-ba1733d30e2c?branch=main I was not able to reproduce it locally. Also, old version of test was very stable on flaky test runner too: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7078 I changed test a bit by re-arranging order of form filling. So, suppression fields will be last. Maybe it can reduce possibility of race condition when form is just rendered and fields being interacted with by Cypress. Also, added assertion if threshold checkbox changed it status to enabled before interacting with other suppression inputs. If this won't help, next step can be using default suppression configuration instead. New version of test: 200 runs w/o failures https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7084 --------- Co-authored-by: Ryland Herrick <ryalnd@gmail.com> (cherry picked from commit 7fdd90d)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Oct 9, 2024
#194866) (#195550) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution][Detection Engine] fix cypress MKI flaky test (#194866)](#194866) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Vitalii Dmyterko","email":"92328789+vitaliidm@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-10-09T09:14:16Z","message":"[Security Solution][Detection Engine] fix cypress MKI flaky test (#194866)\n\n## Summary\r\n\r\nThis an attempt to fix flaky Cypress test:\r\nhttps://buildkite.com/organizations/elastic/analytics/suites/serverless-mki-cypress-detection-engine/tests/9cd134bd-fa8b-8ff3-858e-ba1733d30e2c?branch=main\r\n\r\nI was not able to reproduce it locally.\r\nAlso, old version of test was very stable on flaky test runner too:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7078\r\n\r\nI changed test a bit by re-arranging order of form filling. So,\r\nsuppression fields will be last. Maybe it can reduce possibility of race\r\ncondition when form is just rendered and fields being interacted with by\r\nCypress.\r\nAlso, added assertion if threshold checkbox changed it status to enabled\r\nbefore interacting with other suppression inputs.\r\nIf this won't help, next step can be using default suppression\r\nconfiguration instead.\r\n\r\nNew version of test: 200 runs w/o failures\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7084\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick <ryalnd@gmail.com>","sha":"7fdd90d148660256d822c45f5a4594d82d21af38","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team: SecuritySolution","backport:prev-minor","Team:Detection Engine","8.16 candidate"],"title":"[Security Solution][Detection Engine] fix cypress MKI flaky test","number":194866,"url":"https://github.com/elastic/kibana/pull/194866","mergeCommit":{"message":"[Security Solution][Detection Engine] fix cypress MKI flaky test (#194866)\n\n## Summary\r\n\r\nThis an attempt to fix flaky Cypress test:\r\nhttps://buildkite.com/organizations/elastic/analytics/suites/serverless-mki-cypress-detection-engine/tests/9cd134bd-fa8b-8ff3-858e-ba1733d30e2c?branch=main\r\n\r\nI was not able to reproduce it locally.\r\nAlso, old version of test was very stable on flaky test runner too:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7078\r\n\r\nI changed test a bit by re-arranging order of form filling. So,\r\nsuppression fields will be last. Maybe it can reduce possibility of race\r\ncondition when form is just rendered and fields being interacted with by\r\nCypress.\r\nAlso, added assertion if threshold checkbox changed it status to enabled\r\nbefore interacting with other suppression inputs.\r\nIf this won't help, next step can be using default suppression\r\nconfiguration instead.\r\n\r\nNew version of test: 200 runs w/o failures\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7084\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick <ryalnd@gmail.com>","sha":"7fdd90d148660256d822c45f5a4594d82d21af38"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194866","number":194866,"mergeCommit":{"message":"[Security Solution][Detection Engine] fix cypress MKI flaky test (#194866)\n\n## Summary\r\n\r\nThis an attempt to fix flaky Cypress test:\r\nhttps://buildkite.com/organizations/elastic/analytics/suites/serverless-mki-cypress-detection-engine/tests/9cd134bd-fa8b-8ff3-858e-ba1733d30e2c?branch=main\r\n\r\nI was not able to reproduce it locally.\r\nAlso, old version of test was very stable on flaky test runner too:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7078\r\n\r\nI changed test a bit by re-arranging order of form filling. So,\r\nsuppression fields will be last. Maybe it can reduce possibility of race\r\ncondition when form is just rendered and fields being interacted with by\r\nCypress.\r\nAlso, added assertion if threshold checkbox changed it status to enabled\r\nbefore interacting with other suppression inputs.\r\nIf this won't help, next step can be using default suppression\r\nconfiguration instead.\r\n\r\nNew version of test: 200 runs w/o failures\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7084\r\n\r\n---------\r\n\r\nCo-authored-by: Ryland Herrick <ryalnd@gmail.com>","sha":"7fdd90d148660256d822c45f5a4594d82d21af38"}}]}] BACKPORT--> Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This an attempt to fix flaky Cypress test: https://buildkite.com/organizations/elastic/analytics/suites/serverless-mki-cypress-detection-engine/tests/9cd134bd-fa8b-8ff3-858e-ba1733d30e2c?branch=main
I was not able to reproduce it locally.
Also, old version of test was very stable on flaky test runner too: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7078
I changed test a bit by re-arranging order of form filling. So, suppression fields will be last. Maybe it can reduce possibility of race condition when form is just rendered and fields being interacted with by Cypress.
Also, added assertion if threshold checkbox changed it status to enabled before interacting with other suppression inputs.
If this won't help, next step can be using default suppression configuration instead.
New version of test: 200 runs w/o failures https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7084