[Detection Engine] Instrumenting FTR for flaky test investigation#228850
[Detection Engine] Instrumenting FTR for flaky test investigation#228850rylnd wants to merge 18 commits intoelastic:mainfrom
Conversation
This integration test is executed 100 times, with the single-document archive being loaded and unloaded before/after each test. If this test fails, it will eliminate the detection engine as a culprit.
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
rylnd
force-pushed
the
rylnd/instrument-ftr
branch
from
7604ff2 to
7637e9e
Compare
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8749[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 7/10 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8751[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 0/10 tests passed. |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8753[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 10/10 tests passed. |
rylnd
force-pushed
the
rylnd/instrument-ftr
branch
from
b004266 to
e073939
Compare
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8776[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 10/10 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8779[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 0/20 tests passed. |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8780[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 20/20 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8784[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 66/100 tests passed. |
This should allow us to more clearly trace what's happening via APM data.
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8804[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 5/30 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8806[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 4/20 tests passed. |
As there might be useful info in here, and using the logging service is apparently equivalent to `> /dev/null`
Again, this should offer more visibility into what the tests are doing.
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8808[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 4/30 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8810[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 0/25 tests passed. |
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8813[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 7/20 tests passed. |
|
Running a 200x build with @pheyos's fixes from #229457: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8905 |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8905[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 200/200 tests passed. |
|
The previous 200x build has 100% success, which is nigh-definitive confirmation of a fix. 🎉 |
By specifying an ID in our archive doc, we should no longer get the duplication, but I expect instead to see the index request fail in the state produced by these tests.
|
Attempting to force a |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8958[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 30/30 tests passed. |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8959[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 200/200 tests passed. |
6 participants
Summary
🚧 this branch is not intended to be merged, but instead used to exercise the flaky test runner for investigative purposes.
Previously, #223043 had this role, but that branch has gathered a lot of noise and we've changed approaches/meandered a bit, there. Instead of rewriting history there, I'm starting a new branch here.