[Detection Engine] Investigating CI failures

[rylnd]

Summary

Investigation branch related to #221659 (comment).

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!
• Click to trigger kibana-deploy-cloud-from-pr for this PR!

[rylnd]

Running the failing test 100 times (in isolation) with some debugging: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8310

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#8310

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 100/100 tests passed.

see run history

[rylnd]

No failures. Another 200: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8311

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#8311

[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 200/200 tests passed.

see run history

[rylnd]

No failures. Another 200, but this time running the entire timestamps test suite (instead of just our single failing test): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8343

[kibanamachine]

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#8343

[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 172/200 tests passed.

see run history

[rylnd]

Closing as this work has now been completed with #229457 for es_archiver and #231392 for our internal dataGeneratorFactory.