[Detection Engine] Prevent test data from rolling outside of Cypress rules' query#220923
Merged
rylnd merged 9 commits intoelastic:mainfrom May 23, 2025
Merged
[Detection Engine] Prevent test data from rolling outside of Cypress rules' query#220923rylnd merged 9 commits intoelastic:mainfrom
rylnd merged 9 commits intoelastic:mainfrom
Conversation
Contributor
Author
|
/ci |
As opposed to a rolling window of 50,000h (~5.7 years), this updates our cypress-created rules to query a window with a fixed starting point. The rolling window was problematic because our static test data will eventually fall outside of this window. While the fixed `from` means that we will technically be querying an increasing window over time, this should have little practical effect as the data should remain equally sparse. This is also how our integration tests generate rules, so cypress rules are now aligned in that approach, as well.
See previous commit.
This should not be needed and only lends us to future failures.
These have been fixed by fixing the window in which the rules query, so that our static data will no longer be able to roll "outside" of that window (which was causing these failures). There may be other instances of this type of failure, but these are the obvious ones from the last few days.
rylnd
force-pushed
the
rylnd/fix_outdated_test_data
branch
from
9e7b53e to
f878869
Compare
rylnd
changed the title
rylnd
added
Team:Detection Engine
Conflicts: x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/exceptions/alerts_table_flow/rule_exceptions/closing_all_matching_alerts.cy.ts x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/exceptions/rule_details_flow/add_edit_exception.cy.ts
Contributor
Author
|
/ci |
Contributor
Author
|
/ci |
The new_terms validation requires that the history window be greater than the `from`, which is currently set to 1900. If you're a developer reading this commit in 2050, I apologize.
Contributor
Author
|
/ci |
In moving the default `from` for rules to a _static, absolute_ value (to address a separate class of failures), this assertion started failing. In investigating the cause, I found that we both provide and assert on values based on the duration _from the current time_. For relative values, i.e. `now-5m`, this would always equate to 5 minutes, but with an absolute value it was now increasing as time passed, and subsequent calls with the same absolute value would likely never return the same value. I debated trying to account for this with some "fuzzy" matching (i.e. "interval must be between the provided value and (provided value + some duration)"), but that would still depend on the time between calls not exceeding that value. Ultimately, since: * this assertion is ultimately just asserting that the value we provide to the form is the value that is saved, and * we test this field for other rule types, with more reliable values, and * there is no indicator-match specific logic to this field, I decided that keeping this particular assertion was not worth the effort.
Contributor
Author
|
/ci |
Because this duration is imbedded as a filter in the generated ES queries, 125 years in seconds was causing `number_format_exceptions`
Contributor
Author
|
/ci |
Contributor
💚 Build Succeeded
Metrics [docs]
History
cc @rylnd |
Contributor
|
Starting backport for target branches: 8.17, 8.18, 8.19, 9.0 https://github.com/elastic/kibana/actions/runs/15218470223 |
This was referenced May 23, 2025
Contributor
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
rylnd
added a commit
to rylnd/kibana
that referenced
this pull request
May 23, 2025
…rules' query (elastic#220923) ## Summary Our tests rely on static test data, and our cypress suite was running rules with a large lookback (50000h) against them. Recently, tests have begun failing because we've exceeded 50000h since the data's `@timestamp`. This PR updates the cypress rules to use a fixed `from` field wherever possible, in order to "fix" that query window's starting point, ensuring that this issue does not recur. This is already how our integration tests are creating similar rules, so as an added bonus we're now more consistent across test suites, as well. This unskips and thus resolves the following issues: * elastic#201334 * elastic#220822 * elastic#207913 * elastic#199905 Note that there may be more instances of skipped tests that were failing due to this issue (with e.g. data with an earlier date than the `2019-09-02` here); the above tests were just the most obvious and most recent (within the last week). ### Flaky Runner * Detection Engine Cypress (x150): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8291 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 1f6777c) # Conflicts: # x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/exceptions/rule_details_flow/add_edit_exception_data_view.cy.ts # x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/indicator_match_rule.cy.ts
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
1 task
kibanamachine
added
the
backport missing
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
2 similar comments
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
May 29, 2025
…rules' query (elastic#220923) ## Summary Our tests rely on static test data, and our cypress suite was running rules with a large lookback (50000h) against them. Recently, tests have begun failing because we've exceeded 50000h since the data's `@timestamp`. This PR updates the cypress rules to use a fixed `from` field wherever possible, in order to "fix" that query window's starting point, ensuring that this issue does not recur. This is already how our integration tests are creating similar rules, so as an added bonus we're now more consistent across test suites, as well. This unskips and thus resolves the following issues: * elastic#201334 * elastic#220822 * elastic#207913 * elastic#199905 Note that there may be more instances of skipped tests that were failing due to this issue (with e.g. data with an earlier date than the `2019-09-02` here); the above tests were just the most obvious and most recent (within the last week). ### Flaky Runner * Detection Engine Cypress (x150): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8291 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
rylnd
added a commit
that referenced
this pull request
May 30, 2025
…ypress rules' query (#220923) (#221439) # Backport This will backport the following commits from `main` to `8.19`: - [[Detection Engine] Prevent test data from rolling outside of Cypress rules' query (#220923)](#220923) <!--- Backport version: 10.0.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Ryland Herrick","email":"ryalnd@gmail.com"},"sourceCommit":{"committedDate":"2025-05-23T20:16:26Z","message":"[Detection Engine] Prevent test data from rolling outside of Cypress rules' query (#220923)\n\n## Summary\n\nOur tests rely on static test data, and our cypress suite was running\nrules with a large lookback (50000h) against them. Recently, tests have\nbegun failing because we've exceeded 50000h since the data's\n`@timestamp`. This PR updates the cypress rules to use a fixed `from`\nfield wherever possible, in order to \"fix\" that query window's starting\npoint, ensuring that this issue does not recur. This is already how our\nintegration tests are creating similar rules, so as an added bonus we're\nnow more consistent across test suites, as well.\n\nThis unskips and thus resolves the following issues:\n\n* https://github.com/elastic/kibana/issues/201334\n* https://github.com/elastic/kibana/issues/220822\n* https://github.com/elastic/kibana/issues/207913\n* https://github.com/elastic/kibana/issues/199905\n\nNote that there may be more instances of skipped tests that were failing\ndue to this issue (with e.g. data with an earlier date than the\n`2019-09-02` here); the above tests were just the most obvious and most\nrecent (within the last week).\n\n### Flaky Runner\n* Detection Engine Cypress (x150):\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8291\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios","sha":"1f6777c40130e9804faa1b7a4a33f5e1655053a7","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:prev-minor","backport:prev-major","Team:Detection Engine","backport:current-major","Test:Cypress","v9.1.0"],"title":"[Detection Engine] Prevent test data from rolling outside of Cypress rules' query","number":220923,"url":"https://github.com/elastic/kibana/pull/220923","mergeCommit":{"message":"[Detection Engine] Prevent test data from rolling outside of Cypress rules' query (#220923)\n\n## Summary\n\nOur tests rely on static test data, and our cypress suite was running\nrules with a large lookback (50000h) against them. Recently, tests have\nbegun failing because we've exceeded 50000h since the data's\n`@timestamp`. This PR updates the cypress rules to use a fixed `from`\nfield wherever possible, in order to \"fix\" that query window's starting\npoint, ensuring that this issue does not recur. This is already how our\nintegration tests are creating similar rules, so as an added bonus we're\nnow more consistent across test suites, as well.\n\nThis unskips and thus resolves the following issues:\n\n* https://github.com/elastic/kibana/issues/201334\n* https://github.com/elastic/kibana/issues/220822\n* https://github.com/elastic/kibana/issues/207913\n* https://github.com/elastic/kibana/issues/199905\n\nNote that there may be more instances of skipped tests that were failing\ndue to this issue (with e.g. data with an earlier date than the\n`2019-09-02` here); the above tests were just the most obvious and most\nrecent (within the last week).\n\n### Flaky Runner\n* Detection Engine Cypress (x150):\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8291\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios","sha":"1f6777c40130e9804faa1b7a4a33f5e1655053a7"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/220923","number":220923,"mergeCommit":{"message":"[Detection Engine] Prevent test data from rolling outside of Cypress rules' query (#220923)\n\n## Summary\n\nOur tests rely on static test data, and our cypress suite was running\nrules with a large lookback (50000h) against them. Recently, tests have\nbegun failing because we've exceeded 50000h since the data's\n`@timestamp`. This PR updates the cypress rules to use a fixed `from`\nfield wherever possible, in order to \"fix\" that query window's starting\npoint, ensuring that this issue does not recur. This is already how our\nintegration tests are creating similar rules, so as an added bonus we're\nnow more consistent across test suites, as well.\n\nThis unskips and thus resolves the following issues:\n\n* https://github.com/elastic/kibana/issues/201334\n* https://github.com/elastic/kibana/issues/220822\n* https://github.com/elastic/kibana/issues/207913\n* https://github.com/elastic/kibana/issues/199905\n\nNote that there may be more instances of skipped tests that were failing\ndue to this issue (with e.g. data with an earlier date than the\n`2019-09-02` here); the above tests were just the most obvious and most\nrecent (within the last week).\n\n### Flaky Runner\n* Detection Engine Cypress (x150):\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8291\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios","sha":"1f6777c40130e9804faa1b7a4a33f5e1655053a7"}}]}] BACKPORT-->
rylnd
added a commit
that referenced
this pull request
Jun 2, 2025
…press rules' query (#220923) (#221437) # Backport This will backport the following commits from `main` to `9.0`: - [[Detection Engine] Prevent test data from rolling outside of Cypress rules' query (#220923)](#220923) <!--- Backport version: 10.0.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Ryland Herrick","email":"ryalnd@gmail.com"},"sourceCommit":{"committedDate":"2025-05-23T20:16:26Z","message":"[Detection Engine] Prevent test data from rolling outside of Cypress rules' query (#220923)\n\n## Summary\n\nOur tests rely on static test data, and our cypress suite was running\nrules with a large lookback (50000h) against them. Recently, tests have\nbegun failing because we've exceeded 50000h since the data's\n`@timestamp`. This PR updates the cypress rules to use a fixed `from`\nfield wherever possible, in order to \"fix\" that query window's starting\npoint, ensuring that this issue does not recur. This is already how our\nintegration tests are creating similar rules, so as an added bonus we're\nnow more consistent across test suites, as well.\n\nThis unskips and thus resolves the following issues:\n\n* https://github.com/elastic/kibana/issues/201334\n* https://github.com/elastic/kibana/issues/220822\n* https://github.com/elastic/kibana/issues/207913\n* https://github.com/elastic/kibana/issues/199905\n\nNote that there may be more instances of skipped tests that were failing\ndue to this issue (with e.g. data with an earlier date than the\n`2019-09-02` here); the above tests were just the most obvious and most\nrecent (within the last week).\n\n### Flaky Runner\n* Detection Engine Cypress (x150):\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8291\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios","sha":"1f6777c40130e9804faa1b7a4a33f5e1655053a7","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:prev-minor","backport:prev-major","Team:Detection Engine","backport:current-major","Test:Cypress","v9.1.0"],"title":"[Detection Engine] Prevent test data from rolling outside of Cypress rules' query","number":220923,"url":"https://github.com/elastic/kibana/pull/220923","mergeCommit":{"message":"[Detection Engine] Prevent test data from rolling outside of Cypress rules' query (#220923)\n\n## Summary\n\nOur tests rely on static test data, and our cypress suite was running\nrules with a large lookback (50000h) against them. Recently, tests have\nbegun failing because we've exceeded 50000h since the data's\n`@timestamp`. This PR updates the cypress rules to use a fixed `from`\nfield wherever possible, in order to \"fix\" that query window's starting\npoint, ensuring that this issue does not recur. This is already how our\nintegration tests are creating similar rules, so as an added bonus we're\nnow more consistent across test suites, as well.\n\nThis unskips and thus resolves the following issues:\n\n* https://github.com/elastic/kibana/issues/201334\n* https://github.com/elastic/kibana/issues/220822\n* https://github.com/elastic/kibana/issues/207913\n* https://github.com/elastic/kibana/issues/199905\n\nNote that there may be more instances of skipped tests that were failing\ndue to this issue (with e.g. data with an earlier date than the\n`2019-09-02` here); the above tests were just the most obvious and most\nrecent (within the last week).\n\n### Flaky Runner\n* Detection Engine Cypress (x150):\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8291\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios","sha":"1f6777c40130e9804faa1b7a4a33f5e1655053a7"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/220923","number":220923,"mergeCommit":{"message":"[Detection Engine] Prevent test data from rolling outside of Cypress rules' query (#220923)\n\n## Summary\n\nOur tests rely on static test data, and our cypress suite was running\nrules with a large lookback (50000h) against them. Recently, tests have\nbegun failing because we've exceeded 50000h since the data's\n`@timestamp`. This PR updates the cypress rules to use a fixed `from`\nfield wherever possible, in order to \"fix\" that query window's starting\npoint, ensuring that this issue does not recur. This is already how our\nintegration tests are creating similar rules, so as an added bonus we're\nnow more consistent across test suites, as well.\n\nThis unskips and thus resolves the following issues:\n\n* https://github.com/elastic/kibana/issues/201334\n* https://github.com/elastic/kibana/issues/220822\n* https://github.com/elastic/kibana/issues/207913\n* https://github.com/elastic/kibana/issues/199905\n\nNote that there may be more instances of skipped tests that were failing\ndue to this issue (with e.g. data with an earlier date than the\n`2019-09-02` here); the above tests were just the most obvious and most\nrecent (within the last week).\n\n### Flaky Runner\n* Detection Engine Cypress (x150):\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8291\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios","sha":"1f6777c40130e9804faa1b7a4a33f5e1655053a7"}}]}] BACKPORT-->
kibanamachine
added
v9.0.2
and removed
backport missing
Contributor
|
This PR didn't make it into the latest 9.0.2 BC. Updating the labels. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Our tests rely on static test data, and our cypress suite was running rules with a large lookback (50000h) against them. Recently, tests have begun failing because we've exceeded 50000h since the data's
@timestamp. This PR updates the cypress rules to use a fixedfromfield wherever possible, in order to "fix" that query window's starting point, ensuring that this issue does not recur. This is already how our integration tests are creating similar rules, so as an added bonus we're now more consistent across test suites, as well.This unskips and thus resolves the following issues:
Note that there may be more instances of skipped tests that were failing due to this issue (with e.g. data with an earlier date than the
2019-09-02here); the above tests were just the most obvious and most recent (within the last week).Flaky Runner
Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.