[7.x] [Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)#83967
Merged
FrankHassanabad merged 1 commit intoelastic:7.xfrom Nov 20, 2020
Conversation
…nd 213 tests for exception lists (elastic#83764) ## Summary Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism. The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as: * Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE. * Check REST output of the rule, did it match expected output body and status code? * In some rare cases we check if the the rule can be executed and we get a status of 'succeeded' With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic. Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that: * Detection engine produces signals as expected * Structure of the signals are as expected, including signal on signals * Exceptions to signals are working as expected * Most runtime bugs can be TDD'ed with e2e FTR's and regressions * Whack-a-mole will not happen * Consistency and predictability of signals is validated * Refactoring can occur with stronger confidence * Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors * Scaling tests can happen without failures * Velocity for creating tests increases as the utilities and examples increase Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as: * Creating tests that exercise each rule against a set of data criteria and get signal hits * Creating tests that validate the rule overrides operate as expected against data sets * Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value. These tests follow the pattern of: * Add the smallest data set to a folder in data.json (not gzip format) * Add the smallest mapping to that folder (mapping.json) * Call REST input related to exception lists, value lists, adding prepackaged rules, etc... * Call REST input related endpoint with utilities to create and activate the rule * Wait for the rule to go into the `succeeded` phase * Wait for the N exact signals specific to that rule to be available * Check against the set of signals to ensure that the matches are exactly as expected Example of one runtime test: A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected. ```ts { "type": "doc", "value": { "id": "1", "index": "long", "source": { "@timestamp": "2020-10-28T05:00:53.000Z", "long": 1 }, "type": "_doc" } } { "type": "doc", "value": { "id": "2", "index": "long", "source": { "@timestamp": "2020-10-28T05:01:53.000Z", "long": 2 }, "type": "_doc" } } { "type": "doc", "value": { "id": "3", "index": "long", "source": { "@timestamp": "2020-10-28T05:02:53.000Z", "long": 3 }, "type": "_doc" } } { "type": "doc", "value": { "id": "4", "index": "long", "source": { "@timestamp": "2020-10-28T05:03:53.000Z", "long": 4 }, "type": "_doc" } } ``` Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there. ```ts { "type": "index", "value": { "index": "long", "mappings": { "properties": { "@timestamp": { "type": "date" }, "long": { "type": "long" } } }, "settings": { "index": { "number_of_replicas": "1", "number_of_shards": "1" } } } } ``` Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox. ```ts beforeEach(async () => { await createSignalsIndex(supertest); await createListsIndex(supertest); await esArchiver.load('rule_exceptions/keyword'); }); afterEach(async () => { await deleteSignalsIndex(supertest); await deleteAllAlerts(supertest); await deleteAllExceptions(es); await deleteListsIndex(supertest); await esArchiver.unload('rule_exceptions/keyword'); }); describe('"is" operator', () => { it('should filter 1 single keyword if it is set as an exception', async () => { const rule = getRuleForSignalTesting(['keyword']); const { id } = await createRuleWithExceptionEntries(supertest, rule, [ [ { field: 'keyword', operator: 'included', type: 'match', value: 'word one', }, ], ]); await waitForRuleSuccess(supertest, id); await waitForSignalsToBePresent(supertest, 3, [id]); const signalsOpen = await getSignalsById(supertest, id); const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort(); expect(hits).to.eql(['word four', 'word three', 'word two']); }); }); ``` ### Changes for better determinism To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc... Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules. ```ts export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({ ``` Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 . ```ts export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true) ``` This waits for a rule to succeed before continuing ```ts await waitForRuleSuccess(supertest, id); ``` I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours. ```ts await waitForSignalsToBePresent(supertest, 4, [id]); ``` I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough. ```ts const signalsOpen = await getSignalsById(supertest, id); const signalsOpen = await getSignalsByIds(supertest, [createdId]); const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']); ``` I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing. ```ts deleteAllAlerts() ``` When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are: ```ts installPrePackagedRules() waitForRuleSuccess() importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing ``` Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Contributor
💛 Build succeeded, but was flaky
Test FailuresChrome X-Pack UI Functional Tests.x-pack/test/functional/apps/dashboard/drilldowns/dashboard_to_dashboard_drilldown·ts.dashboard drilldowns Dashboard to dashboard drilldown Copy to space Dashboards linked by a drilldown are both copied to a spaceStandard OutStack TraceMetrics [docs]
To update your PR or re-run it, just comment with: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports the following commits to 7.x: