[Security Solution][Detection Engine] Running API tests in Serverless & ESS using Mocha Tagging by WafaaNasr · Pull Request #166755 · elastic/kibana

WafaaNasr · 2023-09-19T17:05:13Z

Summary

Addresses [Security Solution] Find a way to reuse tests between ESS and Serverless suites #161537

Description

This PR follows the second option defined in this document the Mocha tagging
It introduces a new folder x-pack/test/security_solution_api_integration which will serve as a centralized location to meet all the requirements related to renaming tests appropriately and grouping similar tests together. It will facilitate the management of tests that must be run in Serverless and ESS environments.
Within this folder, there is a "config" subdirectory that stores base configurations specific to both the Serverless and ESS environments, These configurations build upon the base configuration provided by test_serverless and api_integrations, incorporating additional settings such as environment variables and tagging options.
It demonstrates scenarios involving @ess, @serverless, and @brokenInServerless.
The file x-pack/test/security_solution_api_integration/test_suites/detections_response/rule_creation/create_rules.ts is functional in both Serverless and ESS. However, some tests related to roles are currently skipped for Serverless until they are resolved, and these tests are tagged with @brokenInServerless

CI

It includes a new entry in the ftr_configs.yml to execute the newly added tests in the pipeline.
It involves the addition of mochaOptions in both serverless/config.base.ts and ess/config.base.ts. In the case of serverless, it includes @serverless while excluding @brokenInServerless. Similarly, for ess, it includes @ess and excludes @brokenInEss.

from x-pack/test/security_solution_api_integration/config/serverless

Update in x-pack/test/detection_engine_api_integration

The create_rules.ts and create_rule_exceptions files have been relocated from x-pack/test/detection_engine_api_integration/security_and_spaces/group1 to their respective domains within the x-pack/test/security_solution_api_integration folder.
The util files now are copied over from the old folder x-pack/test/detection_engine_api_integration to the new folder and will be removed once all tests are moved to the new folder to don't break the existing tests

…fix'

…thub.com/WafaaNasr/kibana into detection-enginer-ftr-serverless-initial

…-fix'

…thub.com/WafaaNasr/kibana into detection-enginer-ftr-serverless-initial

… reuse them

…on-enginer-ftr-serverless-initial

…tion groups in the new api integration test folder (#169234) ## Summary - Following the initial work in this #166755 - Addresses part of #151902 for actions https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit - Enable migrations of legacy actions to run only in ESS - Add the `@skipInQA` tag to the failing tests in QA env --------- Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…est and actions legacy utils (#170147) ## Summary - Following the initial work in this #166755 - Addresses part of #151902 for rule preview https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit - Remove the `skipInQA` tag from `Migrations` actions test since it runs only in ESS env. ### Moved files from `detection_engine_api_integrations` to `security_solution_api_integration` - utils (Moved and references are updated): - './get_legacy_action_notifications_so_by_id'; - './get_legacy_actions_so_by_id'; - './get_simple_preview_rule'; - './get_simple_rule_output_with_web_hook_action'; - './get_simple_rule_preview_output'; - Group1: - `preview_rules` moved under the `rule_creation` folder

…est and actions legacy utils (elastic#170147) ## Summary - Following the initial work in this elastic#166755 - Addresses part of elastic#151902 for rule preview https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit - Remove the `skipInQA` tag from `Migrations` actions test since it runs only in ESS env. ### Moved files from `detection_engine_api_integrations` to `security_solution_api_integration` - utils (Moved and references are updated): - './get_legacy_action_notifications_so_by_id'; - './get_legacy_actions_so_by_id'; - './get_simple_preview_rule'; - './get_simple_rule_output_with_web_hook_action'; - './get_simple_rule_preview_output'; - Group1: - `preview_rules` moved under the `rule_creation` folder

…170350) ## Summary - Following the initial work in this #166755 - Addresses part of #151902 for Alerts - Introduced a new folder called `alerts` under `security_solution_api_integration` - Moved the utility files associated with alerts to the new directory `security_solution_api_integration`. Files that were not actively used in the previous folder were moved, while any duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old / new groups details and execution time [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) | Action | File | New Path if moved | |--------|------|----------| | Delete| group1/find_rule_exception_references.ts| Already moved in previous PR | | Delete|security_and_spaces/group6 | - | | Move|detection_engine_api_integration/security_and_spaces/group6 |detections_response/default_license/alerts/alerts_compatibility.ts| | Move|detection_engine_api_integration/security_and_spaces/group1| detections_response/default_license/alerts/aliases.ts | | Move |detection_engine_api_integration/security_and_spaces/group1| detections_response/default_license/alerts/create_index.ts| | Move|detection_engine_api_integration/security_and_spaces/group10/create_signals_migrations| detections_response/default_license/alerts/migrations/create_alerts_migrations.ts | | Move|detection_engine_api_integration/security_and_spaces/group10/delete_signals_migrations| detections_response/default_license/alerts/migrations/delete_alerts_migrations.ts| | Move|detection_engine_api_integration/security_and_spaces/group10/finalize_signals_migrations| detections_response/default_license/alerts/migrations/finalize_alerts_migrations.ts | | Move|detection_engine_api_integration/security_and_spaces/group10/get_signals_migration_status| detections_response/default_license/alerts/migrations/get_alerts_migration_status.ts| | Move|detection_engine_api_integration/security_and_spaces/group10/open_close_signals| detections_response/default_license/alerts/open_close_alerts.ts| | Move|detection_engine_api_integration/security_and_spaces/group10/alert_tags| detections_response/default_license/alerts/alert_tags.ts| Note: these tests are skipped on the main branch https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/open_close_signals.ts#L215 https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/open_close_signals.ts#L252 https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/finalize_signals_migrations.ts#L192 https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_index.ts#L42 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…sts (#170530) ## Summary Following the initial work in this #166755 - Addresses part of #151902 for Entity Analytics/Risk Engine - Introduced new folders called `entity_analytics/risk_engine` under `security_solution_api_integration` - Moved the utility files associated with risk_engine to the new directory `security_solution_api_integration`. Files that were not actively used in the previous folder were moved, while any duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old / new groups details and execution time [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) | Action | File | New Path if moved | |--------|------|----------| | Moved| group10/risk_engine/init_and_status_apis|/entity_analytics/default_license/risk_engine/init_and_status_apis.ts | | Moved| group10/risk_engine/risk_score_calculation|/entity_analytics/default_license/risk_engine/risk_score_calculation.ts | | Moved| group10/risk_engine/risk_score_preview|/entity_analytics/default_license/risk_engine/risk_score_preview.ts | | Moved| group10/risk_engine/risk_scoring_task_execution|/entity_analytics/default_license/risk_engine/risk_scoring_task_execution.ts | | Moved| group10/risk_engine/telemetry_usage|/entity_analytics/default_license/risk_engine/telemetry_usage.ts | | Moved| group10/risk_engine/utils|entity_analytics/utils/risk_engine.ts | | Moved| utils/get_stats|entity_analytics/utils/get_risk_engine_stats.ts | Tests skipped on main: https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/risk_engine/init_and_status_apis.ts#L363 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Nikita Khristinin <nkhristinin@gmail.com>

… logic (#170765) ## Summary Following the initial work in this #166755 - Addresses part of #151902 for rule execution logic - Moved the utility files associated with rule execution logic to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old/new group details, decisions, and execution time are mentioned in this[document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) - Added new `Alert` archive for version `8.8.0` - Resolved the issue with the `query.ts` test where the execution logic is executed last, encompassing the "query" test because it was unloading the alerts document and led to failures in subsequent tests. - For `Alert As Data` in **Serverless** the alert ancestor will be a data-stream however in **ESS** will be `.internal.alerts-security.alerts-default-000001'` | Action | File | New Path if moved | |--------|------|----------| | Delete| security_and_spaces/rule_execution_logic| - | | Delete|security_and_spaces/group5 | - | | Move|detection_engine_api_integration/security_and_spaces/group5|detections_response/default_license/rule_execution_logic/keyword_family| | Move|detection_engine_api_integration/security_and_spaces/rule_execution_logic| detections_response/default_license/rule_execution_logic/execution_logic | | Move |detection_engine_api_integration/security_and_spaces/group1/ignore_fields| detections_response/default_license/rule_execution_logic/ignore_fields.ts| | Move|detection_engine_api_integration/security_and_spaces/group1/runtime| detections_response/default_license/rule_execution_logic/runtime.ts | | Move|detection_engine_api_integration/security_and_spaces/group1/timestamps| detections_response/default_license/rule_execution_logic/timestamps.ts| --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…ated tests and Action Throttle (#171397) ## Summary Following the initial work in this #166755 - Addresses part of #151902 for Privileges related tests and Action Throttle - Moved the utility files associated with telemetry to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old/new group details, decisions, and execution time are mentioned in this [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) - The **Privileges** Tests are skipped in Serverless now until the FTR Roles [PR](#170131) gets merged | Action | File | New Path if moved | |--------|------|----------| | Delete| security_and_spaces/group4| - | | Move|detection_engine_api_integration/security_and_spaces/group10/throttle|detections_response/default_license/action/throttle| | Move|detection_engine_api_integration/security_and_spaces/group1/check_privileges|detections_response/default_license/actions/check_privileges| | Move|detection_engine_api_integration/security_and_spaces/group10/read_privileges|detections_response/default_license/user_roles/read_privileges|

#171159) ## Summary Following the initial work in this #166755 - Addresses part of #151902 for Telemetry - Moved the utility files associated with telemetry to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old/new group details, decisions, and execution time are mentioned in this [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) | Action | File | New Path if moved | |--------|------|----------| | Delete| security_and_spaces/group4| - | | Move|detection_engine_api_integration/security_and_spaces/group4|detections_response/default_license/telemetry/keyword_family|

…n engine tests (elastic#171531) ## Summary Following the initial work in this elastic#166755 - Addresses part of elastic#151902 for basic detection engine-related tests - Introduced a new license folder to hold the `Basic` Ess tests and the `Essentials` Serverless tests, is called `basic_essentials_license` - Added new base configurations files for `serverless/config.base.essentials` and `ess/config.base.basic` - Moved the utility files associated with Basic tests to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old/new group details, decisions, and execution time are mentioned in this [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) - The **Privileges** Tests are skipped in Serverless now until the FTR Roles [PR](elastic#170131) gets merged | Action | File | New Path | |--------|------|----------| | Moved|basic/create_rules| basic_essentials_license/detection_engine/rules/create_rules| | Moved|basic/create_rules| basic_essentials_license/detection_engine/rules/create_ml_rules_privileges| | Moved|basic/create_rules| basic_essentials_license/detection_engine/alerts/open_close_alerts| | Moved|basic/create_rules| basic_essentials_license/detection_engine/alerts/query_alerts_backword_compatibility| | Moved|basic/create_rules| basic_essentials_license/detection_engine/alerts/query_alerts|

…astic#171992) ## Summary Following the initial work in this elastic#166755 - Addresses part of elastic#151902 for List APIs tests - Added a new folder under the `security_solution_api_integration` called `lists_and_exception_lists` to hold the lists and exception lists tests, and split the `List` APIs into two groups since the execution time in Serverless was close to 30 mins - Modified the [x-pack/plugins/lists/common/schemas/response/exception_list_item_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-4e3545fdeb8c8d9467cfa1c4aa88194e189193a92fa6f1cf5f859b1ef1beb45c), [x-pack/plugins/lists/common/schemas/response/exception_list_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-c3cc18faf07aab86e307185d41599c3596a3f8b360d3e4829591afa148283238) , [x-pack/plugins/lists/common/schemas/response/list_item_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-608579ca5e65da74f41319a58d81ab12cc3d79d389b087806c7b74949fbc6cc3), [x-pack/plugins/lists/common/schemas/response/list_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-efc64eb35937a8da28fc982c527253c0923650ae4163d4bbc203d3ebc2949835) to accept `elastic user` input because it changes in ESS and Serverless - Deleted the `x-pack/test/lists_api_integration` folder - Moved the utility files associated with Basic tests to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the below files imports from the old `lists_api_integration` folder to the new `lists_and_exception_lists` ``` lists_api_integration/ - exceptions/operators_data_types/date_numeric_types/date.ts - exceptions/operators_data_types/date_numeric_types/double.ts - exceptions/operators_data_types/date_numeric_types/float.ts - exceptions/operators_data_types/date_numeric_types/integer.ts - exceptions/operators_data_types/ips/ip.ts - exceptions/operators_data_types/ips/ip_array.ts - exceptions/operators_data_types/keyword/keyword.ts - exceptions/operators_data_types/keyword/keyword_array.ts - exceptions/operators_data_types/long/long.ts - exceptions/operators_data_types/text/text.ts - exceptions/operators_data_types/text/text_array.ts - exceptions/workflows/create_endpoint_exceptions.ts - exceptions/workflows/create_rule_exceptions.ts - exceptions/workflows/find_rule_exception_references.ts - exceptions/workflows/role_based_add_edit_comments.ts - exceptions/workflows/role_based_rule_exceptions_workflows.ts - exceptions/workflows/rule_exception_synchronizations.ts - rule_execution_logic/execution_logic/esql.ts - rule_execution_logic/execution_logic/machine_learning.ts - rule_execution_logic/execution_logic/new_terms.ts - rule_execution_logic/execution_logic/query.ts - telemetry/task_based/all_types.ts - telemetry/task_based/detection_rules.ts - telemetry/task_based/security_lists.ts ``` ``` These files should be moved too soon to the new `lists_api_integration` detection_engine_api_integration/security_and_spaces/group10 - import_export_rules.ts - import_rules.ts - perform_bulk_actions ``` - Updated the below files imports to the `ftr_provider_context_with_spaces.d.ts` ` - risk_engine/risk_scoring_task/task_execution_nondefault_spaces.ts` - The QA phase concluded with all tests passing successfully. 🟢 - Updated the CodeOwner file for the newly moved tests - Add a new util file to `deleteAllExceptions` under the old `detection_engine_api` folder since the Rule management related-tests are still need to be moved over to the new folder - Old/new group details, decisions, and execution time are mentioned in this [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

WafaaNasr and others added 19 commits September 19, 2023 15:29

add detection_engine folder under test_serverless for testing

c165f2f

running create_rule_exception in serverless

423ccde

running create_rule_exception in serverless

1e0730c

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

2189c61

…fix'

add security_solution_api_integration and config serverless

62d8a68

Merge branch 'detection-enginer-ftr-serverless-initial' of https://gi…

1b95701

…thub.com/WafaaNasr/kibana into detection-enginer-ftr-serverless-initial

remove detection_engine from test_serverless

82429a7

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

c01c148

…-fix'

define commands in package.json

35198f0

latest

5df3507

run tests for both ess and serverless

dbc106e

Merge branch 'main' into detection-enginer-ftr-serverless-initial

925014a

try mocha tags

9bb3bf8

Merge branch 'detection-enginer-ftr-serverless-initial' of https://gi…

b08cace

…thub.com/WafaaNasr/kibana into detection-enginer-ftr-serverless-initial

use suiteTags in the config

b68b981

move serverless config to serverless config until figuring out how to…

ed96e45

… reuse them

add rule_creation test with utils

16aff77

Merge branch 'main' of https://github.com/elastic/kibana into detecti…

fd59ac9

…on-enginer-ftr-serverless-initial

only allow exceptions to be for serverless and disable rule

591f0fe

WafaaNasr self-assigned this Sep 22, 2023

WafaaNasr added FTR release_note:skip Skip the PR/issue when compiling release notes labels Sep 22, 2023

WafaaNasr added 2 commits September 25, 2023 15:43

apply the mocha tagging with introducing @brokenInServerless

bcbabbf

Merge branch 'main' of https://github.com/elastic/kibana into detecti…

efa734d

…on-enginer-ftr-serverless-initial

WafaaNasr changed the title ~~[Security Solution][Detection Engine] Running API tests in Serverless~~ [Security Solution][Detection Engine] Running API tests in Serverless using Mocha Tagging Sep 25, 2023

WafaaNasr added 5 commits September 26, 2023 17:38

Merge branch 'main' of https://github.com/elastic/kibana into detecti…

6e12eba

…on-enginer-ftr-serverless-initial

reuse config.base from test_serverless

7532f52

reuse test_serverless configs and only add commands for detectionegine

9199d72

Merge branch 'main' of https://github.com/elastic/kibana into detecti…

c868a08

…on-enginer-ftr-serverless-initial

add config base in ftr ci

d1970aa

WafaaNasr mentioned this pull request Oct 30, 2023

[Security Solution][API testing] Move and restructures rule preview test and actions legacy utils #170147

Merged

gergoabraham mentioned this pull request Oct 30, 2023

[Defend Workflows] Enable FTRs for serverless using tags #169946

Merged

This was referenced Nov 1, 2023

[Security Solution][API testing] Move and restructures Alerts' tests #170350

Merged

[Security Solution][API testing] Move and restructures Risk Engine tests #170530

Merged

WafaaNasr mentioned this pull request Nov 7, 2023

[Security Solution][API testing] Move and restructures Rule execution logic #170765

Merged

This was referenced Nov 14, 2023

[Security Solution][API testing] Move and restructures Telemetry logic #171159

Merged

[Security Solution][API testing] Move and restructures Privileges related tests and Action Throttle #171397

Merged

WafaaNasr mentioned this pull request Nov 20, 2023

[Security Solution][API testing] Move and restructures Basic detection engine tests #171531

Merged

WafaaNasr mentioned this pull request Nov 27, 2023

[Security Solution][API testing] Move and restructures Lists APIS #171992

Merged

WafaaNasr mentioned this pull request Nov 29, 2023

[Security Solution][API testing][Rule Management] Move and restructures remaining rule managements trail-tests #172173

Merged

yctercero mentioned this pull request Nov 29, 2023

[Detections Response][FTR] Move remaining basic license FTRs to new folder #172132

Closed

WafaaNasr deleted the detection-enginer-ftr-serverless-initial branch February 6, 2024 10:26

WafaaNasr changed the title ~~[Security Solution][Detection Engine] Running API tests in Serverless using Mocha Tagging~~ [Security Solution][Detection Engine] Running API tests in Serverless & ESS using Mocha Tagging Feb 9, 2024

mgiota mentioned this pull request May 10, 2024

[Observability solution] [SLO] Run burn rate api tests in serverless & ess using mocha tagging #183113

Closed

4 tasks

elasticmachine unassigned WafaaNasr Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detection Engine] Running API tests in Serverless & ESS using Mocha Tagging#166755

[Security Solution][Detection Engine] Running API tests in Serverless & ESS using Mocha Tagging#166755
WafaaNasr merged 84 commits intoelastic:mainfrom
WafaaNasr:detection-enginer-ftr-serverless-initial

WafaaNasr commented Sep 19, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

WafaaNasr commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Description

CI

Update in x-pack/test/detection_engine_api_integration

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

WafaaNasr commented Sep 19, 2023 •

edited

Loading