Fixed parallel script for cypress tests in QA and buildkite by dkirchan · Pull Request #169311 · elastic/kibana

dkirchan · 2023-10-18T20:27:19Z

Summary

A new parallel script is introduced, specifically for QA - Serverless environment and Cypress tests of security solution.
To be extended for:

Prod
Dev-Test
Potentially to be working with FTR tests.

A new target is created in package.json of security_solution_cypress in order to run the tests. With the introduced parallel script, the following steps are handled by the script during the test runtime.

Create Environment
Reset Credentials
Delete Environment

TEST RUNTIME
With this change any new test development can be directly tested by the kibana serverless pipeline providing the branch/fork name and the commit hash in case a fork is under test.

FOR LOCAL RUN
The developer needs to have an API key configured for the QA environment. It can either live in ~/.elastic/cloud.json file or be provided as an env var :
API_KEY=... yarn run cypress:run:qa:serverless:parallel

If the credentials of the required environment are needed to be crosschecked then run the yarn target with the DEBUG env var:
DEBUG=1 yarn run cypress:run:qa:serverless:parallel

As mentioned above, at the time being, the only environment where we run the suites and this script is QA.

Succesful Buildkite run for serverless tests and the specific test functionality

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

ghost · 2023-10-18T20:27:36Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-10-18T20:29:39Z

Pinging @elastic/security-solution (Team: SecuritySolution)

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

.buildkite/scripts/pipelines/security_solution_quality_gate/pipeline.sh

jbudz

.buildkite LGTM

… the job is finalised

maximpn

@dkirchan thank you for addressing my comments and making the script better 👍

I tested locally and it works as expected. The only problem it takes a lot of time to run the tests. On top of that I left some extra comments.

Overall the PR looks like almost finalized. I approve it in advance to unblock.

maximpn · 2023-11-03T15:25:27Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

+      product: response.data.type,
+    };
+  } catch (error) {
+    log.error(`${error}`);


Can you log an error.message instead of implicit error.toString()? It's not transparent what's error.toString() outputs.

Fixed with d25ab4e

maximpn · 2023-11-03T15:25:33Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

+    });
+    log.info(`Environment ${projectName} was successfully deleted!`);
+  } catch (error) {
+    log.error(`${error}`);


Can you log an error.message instead of implicit error.toString()? It's not transparent what's error.toString() outputs.

Fixed with d25ab4e

maximpn · 2023-11-03T15:25:39Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

+      username: response.data.username,
+    };
+  } catch (error) {
+    throw new Error(`${error}`);


Can you throw new Error(error.message) instead of implicit error.toString()? It's not transparent what's error.toString() outputs.

Fixed with d25ab4e

maximpn · 2023-11-03T15:31:42Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

+      throw new Error(`${runnerId} - ${error}`);
+    },
+    retries: 50,
+    factor: 2,


It looks like we can have a delay before attempting to fetch the status. As far as I know an MKI env takes around 6 minutes to be up and running. A delay can be a minute or two or the other amount of time we certain about.

Additionally we can "play" with params like factor, minTimeout and maxTimeout to find an optimal approach. Most probably a linear or fixed attempt interval with a delay before will work better.

It doesn't have to be addresses in this PR, just a general though.

maximpn · 2023-11-03T15:32:50Z

x-pack/test/security_solution_cypress/cypress/cypress_ci_serverless_qa.config.ts

    configFile: './cypress/reporter_config.json',
  },
-  defaultCommandTimeout: 150000,
+  defaultCommandTimeout: 300000,


Why the timeout was increased?

@MadameSheema can you respond this?

maximpn · 2023-11-03T15:41:10Z

x-pack/test/security_solution_cypress/package.json

    "cypress:explore:run:serverless": "yarn cypress:serverless --spec './cypress/e2e/explore/**/*.cy.ts'",
    "cypress:changed-specs-only:serverless": "yarn cypress:serverless --changed-specs-only --env burn=5",
-    "cypress:burn:serverless": "yarn cypress:serverless --env burn=5"
+    "cypress:burn:serverless": "yarn cypress:serverless --env burn=5",


Let't set burn to 2 instead of 5. It should be enough to verify the tests doesn't fail due to artefacts left.

Fixed with d25ab4e

maximpn · 2023-11-03T15:43:12Z

.buildkite/pipelines/security_solution/base.yml

@@ -0,0 +1,11 @@
+steps:


How often to we plan to run it?

On demand, when triggered by the quality gate, maybe in different PRs..... Not yet strictly defined

MadameSheema · 2023-11-06T10:54:49Z

.buildkite/pipelines/security_solution/base.yml

+    agents:
+      queue: n2-4-spot
+    timeout_in_minutes: 300
+    parallelism: 6


Why 6 as parallelism?

I split the test suites to non explore/investigations and these two categories on their own.

MadameSheema · 2023-11-06T16:46:53Z

Note that once this PR is merged, there is more work we need to do. We want to merge this PR as it is because we are not breaking any existing or new flow and if we continue working on it is going to become a huge/monster PR and we may face the risk of having huge conflicts. On following PRs:

We'll continue stabilizing our Cypress tests on MKI to make them more robust and reliable.
We'll continue improving our pipeline
We'll work on the requirements we need to meet in order to have the quality gate in the release pipeline
Once everything is stabilized and we feel ready, we'll integrate our quality gate with the release pipeline

kibana-ci · 2023-11-06T17:21:02Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: f10e5ff

Failed CI Steps

FTR Configs #48

Test Failures

[job] [logs] FTR Configs #48 / EPM Endpoints EPM - list list api tests lists all limited packages from the registry

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	472	478	+6

Total ESLint disabled count

id	before	after	diff
`securitySolution`	540	546	+6

History

💚 Build #173424 succeeded b73cf3c
💔 Build #173266 failed 52a7b74
💔 Build #173216 failed 691d8c7
💔 Build #173092 failed 3077982
💔 Build #173039 failed 3006b32
💔 Build #173037 failed d7613c8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @dkirchan

kibanamachine · 2023-11-06T17:24:51Z

💔 All backports failed

Status	Branch	Result
❌	8.11	Backport failed because of merge conflicts You might need to backport the following PRs to 8.11: - [Security Solution] Unskip and enable for Serverless `shared_exception_lists_management` Cypress tests (#169182) - [Security Solution] fix cypress config to run all tests (#169942) - [Security Solution] Adding serverlessQA tag (#167494)

Manual backport

To create the backport manually run:

node scripts/backport --pr 169311

Questions ?

Please refer to the Backport tool documentation

dkirchan requested review from a team, MadameSheema, banderror, maximpn, oatkiller and patrykkopycinski as code owners October 18, 2023 20:27

dkirchan requested a review from a team October 18, 2023 20:27

dkirchan added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.11.0 v8.12.0 labels Oct 18, 2023

dkirchan self-assigned this Oct 18, 2023

dkirchan force-pushed the security/dkirchan-create-envs branch from a59a310 to 3a1327e Compare October 18, 2023 20:33

jbudz reviewed Oct 18, 2023

View reviewed changes

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts Outdated Show resolved Hide resolved

jbudz reviewed Oct 18, 2023

View reviewed changes

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts Outdated Show resolved Hide resolved

jbudz reviewed Oct 18, 2023

View reviewed changes

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts Outdated Show resolved Hide resolved

jbudz reviewed Oct 18, 2023

View reviewed changes

.buildkite/scripts/pipelines/security_solution_quality_gate/pipeline.sh Outdated Show resolved Hide resolved

dkirchan force-pushed the security/dkirchan-create-envs branch 7 times, most recently from 1431a92 to 9dc4e91 Compare October 19, 2023 15:43

jbudz approved these changes Oct 19, 2023

View reviewed changes

dkirchan force-pushed the security/dkirchan-create-envs branch from 244f289 to a53ab3a Compare October 22, 2023 15:26

dkirchan requested a review from a team as a code owner October 23, 2023 09:08

maximpn and others added 6 commits November 3, 2023 12:12

simplify waitForEsStatusGreen function

7b9a157

Fixed wait for kibana status available.

3077982

Enabled tests to run on proper kibana nodes with parallelism

cda8ecd

Removed parallel count as it is defined in the yaml file

2482326

Introducing a long timeout in and we will be able to redefine it when…

fb18ec6

… the job is finalised

Fixed the ts script that is targeted in the preparation of the build

954c08f

maximpn approved these changes Nov 3, 2023

View reviewed changes

dkirchan and others added 3 commits November 3, 2023 18:33

Fixed delay before environment ready

691d8c7

Addressed comments around errors and burn number

d25ab4e

Merge branch 'main' into security/dkirchan-create-envs

52a7b74

MadameSheema reviewed Nov 6, 2023

View reviewed changes

MadameSheema and others added 5 commits November 6, 2023 11:56

Merge branch 'main' into security/dkirchan-create-envs

756665e

Merge branch 'main' into security/dkirchan-create-envs

b73cf3c

Split all tests to non investigations/explore

46649ed

Removed dependencies in Buildkite jobs

91708d6

Wait for 8 mins

f10e5ff

angorayc approved these changes Nov 6, 2023

View reviewed changes

MadameSheema enabled auto-merge (squash) November 6, 2023 16:47

MadameSheema merged commit ed4ef2a into main Nov 6, 2023

MadameSheema deleted the security/dkirchan-create-envs branch November 6, 2023 17:21

MadameSheema removed the v8.11.0 label Nov 6, 2023

kibanamachine added the backport:skip This PR does not require backporting label Nov 6, 2023

This was referenced Nov 6, 2023

[Security Solution] Removing cleanKibana method from Cypress #170636

Merged

[Security Solution] Removing cy.session from Cypress tests #170969

Merged

MadameSheema mentioned this pull request Nov 14, 2023

[Security Solution] [Serverless] Integrates Cypress in visual mode with QA environment #171107

Merged

MadameSheema mentioned this pull request Dec 13, 2023

[Security Solution] Preparing Cypress for the second quality gate - latest steps #173327

Open

43 tasks

mgiota mentioned this pull request May 27, 2024

[Observability solution] [SLO] Run burn rate api tests in serverless & ess using mocha tagging #183113

Closed

4 tasks

Conversation

dkirchan commented Oct 18, 2023 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Risk Matrix

For maintainers

Uh oh!

ghost commented Oct 18, 2023

🤖 GitHub comments

Uh oh!

elasticmachine commented Oct 18, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbudz left a comment

Choose a reason for hiding this comment

Uh oh!

maximpn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadameSheema commented Nov 6, 2023

Uh oh!

kibana-ci commented Nov 6, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

kibanamachine commented Nov 6, 2023

💔 All backports failed

Manual backport

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

dkirchan commented Oct 18, 2023 •

edited by kibanamachine

Loading