[Security Solution] [Serverless] Integrates Cypress in visual mode with QA environment

[MadameSheema]

Summary

We are working in order to have our Cypress tests ready to be executed on MKI projects. Our first goal is to be able to execute the tests in the QA environment.

So far we have created all the code needed to be able to execute them in headless mode:

• Fixed parallel script for cypress tests in QA and buildkite #169311
• [Security Solution] Parallel serverless script adjustments #171074

With all the above code, what we are doing is mimicking what we currently have for ESS and serverless on the first quality gate but for MKI projects in the QA environment. We create a project for each spec file following the indications of the Cypress spec and we execute the tests on that specific project.

That is really convenient for CI, but what happens when a test fails? How can we debug what is happening there?

In this PR

Continue mimicking our current flows, in this PR we have created the cypress:open:qa:serverless command.

When the command is executed:

1. A security project will be created in QA environment with the following characteristics:

   {
      "product_line":"security",
      "product_tier":"complete"
   },
   {
      "product_line":"endpoint",
      "product_tier":"complete"
   },
   {
      "product_line":"cloud",
      "product_tier":"complete"
   }

2. Opens the Cypress UI with all tests in the e2e directory tagged as @serverless.

3. Tests are executed using the recently created environment

4. The project is automatically deleted when:

• fn resolve/rejects
• Something calls process.exit()
• ctrl+c is used in the terminal causing the SIGINT signal to be sent to the process

How to test it

Setup a valid Elastic Cloud API key for QA environment:

1. Navigate to QA environment.
2. Click on the User menu button located on the top right of the header.
3. Click on Organization.
4. Click on the API keys tab.
5. Click on Create API key button.
6. Add a name, set an expiration date, assign an organization owner role.
7. Click on Create API key
8. Save the value of the key
   Store the saved key on ~/.elastic/cloud.json using the following format:

{
  "api_key": {
    "qa": "<API_KEY>"
  }
}

From the x-pack/test/security_solution_cypress you can execute the following commands:

yarn cypress:open:qa:serverless

The above command will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"complete"
   },
   {
      "product_line":"endpoint",
      "product_tier":"complete"
   },
   {
      "product_line":"cloud",
      "product_tier":"complete"
   }

yarn cypress:open:qa:serverless --tier=essentials

OR

yarn cypress:open:qa:serverless -t essentials

The above commands will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"essentials"
   },
   {
      "product_line":"endpoint",
      "product_tier":"essentials"
   },
   {
      "product_line":"cloud",
      "product_tier":"essentials"
   }

yarn cypress:open:qa:serverless --no-endpoint-addon

OR

yarn cypress:open:qa:serverless --endpoint-addon=false

The above command will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"complete"
   },
   {
      "product_line":"cloud",
      "product_tier":"complete"
   }

yarn cypress:open:qa:serverless --no-cloud-addon

OR

yarn cypress:open:qa:serverless --cloud-addon=false

The above commands will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"complete"
   },
   {
      "product_line":"endpoint",
      "product_tier":"complete"
   }

Note that tier and addons flags can be combined, for instance:

yarn cypress:open:qa:serverless --tier=essentials --no-endpoint-addon 

The above command will create a project with the following characteristics

   {
      "product_line":"security",
      "product_tier":"essentials"
   },
   {
      "product_line":"cloud",
      "product_tier":"essentials"
   }

To take into consideration

• RBAC cannot be currently tested on QA
• Cypress in visual mode does not have the logic to modify/create a project with the product tier and line(s) needed, you should be the one providing that information when executing the command
• The code available is the one deployed in QA environment
• The setup of the project takes time, there is nothing we can do to speed-up the process since is outside our control

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[cavokz]

As far as I can understand, LGTM.

[semd]

LGTM! Thanks for doing this Glo 💯

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 7156604

Metrics [docs]

✅ unchanged

History

• 💔 Build #176164 failed 073d7ec
• 💔 Build #176061 failed 5a6e17d
• 💚 Build #175869 succeeded 7e2f4e0
• 💚 Build #175561 succeeded eb0de0f
• 💚 Build #175447 succeeded dabe482

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @MadameSheema

[maximpn]

@MadameSheema thank you for adding an ability to run Cypress against QA in visual mode 👍

The changes look good overall. I left some comments regarding code structure and naming. On top of this I believe we should stick to first capital letter in texts for Kibana and Serverless but it's a minor note.

[maximpn]

nit: Instead of filtering out product types I'd recommend to add product types if a corresponding flag is set. It will make the implementation clearer.

[maximpn]

nit: tier, endpointAddon and cloudAddon are only specific for running Cypress against QA env in visual mode but script consumer can specify these props in the other modes as well though it has no effect. Since yargs supports commands clearer solution would to split functionality in commands with unique params.

[maximpn]

getProductTypes sounds too generic and doesn't reflect its purpose. Something like getVisualModeProductTypes should work better.

[maximpn]

Just curious. Why did you decide to run curl command instead of axios?

[MadameSheema]

@maximpn this addCleanupTask() didn't accept async, that was the reason behind if there is any other solution/way, more than happy to implement it :)

[maximpn]

kibana -> Kibana

[maximpn]

Suggested change

	### Running serverless tests locally pointing to FTR serverless (First Quality Gate)
	### Running serverless tests locally against FTR Serverless (First Quality Gate)

[maximpn]

Suggested change

	### Running serverless tests locally pointing to a MKI project created in QA environment (Second Quality Gate)
	### Running serverless tests locally against a MKI project created in QA environment (Second Quality Gate)

[maximpn]

kibana -> Kibana

[maximpn]

I'm curious about command naming. We have only one separate QA environment we run tests against. It turns out qa is a subset of serverless in some sense. I can imagine we have ESS QA environment in the future and this naming will have more sense.

Have you considered the other options like cypress:open:serverless-qa?

Naming in this file isn't consistent but we can improve it by sticking to one format something like cypress:mode:env[:suites name].