[Security Solution] Reusing Cypress tests POC1

[MadameSheema]

Summary

With the aim of speed-up the test automation on serverless and reusing the work we have done so far with Cypress, we are doing different POC to check which one fits better our needs(#161537).

NOTE: This PR is just a POC, the code is not going to be merged, the idea behind is to open a discussion to see which solution should be implemented and then create a proper PR for it.

Idea

To have different folders. One for serverless specific, one for ESS specific, and one for tests that can be executed in both environments.

Implementation for the POC

• x-pack/plugins/security_solution/cypress/e2e is used for ESS tests
• x-pack/test_serverless/functional/test_suites/security/cypress/e2e is used for specific serverless tests
• x-pack/test/security_solution_cypress/e2e is used for tests that can be executed on both environments

The idea is that when we execute the ESS or serverless tests, to include as well the ones on the x-pack/test/security_solution_cypress/e2e security folder. In order to do so, we have modified the specPattern on each one of the executions. To demonstrate that the idea works, we have used the user_details.cy.ts.

As part of this POC we have also:

• Removed the waitForPageToBeLoaded method for investigation purposes. The serverless environment does not have that locator and cypress by default waits for an element to be present on the UI before performing an action, so just checking if is really necessary for the current tests.
• Added a modification to the parallel.ts script used currently for the ESS tests to spin-up the environment to take into consideration the specPattern when using Cypress on visual mode.
• Loaded the default audibteat archive when starting the FTR serverless environment and added some missing environment variables to make the user_details.cy.ts test work when is executed from serverless.
• Added missing commands on the serverless folder to make the user_details.cy.ts test work when is executed from serverless.

PROS

• Easy way of having a clear picture of the belonging of each test due to the folder structure.

CONS

• It might be slow to move all the tests.
• It might be complex or not quite clear how to manage the shared screens and tasks

Takeaways

If we want to have a clear picture of the belonging of each test, that might be our solution. If we finally go for this we should also decide:

• If the proposed folder for the environment agnostic tests is the correct one.
• If the proposed naming is the correct one.
• What do we do with the screens and tasks? Should we move them into that folder as well? Should all the tasks and screens live in that folder?

[kibana-ci]

💔 Build Failed

• Buildkite Build
• Commit: 00f3a69
• Interpreting CI Failures

Failed CI Steps

• FTR Configs #19
• Security Solution Tests #6
• Explore - Security Solution Tests #1
• Explore - Security Solution Tests #1
• Explore - Security Solution Tests #1
• Explore - Security Solution Tests #1
• Explore - Security Solution Tests #2
• Explore - Security Solution Tests #2
• Explore - Security Solution Tests #2
• Explore - Security Solution Tests #2

Test Failures

• [job] [logs] FTR Configs #19 / management Files management should render an empty prompt

Metrics [docs]

✅ unchanged

History

• 💔 Build #143080 failed 23f0392
• 💔 Build #143070 failed 2effd3a
• 💔 Build #143000 failed e78f467
• 💔 Build #142933 failed e6b07a9b76ef9a91f911c6ab3f7319dfbecae3d3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[banderror]

@MadameSheema I'm in favor of the 2nd POC that uses tagging.

This POC proposes to split tests into this kind of three folders:

• x-pack/plugins/security_solution/cypress/e2e for ESS tests
• x-pack/test_serverless/functional/test_suites/security/cypress/e2e for Serverless tests
• x-pack/test/security_solution_cypress/e2e for tests that can run both against ESS and Serverless

I think this would:

• make it hard to reason about what tests are kept where, just because 2/3 of path names are not explicit about that
• force us to jump between folders all the time; jumping between distant folders is poor DX and takes more time usually
• as you mentioned in the description, this would bring the problem of implementing and reusing helper code for tests ("screens", "tasks", "commands", fixtures, page objects, and whatnot)

If we add other potential "dimensions" for tests in the future, such as licensing, we would probably need to add even more folder hierarchies. This doesn't seem scalable.

[MadameSheema]

Closing in favor of: #162698