[Defend Workflows] Add tags for mocked and real Endpoint Cypress tests for Serverless by gergoabraham · Pull Request #165094 · elastic/kibana

gergoabraham · 2023-08-29T13:35:29Z

Summary

introduces tags for [Defend Workflows] cypress tests (similarly to [Security Solution][Serverless] Reusing Cypress tests for Serverless infrastructure #162698)
adds scripts to Security Solution:
- cypress:dw:serverless:open and :run
- cypress:dw:endpoint:serverless:open and :run
adds CI jobs to run these scripts
so far most of the expected tests got both @serverless and @brokenInServerless tests, because of other issues to be solved,
one test is able to run against serverless: x-pack/plugins/security_solution/public/management/cypress/e2e/mocked_data/policy_details.cy.ts

…a workaround

elasticmachine · 2023-09-14T13:57:20Z

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

...ck/plugins/security_solution/public/management/cypress/cypress_endpoint_serverless.config.ts

paul-tavares

awesome job here. Thansk for taking this on and for all the changes to enable this.

Left a round of feedback - let me know if you have any questions.

Once thing I wanted to follow up on - perhaps best in our team channel or during our weekly eng. hour:

What are all of the approved tags to use? are they documented somewhere? Also - might be good to add documentation it to our cypress README file.
What happens to a test that has no tags? I had (incorrectly?) assumed that all tests would continue to be picked up and ran in ESS, only those with @serverless would be executed in serverless. but given this PR, it leads me to belive that tags are always required? If so, is there a way to catch and error a test that has no tags?

paul-tavares · 2023-09-14T19:02:01Z

x-pack/plugins/security_solution/public/management/cypress/e2e/endpoint/artifacts.cy.ts

 const parseRevNumber = (revString: string) => Number(revString.match(/\d+/)?.[0]);

-describe('Artifact pages', () => {
+describe('Artifact pages', { tags: ['@ess', '@serverless', '@brokenInServerless'] }, () => {


It would be great if we could type this new property (tags) so that we get content assist. Can you add that to our cypress.d.ts file here:

kibana/x-pack/plugins/security_solution/public/management/cypress/cypress.d.ts

Lines 38 to 42 in 054cdba

interface SuiteConfigOverrides {

env?: {

ftrConfig: SecuritySolutionDescribeBlockFtrConfig;

};

}

Actually - looks like tags is already defined via @cypress types so need to add to the our cypress.d.ts file. I do wonder if there is a way to make the type more specific in that it can suggest our "known" tags 🤔

I did some experiments with types to be able to have suggestions, but didn't work so far, exactly because tags are already defined in cypress.d.ts, and therefore it cannot be overriden by a more exact type.

one solution would be to use our own security_tags or something, but I'm not sure if grep can pick it up - I'll look into this, but this is definitely an open question.

no big deal. it would just help with DX. Don't hold this PR because of this.

x-pack/plugins/security_solution/public/management/cypress/e2e/mocked_data/artifacts.cy.ts

x-pack/plugins/security_solution/public/management/cypress/.eslintrc.json

x-pack/plugins/security_solution/public/management/cypress/cypress.config.ts

...ck/plugins/security_solution/public/management/cypress/cypress_endpoint_serverless.config.ts

jbudz · 2023-09-14T20:04:47Z

.buildkite/pipelines/pull_request/defend_workflows.yml

        - exit_status: '*'
          limit: 1
+
+  - command: .buildkite/scripts/steps/functional/defend_workflows_serverless.sh


Do we only want this to run conditionally on pull requests?

on-merge at https://github.com/elastic/kibana/blob/main/.buildkite/pipelines/on_merge.yml#L82
all pull requests at https://github.com/elastic/kibana/blob/main/.buildkite/pipelines/pull_request/base.yml#L71

thanks for the question! I've added it to our tracking issue for serverless tests, to make a team decision - until then, for this PR, I'd keep it this way to merge this as soon as possible.

x-pack/test_serverless/shared/config.base.ts

...ck/plugins/security_solution/public/management/cypress/cypress_endpoint_serverless.config.ts

paul-tavares

👍

paul-tavares · 2023-09-19T19:39:36Z

x-pack/plugins/security_solution/public/management/cypress/cypress_base.config.ts

-import { dataLoaders } from './cypress/support/data_loaders';
-
-export default defineCypressConfig({
+export const CY_BASE_CONFIG = {


Suggestion: turn this into a function that returns the config object so that there is no concern around someone accidentally mutating this object. Or, use Object.freeze() on it (although, tht is also not a deep freeze)

thanks for the suggestion!
75cde75

kibana-ci · 2023-09-19T21:11:05Z

💛 Build succeeded, but was flaky

Failed CI Steps

Investigations - Security Solution Cypress Tests #6

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`securitySolution`	66	67	+1

ESLint disabled line counts

id	before	after	diff
`securitySolution`	453	458	+5

Total ESLint disabled count

id	before	after	diff
`securitySolution`	519	525	+6

History

💛 Build #160002 was flaky a2dca01
💔 Build #159910 failed 08fab69
💛 Build #159116 was flaky 4ac5a03
💔 Build #159047 failed d556353016da393c0ad371aacb136110eb6409e5
💔 Build #158724 failed 0b52f93552ff1750e591e9a67fd0a5d6519dd33b
💔 Build #158349 failed bd4a108093b8bb41005cbdbfd700c12cfd01ebd8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @gergoabraham

… to elastic#165094)

gergoabraham added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:Defend Workflows “EDR Workflows” sub-team of Security Solution labels Aug 29, 2023

gergoabraham self-assigned this Aug 29, 2023

gergoabraham force-pushed the test/olm-7133-reuse-tests-for-serverless branch 2 times, most recently from 2279454 to f6e7a5c Compare September 4, 2023 08:51

gergoabraham force-pushed the test/olm-7133-reuse-tests-for-serverless branch 6 times, most recently from 0b52f93 to b4d1246 Compare September 14, 2023 10:20

gergoabraham added 18 commits September 14, 2023 15:26

add cypress:dw:serverless:* scripts

b6642b6

add Elastic-Api-Version to all cy requests

8cc56e7

add http.host to shared setting, so ES is accessible via local IP

8645b2c

add config and scripts for real endpoint serverless

db6b729

fix test scripts (removing ts and open from shared scripts)

b51cdbf

add @ess tag for every test

774c548

fix eslint errors

c85ec25

add @ess grep to ess test configs

26f13f1

fine tune grep config

96681c4

move cypress configs into cypress folder

171d093

run tests on CI

d012eaf

add @serverless tag to selected cypress tests

5bd9599

fix eslint errors

bdabdfa

improve earlier eslint fix: remove unnecessary click

df6c6f6

add @brokenInServerless to all "real endpoint" tests, until we got …

5b3f9e5

…a workaround

add reporter config to cypress configs

f01466d

fix lint errors

0715a0c

remove CI job for real endpoints against serverless

783402a

gergoabraham commented Sep 14, 2023

View reviewed changes

...ck/plugins/security_solution/public/management/cypress/cypress_endpoint_serverless.config.ts Outdated Show resolved Hide resolved

paul-tavares reviewed Sep 14, 2023

View reviewed changes

jbudz reviewed Sep 14, 2023

View reviewed changes

x-pack/test_serverless/shared/config.base.ts Outdated Show resolved Hide resolved

paul-tavares reviewed Sep 18, 2023

View reviewed changes

...ck/plugins/security_solution/public/management/cypress/cypress_endpoint_serverless.config.ts Show resolved Hide resolved

gergoabraham added 8 commits September 19, 2023 13:01

Merge branch 'main' into test/olm-7133-reuse-tests-for-serverless

08fab69

remove http.host server arg from shared config

4a0cea8

prepare runEndpointLoaderScript() for serverless

11132d6

add dataLoaders() back to serverless config

63e5342

enable Policy Details cy test for serverless

a2dca01

re-enable configurations that have problems fixed

6ae9f21

update readme

b08267a

add reusable base config for cypress

ad1c3fe

gergoabraham removed the request for review from a team September 19, 2023 19:32

Merge branch 'main' into test/olm-7133-reuse-tests-for-serverless

6f90e66

gergoabraham requested a review from paul-tavares September 19, 2023 19:36

add IS_SERVERLESS to cy config

7ec96a3

paul-tavares approved these changes Sep 19, 2023

View reviewed changes

convert base config to a function instead of const

75cde75

gergoabraham requested a review from jbudz September 19, 2023 20:33

jbudz approved these changes Sep 19, 2023

View reviewed changes

joeypoon approved these changes Sep 20, 2023

View reviewed changes

gergoabraham merged commit aa36fe6 into elastic:main Sep 20, 2023

gergoabraham deleted the test/olm-7133-reuse-tests-for-serverless branch September 20, 2023 08:36

kibanamachine added the v8.11.0 label Sep 20, 2023

delanni added a commit to delanni/kibana that referenced this pull request Sep 29, 2023

chore(ss-management): Move cypress configs to cypress folder (similar…

825758e

… to elastic#165094)

	interface SuiteConfigOverrides {
	env?: {
	ftrConfig: SecuritySolutionDescribeBlockFtrConfig;
	};
	}

Conversation

gergoabraham commented Aug 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

elasticmachine commented Sep 14, 2023

Uh oh!

Uh oh!

paul-tavares left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

paul-tavares left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Sep 19, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

gergoabraham commented Aug 29, 2023 •

edited

Loading