Support running tests in full VM's #1740

adam-stokes · 2021-10-28T16:39:59Z

Orchestrates deploying pure vm's for both amd64/arm64, utilizing ansible. Windows support coming next in followup
Provides uniqueness with policies and integrations so that all test suites can run against a single deployed stack
Update the step definitions to be a little more platform agnostic, still more work to do there in followup pr's

Signed-off-by: Adam Stokes [email protected]

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests (make unit-test), and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

How to test this PR locally

Please see .ci/README.md

Related issues

Closes https://github.com/elastic/observability-robots/issues/720

This gives us full vm's to run our e2e tests in. Supports both Linux amd64/arm64, and will support Windows. Signed-off-by: Adam Stokes <[email protected]>

mergify · 2021-10-28T16:40:26Z

This pull request does not have a backport label. Could you fix it @adam-stokes? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit
NOTE: backport-skip has been added to this pull request.

kuisathaverat · 2021-10-28T16:46:03Z

.ci/Jenkinsfile

-def checkTestSuite(Map parallelTasks = [:], Map item = [:]) {
-    def suite = item.suite
-    def platforms = item.platforms
+def sshexec(connection, cmd){


candidate to be a step in the library :D

kuisathaverat · 2021-10-28T16:46:12Z

.ci/Jenkinsfile

+  sh "ssh -tt -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${env.WORKSPACE}/${env.BASE_DIR}/e2essh ${connection} -- '${cmd}'"
+}
+
+def scpr(connection, remote_src, local_dst){


candidate to be a step in the library :D

.ci/Jenkinsfile

.ci/ansible/playbook.yml

elasticmachine · 2021-10-28T17:42:27Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-30T16:05:37.301+0000
Duration: 43 min 18 sec
Commit: 6cd07fa

Test stats 🧪

Test	Results
Failed	0
Passed	86
Skipped	0
Total	86

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

adam-stokes · 2021-11-01T12:28:30Z

/test

adam-stokes · 2021-11-02T13:21:08Z

/test

.ci/.e2e-tests.yaml

.ci/Jenkinsfile

mdelapenya · 2021-11-02T17:32:57Z

.ci/Jenkinsfile

+/*
+ * Runs python in venv
+ */
+def pyrun(args){


Another candidate for the library?

.ci/Jenkinsfile

.ci/ansible/roles/common/tasks/setup_test_script.yml

Signed-off-by: Adam Stokes <[email protected]>

Co-authored-by: Manuel de la Peña <[email protected]>

Signed-off-by: Adam Stokes <[email protected]>

The expression in the e2e-tests already handles both skip_tests and force_skip_git_checks. We make use of the skip_tests env here to skip those test suites if no files are changed Signed-off-by: Adam Stokes <[email protected]>

adam-stokes · 2021-11-30T15:06:02Z

/test

Signed-off-by: Adam Stokes <[email protected]>

…vm-tester-feat

* Support running e2e in VM's Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> (cherry picked from commit 8a94011) # Conflicts: # .ci/Jenkinsfile

* Support running e2e in VM's Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> (cherry picked from commit 8a94011) # Conflicts: # .ci/Jenkinsfile # .gitignore

* Support running e2e in VM's Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> (cherry picked from commit 8a94011) # Conflicts: # .ci/.e2e-tests.yaml # .ci/Jenkinsfile # .gitignore # e2e/_suites/fleet/features/fleet_mode_agent.feature # e2e/_suites/fleet/features/preconfigured_policies.feature # e2e/_suites/fleet/fleet.go # go.sum

* Support running tests in full VM's (#1740) * Support running e2e in VM's Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> (cherry picked from commit 8a94011) # Conflicts: # .ci/Jenkinsfile * Fix conflict Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Adam Stokes <[email protected]>

* Support running tests in full VM's (#1740) * Support running e2e in VM's Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> (cherry picked from commit 8a94011) # Conflicts: # .ci/Jenkinsfile # .gitignore * Fix conflicts Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Adam Stokes <[email protected]>

* Support running tests in full VM's (#1740) * Support running e2e in VM's Signed-off-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> (cherry picked from commit 8a94011) # Conflicts: # .ci/.e2e-tests.yaml # .ci/Jenkinsfile # .gitignore # e2e/_suites/fleet/features/fleet_mode_agent.feature # e2e/_suites/fleet/features/preconfigured_policies.feature # e2e/_suites/fleet/fleet.go # go.sum * Fix conflicts Signed-off-by: Adam Stokes <[email protected]> * fix: remove not implemented function call The backport added a reference to code implemented in: Test preconfigured agent policies and output (#1680) But this PR was not backported to 7.15 because there is no support to preconfigured policies. Co-authored-by: Adam Stokes <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]>

otherwise it fails in the ephemeral workers relative path was added in elastic#1740

Support running e2e in VM's

d176f2d

This gives us full vm's to run our e2e tests in. Supports both Linux amd64/arm64, and will support Windows. Signed-off-by: Adam Stokes <[email protected]>

adam-stokes requested a review from mdelapenya October 28, 2021 16:39

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 28, 2021

kuisathaverat reviewed Oct 28, 2021

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

kuisathaverat reviewed Oct 28, 2021

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

kuisathaverat reviewed Oct 28, 2021

View reviewed changes

.ci/ansible/playbook.yml Outdated Show resolved Hide resolved

This was referenced Nov 2, 2021

Step definitions need to be reworded to support multiple os/installtype matrix #1511

Closed

Enrolling on Windows is flaky #1557

Closed

Merge branch 'master' into vm-tester-feat

beae168

mdelapenya reviewed Nov 2, 2021

View reviewed changes

.ci/.e2e-tests.yaml Outdated Show resolved Hide resolved

Merge branch 'master' into vm-tester-feat

d141a7f

mdelapenya reviewed Nov 2, 2021

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

mdelapenya reviewed Nov 2, 2021

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

mdelapenya reviewed Nov 2, 2021

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

mdelapenya reviewed Nov 2, 2021

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

mdelapenya reviewed Nov 2, 2021

View reviewed changes

.ci/ansible/roles/common/tasks/setup_test_script.yml Outdated Show resolved Hide resolved

adam-stokes and others added 7 commits November 3, 2021 07:29

Merge branch 'master' into vm-tester-feat

eba6d47

bump log level to trace

320fe92

Signed-off-by: Adam Stokes <[email protected]>

Update .ci/Jenkinsfile

ef41720

Co-authored-by: Manuel de la Peña <[email protected]>

Update .ci/Jenkinsfile

8d2e855

Co-authored-by: Manuel de la Peña <[email protected]>

Update .ci/Jenkinsfile

2775cf7

Co-authored-by: Manuel de la Peña <[email protected]>

use provision-stack as tag instead

91b96b8

Signed-off-by: Adam Stokes <[email protected]>

dont test ubi8

ab742ac

Signed-off-by: Adam Stokes <[email protected]>

adam-stokes added 2 commits November 30, 2021 07:50

move elasticcredentials into es endpoint struct

27cbd73

Signed-off-by: Adam Stokes <[email protected]>

make notice

9f7caeb

Signed-off-by: Adam Stokes <[email protected]>

This was referenced Nov 30, 2021

Support creating policy with optional integrations #1853

Closed

[automation] update elastic stack version for testing 8.0.0-5dc82a0c #1848

Merged

adam-stokes added 3 commits November 30, 2021 08:28

move force git check into e2e stage prior to checktestsuite

58cc2cf

Signed-off-by: Adam Stokes <[email protected]>

utilize skip_tests in matching changed git files

e91746c

The expression in the e2e-tests already handles both skip_tests and force_skip_git_checks. We make use of the skip_tests env here to skip those test suites if no files are changed Signed-off-by: Adam Stokes <[email protected]>

Merge branch 'master' into vm-tester-feat

cbfdf91

adam-stokes added 2 commits November 30, 2021 11:04

add force skip git checks on unit test stage

7994852

Signed-off-by: Adam Stokes <[email protected]>

Merge branch 'vm-tester-feat' of github.com:elastic/e2e-testing into …

6cd07fa

…vm-tester-feat

adam-stokes merged commit 8a94011 into master Nov 30, 2021

mergify bot deleted the vm-tester-feat branch November 30, 2021 16:56

adam-stokes added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Nov 30, 2021

mergify bot mentioned this pull request Nov 30, 2021

[8.0](backport #1740) Support running tests in full VM's #1854

Merged

mergify bot mentioned this pull request Nov 30, 2021

[7.16](backport #1740) Support running tests in full VM's #1855

Merged

mergify bot mentioned this pull request Nov 30, 2021

[7.15](backport #1740) Support running tests in full VM's #1856

Merged

adam-stokes mentioned this pull request Nov 30, 2021

chore: run k8s and helm test suites on ARM #1471

Closed

8 tasks

v1v added a commit to v1v/e2e-testing that referenced this pull request Jun 16, 2022

e2e: use relative path

8c95942

otherwise it fails in the ephemeral workers relative path was added in elastic#1740

Support running tests in full VM's #1740

Support running tests in full VM's #1740

Uh oh!

Conversation

adam-stokes commented Oct 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

How to test this PR locally

Related issues

Uh oh!

mergify bot commented Oct 28, 2021

Uh oh!

kuisathaverat Oct 28, 2021

Choose a reason for hiding this comment

Uh oh!

kuisathaverat Oct 28, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Oct 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

adam-stokes commented Nov 1, 2021

Uh oh!

adam-stokes commented Nov 2, 2021

Uh oh!

Uh oh!

Uh oh!

mdelapenya Nov 2, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adam-stokes commented Nov 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

adam-stokes commented Oct 28, 2021 •

edited

Loading

elasticmachine commented Oct 28, 2021 •

edited

Loading