tests: introduce end-to-end test framework #226

willfindlay · 2022-07-11T16:26:33Z

This PR introduces a new end-to-end test framework in tests/e2e which we can use to
write cross-provider end-to-end tests for Tetragon. The tests themselves live in
tests/e2e/tests. I have added a skeleton file with lots of comments that folks can use
as a starting point for writing new tests.

Updated the Makefile to support e2e-tests by adding a separate e2e-test target and making
sure that regular make test will not call into the e2e-tests by pointing it directly at
./pkg/.... Also added an EXTRA_TESTFLAGS variable to the Makefile that can be used to
pass flags directly to the test binary from either a make test or make e2e-test.

The plan is to use this new framework to start writing e2e tests for Tetragon features.

kkourt

Thanks William.

I have some minor questions/comments, but otherwise LGTM.

tests/e2e/checker/doc.go

tests/e2e/checker/multiplexer.go

tests/e2e/checker/rpcchecker.go

kkourt · 2022-07-19T12:14:21Z

tests/e2e/checker/rpcchecker.go

+func (rc *RPCChecker) CheckWithFilters(connTimeout time.Duration, allowList, denyList []*tetragon.Filter) features.Func {
+	return func(ctx context.Context, t *testing.T, _ *envconf.Config) context.Context {
+		rc.lock.Lock()
+		defer rc.lock.Unlock()


I'm wondering why is this lock needed? AFAICT, it is only used in this function. Can this function be called concurrently from different goroutines? If so, this seems weird to me because I assumed we had one checker that talks to the multiplexer.

If memory serves I was concerned about test authors using the same checker multiple times in a single testenv.TestInParallel. (Which runs its feature funcs concurrently.)

So maybe a comment here? to prevent accidental changes

I was concerned about test authors using the same checker multiple times in a single testenv.TestInParallel.

I see. Out of curiosity, how would one go about doing this?

@kkourt testenv.TestInParallel(workloadFeature, checkerFeature, checkerFeature, checkerFeature /* etc... */)

kkourt · 2022-07-19T12:17:04Z

tests/e2e/checker/rpcchecker.go

+			addrs = append(addrs, fmt.Sprintf("localhost:%d", port))
+		}
+
+		if rc.getEvents == nil {


Could this be not nil? Can we enter the CheckWithFilters with .connect() already called? (AFAICT, this is the only place that getEvents is set)

In the case where we want to use the same checker more than once we keep the original connection open. Useful for example when we want to run the same workload multiple times in a stress test.

In the case where we want to use the same checker more than once we keep the original connection open. Useful for example when we want to run the same workload multiple times in a stress test.

Similar to my previous comment. How would one go about doing this?

@kkourt just stick the workload + checker features in a for loop. For example you could imagine a TestKprobeNTimes test or something similar that runs the same test 1, 10, 20, etc. times in a row to check for flakes.

tests/e2e/helpers/dumpinfo.go

willfindlay · 2022-07-19T14:03:44Z

Looks like 4dac275 causes Tetragon to fail the skeleton test :(

tests/e2e/checker/rpcchecker.go

tests/e2e/helpers/dumpinfo.go

tests/e2e/tests/skeleton/skeleton_test.go

kkourt · 2022-07-26T08:17:42Z

@willfindlay since you seem to be still working on it, I'll move it to draft. Please mark it as ready when you feel it's ready to be merged.

willfindlay · 2022-07-26T13:22:23Z

@kkourt I think it's basically ready tbh (needs a squash/rebase obviously), but we should wait until the bug causing the new e2e job to fail has been fixed. @jrfastab says he is working on it atm. Let's keep it as draft until then 👍

willfindlay · 2022-07-27T01:11:06Z

@kkourt I think it's in a good state to ship it now. I'll leave it for you just in case you want to do another round of comments in the morning.

This PR introduces a new end-to-end test framework in `tests/e2e` which we can use to write cross-provider end-to-end tests for Tetragon. The tests themselves live in `tests/e2e/tests`. I have added a skeleton file with lots of comments that folks can use as a starting point for writing new tests. Updated the Makefile to support e2e-tests by adding a separate e2e-test target and making sure that regular `make test` will not call into the e2e-tests by pointing it directly at `./pkg/...`. Also added an EXTRA_TESTFLAGS variable to the Makefile that can be used to pass flags directly to the test binary from either a `make test` or `make e2e-test`. Signed-off-by: William Findlay <[email protected]>

Add a basic job to run end-to-end tests in CI using a KinD cluster. Signed-off-by: William Findlay <[email protected]>

willfindlay requested a review from a team as a code owner July 11, 2022 16:26

willfindlay requested review from kevsecurity, tixxdz, kkourt and jrfastab and removed request for kevsecurity July 11, 2022 16:26

willfindlay force-pushed the pr/willfindlay/e2e-framework branch 2 times, most recently from 158ff4a to dbcfacb Compare July 11, 2022 17:53

willfindlay mentioned this pull request Jul 12, 2022

tests/e2e: write some end-to-end tests with the new e2e-framework #230

Open

5 tasks

willfindlay force-pushed the pr/willfindlay/e2e-framework branch 3 times, most recently from a75c224 to 25fb560 Compare July 13, 2022 17:52

kkourt added the needs-rebase This PR needs to be rebased because it has merge conflicts. label Jul 14, 2022

willfindlay force-pushed the pr/willfindlay/e2e-framework branch from 25fb560 to d578a52 Compare July 14, 2022 15:57

willfindlay removed the needs-rebase This PR needs to be rebased because it has merge conflicts. label Jul 14, 2022

kkourt added the area/testing Related to testing label Jul 15, 2022

kkourt approved these changes Jul 19, 2022

View reviewed changes

willfindlay force-pushed the pr/willfindlay/e2e-framework branch from d578a52 to 2bd41df Compare July 19, 2022 14:02

willfindlay force-pushed the pr/willfindlay/e2e-framework branch 9 times, most recently from 8725bb1 to ec08611 Compare July 19, 2022 20:46

willfindlay added area/ci Related to CI area/e2e Related to the e2e-framework labels Jul 19, 2022

tixxdz reviewed Jul 22, 2022

View reviewed changes

tests/e2e/checker/rpcchecker.go Outdated Show resolved Hide resolved

tixxdz reviewed Jul 22, 2022

View reviewed changes

tests/e2e/helpers/dumpinfo.go Show resolved Hide resolved

tixxdz reviewed Jul 22, 2022

View reviewed changes

tests/e2e/tests/skeleton/skeleton_test.go Show resolved Hide resolved

tixxdz approved these changes Jul 25, 2022

View reviewed changes

willfindlay force-pushed the pr/willfindlay/e2e-framework branch 3 times, most recently from f5db4e3 to 6d63a18 Compare July 25, 2022 20:45

kkourt marked this pull request as draft July 26, 2022 08:17

willfindlay force-pushed the pr/willfindlay/e2e-framework branch from 6d63a18 to 02172ca Compare July 26, 2022 15:14

willfindlay changed the base branch from main to pr/willfindlay/fix-broken-cache July 26, 2022 20:12

willfindlay force-pushed the pr/willfindlay/e2e-framework branch from 02172ca to 94c965f Compare July 26, 2022 20:12

Base automatically changed from pr/willfindlay/fix-broken-cache to main July 27, 2022 01:08

willfindlay marked this pull request as ready for review July 27, 2022 01:09

willfindlay added 2 commits July 27, 2022 16:53

ci: add an e2e test job

158dd23

Add a basic job to run end-to-end tests in CI using a KinD cluster. Signed-off-by: William Findlay <[email protected]>

willfindlay force-pushed the pr/willfindlay/e2e-framework branch from 94c965f to 158dd23 Compare July 27, 2022 20:53

kkourt merged commit a9e4a9b into main Jul 28, 2022

kkourt deleted the pr/willfindlay/e2e-framework branch July 28, 2022 15:42

willfindlay mentioned this pull request Jul 28, 2022

VM multi-kernel testing using little-vm-helper #256

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: introduce end-to-end test framework #226

tests: introduce end-to-end test framework #226

willfindlay commented Jul 11, 2022

kkourt left a comment

kkourt Jul 19, 2022

willfindlay Jul 19, 2022 •

edited

Loading

tixxdz Jul 22, 2022

kkourt Jul 26, 2022

willfindlay Jul 26, 2022

kkourt Jul 19, 2022

willfindlay Jul 19, 2022

kkourt Jul 26, 2022

willfindlay Jul 26, 2022

willfindlay commented Jul 19, 2022

kkourt commented Jul 26, 2022

willfindlay commented Jul 26, 2022 •

edited

Loading

willfindlay commented Jul 27, 2022

tests: introduce end-to-end test framework #226

tests: introduce end-to-end test framework #226

Conversation

willfindlay commented Jul 11, 2022

kkourt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willfindlay Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willfindlay commented Jul 19, 2022

kkourt commented Jul 26, 2022

willfindlay commented Jul 26, 2022 • edited Loading

willfindlay commented Jul 27, 2022

willfindlay Jul 19, 2022 •

edited

Loading

willfindlay commented Jul 26, 2022 •

edited

Loading