Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VM multi-kernel testing using little-vm-helper #256

Merged
merged 25 commits into from
Aug 10, 2022
Merged

VM multi-kernel testing using little-vm-helper #256

merged 25 commits into from
Aug 10, 2022

Conversation

kkourt
Copy link
Contributor

@kkourt kkourt commented Jul 21, 2022 •
edited
Loading

This PR adds multi-kernel testing to tetragon.

We use https://github.com/cilium/little-vm-helper/ and https://github.com/cilium/little-vm-helper-images to implement this.
The PR builds functionality gradually. This both because this is how it was developed, but also to ease reviewing.

TODO:

  • remove --force-rebuild-image and always rebuid the image
  • list results for each test (rather than each package)
  • use tests/vmtests/ instead of testing/
  • add ignore file pattern
    • ignore md files (docs) for now
  • add 4.19 kernel: requires btf image

Followups:

  • try out MacOS GH actions runners
  • manually specify tests to be skipped (can be done in the split program)
  • getting e2e-framework running on lvh
  • Add something like:
concurrency:
group: ${{ github.ref }}-kvm-e2e
cancel-in-progress: true

@kkourt kkourt changed the title VM testing VM multi-kernel testing using little-vm-helper Jul 21, 2022
@kkourt kkourt force-pushed the pr/kkourt/lvhtest branch 29 times, most recently from 8552f0e to 51a4bb1 Compare July 25, 2022 13:57
kkourt added 4 commits August 10, 2022 08:14
@kkourt
pkg/observer: remove dfltVerbosity
5bc3e8e 
Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
alignchecker_test: fix bpf-lib default argument
24e0419 
Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
multichecker: improve unordered checker messages
9356f00 
Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
testKprobeObjectFiltered: del export file on fail
5eb4103 
If we are expecting the test to fail, mark the export file so that it is
not kept.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt kkourt force-pushed the pr/kkourt/lvhtest branch from 97e79dd to f48e3bb Compare August 10, 2022 06:28
kkourt added 11 commits August 10, 2022 09:19
@kkourt
observer test: pass context from test
f6f7428 
functions loadExporter and loadObserver were using context.Background()
and/or context.TODO(). However, in the majority of the testing cases
there is already a context build in the test function. Modify the code
to pass this context.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
testing: fix "file already closed" error messages
1107103 
In the previous PR we threaded the ctx from the test cases to the
loadObserver and loadExporter functions. This means that we can now use
the Close() method to cleanup the export files used in our tests.

This wil fix an error condition that caused messages such as:
fix time="2022-07-22T11:23:08Z" level=warning msg="Failed to JSON encode" error="write /tmp/tetragon.gotest.TestKprobeObjectWriteReadHostNs.3694345681.json: file already closed"
to appear.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
testutils/filenames: better error handling
ff5ea5e 
using t.Fatal() leads to confusing logs. Return an error instead.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
pkg/grpc: properly initialize tests
079dab6 
The following tests:
 - TestProcessManager_GetProcessExec
 - TestProcessManager_getPodInfoMaybeExecProbe
 - TestProcessManager_GetProcessID

work only if the state is initialized by another test before them. Fix
this, so that we can run the tests on their own.

As an example of a failed test before this patch:
$ ./go-tests/pkg.grpc  -test.run TestProcessManager_getPodInfoMaybeExecProbe
--- FAIL: TestProcessManager_getPodInfoMaybeExecProbe (0.00s)
    process_manager_test.go:128:
                Error Trace:    process_manager_test.go:128
                Error:          Received unexpected error:
                                ciliumState must be initialized before process cache
                Test:           TestProcessManager_getPodInfoMaybeExecProbe
panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x153241b]

goroutine 66 [running]:
testing.tRunner.func1.2({0x16fdd60, 0x275a5f0})
        /opt/go/src/testing/testing.go:1389 +0x24e
testing.tRunner.func1()
        /opt/go/src/testing/testing.go:1392 +0x39f
panic({0x16fdd60, 0x275a5f0})
        /opt/go/src/runtime/panic.go:838 +0x207
github.com/cilium/tetragon/pkg/process.(*Cache).Purge(...)
        /home/kkourt/src/tetragon/pkg/process/cache.go:130
github.com/cilium/tetragon/pkg/process.FreeCache()
        /home/kkourt/src/tetragon/pkg/process/process.go:82 +0x1b
panic({0x16fdd60, 0x275a5f0})
        /opt/go/src/runtime/panic.go:838 +0x207
github.com/cilium/tetragon/pkg/process.GetPodInfo(...)
        /home/kkourt/src/tetragon/pkg/process/process.go:209
github.com/cilium/tetragon/pkg/grpc.TestProcessManager_getPodInfoMaybeExecProbe(0xc00042eb60?)
        /home/kkourt/src/tetragon/pkg/grpc/process_manager_test.go:130 +0x388
testing.tRunner(0xc00042eb60, 0x1a17f28)
        /opt/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
        /opt/go/src/testing/testing.go:1486 +0x35f

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
introduce pkg/vmtests
5b7b2d4 
Package vmtests includes utilities for running tetragon tests inside
VMs. It allows the program that manages the VM to coordinate with the
program that runs the tests inside the VM (tester). The former
will provide a configuration file (TesterConf) to the latter, and the
latter will produce a set of results (Result[]) to the former via a
json file.

The tester requires access to a tetragon source directory. It executes
the the compiled go tests in the go-tests directory. It then produces a
Result for each of them.

The configuration file can be used to for further configuration in the
future, e.g., to pass specific options for all or individual tests
(e.g., timeouts).

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
introduce cmd/tetragon-tester
6ee2cdc 
cmd/tetragon-tester is meant to be used as a program that runs tetragon
tests inside a VM.

It has some limited support for running as init (pid=1), but this does
not seem to be useful at this point because booting is not the
bottleneck. It uses pkg/vmtests to run the tests.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
introduce a tests/vmtests dir
b77d278 
Create a testing dir keep the top directory clean(er). Include a
Makefile for building tetragon-tester and a script for fetching rootfs
and kernel data.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
introduce cmd/tetragon-vmtests-run
eec3588 
tetragon-vmtests-run is responsible for:
 - creating a VM image to run tetragon-tests
 - starting the VM using qemu to do the tests
 - collect the results at the end

It uses kernels and root iamges build using:
https://github.com/cilium/little-vm-helper.

It can be used as follows:
$ make test-compile                  # compile the test programs
$ make -C tests/vmtests              # compile the necessary utilities
$ ./tests/vmtests/fetch-data.sh 5.10 # fetch kernel and rootfs
$ ./testing/tetragon-vmtests-run \
	--kernel ./tests/vmtests/test-data/kernels/5.10/boot/vmlinuz-5.10* \
	--base ./tests/vmtests/test-data/images/base.qcow2
...
results directory: /home/kkourt/src/tetragon/tests/vmtests/vmtests-results-3899614280
✅      pkg.aggregator.Test_getNameOrIp                                 40ms    (40ms)
✅      pkg.alignchecker.TestStructAlignments                           60ms    (100ms)
✅      pkg.bpf.TestNewBtfError                                         18ms    (119ms)
✅      pkg.bpf.TestBtf54                                               18ms    (137ms)
✅      pkg.btf.TestSpecs                                               49ms    (186ms)
✅      pkg.bugtool.TestSaveAndLoad                                     33ms    (219ms)
✅      pkg.bugtool.TestMultiLog                                        30ms    (249ms)
✅      pkg.config.TestYamlWritev                                       58ms    (307ms)
✅      pkg.config.TestYamlData                                         54ms    (361ms)
✅      pkg.config.TestYamlLseek                                        42ms    (403ms)
✅      pkg.config.TestExamplesSmoke                                    88ms    (491ms)
✅      pkg.encoder.TestCompactEncoder_InvalidEventToString             52ms    (543ms)
✅      pkg.encoder.TestCompactEncoder_ExecEventToString                40ms    (583ms)
✅      pkg.encoder.TestCompactEncoder_ExitEventToString                57ms    (640ms)
✅      pkg.encoder.TestCompactEncoder_KprobeEventToString              42ms    (682ms)
✅      pkg.encoder.TestCompactEncoder_KprobeOpenEventToString          39ms    (721ms)
✅      pkg.encoder.TestCompactEncoder_KprobeWriteEventToString         42ms    (763ms)
✅      pkg.encoder.TestCompactEncoder_KprobeCloseEventToString         41ms    (804ms)
✅      pkg.encoder.TestCompactEncoder_Encode                           40ms    (844ms)
✅      pkg.eventcheckertests.yaml.TestExamplesSmoke                    53ms    (896ms)
✅      pkg.filters.TestArgumentsRegexFilterBasic                       60ms    (956ms)
✅      pkg.filters.TestArgumentsRegexFilterInvalidRegex                43ms    (999ms)
✅      pkg.filters.TestArgumentsRegexFilterInvalidEvent                42ms    (1.041s)
✅      pkg.filters.TestBinaryRegexFilterBasic                          53ms    (1.095s)
✅      pkg.filters.TestBinaryRegexFilterAdvanced                       50ms    (1.145s)
✅      pkg.filters.TestBinaryRegexFilterInvalidRegex                   42ms    (1.187s)
✅      pkg.filters.TestBinaryRegexFilterInvalidEvent                   42ms    (1.229s)
✅      pkg.filters.TestParseFilterList                                 46ms    (1.275s)
✅      pkg.filters.Test_canBeHealthCheck                               57ms    (1.332s)
✅      pkg.filters.Test_maybeExecProbe                                 43ms    (1.375s)
✅      pkg.filters.Test_healthCheckFilter                              58ms    (1.434s)
✅      pkg.filters.TestLabelSelectorFilterInvalidFilter                45ms    (1.479s)
✅      pkg.filters.TestLabelSelectorFilterInvalidEvent                 44ms    (1.522s)
✅      pkg.filters.TestLabelSelectorFilterNoValue                      42ms    (1.564s)
✅      pkg.filters.TestLabelSelectorFilterWithValue                    41ms    (1.605s)
✅      pkg.filters.TestLabelSelectorFilterEmptySelector                43ms    (1.648s)
✅      pkg.filters.TestLabelSelectorFilterSetSelector                  42ms    (1.689s)
✅      pkg.filters.TestNamespace                                       56ms    (1.746s)
✅      pkg.filters.TestPodRegexFilterBasic                             42ms    (1.788s)
✅      pkg.filters.TestPodRegexFilterAdvanced                          42ms    (1.83s)
✅      pkg.filters.TestPodRegexFilterInvalidRegex                      43ms    (1.873s)
✅      pkg.filters.TestPodRegexFilterInvalidEvent                      67ms    (1.94s)
✅      pkg.grpc.TestProcessManager_getPodInfo                          124ms   (2.063s)
✅      pkg.grpc.TestProcessManager_getPodInfoMaybeExecProbe            121ms   (2.184s)
✅      pkg.grpc.TestProcessManager_GetProcessExec                      128ms   (2.312s)
✅      pkg.grpc.Test_getNodeNameForExport                              126ms   (2.438s)
✅      pkg.grpc.TestProcessManager_GetProcessID                        130ms   (2.569s)
✅      pkg.grpc.exec.TestGrpcExecOutOfOrder                            1.141s  (3.709s)
✅      pkg.grpc.exec.TestGrpcExecInOrder                               1.169s  (4.879s)
✅      pkg.idtable.TestOps                                             16ms    (4.895s)
✅      pkg.kernels.TestKernelStringToNumeric                           24ms    (4.918s)
✅      pkg.kernels.TestGetKernelVersion                                35ms    (4.953s)
✅      pkg.matchers.bytesmatcher.TestBytesMatcherFullSmoke             28ms    (4.981s)
✅      pkg.matchers.bytesmatcher.TestBytesMatcherPrefixSmoke           30ms    (5.011s)
✅      pkg.matchers.bytesmatcher.TestBytesMatcherSuffixSmoke           29ms    (5.04s)
✅      pkg.matchers.bytesmatcher.TestBytesMatcherContainsSmoke         38ms    (5.078s)
✅      pkg.matchers.durationmatcher.TestDurationMatcherFullSmoke       35ms    (5.112s)
✅      pkg.matchers.durationmatcher.TestDurationMatcherLessSmoke       38ms    (5.15s)
✅      pkg.matchers.durationmatcher.TestDurationMatcherGreaterSmoke    44ms    (5.195s)
✅      pkg.matchers.durationmatcher.TestDurationMatcherBetweenSmoke    36ms    (5.23s)
✅      pkg.matchers.stringmatcher.TestStringMatcherFullSmoke           28ms    (5.259s)
✅      pkg.matchers.stringmatcher.TestStringMatcherPrefixSmoke         27ms    (5.286s)
✅      pkg.matchers.stringmatcher.TestStringMatcherSuffixSmoke         28ms    (5.314s)
✅      pkg.matchers.stringmatcher.TestStringMatcherContainsSmoke       28ms    (5.342s)
✅      pkg.matchers.stringmatcher.TestStringMatcherRegexSmoke          27ms    (5.369s)
✅      pkg.matchers.timestampmatcher.TestTimestampMatcherTimeYaml      35ms    (5.404s)
✅      pkg.matchers.timestampmatcher.TestTimestampMatcherDaySmoke      34ms    (5.438s)
✅      pkg.matchers.timestampmatcher.TestTimestampMatcherHourSmoke     35ms    (5.473s)
✅      pkg.matchers.timestampmatcher.TestTimestampMatcherMinuteSmoke   35ms    (5.509s)
✅      pkg.matchers.timestampmatcher.TestTimestampMatcherSecondSmoke   37ms    (5.546s)
✅      pkg.matchers.timestampmatcher.TestTimestampMatcherBeforeSmoke   49ms    (5.595s)
✅      pkg.matchers.timestampmatcher.TestTimestampMatcherAfterSmoke    34ms    (5.629s)
✅      pkg.matchers.timestampmatcher.TestTimestampMatcherBetweenSmoke  34ms    (5.663s)
✅      pkg.metrics.eventmetrics.TestHandleProcessedEvent               50ms    (5.713s)
✅      pkg.metrics.eventmetrics.TestHandleOriginalEvent                68ms    (5.781s)
✅      pkg.mountinfo.TestIsMountFS                                     32ms    (5.814s)
✅      pkg.process.TestProcessCache                                    189ms   (6.003s)
✅      pkg.ratelimit.Test_getLimit                                     28ms    (6.03s)
✅      pkg.ratelimit.Test_rateLimitJSON                                23ms    (6.054s)
✅      pkg.reader.exec.TestDecodeCommonFlags                           16ms    (6.069s)
✅      pkg.reader.path.Test_getBinaryAbsolutePath                      21ms    (6.091s)
✅      pkg.reader.proc.TestGetProcStatStrings                          38ms    (6.129s)
✅      pkg.selectors.TestWriteSelectorUint32                           49ms    (6.177s)
✅      pkg.selectors.TestWriteSelectorLength                           77ms    (6.255s)
✅      pkg.selectors.TestWriteSelectorByteArray                        52ms    (6.306s)
✅      pkg.selectors.TestArgSelectorValue                              48ms    (6.354s)
✅      pkg.selectors.TestSelectorOp                                    48ms    (6.402s)
✅      pkg.selectors.TestPidSelectorFlags                              47ms    (6.449s)
✅      pkg.selectors.TestPidSelectorValue                              54ms    (6.503s)
✅      pkg.selectors.TestNamespaceValue                                48ms    (6.551s)
✅      pkg.selectors.TestNamespaceValueStr                             48ms    (6.599s)
✅      pkg.selectors.TestParseMatchArg                                 66ms    (6.664s)
✅      pkg.selectors.TestParseMatchPid                                 47ms    (6.711s)
✅      pkg.selectors.TestParseMatchNamespaces                          47ms    (6.758s)
✅      pkg.selectors.TestParseMatchNamespaceChanges                    49ms    (6.807s)
✅      pkg.selectors.TestParseMatchCapabilities                        56ms    (6.863s)
✅      pkg.selectors.TestParseMatchAction                              76ms    (6.939s)
✅      pkg.selectors.TestInitKernelSelectors                           48ms    (6.986s)
✅      pkg.sensors.exec.Test_msgToExecveUnix                           158ms   (7.144s)
✅      pkg.sensors.exec.TestNamespaces                                 1.423s  (8.568s)
✅      pkg.sensors.exec.TestEventExecve                                7.044s  (15.612s)
✅      pkg.sensors.exec.TestEventExecveLongPath                        4.972s  (20.583s)
✅      pkg.sensors.exec.TestEventExecveLongArgs                        1.498s  (22.081s)
✅      pkg.sensors.exec.TestEventExecveLongPathLongArgs                1.511s  (23.593s)
✅      pkg.sensors.exec.TestFork                                       2.573s  (26.166s)
✅      pkg.sensors.exec.TestForkTesterParser                           152ms   (26.318s)
✅      pkg.sensors.exec.procevents.TestProcsContainerIdOffset          136ms   (26.454s)
✅      pkg.sensors.exec.procevents.TestProcsContainerId                178ms   (26.632s)
✅      pkg.sensors.exec.procevents.TestProcsFindContainerId            145ms   (26.777s)
✅      pkg.sensors.test.TestTestChecker                                3.506s  (30.283s)
✅      pkg.sensors.test.TestSensorLseekLoad                            1.553s  (31.836s)
✅      pkg.sensors.test.TestSensorLseekEnable                          1.504s  (33.34s)
✅      pkg.sensors.tracing.TestCopyFd                                  12.028s (45.368s)
✅      pkg.sensors.tracing.TestKprobeNSChanges                         5.968s  (51.336s)
✅      pkg.sensors.tracing.TestKprobeCapChangesIn                      5.867s  (57.202s)
✅      pkg.sensors.tracing.TestKprobeCapChangesNotIn                   5.821s  (1m3.023s)
✅      pkg.sensors.tracing.TestKprobeSigkill                           4.541s  (1m7.563s)
✅      pkg.sensors.tracing.TestKprobeObjectLoad                        3.772s  (1m11.336s)
✅      pkg.sensors.tracing.TestKprobeLseek                             3.562s  (1m14.898s)
✅      pkg.sensors.tracing.TestKprobeObjectWriteReadHostNs             3.616s  (1m18.513s)
✅      pkg.sensors.tracing.TestKprobeObjectWriteRead                   13.323s (1m31.836s)
✅      pkg.sensors.tracing.TestKprobeObjectWriteCapsNotIn              3.564s  (1m35.4s)
✅      pkg.sensors.tracing.TestKprobeObjectWriteReadNsOnly             3.605s  (1m39.005s)
✅      pkg.sensors.tracing.TestKprobeObjectWriteReadPidOnly            3.523s  (1m42.528s)
✅      pkg.sensors.tracing.TestKprobeObjectRead                        7.317s  (1m49.845s)
✅      pkg.sensors.tracing.TestKprobeObjectReadReturn                  3.835s  (1m53.68s)
✅      pkg.sensors.tracing.TestKprobeObjectOpen                        6.804s  (2m0.484s)
✅      pkg.sensors.tracing.TestKprobeObjectOpenMount                   3.682s  (2m4.166s)
✅      pkg.sensors.tracing.TestKprobeObjectMultiValueOpen              6.785s  (2m10.951s)
✅      pkg.sensors.tracing.TestKprobeObjectMultiValueOpenMount         3.639s  (2m14.59s)
✅      pkg.sensors.tracing.TestKprobeObjectFilterOpen                  3.537s  (2m18.127s)
✅      pkg.sensors.tracing.TestKprobeObjectMultiValueFilterOpen        5.845s  (2m23.973s)
✅      pkg.sensors.tracing.TestKprobeObjectFilterPrefixOpen            6.977s  (2m30.95s)
✅      pkg.sensors.tracing.TestKprobeObjectFilterPrefixOpenMount       3.671s  (2m34.621s)
✅      pkg.sensors.tracing.TestKprobeObjectFilterPrefixExactOpen       6.842s  (2m41.463s)
✅      pkg.sensors.tracing.TestKprobeObjectFilterPrefixExactOpenMount  3.638s  (2m45.1s)
✅      pkg.sensors.tracing.TestKprobeObjectFilterPrefixSubdirOpen      6.701s  (2m51.801s)
✅      pkg.sensors.tracing.TestKprobeObjectFilterPrefixSubdirOpenMount 3.574s  (2m55.375s)
✅      pkg.sensors.tracing.TestKprobeObjectFilterPrefixMissOpen        5.71s   (3m1.085s)
✅      pkg.sensors.tracing.TestKprobeObjectPostfixOpen                 3.716s  (3m4.801s)
✅      pkg.sensors.tracing.TestKprobeObjectWriteVRead                  3.666s  (3m8.466s)
✅      pkg.sensors.tracing.TestKprobeObjectFilenameOpen                3.517s  (3m11.984s)
✅      pkg.sensors.tracing.TestKprobeObjectReturnFilenameOpen          3.891s  (3m15.875s)
✅      pkg.sensors.tracing.TestKprobeObjectFileWrite                   23.85s  (3m39.725s)
✅      pkg.sensors.tracing.TestKprobeObjectFileWriteFiltered           5.596s  (3m45.322s)
✅      pkg.sensors.tracing.TestKprobeObjectFileWriteMount              10.81s  (3m56.131s)
✅      pkg.sensors.tracing.TestKprobeObjectFileWriteMountFiltered      5.58s   (4m1.711s)
✅      pkg.sensors.tracing.TestMultipleMountsFiltered                  7.718s  (4m9.429s)
✅      pkg.sensors.tracing.TestMultiplePathComponents                  5.781s  (4m15.21s)
✅      pkg.sensors.tracing.TestMultipleMountPath                       10.945s (4m26.155s)
✅      pkg.sensors.tracing.TestMultipleMountPathFiltered               5.654s  (4m31.809s)
✅      pkg.sensors.tracing.TestKprobeArgValues                         3.543s  (4m35.352s)
✅      pkg.sensors.tracing.TestKprobeOverride                          427ms   (4m35.779s)
✅      pkg.sensors.tracing.TestKprobeOverrideNonSyscall                466ms   (4m36.244s)
✅      pkg.sensors.tracing.TestKprobe_char_iovec                       10.424s (4m46.669s)
✅      pkg.sensors.tracing.TestKprobe_char_iovec_overflow              3.618s  (4m50.287s)
✅      pkg.sensors.tracing.TestKprobe_char_iovec_returnCopy            3.756s  (4m54.043s)
✅      pkg.sensors.tracing.TestKprobeMatchArgsFileEqual                3.539s  (4m57.582s)
✅      pkg.sensors.tracing.TestKprobeMatchArgsFilePostfix              3.56s   (5m1.142s)
✅      pkg.sensors.tracing.TestKprobeMatchArgsFilePrefix               3.629s  (5m4.771s)
✅      pkg.sensors.tracing.TestKprobeMatchArgsFdEqual                  7.675s  (5m12.446s)
✅      pkg.sensors.tracing.TestKprobeMatchArgsFdPostfix                7.639s  (5m20.085s)
✅      pkg.sensors.tracing.TestKprobeMatchArgsFdPrefix                 7.609s  (5m27.694s)
✅      pkg.sensors.tracing.TestGenericTracepointSimple                 4.278s  (5m31.972s)
✅      pkg.sensors.tracing.TestGenericTracepointPidFilterLseek         3.377s  (5m35.349s)
✅      pkg.sensors.tracing.TestGenericTracepointArgFilterLseek         3.257s  (5m38.606s)
✅      pkg.sensors.tracing.TestGenericTracepointMeta                   3.273s  (5m41.879s)
✅      pkg.sensors.tracing.TestGenericTracepointRawSyscall             3.314s  (5m45.193s)
✅      pkg.stacktracetree.TestSimple                                   55ms    (5m45.248s)
✅      pkg.syscallinfo.TestSycallInfo                                  35ms    (5m45.283s)
✅      pkg.timer.TestTimer                                             2.393s  (5m47.676s)
✅      pkg.tracepoint.TestTracepointFieldParsing                       20ms    (5m47.696s)
✅      pkg.tracepoint.TestTracepointLoadFormat                         16ms    (5m47.712s)
✅      pkg.tracepoint.TestTracepointsAll                               281ms   (5m47.993s)
✅      pkg.vtuplefilter.TestLines                                      22ms    (5m48.015s)
✅      pkg.watcher.TestK8sWatcher_GetPodInfo                           271ms   (5m48.286s)
All 176 tests succeeded! 🎉🚢🍕 (took: 5m59.941s)

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
cmd/tetragon-vmtests-run: add --testfile switch
bfd0848 
This argument allows users to provide a file with the list of tests to
run.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
tests/vmtests: add ignore files
caf8d48 
Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
testutils/sensors: increase timeout to 5'
8dbba46 
This is meant to accommodate slow and loaded platforms, such as GH
actions.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt kkourt force-pushed the pr/kkourt/lvhtest branch from f48e3bb to 2a8b8f8 Compare August 10, 2022 07:20
kkourt added 10 commits August 10, 2022 09:23
@kkourt
gh/vmtests: add a vmtests gh workflow
04d344c 
The workflow currenlty supports only a single kernel. Subsequent patches
will add support for multiple kernels.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
gh/vmtests: multi-job testing
725d435 
We want to test multiple kernels. So this patch spits the testing into
two jobs: build and test. The idea is that build will produce a tarball
that can be used by each test, allowing us to test multiple kernels in
parallel.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
gh/vmtests: add 5.4 kernel
c77def5 
Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
gh/vmtests: panic on RCU stalls
79064b5 
We have been hitting an issue, where the test will continuously print
RCU stalls. Set an option to just panic if that happens

[ 1109.837053] rcu: 	1-...!: (20984 ticks this GP) idle=a4a/1/0x4000000000000002 softirq=46388/46388 fqs=1
[ 1109.837053] 	(t=21001 jiffies g=140697 q=9)
[ 1109.837053] rcu: rcu_sched kthread starved for 20995 jiffies! g140697 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
[ 1109.837053] rcu: RCU grace-period kthread stack dump:
[ 1109.837053] rcu_sched       R  running task    14904    11      2 0x90004000
[ 1109.837053] Call Trace:
[ 1109.837053]  __schedule+0x237/0x610
[ 1109.837053]  ? __mod_timer+0x19d/0x3c0
[ 1109.837053]  schedule+0x34/0xa0
[ 1109.837053]  schedule_timeout+0x84/0x150
[ 1109.837053]  ? __next_timer_interrupt+0xc0/0xc0
[ 1109.837053]  rcu_gp_kthread+0x4f4/0xd50
[ 1109.837053]  ? kfree_call_rcu+0x10/0x10
[ 1109.837053]  kthread+0x112/0x130
[ 1109.837053]  ? __kthread_bind_mask+0x60/0x60
[ 1109.837053]  ret_from_fork+0x35/0x40
[ 1109.837053] NMI backtrace for cpu 1
[ 1109.837053] CPU: 1 PID: 533 Comm: pkg.sensors.tra Not tainted 5.4.206 #1
[ 1109.837053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 1109.837053] Call Trace:
[ 1109.837053]  <IRQ>
[ 1109.837053]  dump_stack+0x50/0x63
[ 1109.837053]  nmi_cpu_backtrace.cold+0x14/0x53
[ 1109.837053]  ? lapic_can_unplug_cpu+0x70/0x70
[ 1109.837053]  nmi_trigger_cpumask_backtrace+0x7c/0x90
[ 1109.837053]  rcu_dump_cpu_stacks+0x7c/0xaa
[ 1109.837053]  rcu_sched_clock_irq.cold+0x1b3/0x39e
[ 1109.837053]  update_process_times+0x56/0x90
[ 1109.837053]  tick_sched_handle+0x2f/0x40
[ 1109.837053]  tick_sched_timer+0x4c/0xb0
[ 1109.837053]  ? can_stop_idle_tick+0x90/0x90
[ 1109.837053]  __hrtimer_run_queues+0x123/0x2a0
[ 1109.837053]  hrtimer_interrupt+0x10b/0x2c0
[ 1109.837053]  smp_apic_timer_interrupt+0x61/0x130
[ 1109.837053]  apic_timer_interrupt+0xf/0x20
[ 1109.837053]  </IRQ>
[ 1109.837053] RIP: 0010:syscall_trace_enter+0x1f1/0x290
[ 1109.837053] Code: 01 00 48 c7 80 88 07 00 00 00 00 00 00 48 8b 10 83 e2 04 74 af f6 80 c9 06 00 00 01 74 a6 48 c7 c0 ff ff ff ff e9 25 ff ff ff <e9> 40 00 00 00 e9 ec fe ff ff 4c 8b 4b 58 48 8b 73 28 49 89 d0 4c
[ 1109.837053] RSP: 0018:ffffbb0f40247ec8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
[ 1109.837053] RAX: 0000000010000000 RBX: ffffbb0f40247f58 RCX: 0000000000000000
[ 1109.837053] RDX: 0000000000000000 RSI: ffffbb0f40247f58 RDI: 00000000000000e4
[ 1109.837053] RBP: 00000000c000003e R08: 0000000000000000 R09: 0000000000000000
[ 1109.837053] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1109.837053] R13: 00000000000000e4 R14: 0000000010000000 R15: 0000000000000000
[ 1109.837053]  ? _copy_to_user+0x28/0x30
[ 1109.837053]  ? put_timespec64+0x35/0x60
[ 1109.837053]  do_syscall_64+0xc8/0x110
[ 1109.837053]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 1109.837053] RIP: 0033:0x7ffd2f7bb7ff

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
tools/split-tetragon-gotests: split tests utility
068a166 
We want to split tests into groups so that we can execute them in
paralle using GH actions. This tool retrieves the tests, and splits them
into tests/vmtests/test-group-%d files.

Have a CI flag to implement blacklisting.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
gh/vmtests: split tests into multiple actions
b52bd11 
Currently, running the whole suite in a VM takes about 1h.  Split the
tests into three groups, and run them concurrently. Even if they do not
run concurrently, there is stil a benefit in doing this because dealing
with flakes means that we only have to restart a single group.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
tests/vmtests: README.md
2e23a26 
Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
vmtests: support passing a BTF file
31c56d9 
Older kernels (e.g., 4.19) do not support /sys/kernel/btf/vmlinux. For
those, we need to pass the btf file separately to tetragon. Add support
for this in vmtests.

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
gh/vmtests: add 4.19 kernel
606aa5c 
Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt
gh/vmtests: truncate artifacts if run was successful
9dd0dc8 
There does not seem to be an easy way to delete artifacts of the
existing run (see details below). Instead, this patch truncates the
build artifact so at least we do not waste space.

I tried the following:
- name: Delete artifacts
uses: actions/github-script@v3
with:
  github-token: ${{ github.token }}
  script: |
    const {GITHUB_RUN_ID, GITHUB_REPOSITORY} = process.env
    const artifactsURL = `/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/artifacts`
    console.log(`Deleting artifacs from ${artifactsURL}`)
    const response = await github.request(`GET ${artifactsURL}`)
    console.log(response.data)
    for (artifact of response.data.artifacts) {
	  console.log(`Deleting artifact with name=${artifact.name}, id=${artifact.id}`)
	  const deleteResponse = await github.request(`DELETE /repos/${repo}/actions/artifacts/${artifact.id}`)
    }

But it did not work, because the GITHUB_RUN_ID does not seem to be available over the REST API until
after the action is finished.

The logs from my action printed:
Deleting artifacs from /repos/cilium/tetragon/actions/runs/2817658181/artifacts
...
{ total_count: 0, artifacts: [] }

But if I curl the above URL after the action, I get:
{
  "total_count": 1,
  "artifacts": [
    {
      "id": 322874447,
      "node_id": "MDg6QXJ0aWZhY3QzMjI4NzQ0NDc=",
      "name": "tetragon-build",
      "size_in_bytes": 6,
      "url": "https://api.github.com/repos/cilium/tetragon/actions/artifacts/322874447",
      "archive_download_url": "https://api.github.com/repos/cilium/tetragon/actions/artifacts/322874447/zip",
      "expired": false,
      "created_at": "2022-08-08T11:50:54Z",
      "updated_at": "2022-08-08T11:50:54Z",
      "expires_at": "2022-08-13T11:49:30Z",
      "workflow_run": {
        "id": 2817658181,
        "repository_id": 473137633,
        "head_repository_id": 473137633,
        "head_branch": "pr/kkourt/lvhtest",
        "head_sha": "ea22990af2283c38aac92f7e6febacabf3786280"
      }
    }
  ]
}

For (even) more context, see: actions/upload-artifact#5

Signed-off-by: Kornilios Kourtis <[email protected]>
@kkourt kkourt force-pushed the pr/kkourt/lvhtest branch from 2a8b8f8 to 9dd0dc8 Compare August 10, 2022 07:23
@kkourt kkourt merged commit 8d35b43 into main Aug 10, 2022
@kkourt kkourt deleted the pr/kkourt/lvhtest branch August 10, 2022 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@olsajiri olsajiri olsajiri left review comments

@willfindlay willfindlay willfindlay approved these changes

Assignees

@kkourt kkourt

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kkourt @willfindlay @olsajiri