Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

End-to-end tests fail cleaning up node pools #258

Open
rhmdnd opened this issue Mar 13, 2023 · 1 comment
Open

End-to-end tests fail cleaning up node pools #258

rhmdnd opened this issue Mar 13, 2023 · 1 comment

Comments

@rhmdnd
Copy link

rhmdnd commented Mar 13, 2023

Recently, we've been seeing e2e failures that look like:

     helpers.go:1140: 2023-03-10T16:16:17Z: All machines updated
    helpers.go:180: 2023-03-10T16:16:17Z: Removing pool e2e
    helpers.go:191: 2023-03-10T16:16:17Z: Removing pool e2e-invalid
    helpers.go:194: E2E-FAILURE: 2023-03-10T16:16:17Z: Could not remove pool e2e-invalid: Delete "https://api.ci-op-80s6kmwz-f837a.origin-ci-int-aws.dev.rhcloud.com:6443/apis/machineconfiguration.openshift.io/v1/machineconfigpools/e2e-invalid": read tcp 10.131.94.172:55938->50.18.129.189:6443: read: connection reset by peer
time="2023-03-10T16:16:17Z" level=info msg="Skipping cleanup function since --skip-cleanup-error is true"
--- FAIL: TestE2E (2289.37s)
    --- PASS: TestE2E/Serial_tests (2011.23s)
        --- PASS: TestE2E/Serial_tests/TestScanStorageOutOfQuotaRangeFails (5.34s)
        --- PASS: TestE2E/Serial_tests/TestSuiteScan (126.38s)
        --- PASS: TestE2E/Serial_tests/TestTolerations (65.55s)
        --- PASS: TestE2E/Serial_tests/TestAutoRemediate (327.29s)
        --- PASS: TestE2E/Serial_tests/TestUnapplyRemediation (377.71s)
        --- PASS: TestE2E/Serial_tests/TestInconsistentResult (65.89s)
        --- PASS: TestE2E/Serial_tests/TestPlatformAndNodeSuiteScan (166.40s)
        --- PASS: TestE2E/Serial_tests/TestUpdateRemediation (377.89s)
        --- PASS: TestE2E/Serial_tests/TestProfileBundleDefaultIsKept (165.67s)
        --- PASS: TestE2E/Serial_tests/TestVariableTemplate (151.26s)
        --- PASS: TestE2E/Serial_tests/TestKubeletConfigRemediation (181.85s)
FAIL
time="2023-03-10T16:16:19Z" level=warning msg="13 scans not cleaned up"
FAIL	github.com/ComplianceAsCode/compliance-operator/tests/e2e	2292.253s
FAIL
make[1]: *** [Makefile:568: e2e] Error 1
make[1]: Leaving directory '/go/src/github.com/ComplianceAsCode/compliance-operator'
make: *** [Makefile:576: e2e-serial] Error 2
{"component":"entrypoint","error":"wrapped process failed: exit status 2","file":"k8s.io/test-infra/prow/entrypoint/run.go:79","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2023-03-10T16:16:19Z"}
error: failed to execute wrapped command: exit status 2 
INFO[2023-03-10T16:16:28Z] Step e2e-aws-serial-test failed after 39m20s. 
INFO[2023-03-10T16:16:28Z] Step phase test failed after 39m20s.         

Even though all the tests pass, the entire suite fails (presumably because the cleanup fails).

@rhmdnd
Copy link
Author

rhmdnd commented Mar 13, 2023

This cropped up in #223

It's cropped up in other PRs too, but I don't have a list.

It would be nice to make the cleanup code more robust (if that is indeed the problem) so we don't have to recheck this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant