WIP: test/e2e: Validate that no nodes went degraded

[cgwalters]

I think we have this in CI but we're not noticing it.
If it's happening we need to fix it.

Ref: #301

[cgwalters]

(Tested this locally by forcing a node degrade and the test failed as expected)

[cgwalters]

/test e2e-aws-op

[ashcrow]

/test e2e-aws

[jlebon]

/test e2e-aws-op

[cgwalters]

Hm, looks like the cluster in e2e-aws-op here lost a node;

$ oc get pods --all-namespaces | grep NodeLost
openshift-cluster-network-operator           cluster-network-operator-qsztm                                  1/1       NodeLost    2          1h
openshift-cluster-node-tuning-operator       tuned-b4pq6                                                     1/1       NodeLost    0          1h
openshift-dns                                dns-default-gn9jz                                               2/2       NodeLost    0          1h
openshift-image-registry                     node-ca-4x9jw                                                   1/1       NodeLost    0          1h
openshift-machine-config-operator            machine-config-daemon-6t8tn                                     1/1       NodeLost    0          1h
openshift-machine-config-operator            machine-config-server-xt7r4                                     1/1       NodeLost    0          1h
openshift-sdn                                ovs-qvcsd                                                       1/1       NodeLost    0          1h
openshift-sdn                                sdn-controller-mprjl                                            1/1       NodeLost    0          1h
openshift-sdn                                sdn-jg9hv                                                       1/1       NodeLost    0          1h

And it looks like the previous MCC was there, so we don't have logs from it. Ah, and the master pool is updating:

$ oc get machineconfigpool
NAME      CONFIG                                    UPDATED   UPDATING   DEGRADED
master    master-73ad249a9f5189df75bec7acb767cc01   False     True       False
worker    worker-ad3f99702752cb7f13d04e3d1b7202df   True      False      False

And...the cluster got GC'd, damn.

[ashcrow]

Failed to connect events watcher ... connection refused

Flake.

[cgwalters]

(Any reason no one is lgtm this?)

[jlebon]

I was waiting to see if e2e-aws-op was failing or not, but it's just flaking a lot right now. One thing that was confusing me was actually finding the test results from make test-e2e. It turns out it's in "artifacts" and then artifacts/e2e-aws-op/container-logs/test.log.gz.

The test itself looks good to me.
/lgtm

[ashcrow]

More Terraform + aws flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[cgwalters]

/test e2e-aws-op

[cgwalters]

Cool:

go test -timeout 20m -v${WHAT:+ -run="$WHAT"} ./test/e2e/
=== RUN   TestOperatorLabel
--- PASS: TestOperatorLabel (0.18s)
=== RUN   TestNoDegraded
--- PASS: TestNoDegraded (0.11s)
PASS
ok      github.com/openshift/machine-config-operator/test/e2e   0.327s

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[cgwalters]

/lgtm cancel

[ashcrow]

/retest

[cgwalters]

No point in landing this until we fix #367

(Though a PR to fix that could roll in this one)

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlebon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jlebon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ashcrow]

=== RUN   TestNoDegraded
--- FAIL: TestNoDegraded (0.06s)
	sanity_test.go:67: 3 degraded nodes found
FAIL
FAIL	github.com/openshift/machine-config-operator/test/e2e	0.370s

[openshift-ci-robot]

@cgwalters: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-op	32b226a	link	/test e2e-aws-op
ci/prow/e2e-aws	32b226a	link	/test e2e-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[runcom]

with #386 do we still need a test like this?

[cgwalters]

Yeah, I think this one is obsoleted.