e2e: add ssh mcd test

[kikisdeliveryservice]

Test verifies that MCD updates MCP, daemons and
writes new ssh keys to node filesystems.

Closes #546

[kikisdeliveryservice]

hmm passed before added the done check but seeing lots of timeouts in ci that seem unrelated to mco.

/retest

[kikisdeliveryservice]

/retest

[cgwalters]

This is probably OK, but we could go the extra mile and actually verify that the SSH key ends up on the node's filesystem. Basically the API equivalent of oc rsh pods/machine-config-daemon-xyz ls /rootfs/var/roothome/.ssh/authorized_keys.

[kikisdeliveryservice]

@cgwalters Yep, I was thinking doing a real check would make sense too! First pass was me poking around with the e2e for the first time. Ty!

[kikisdeliveryservice]

Ok so I think I was checking the Done annotation incorrectly before. Trying something new.

Update: Ok cool, that worked!

[ashcrow]

level=info msg="Waiting up to 30m0s for the cluster to initialize..."
level=fatal msg="failed to initialize the cluster: timed out waiting for the condition"

[kikisdeliveryservice]

@ashcrow we are hitting aws issues all over openshift rn.

[kikisdeliveryservice]

For tomorrow: need to figure out a way to do this: (oc debug isn't working on my local cluster)

• oc get pods -n openshift-machine-config-operator --field-selector spec.nodeNAME=node.Name
• oc rsh -n openshift-machine-config-operator machine-config-daemon-zt85f
• cat /rootfs/home/core/.ssh/authorized_keys to search file for test key

[kikisdeliveryservice]

basically i have a worker node name. so i need to get the daemon pod name that goes along with it, then rsh into that daemon and grab the authorized_keys file and check that the test key is inside.

the way to get a list of all the mcd is here:

machine-config-operator/test/e2e/mcd_test.go

Line 28 in ad18b8f

mcdList, err := cs.Pods("openshift-machine-config-operator").List(listOptions)

should i should be able to somehow filter that list to get that daemon pod whose spec.nodeName matches worker node.Name without having to use exec.Command to execute oc get pods -n openshift-machine-config-operator --field-selector ...

maybe some sort of filter for spec.nodeName on cs.Pods().List or cs.Pods().Get()?

[cgwalters]

You want to use https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1#ListOptions

[runcom]

/approve

pending LGTM until this works and passes and others review this

(great work with this!)

[kikisdeliveryservice]

thanks for your suggestions @runcom - they are really helpful!!!

ill squash once i get this working.

[kikisdeliveryservice]

level=info msg="Waiting up to 30m0s for the cluster at https://api.ci-op-jfqwkc5d-57a9f.origin-ci-int-aws.dev.rhcloud.com:6443 to initialize..."
level=fatal msg="failed to initialize the cluster: timed out waiting for the condition"

looking through the logs the timeout might be due to InstallerControllerFailed (kube-apiserver), so going to try retest

/test e2e-aws-op

[kikisdeliveryservice]

pretty sure grep is failing bc it's being done in a different shell than the oc rsh.

[kikisdeliveryservice]

cc: @runcom

GOCACHE=off go test -timeout 50m -v${WHAT:+ -run="$WHAT"} ./test/e2e/
=== RUN   TestMCDToken
--- PASS: TestMCDToken (0.45s)
=== RUN   TestMCDeployed
--- PASS: TestMCDeployed (2418.86s)
=== RUN   TestUpdateSSH
panic: test timed out after 50m0s

goroutine 2196 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:1240 +0xfc
created by time.goFunc
	/usr/local/go/src/time/sleep.go:172 +0x44

goroutine 1 [chan receive, 9 minutes]:

Because TestMCDeployed takes 40 minutes, it's blocking my test from ever passing bc only 10 minutes are left for all of the other tests in our suite to run.

I believe that my test might be working correctly as I see no error output on this pass. I will change TestMCDeployed from testing 10 to 2 MCs my branch test file to allow my PR more time to run. If it works, we'll have to decide what to do bc I've seen TestMCDeployed take ~40 min for 10 MCs consistently.

cc:@runcom

[kikisdeliveryservice]

/retest

[kikisdeliveryservice]

Awesome!! Got my e2e ssh test working! Will clean up commits tomorrow and pick up #563 to see if it fixes the time out issue.

[runcom]

Awesome!!!!!!

[kikisdeliveryservice]

@runcom timeout issues fixed, I think your PR did the trick!

[runcom]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kikisdeliveryservice, runcom

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [kikisdeliveryservice,runcom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment