controller: Add a 5s delay before rendering MCs

[cgwalters]

To reduce churn if MCs are being created rapidly - both on general
principle, and also to reduce our exposure to the current bug
that a booting node may fail to find a GC'd MachineConfig:
#301

[ashcrow]

TF failures

/retest

[jlebon]

Hmm, if I'm reading this right, even if we translate three MC creation events by 5s, we're still regenerating three times, right? Though I guess they'll hash to the same name now at least, so we won't get a generated MC quickly appearing then getting deleted.

LGTM though will defer to folks more familiar with the workqueue API.

/approve

[cgwalters]

we're still regenerating three times, right?

See https://godoc.org/k8s.io/client-go/util/workqueue
specifically:

and if an item is added multiple times before it can be processed, it will only be processed once.

[ashcrow]

tf failures again... looks like somethings up. Let's retest shortly.

[jlebon]

OK, after reading up some more on the workqueue API, I'm more confident this works now. I've also just tested it!

/lgtm

Re.

and if an item is added multiple times before it can be processed, it will only be processed once.

Right, I see that at https://github.com/kubernetes/client-go/blob/b831b8de7155117e51afaffeb647007a756ddc92/util/workqueue/queue.go#L114. But this happens at Add() time, and AddAfter() just delays the Add() call: https://github.com/kubernetes/client-go/blob/b831b8de7155117e51afaffeb647007a756ddc92/util/workqueue/delaying_queue.go#L162. So Add() is still called e.g. three times. So if the previous work item has already been processed, we'll just sync again. One test I did to verify this was oc create -f hello1.yaml && sleep 3.5 && oc create -f hello2.yaml.

Anyway, this still mitigates the issue fine since the MCs we're concerned about definitely happen within a 5s window. So even on the earliest event fired, we've already got all the MCs. Though of course closing the race window completely will still require some work.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, jlebon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [cgwalters,jlebon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ashcrow]

/test e2e-aws

[ashcrow]

/retest

[ashcrow]

/test e2e-aws