Bug 2041971: Bump library-go to fix mutating webhook reconcile by damdo · Pull Request #981 · openshift/machine-api-operator

damdo · 2022-01-20T15:07:22Z

To include the fix in openshift/library-go#1293 for an issue causing the reconciliation of mutating webhooks to not happen.
Also uses the improved version of Apply*WebhookConfiguration with cache.
For more details on the bug: https://bugzilla.redhat.com/show_bug.cgi?id=2041971

Here is an attempt to reproduce the issue with this fix
(the reproduction steps are available on the linked Bugzilla issue):

# check the paths of the mutatingwebhookconfiguration machine-api config
[~] $ oc -n openshift-machine-api get mutatingwebhookconfiguration machine-api -o yaml | grep path
      path: /mutate-machine-openshift-io-v1beta1-machine
      path: /mutate-machine-openshift-io-v1beta1-machineset
      
[~] $ oc -n openshift-machine-api get pods
NAME                                          READY   STATUS    RESTARTS   AGE
cluster-autoscaler-operator-f8f5c6d79-hx5qn   2/2     Running   0          143m
cluster-baremetal-operator-86d698c796-hq6vw   2/2     Running   0          143m
machine-api-controllers-676954949c-8w5rc      7/7     Running   0          142m
machine-api-operator-869df657b8-mwrns         2/2     Running   0          143m

# scale down the machine-api-operator
[~] $ oc scale deployment machine-api-operator -n openshift-machine-api --replicas=0
deployment.apps/machine-api-operator scaled

# edit the machine-api mutatingwebhookconfiguration to change some bits of the config to check the reconciliation mechanism
[~] $ oc -n openshift-machine-api edit mutatingwebhookconfiguration machine-api
mutatingwebhookconfiguration.admissionregistration.k8s.io/machine-api edited

# the two paths of the mutatingwebhookconfiguration  have been changed with the editor to: 
# path: /mutate-machine-openshift-io-v1beta1-machineset2
# path: /mutate-machine-openshift-io-v1beta1-machine2

# check that the change has happened
[~] $ oc -n openshift-machine-api get mutatingwebhookconfiguration machine-api -o yaml | grep path
      path: /mutate-machine-openshift-io-v1beta1-machine2
      path: /mutate-machine-openshift-io-v1beta1-machineset2

# scale the machine-api-operator back up
[~] $ oc scale deployment machine-api-operator -n openshift-machine-api --replicas=1
deployment.apps/machine-api-operator scaled

# wait some time for the operator to come back and get leadership
# then check again the machine-api mutatingwebhookconfiguration
[~] $ oc -n openshift-machine-api get mutatingwebhookconfiguration machine-api -o yaml | grep path
      path: /mutate-machine-openshift-io-v1beta1-machine
      path: /mutate-machine-openshift-io-v1beta1-machineset

# the paths have been brought back to the default values by the machine-api-operator

After the fix, the issue is no longer reproducible.

damdo · 2022-01-21T11:47:01Z

/test unit

openshift-ci · 2022-01-21T11:55:34Z

@damdo: This pull request references Bugzilla bug 2041971, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @sunzhaohua2

Details

In response to this:

Bug 2041971: Bump library-go to fix mutating webhook reconcile

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

damdo · 2022-01-21T13:59:22Z

/retest-required

damdo · 2022-01-21T14:30:53Z

/retest-required

openshift-ci · 2022-01-21T15:47:01Z

@damdo: This pull request references Bugzilla bug 2041971, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @sunzhaohua2

Details

In response to this:

Bug 2041971: Bump library-go to fix mutating webhook reconcile

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-01-21T15:48:09Z

@damdo: This pull request references Bugzilla bug 2041971, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @sunzhaohua2

Details

In response to this:

Bug 2041971: Bump library-go to fix mutating webhook reconcile

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-01-21T15:48:45Z

@damdo: This pull request references Bugzilla bug 2041971, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @sunzhaohua2

Details

In response to this:

Bug 2041971: Bump library-go to fix mutating webhook reconcile

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

damdo · 2022-01-21T15:49:20Z

/retest-required

openshift-ci · 2022-01-21T15:50:40Z

@damdo: This pull request references Bugzilla bug 2041971, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @sunzhaohua2

Details

In response to this:

Bug 2041971: Bump library-go to fix mutating webhook reconcile

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

to include the fix in openshift/library-go#1293 causing the issue at: https://bugzilla.redhat.com/show_bug.cgi?id=2041971

damdo · 2022-01-21T17:57:10Z

/retest-required

damdo · 2022-01-21T19:51:59Z

/retest-required

damdo · 2022-01-22T11:53:55Z

/retest

damdo · 2022-01-22T14:54:53Z

/retest

damdo · 2022-01-22T17:52:34Z

/retest-required

damdo · 2022-01-22T19:55:32Z

/retest-required

damdo · 2022-01-22T21:52:04Z

/retest-required

damdo · 2022-01-23T01:07:18Z

/retest-required

damdo · 2022-01-23T13:03:02Z

/retest-required

JoelSpeed · 2022-01-24T08:46:19Z

/approve
/lgtm
/retest-required

openshift-ci · 2022-01-24T08:48:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [JoelSpeed]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2022-01-24T10:04:11Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-01-24T10:16:12Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

JoelSpeed · 2022-01-24T10:20:12Z

/skip

Clearing optional statuses to prevent spurious retest comments

openshift-ci · 2022-01-24T10:36:00Z

@damdo: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-metal-ipi-upgrade	`1e6760a`	link	false	`/test e2e-metal-ipi-upgrade`
ci/prow/e2e-gcp	`1e6760a`	link	false	`/test e2e-gcp`
ci/prow/e2e-vsphere-upgrade	`1e6760a`	link	false	`/test e2e-vsphere-upgrade`
ci/prow/e2e-openstack	`1e6760a`	link	false	`/test e2e-openstack`
ci/prow/e2e-metal-ipi-ovn-dualstack	`1e6760a`	link	false	`/test e2e-metal-ipi-ovn-dualstack`
ci/prow/e2e-gcp-operator	`1e6760a`	link	false	`/test e2e-gcp-operator`
ci/prow/e2e-azure-operator	`1e6760a`	link	false	`/test e2e-azure-operator`
ci/prow/e2e-aws-disruptive	`1e6760a`	link	false	`/test e2e-aws-disruptive`
ci/prow/e2e-vsphere-operator	`1e6760a`	link	false	`/test e2e-vsphere-operator`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

JoelSpeed · 2022-01-24T10:44:34Z

/override ci/prow/e2e-aws-operator

The autoscaler failures are unrelated to this PR and we have a fix in progress for them

openshift-ci · 2022-01-24T10:45:19Z

@JoelSpeed: Overrode contexts on behalf of JoelSpeed: ci/prow/e2e-aws-operator

Details

In response to this:

/override ci/prow/e2e-aws-operator

The autoscaler failures are unrelated to this PR and we have a fix in progress for them

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-01-24T10:46:27Z

@damdo: All pull requests linked via external trackers have merged:

openshift/machine-api-operator#981

Bugzilla bug 2041971 has been moved to the MODIFIED state.

Details

In response to this:

Bug 2041971: Bump library-go to fix mutating webhook reconcile

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot requested review from Fedosin and elmiko January 20, 2022 15:19

damdo changed the title ~~Bump library-go to fix mutating webhook reconcile~~ Bug 2041971: Bump library-go to fix mutating webhook reconcile Jan 21, 2022

openshift-ci bot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label Jan 21, 2022

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jan 21, 2022

openshift-ci bot requested a review from sunzhaohua2 January 21, 2022 11:55

damdo added 2 commits January 21, 2022 15:55

bump library-go to fix mutating webhook reconcile

5076a82

to include the fix in openshift/library-go#1293 causing the issue at: https://bugzilla.redhat.com/show_bug.cgi?id=2041971

use cache improved version of Apply*WebhookConfiguration

1e6760a

damdo force-pushed the bump-library-go-fix-mutating-webhook-reconcile branch from a233fd3 to 1e6760a Compare January 21, 2022 15:56

JoelSpeed mentioned this pull request Jan 24, 2022

Bug 2041971: Revert use webhookApply implementation from Library Go #980

Closed

openshift-ci bot assigned JoelSpeed Jan 24, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 24, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 24, 2022

openshift-merge-robot merged commit 668c5b5 into openshift:master Jan 24, 2022

Comments

Conversation

damdo commented Jan 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

damdo commented Jan 21, 2022

Uh oh!

openshift-ci bot commented Jan 21, 2022

Uh oh!

damdo commented Jan 21, 2022

Uh oh!

damdo commented Jan 21, 2022

Uh oh!

openshift-ci bot commented Jan 21, 2022

Uh oh!

openshift-ci bot commented Jan 21, 2022

Uh oh!

openshift-ci bot commented Jan 21, 2022

Uh oh!

damdo commented Jan 21, 2022

Uh oh!

openshift-ci bot commented Jan 21, 2022

Uh oh!

damdo commented Jan 21, 2022

Uh oh!

damdo commented Jan 21, 2022

Uh oh!

damdo commented Jan 22, 2022

Uh oh!

damdo commented Jan 22, 2022

Uh oh!

damdo commented Jan 22, 2022

Uh oh!

damdo commented Jan 22, 2022

Uh oh!

damdo commented Jan 22, 2022

Uh oh!

damdo commented Jan 23, 2022

Uh oh!

damdo commented Jan 23, 2022

Uh oh!

JoelSpeed commented Jan 24, 2022

Uh oh!

openshift-ci bot commented Jan 24, 2022

Uh oh!

openshift-bot commented Jan 24, 2022

Uh oh!

openshift-bot commented Jan 24, 2022

Uh oh!

JoelSpeed commented Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoelSpeed commented Jan 24, 2022

Uh oh!

openshift-ci bot commented Jan 24, 2022

Uh oh!

openshift-ci bot commented Jan 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

damdo commented Jan 20, 2022 •

edited

Loading

JoelSpeed commented Jan 24, 2022 •

edited

Loading

openshift-ci bot commented Jan 24, 2022 •

edited

Loading