Bug 2053359: Feature gate initialization #740

wking · 2022-02-11T04:33:45Z

Bringing #736 and #739 back to 4.10.

…ingTechPreviewState As described in [1], some 4.10 jobs that set TechPreviewNoUpgrade very early during install are running into trouble like: 1. Early in bootstrap, something sets TechPreviewNoUpgrade. 2. Cluster-version operator comes up, and attempts to figure out the current featureSet. But because the Kubernetes API is also still coming up, that fails on an error like [2]: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-e2e-aws-techpreview/1489537240471179264/artifacts/e2e-aws-techpreview/gather-extra/artifacts/pods/openshift-cluster-version_cluster-version-operator-76cd65b7bb-4p945_cluster-version-operator.log | grep 'Error getting featuregate value\|tech' W0204 10:10:40.126809 1 start.go:142] Error getting featuregate value: Get "https://127.0.0.1:6443/apis/config.openshift.io/v1/featuregates/cluster": dial tcp 127.0.0.1:6443: connect: connection refused I0204 10:19:53.097129 1 techpreviewchangestopper.go:97] Starting stop-on-techpreview-change controller with TechPreviewNoUpgrade false. 3. The TechPreviewChangeStopper waits for any FeatureGate changes, but we don't get any. 4. CVO happily spends hours without synchronizing any of the requested TechPreviewNoUpgrade manifests. Step 2 was originally fatal, but I'd softened it in 90b1454 (pkg/start: Log and continue when we fail to retrieve the feature gate, 2021-12-06, openshift#706). Here are the relevant cases, and how they'd behave with the different approaches: a. No Kube-API hiccup on the initial FeatureGate fetch. All implementations handle this well. b. Kube-API hiccup on the initial FeatureGate fetch. i. And the actual FeatureGate value was not TechPreviewNoUpgrade. Before 90b1454, this would have caused a useless CVO container restart. Since 90b1454, and unchanged in this commit, the CVO container's default: includeTechPreview := false is correct, and we correctly ignore the hiccup. ii. The actual FeatureGate value was TechPreviewNoUpgrade. Before 90b1454, this would cause a useful CVO container restart. From 90b1454 until this commit, we'd hit the bug case where we'd go an unbounded amount of time failing to reconcile the TechPreviewNoUpgrade manifests the user was asking for. With this commit, we notice the divergence right after the informer caches sync, and restart the CVO container. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=2050946#c0 [2]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-e2e-aws-techpreview/1489537240471179264

Not Found errors are a clear configuration for "no feature set", so syncHandler should treat them as successful results. This avoids continually requeuing for a new GET call, now that 79742d7 (pkg/featurechangestopper: Seed queue to guard against incorrect startingTechPreviewState, 2022-02-04, openshift#736) is seeding the queue, even in clusters where there is no FeatureSet clusters. And it also allows us to detect set -> removed transitions if those were allowed, although the API-server is supposed to make it impossible to remove TechPreviewNoUpgrade once it has been set [1]. [1]: https://docs.openshift.com/container-platform/4.9/nodes/clusters/nodes-cluster-enabling-features.html#nodes-cluster-enabling-features-about_nodes-cluster-enabling

cvInformer does not make sense for FeatureGate due to it filters out resources with name other than `version` by default.

openshift-ci · 2022-02-11T04:33:50Z

@wking: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs

/lgtm

openshift-ci · 2022-02-22T10:26:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrutkovs, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [vrutkovs,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2022-02-22T11:22:14Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2022-02-22T13:27:31Z

@wking: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2022-02-23T01:01:00Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-02-23T01:01:05Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-02-24T01:00:23Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-02-24T01:00:30Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-02-25T01:00:24Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-02-25T01:00:33Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-02-26T01:00:19Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-02-26T01:00:26Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-02-27T01:00:18Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-02-27T01:00:29Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-02-28T01:00:23Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-02-28T01:00:30Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-01T01:01:11Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-01T01:01:24Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-02T01:00:52Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-02T01:01:03Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-03T01:00:51Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-03T01:01:08Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-04T01:00:50Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-04T01:01:12Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-05T01:00:48Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-05T01:01:16Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-06T01:00:29Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-06T01:00:56Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-07T01:00:33Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-07T01:00:58Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-08T01:00:31Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-08T01:00:55Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-09T01:01:15Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-09T01:01:37Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2022-03-10T01:00:43Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2022-03-10T01:01:07Z

@openshift-bot: This pull request references Bugzilla bug 2053359, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "4.10.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking · 2022-03-10T23:06:49Z

/bugzilla refresh

openshift-ci · 2022-03-10T23:07:01Z

@wking: This pull request references Bugzilla bug 2053359, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.z) matches configured target release for branch (4.10.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 2050946 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 2050946 targets the "4.11.0" release, which is one of the valid target releases: 4.11.0
bug has dependents

Requesting review from QA contact:
/cc @jiajliu

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

LalatenduMohanty · 2022-03-11T04:46:10Z

/label backport-risk-assessed

jiajliu · 2022-03-11T06:02:46Z

/label cherry-pick-approved

openshift-ci · 2022-03-11T06:10:32Z

@wking: All pull requests linked via external trackers have merged:

openshift/cluster-version-operator#740

Bugzilla bug 2053359 has been moved to the MODIFIED state.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking and others added 3 commits February 10, 2022 20:32

Pass sharedInfromer instead of cvInformer to feature-gate-stopper

e8f2658

cvInformer does not make sense for FeatureGate due to it filters out resources with name other than `version` by default.

openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Feb 11, 2022

openshift-ci bot requested review from LalatenduMohanty and vrutkovs February 11, 2022 04:34

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 11, 2022

vrutkovs approved these changes Feb 22, 2022

View reviewed changes

openshift-ci bot assigned vrutkovs Feb 22, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 22, 2022

openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Mar 10, 2022

openshift-ci bot requested a review from jiajliu March 10, 2022 23:07

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Mar 11, 2022

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Mar 11, 2022

openshift-merge-robot merged commit 50246cb into openshift:release-4.10 Mar 11, 2022

wking deleted the feature-gate-initialization branch March 11, 2022 06:49

Bug 2053359: Feature gate initialization #740

Bug 2053359: Feature gate initialization #740

Uh oh!

Conversation

wking commented Feb 11, 2022

Uh oh!

openshift-ci bot commented Feb 11, 2022

Uh oh!

vrutkovs left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Feb 22, 2022

Uh oh!

openshift-bot commented Feb 22, 2022

Uh oh!

openshift-ci bot commented Feb 22, 2022

Uh oh!

openshift-bot commented Feb 23, 2022

Uh oh!

openshift-ci bot commented Feb 23, 2022

Uh oh!

openshift-bot commented Feb 24, 2022

Uh oh!

openshift-ci bot commented Feb 24, 2022

Uh oh!

openshift-bot commented Feb 25, 2022

Uh oh!

openshift-ci bot commented Feb 25, 2022

Uh oh!

openshift-bot commented Feb 26, 2022

Uh oh!

openshift-ci bot commented Feb 26, 2022

Uh oh!

openshift-bot commented Feb 27, 2022

Uh oh!

openshift-ci bot commented Feb 27, 2022

Uh oh!

openshift-bot commented Feb 28, 2022

Uh oh!

openshift-ci bot commented Feb 28, 2022

Uh oh!

openshift-bot commented Mar 1, 2022

Uh oh!

openshift-ci bot commented Mar 1, 2022

Uh oh!

openshift-bot commented Mar 2, 2022

Uh oh!

openshift-ci bot commented Mar 2, 2022

Uh oh!

openshift-bot commented Mar 3, 2022

Uh oh!

openshift-ci bot commented Mar 3, 2022

Uh oh!

openshift-bot commented Mar 4, 2022

Uh oh!

openshift-ci bot commented Mar 4, 2022

Uh oh!

openshift-bot commented Mar 5, 2022

Uh oh!

openshift-ci bot commented Mar 5, 2022

Uh oh!

openshift-bot commented Mar 6, 2022

Uh oh!

openshift-ci bot commented Mar 6, 2022

Uh oh!

openshift-bot commented Mar 7, 2022

Uh oh!

openshift-ci bot commented Mar 7, 2022

Uh oh!

openshift-bot commented Mar 8, 2022

Uh oh!

openshift-ci bot commented Mar 8, 2022

Uh oh!

openshift-bot commented Mar 9, 2022

Uh oh!

openshift-ci bot commented Mar 9, 2022

Uh oh!

openshift-bot commented Mar 10, 2022

Uh oh!

openshift-ci bot commented Mar 10, 2022