Skip to content

Conversation

@sdodson
Copy link
Member

@sdodson sdodson commented Apr 16, 2020

4.3.8 introduced a change which blocks upgrades when the default SCCs are
mutated. That change was merged under the assumption that the CVO would not
block z-stream upgrades on that condition. However that change had not
merged yet and thus ~40% of clusters which have upgraded to 4.3.8-4.3.12
are now blocked from upgrading with out specifying the --force flag. 4.3.13
relaxes the Upgradeable=False such that you can now apply z-stream updates
when that condition is set. A future update, likely 4.3.14 will revert the change
which began setting that condition.

Because the slim majority of subscribed clusters had already upgraded into
this problem and because of the numerous previously blocked upgrades we
had decided to live with the problem until 4.3.13 ships. So once 4.3.13 is
promoted into stable-4.3 we should stop additional clusters from falling
into the problem.

Reference https://bugzilla.redhat.com/show_bug.cgi?id=1821905#c22

@sdodson
Copy link
Member Author

sdodson commented Apr 16, 2020

/hold
For #176

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 16, 2020
@sdodson
Copy link
Member Author

sdodson commented Apr 16, 2020

Hmm, this will create a gap where 4.3.0 cannot upgrade to 4.3.13. Need to consider what to do there.

@sdodson
Copy link
Member Author

sdodson commented Apr 16, 2020

This will also raise the minimum version to upgrade from 4.2 to 4.3 from 4.2.21 to 4.2.29 which is undesirable but not unworkable. Which currently moves us from ~ 26% of the 4.2 fleet to < 1%. This could be mitigated by ensuring that 4.3.14 includes upgrades back to 4.2.21 but this seems like we're stretching and would introduce additional confusion.

@wking
Copy link
Member

wking commented Apr 16, 2020

This will also raise the minimum version to upgrade from 4.2 to 4.3 from 4.2.21 to 4.2.29 which is undesirable but not unworkable.

Should just be 4.2.{current} -> 4.2.29 -> 4.3, which doesn't seem that bad? Even from 4.2.0, it would be 4.2.0 -> 4.2.13 -> 4.2.21 -> 4.2.29 -> 4.3, so still just one extra hop.

@wking
Copy link
Member

wking commented Apr 16, 2020

Hmm, this will create a gap where 4.3.0 cannot upgrade to 4.3.13.

So we should include 4.3.0 in 4.3.14.

@sdodson sdodson force-pushed the block-4.3.8-through-4.3.12 branch from d7b57ec to b019810 Compare April 16, 2020 23:17
4.3.8 introduced a change which blocks upgrades when the default SCCs are
mutated. That change was merged under the assumption that the CVO would not
block z-stream upgrades on that condition. However that change had not
merged yet and thus ~40% of clusters which have upgraded to 4.3.8-4.3.12
are now blocked from upgrading with out specifying the --force flag. 4.3.13
relaxes the Upgradeable=False such that you can now apply z-stream updates
when that condition is set. 4.3.14 will revert the change which began
setting that condition.

Because the majority of subscribed clusters had already upgraded into this
problem and because of the numerous previously blocked upgrades we had
decided to live with the problem until 4.3.13 ships. So once 4.3.13 is
promoted into stable-4.3 we should stop additional clusters from falling
into the problem.
@sdodson sdodson force-pushed the block-4.3.8-through-4.3.12 branch from b019810 to 3be2bd2 Compare April 16, 2020 23:18
@wking
Copy link
Member

wking commented Apr 17, 2020

/lgtm

Just need to wait on #176, or as close to that landing as we feel we need to get so we can land this without ruffling feathers about "where did my update go?". I'm personally fine landing this today, even if lots of folks are already up in the affected versions, because waiting a few days for #176 to land doesn't seem like a big deal. But I also understand that 4.2 -> 4.3 connectivity has not been great so far, and that there are clusters out there who have not mutated their default MCCs. Speaking of which:

/retitle Block edges to 4.3.8 through 4.3.12 on DefaultSecurityContextConstraints_Mutated

should make it a bit easier to find this in the Git logs.

@openshift-ci-robot openshift-ci-robot changed the title Block edges to 4.3.8 - 4.3.12 Block edges to 4.3.8 through 4.3.12 on DefaultSecurityContextConstraints_Mutated Apr 17, 2020
@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 17, 2020
@wking wking removed their assignment Apr 19, 2020
@sdodson
Copy link
Member Author

sdodson commented Apr 20, 2020

Still holding on this until we work through some decisions internally.

This was referenced Apr 20, 2020
@sdodson
Copy link
Member Author

sdodson commented Apr 20, 2020

/hold cancel
This is ready to go now.

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 20, 2020
@eparis
Copy link
Member

eparis commented Apr 21, 2020

/approve

@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eparis, sdodson, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 21, 2020
@openshift-merge-robot openshift-merge-robot merged commit 86b5397 into openshift:master Apr 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants