Bug 1774156: 4.2 cherry-picks in support of automated service ca rotation #105

marun · 2020-01-28T08:16:13Z

NOTE This PR is not mergeable until the updates to library-go and api have merged.

This is required to allow `spec.forceRotation` to be set by an administrator to force CA rotation.

The CA cert will be provided to clients via the CA bundle.

Co-authored-by: Matt Rogers <mrogers@redhat.com> Co-authored-by: Stanislav Laznicka <slaznick@redhat.com>

This ensures that the inclusion of the CA will allow legacy clients to continue reusing a serving cert as a ca bundle.

openshift-ci-robot · 2020-01-28T08:16:18Z

@marun: This pull request references Bugzilla bug 1774156, which is invalid:

expected dependent Bugzilla bug 1774121 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ON_QA instead
expected dependent Bugzilla bug 1788179 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ASSIGNED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

WIP Bug 1774156: 4.2 cherry-picks in support of automated service ca rotation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

marun · 2020-02-11T02:28:16Z

/retest

marun · 2020-02-21T05:51:22Z

/bugzilla refresh

openshift-ci-robot · 2020-02-21T05:51:28Z

@marun: This pull request references Bugzilla bug 1774156, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

deads2k · 2020-02-21T16:04:59Z

glide.yaml

+  repo: git@github.com:marun/api
+  version: 4.2-automated-rotation
 - package: github.com/openshift/client-go
-  version: master


phew, we have to do a better job :(

deads2k · 2020-02-21T16:05:27Z

/lgtm

don't release the hold until you really bump

openshift-ci-robot · 2020-02-22T15:48:49Z

@marun: This pull request references Bugzilla bug 1774156, which is valid.

Details

In response to this:

Bug 1774156: 4.2 cherry-picks in support of automated service ca rotation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

marun · 2020-02-22T15:49:07Z

Bumped and ready for review.

marun · 2020-02-23T22:21:43Z

/retest

marun · 2020-02-26T01:28:53Z

@sttts @stlaz merge plz?

sttts · 2020-02-26T09:01:41Z

/lgtm

openshift-ci-robot · 2020-02-26T09:02:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, marun, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [deads2k,sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

marun · 2020-02-26T21:47:41Z

/retest

marun · 2020-02-26T23:14:40Z

/retest

openshift-bot · 2020-02-26T23:33:23Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-26T23:46:20Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-02-27T01:11:24Z

@marun: All pull requests linked via external trackers have merged. Bugzilla bug 1774156 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1774156: 4.2 cherry-picks in support of automated service ca rotation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…1810036 Also tombstone affected releases to avoid further channel promotion for affected releases. Details on the bug: * 4.5: Fixed by [1], service-ca-operator 74b5ce2 [2], which included library-go d9c73bb [3]. * 4.4: Introduced by [4] (no PR?). Fixed by [5], service-ca-operator e5a04d6 [6], which included library-go 3c25293 [7]. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed $ git --no-pager log -2 --first-parent --oneline origin/release-4.4 e5a04d6a (origin/release-4.4) Merge pull request openshift#111 from marun/4.4-unique-ca-serial 094a9ad0 Merge pull request #95 from vareti/signer-ca-metrics So both RCs are affected. * 4.3: Introduced by [8], service-ca-operator 8395d65 [9]. Fixed by [10], service-ca-operator dd7235b [11], which includes library-go 5844159 [12]. Fix has not been released yet. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 774c394da334dec446703545d4baaf89611ccb9d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 8395d65888b0a4249277989f18ee03f45383e409 So this was introduced in 4.3.5 (there was no 4.3.4). * 4.2: Introduced by [13], service-ca-operator 0324055 [14], which includes library-go 2cf86bb [15] and API 8ce0047 [16]. Fix in flight with [17,18]. [19] has already landed with library-go d58edcb. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator f6720573b9b63147436374e51e6fda44683b1e9f $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.22-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 0324055c3bad3a857dcf3471c024bf42c20d549e So this was introduced in 4.2.22. * 4.1: Backport stream introducing the bug is still ASSIGNED [20], so no 4.1 impact yet. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1810036 [2]: openshift/service-ca-operator#110 (comment) [3]: openshift/library-go#726 (comment) [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1774121 [5]: https://bugzilla.redhat.com/show_bug.cgi?id=1810418 [6]: openshift/service-ca-operator#111 (comment) [7]: openshift/library-go#728 (comment) [8]: https://bugzilla.redhat.com/show_bug.cgi?id=1788179 [9]: openshift/service-ca-operator#104 (comment) [10]: https://bugzilla.redhat.com/show_bug.cgi?id=1810420 [11]: openshift/service-ca-operator#112 (comment) [12]: openshift/library-go#729 (comment) [13]: https://bugzilla.redhat.com/show_bug.cgi?id=1774156 [14]: openshift/service-ca-operator#105 (comment) [15]: openshift/library-go#684 (comment) [16]: openshift/api#577 (comment) [17]: https://bugzilla.redhat.com/show_bug.cgi?id=1810421 [18]: openshift/service-ca-operator#113 [19]: openshift/library-go#730 (comment) [20]: https://bugzilla.redhat.com/show_bug.cgi?id=1774157

…1810036 The bugs were introduced by the [1] series, and fixed by the combination of [2,3]. This commit also tombstones affected releases to avoid further channel promotion. Details on the bug: * 4.5: Introduced by [1] (no PR?). Fixed by [2], service-ca-operator 74b5ce2 [4], which included library-go d9c73bb [5]. Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the 4.4/4.5 split. * 4.4: Introduced by [1] (no PR?). Fixed by [7], service-ca-operator e5a04d6 [7], which included library-go 3c25293 [9]. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed $ git --no-pager log -2 --first-parent --oneline origin/release-4.4 e5a04d6a (origin/release-4.4) Merge pull request openshift#111 from marun/4.4-unique-ca-serial 094a9ad0 Merge pull request #95 from vareti/signer-ca-metrics So both RCs are affected. Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the 4.4/4.5 split. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep oauth-proxy oauth-proxy https://github.com/openshift/oauth-proxy 3d0621eb72c9dd1c036505363032468a9016f381 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep oauth-proxy oauth-proxy https://github.com/openshift/oauth-proxy 3d0621eb72c9dd1c036505363032468a9016f381 So both RCs have OAuth fix, but neither has the service-ca-operator fix. * 4.3: Introduced by [10], service-ca-operator 8395d65 [11]. Fixed by [12], service-ca-operator dd7235b [13], which includes library-go 5844159 [14]. Fix has not been released yet. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 774c394da334dec446703545d4baaf89611ccb9d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 8395d65888b0a4249277989f18ee03f45383e409 So this was introduced in 4.3.5 (there was no 4.3.4). Fix also requires the OAuth proxy fix [15,16], which is still in flight. * 4.2: Introduced by [17], service-ca-operator 0324055 [18], which includes library-go 2cf86bb [19] and API 8ce0047 [20]. Fix in flight with [21,22]. [23] has already landed with library-go d58edcb. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator f6720573b9b63147436374e51e6fda44683b1e9f $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.22-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 0324055c3bad3a857dcf3471c024bf42c20d549e So this was introduced in 4.2.22. Fix also requires the OAuth proxy fix [24,25], which is still in flight. * 4.1: Backport stream introducing the bug is still ASSIGNED [26], so no 4.1 impact yet. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774121 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1810036 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1801573 [4]: openshift/service-ca-operator#110 (comment) [5]: openshift/library-go#726 (comment) [6]: openshift/oauth-proxy#152 (comment) [7]: https://bugzilla.redhat.com/show_bug.cgi?id=1810418 [8]: openshift/service-ca-operator#111 (comment) [9]: openshift/library-go#728 (comment) [10]: https://bugzilla.redhat.com/show_bug.cgi?id=1788179 [11]: openshift/service-ca-operator#104 (comment) [12]: https://bugzilla.redhat.com/show_bug.cgi?id=1810420 [13]: openshift/service-ca-operator#112 (comment) [14]: openshift/library-go#729 (comment) [15]: https://bugzilla.redhat.com/show_bug.cgi?id=1809253 [16]: openshift/oauth-proxy#160 [17]: https://bugzilla.redhat.com/show_bug.cgi?id=1774156 [18]: openshift/service-ca-operator#105 (comment) [19]: openshift/library-go#684 (comment) [20]: openshift/api#577 (comment) [21]: https://bugzilla.redhat.com/show_bug.cgi?id=1810421 [22]: openshift/service-ca-operator#113 [23]: openshift/library-go#730 (comment) [24]: https://bugzilla.redhat.com/show_bug.cgi?id=1809258 [25]: openshift/oauth-proxy#164 [26]: https://bugzilla.redhat.com/show_bug.cgi?id=1774157

…1810036 The bugs were introduced by the [1] series, and fixed by the combination of [2,3]. This commit also tombstones affected releases to avoid further channel promotion. Details on the bug: * 4.5: Introduced by [1] (no linked PR, so not sure exactly when it was introduced). Fixed by [2], service-ca-operator 74b5ce2 [4], which included library-go d9c73bb [5]. Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the 4.4/4.5 split. * 4.4: Introduced by [1] (no linked PR, so not sure exactly when it was introduced). Fixed by [7], service-ca-operator e5a04d6 [7], which included library-go 3c25293 [9]. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed $ git --no-pager log -2 --first-parent --oneline origin/release-4.4 e5a04d6a (origin/release-4.4) Merge pull request openshift#111 from marun/4.4-unique-ca-serial 094a9ad0 Merge pull request #95 from vareti/signer-ca-metrics So both RCs are affected. Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the 4.4/4.5 split. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep oauth-proxy oauth-proxy https://github.com/openshift/oauth-proxy 3d0621eb72c9dd1c036505363032468a9016f381 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep oauth-proxy oauth-proxy https://github.com/openshift/oauth-proxy 3d0621eb72c9dd1c036505363032468a9016f381 So both RCs have OAuth fix, but neither has the service-ca-operator fix. * 4.3: Introduced by [10], service-ca-operator 8395d65 [11]. Fixed by [12], service-ca-operator dd7235b [13], which includes library-go 5844159 [14]. Fix has not been released yet. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 774c394da334dec446703545d4baaf89611ccb9d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 8395d65888b0a4249277989f18ee03f45383e409 So this was introduced in 4.3.5 (there was no 4.3.4). Fix also requires the OAuth proxy fix [15,16], which is still in flight. * 4.2: Introduced by [17], service-ca-operator 0324055 [18], which includes library-go 2cf86bb [19] and API 8ce0047 [20]. Fix in flight with [21,22]. [23] has already landed with library-go d58edcb. $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator f6720573b9b63147436374e51e6fda44683b1e9f $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.22-x86_64 | grep service-ca-operator service-ca-operator https://github.com/openshift/service-ca-operator 0324055c3bad3a857dcf3471c024bf42c20d549e So this was introduced in 4.2.22. Fix also requires the OAuth proxy fix [24,25], which is still in flight. * 4.1: Backport stream introducing the bug is still ASSIGNED [26], so no 4.1 impact yet. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774121 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1810036 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1801573 [4]: openshift/service-ca-operator#110 (comment) [5]: openshift/library-go#726 (comment) [6]: openshift/oauth-proxy#152 (comment) [7]: https://bugzilla.redhat.com/show_bug.cgi?id=1810418 [8]: openshift/service-ca-operator#111 (comment) [9]: openshift/library-go#728 (comment) [10]: https://bugzilla.redhat.com/show_bug.cgi?id=1788179 [11]: openshift/service-ca-operator#104 (comment) [12]: https://bugzilla.redhat.com/show_bug.cgi?id=1810420 [13]: openshift/service-ca-operator#112 (comment) [14]: openshift/library-go#729 (comment) [15]: https://bugzilla.redhat.com/show_bug.cgi?id=1809253 [16]: openshift/oauth-proxy#160 [17]: https://bugzilla.redhat.com/show_bug.cgi?id=1774156 [18]: openshift/service-ca-operator#105 (comment) [19]: openshift/library-go#684 (comment) [20]: openshift/api#577 (comment) [21]: https://bugzilla.redhat.com/show_bug.cgi?id=1810421 [22]: openshift/service-ca-operator#113 [23]: openshift/library-go#730 (comment) [24]: https://bugzilla.redhat.com/show_bug.cgi?id=1809258 [25]: openshift/oauth-proxy#164 [26]: https://bugzilla.redhat.com/show_bug.cgi?id=1774157

marun and others added 6 commits January 27, 2020 22:57

Refactor e2e component check to improve error reporting

03f246c

Prevent breakage if log level is not configured in operator cfg

4923d37

Remove duplicate manifest CR to ensure creation but not management

d51878d

This is required to allow `spec.forceRotation` to be set by an administrator to force CA rotation.

Stop including the issuing CA's primary cert with the serving cert

ad2ce17

The CA cert will be provided to clients via the CA bundle.

Automate rotation of the service CA certificate and signing key

4dc48c2

Co-authored-by: Matt Rogers <mrogers@redhat.com> Co-authored-by: Stanislav Laznicka <slaznick@redhat.com>

Revert the removal of the service CA from serving certs

aa18d21

This ensures that the inclusion of the CA will allow legacy clients to continue reusing a serving cert as a ca bundle.

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jan 28, 2020

openshift-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jan 28, 2020

marun requested review from stlaz and sttts January 28, 2020 08:16

openshift-ci-robot requested a review from deads2k January 28, 2020 08:17

Update CA duration to 26 months to ensure upgrade after rotation

c3619d7

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Feb 21, 2020

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Feb 21, 2020

deads2k reviewed Feb 21, 2020

View reviewed changes

openshift-ci-robot assigned deads2k Feb 21, 2020

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 21, 2020

marun added 2 commits February 22, 2020 07:47

bump(library-go,api)

481ba7e

Update bindata

15bca69

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Feb 22, 2020

marun changed the title ~~WIP Bug 1774156: 4.2 cherry-picks in support of automated service ca rotation~~ Bug 1774156: 4.2 cherry-picks in support of automated service ca rotation Feb 22, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 22, 2020

openshift-ci-robot assigned sttts Feb 26, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 26, 2020

knobunc added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Feb 26, 2020

openshift-merge-robot merged commit 0324055 into openshift:release-4.2 Feb 27, 2020

Bug 1774156: 4.2 cherry-picks in support of automated service ca rotation #105

Bug 1774156: 4.2 cherry-picks in support of automated service ca rotation #105

Uh oh!

Conversation

marun commented Jan 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jan 28, 2020

Uh oh!

marun commented Feb 11, 2020

Uh oh!

marun commented Feb 21, 2020

Uh oh!

openshift-ci-robot commented Feb 21, 2020

Uh oh!

deads2k Feb 21, 2020

Choose a reason for hiding this comment

Uh oh!

deads2k commented Feb 21, 2020

Uh oh!

openshift-ci-robot commented Feb 22, 2020

Uh oh!

marun commented Feb 22, 2020

Uh oh!

marun commented Feb 23, 2020

Uh oh!

marun commented Feb 26, 2020

Uh oh!

sttts commented Feb 26, 2020

Uh oh!

openshift-ci-robot commented Feb 26, 2020

Uh oh!

marun commented Feb 26, 2020

Uh oh!

marun commented Feb 26, 2020

Uh oh!

openshift-bot commented Feb 26, 2020

Uh oh!

openshift-bot commented Feb 26, 2020

Uh oh!

openshift-ci-robot commented Feb 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

marun commented Jan 28, 2020 •

edited

Loading