Add proposal for credentials management outside the openshift cluster for new platforms #709
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| --- | ||
| title: Credentials Management outside OpenShift Cluster | ||
| authors: | ||
| - "@akhil-rane" | ||
| reviewers: | ||
| - "@abhinavdahiya" | ||
| - "@dgoodwin" | ||
| - "@joelddiaz" | ||
| approvers: | ||
| - "@derekwaynecarr" | ||
| - "@sdodson" | ||
| creation-date: 2021-03-25 | ||
| last-updated: 2021-03-25 | ||
| status: provisional | ||
| --- | ||
|
|
||
| # Credentials Management outside OpenShift Cluster | ||
|
|
||
| ## Release Signoff Checklist | ||
|
|
||
| - [x] Enhancement is `partially implemented` | ||
| - [ ] Design details are appropriately documented from clear requirements | ||
| - [ ] Test plan is defined | ||
| - [ ] Operational readiness criteria is defined | ||
| - [ ] Graduation criteria for dev preview, tech preview, GA | ||
| - [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/) | ||
|
|
||
| ## Summary | ||
|
|
||
| The intent of this enhancement is to take the process of credentials management outside the OpenShift cluster for new | ||
| platforms. We propose to make *manual* mode as default for clusters on new platforms, optionally supported by CLI tooling | ||
| that follows the best practices of the hosting infrastructure at that time. | ||
|
|
||
| ## Motivation | ||
|
|
||
| The main motivation behind this enhancement is to satisfy the customer requirement to follow the best security practices | ||
| by not storing admin credentials inside the cluster | ||
|
|
||
| ### Goals | ||
|
|
||
| As part of this enhancement we plan to do the following: | ||
| * Guide new platforms to start with Manual mode rather than Mint mode. | ||
| * Provide optional tooling (ccoctl) for administrators to manage their credentials outside the cluster. (prep for install, reconcile, prep for upgrade) | ||
| * Establish precedent for future direction of preferring *Manual* over *Mint*. | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| * Removing Mint mode for any currently supported cloud. (AWS, Azure, GCP) | ||
| * Changing the default for any provider currently defaulting to Mint mode, though we may revisit in the future. | ||
|
|
||
| ## Proposal | ||
|
|
||
| Currently, *mint* is a default credentials mode for OpenShift. In this mode we run OpenShift installer with an admin | ||
| level cloud credential. The admin credential is stored in kube-system namespace and then used by the cloud credential | ||
| operator to process the CredentialRequests in the cluster and create new users with fine-grained permissions. | ||
| The customers have reported that the need to store admin credentials inside the cluster is a major disadvantage. Based | ||
| on the feedback, we propose to run credentials related setup outside a cluster and then start installation process in a | ||
| *manual* mode without providing admin credentials to the installer. | ||
|
|
||
| ### User Stories | ||
|
|
||
| #### Story 1 | ||
| As an OpenShift cluster administrator I should be able to extract the CredentialsRequest manifests from the release image | ||
| and create required manifests/infrastructure to satisfy all in-cluster component's CredentialsRequests. This has been | ||
| already [implemented](https://github.com/openshift/cloud-credential-operator/blob/master/docs/ccoctl.md). | ||
|
|
||
| #### Story 2 | ||
| As an OpenShift cluster administrator, I should be able to reconcile my credentials manually using local CLI tooling on | ||
| an on-going basis. This is in [backlog](https://issues.redhat.com/browse/CCO-106) and targeted for 4.9. | ||
|
|
||
| #### Story 3 | ||
| As an OpenShift cluster administrator, I should be able to prep credentials for a cluster upgrade. This is in [backlog](https://issues.redhat.com/browse/CCO-106) | ||
| and targeted for 4.9. | ||
|
|
||
| ### Risks and Mitigations | ||
|
|
||
| ## Design Details | ||
|
|
||
| We intend to build an optional tool **ccoctl** which will handle credentials management of the cluster in *manual* mode. | ||
| The following is the set of requirements for the current prototype (for AWS). We can have something similar for other | ||
| platforms. | ||
|
Contributor
There was a problem hiding this comment. I think we need a plan here for how ccoctl is distributed. I am really wondering if this should be exposed in oc adm.
Member
There was a problem hiding this comment. i assume distribution of |
||
|
|
||
| * ccoctl should be able to setup Identity Provider to authenticate OpenShift components | ||
| * ccoctl should be able to take a list of CredentialsRequests from the release image and create/update | ||
| Roles/Users/ServicePrincipals/ServiceAccounts with appropriate permissions. | ||
| * ccoctl should be able to take list of CredentialsRequests for a release image, and the Identity Provider URL to generate | ||
| the objects that need to be passed to the installer for successful installation | ||
| * ccoctl should be able to delete all the resources that it had created in the cloud | ||
|
|
||
| We envision **ccoctl** as a recommended tool to setup credentials on new platforms but customers are free to use other | ||
| tools like Terraform/AWS CloudFormation to do the above mentioned tasks. Read more about AWS implementation details [here](https://github.com/openshift/cloud-credential-operator/blob/master/docs/ccoctl.md) | ||
|
|
||
| We also plan to have a detailed documentation in place to guide new cloud providers to implement *manual* mode. | ||
|
|
||
| To upgrade a cluster we need to execute following steps: | ||
| * Examine the CredentialsRequests in the new OpenShift release. Check if permissions in the existing CredentialsRequest | ||
| have changed. | ||
| * Create/update credentials in the underlying cloud provider, and also create/update Kubernetes Secrets in the correct | ||
| namespaces to satisfy all CredentialsRequests in the new release. | ||
| * CCO sets `Upgradeable=False` when in manual mode. Set an appropriate annotation `cloudcredential.openshift.io/upgradeable-to` | ||
| to a new upgradable version to allow upgrade. | ||
|
|
||
|
Member
There was a problem hiding this comment. Can we add the exact steps for installation and upgrades (keeping the POC for AWS in mind) here?
Member
There was a problem hiding this comment. It will give more clarity around how the user experience is changing because of this.
Contributor
There was a problem hiding this comment. there is no requirement to use |
||
| All the above task will also be executable by *ccoctl*. Details in [this](https://issues.redhat.com/browse/CCO-84) card. | ||
|
|
||
| ### Open Questions | ||
|
|
||
| ### Test Plan | ||
|
|
||
| We plan to have a e2e test that will externally set up a credentials management infrastructure and then kickstart | ||
| install in a *manual* mode. | ||
|
|
||
| ### Graduation Criteria | ||
|
|
||
| #### Dev Preview -> Tech Preview | ||
|
|
||
| - Ability to utilize the enhancement end to end | ||
| - End user documentation, relative API stability | ||
| - Sufficient test coverage | ||
| - Gather feedback from users rather than just developers | ||
| - Enumerate service level indicators (SLIs), expose SLIs as metrics | ||
| - Write symptoms-based alerts for the component(s) | ||
|
|
||
| #### Tech Preview -> GA | ||
|
|
||
| - More testing (upgrade, downgrade, scale) | ||
| - Sufficient time for feedback | ||
| - Available by default | ||
| - Backhaul SLI telemetry | ||
| - Document SLOs for the component | ||
| - Conduct load testing | ||
|
|
||
| **For non-optional features moving to GA, the graduation criteria must include | ||
| end to end tests.** | ||
|
|
||
| #### Removing a deprecated feature | ||
|
|
||
| - Announce deprecation and support policy of the existing feature | ||
| - Deprecate the feature | ||
|
|
||
| ### Upgrade / Downgrade Strategy | ||
|
||
|
|
||
| ### Version Skew Strategy | ||
|
|
||
| ## Implementation History | ||
|
|
||
| We currently have a work-in-progress CLI tool [ccoctl](https://github.com/openshift/cloud-credential-operator/blob/master/docs/ccoctl.md) | ||
| to create and manage cloud credentials outside the cluster for AWS cloud. The design details of this tool is discussed above. | ||
|
|
||
| ## Drawbacks | ||
|
|
||
| * Taking the credentials management outside the cluster will create additional overhead for the customer to make sure all | ||
| the required infrastructure is in place before starting the installation process. Current tooling we have only supports | ||
| the AWS cloud, we do not have anything planned for other cloud providers. | ||
| * Push-button upgrades will not work in *manual* mode as the cluster no longer has the admin credentials to mint credentials | ||
| (with fine-grained permissions) for in-cluster components. | ||
|
|
||
| ## Alternatives | ||
|
|
||
| ## Infrastructure Needed [optional] | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these really three separate stories? They seem more like steps of the same story. In other words, is it meaningful for a user to do Story 1 by itself?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, I agree that this seems like a single "extract and locally reconcile" use-case. Probably also need an "and destroy when I'm done with them" tail for that use case.