Skip to content

IBM Public Cloud Support #202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 2 commits into from
Aug 4, 2020
Merged

IBM Public Cloud Support #202

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
46301f4
IBM Public Cloud Support
csrwng Feb 4, 2020
cb7728e
Add more reviewers
csrwng Feb 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
243 changes: 243 additions & 0 deletions enhancements/ibm-public-cloud/ibm-public-cloud-support.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,243 @@
---
title: IBM Public Cloud Support
authors:
- "@csrwng"
reviewers:
- "@derekwaynecarr"

@sttts sttts Feb 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expecting the owners of the control plane operators here, compare https://github.com/openshift/enhancements/pull/202/files#diff-8d49ecd990d312a72c0ffcdd1784ad05R92.

@csrwng csrwng Feb 5, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sttts did you mean a different pr? the link above is for this one

@sttts sttts Feb 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, meant this one. Just expected the owners of the mentioned operators to be informed about these plans by being reviewer of the enhancement.

@csrwng csrwng Feb 5, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, will add more reviewers

approvers:
- "@derekwaynecarr"
creation-date: 2020-02-04
last-updated: 2020-02-04
status: implementable
see-also:
replaces:
superseded-by:
---

# IBM Public Cloud Support

## Release Signoff Checklist

- [x] Enhancement is `implementable`
- [ ] Design details are appropriately documented from clear requirements
- [ ] Test plan is defined
- [ ] Graduation criteria for dev preview, tech preview, GA
- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)

## Summary

IBM Public Cloud deploys Kubernetes/OpenShift clusters by hosting their control plane
as pods in a central management cluster. OpenShift v3.11 is currently offered as a managed
service using this model. This type of setup allows the hosting of many
cluster control planes on a single Kubernetes cluster. Only the worker nodes are exclusive
to an individual user cluster and may exist in a different account than the central
management cluster.

Supporting this type of deployment in OpenShift 4.x requires a different approach to
installing OpenShift and managing its control plane. Instead of bootstrapping a cluster
control plane from scratch on virtual machines, a cluster control plane can be created
by applying a set of manifests to an existing cluster in a new namespace.

Changes are required in different areas of the product in order to make clusters deployed
using this method viable. These include changes to the cluster version operator (CVO), web
console, second level operators (SLOs) deployed by the CVO, and RHCOS.

@s-urbaniak s-urbaniak Aug 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please don't use SLO as an acronym here. It is very commonly used for "Service Level Objectives".

@csrwng csrwng Aug 5, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack, will do a follow-up to remove.


## Motivation

Enable OpenShift 4.x deployment on IBM Cloud.

### Goals

- Enable the deployment of OpenShift 4.x on IBM Public Cloud by providing the IBM
Public Cloud team the necessary tools to generate manifests needed for a hosted
control plane.
- Ensure that this deployment model remains functional through regular e2e testing
on IBM Public Cloud.

@sttts sttts Feb 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"regular" means we get a normal CI job in the openshift org making sure our control plane code changes don't break it?

@sttts sttts Feb 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the CI job be blocking for OpenShift PRs? E.g. if deployment topology changes and some apiserver suddenly does not serve certain APIs because they moves, their CI will break.

@csrwng csrwng Feb 5, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"regular" means we get a normal CI job in the openshift org making sure our control plane code changes don't break it?

That is the plan. We currently have a periodic job that creates clusters based on the 4.3 branches. One will be added for the master/4.4 branch.

Will the CI job be blocking for OpenShift PRs? E.g. if deployment topology changes and some apiserver suddenly does not serve certain APIs because they moves, their CI will break.

That will be harder. We would require capacity on the IBM Cloud to run that many jobs. Not sure that is feasible right now. The periodic job should block a release, but not individual PRs

@sttts sttts Feb 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We describe those apiserver changes (they are happening, now for oauth) in enhancements. The IBM team has to watch that repo to be informed.

- Make the necessary product changes to make IBM Public Cloud a fully supported
cloud provider.
- Ensure that clusters running on IBM Public Cloud are conformant OpenShift clusters
(pass Kubernetes and OpenShift conformance e2e tests). Fully document any tests that
must be skipped due to differences from a traditional deployment.

### Non-Goals

- Make hosted control planes a supported deployment model outside of IBM Public Cloud.

@cgwalters cgwalters Feb 7, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this confusing...IBM Public Cloud means a lot of different things, and a sort of baseline obvious one is having OpenShift support the default "self driving" path in their existing IaaS. But I guess we're doing hosted control plane first?

Maybe the enhancement should be called: "IBM Public Cloud Hosted Control Plane" ?

And one thing I would say here is that we should think of this "fairly" - if some other IaaS showed up and was willing to commit significant resources to maintaining a similar thing... clearly vast amounts of the design would likely be shared. But that can come later.

@csrwng csrwng Feb 7, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely, I think the things we learn from this work is something we can likely reuse in other cases. And perhaps at the proper time, make the pattern something standalone that's configured per provider. So definitely, this non-goal is a point-in-time statement.

@csrwng csrwng Feb 7, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this confusing...IBM Public Cloud means a lot of different things, and a sort of baseline obvious one is having OpenShift support the default "self driving" path in their existing IaaS. But I guess we're doing hosted control plane first?

At least in the foreseeable future, supporting the self-hosted path is not a priority afaik, but @derekwaynecarr can likely provide more insight into that.

- Define the automation needed on the IBM Public Cloud side to provision new clusters.
- Create an alternate installer for OpenShift.

## Proposal

Support for IBM Public Cloud will be added in 3 phases:

1. Beta: Initial rollout. Enables users to create OpenShift clusters
using RHEL 7 worker nodes. Nodes are labeled as both worker and master to support
SLOs that have masters in their node selectors. IBM provides out of tree cloud support
for storage, service load balancers. Ingress, image registry, storage and auth are
configured after initial provisioning (day 2). Calico is used for the SDN.
2. GA: In-tree support for ingress, image registry, storage, auth, load balancers.
3. Post-GA: Add support for RHCOS workers

In more detail:

### Beta

Enables an OpenShift cluster to be hosted on top of a Kubernetes/OpenShift cluster.

Given a release image, a CLI tool generates manifests that instantiate the control plane

@cgwalters cgwalters Apr 8, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OS upgrades for the worker nodes is owned by the customer? Or does IBM provide tooling for that? Are they using openshift-ansible?

In the "Post-GA" world with RHCOS...do we forsee trying to enable the MCO to manage upgrades for the workers w/RHCOS?

@wking wking Apr 8, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this covered in the Managed Workers section below?

@cgwalters cgwalters Apr 8, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second half is yes, thanks!

components so that they can be applied to a namespace. IBM provides the automation
that applies these manifests and instantiates worker nodes that add themselves to
the new cluster. Minting of kubelet certificates for these worker nodes is handled
by IBM automation.

Components that run on the management cluster include:

@s-urbaniak s-urbaniak Aug 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for cluster-monitoring: as apiserver monitoring is effectively being disabled when running in "ROKS mode" what is monitoring control plane components on the management cluster?

@csrwng csrwng Aug 5, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IBM is running their own monitoring solution on their management/tugboat clusters.

- etcd\+
- kubernetes apiserver
- kubernetes controller manager
- kubernetes scheduler
- openshift apiserver

@sttts sttts Feb 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has it been discussed what is needed to move that into the customer cluster?

@csrwng csrwng Feb 5, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It hasn't... when we first looked at this, however, we were in a catch 22 situation, where in order to be able to schedule pods we needed the openshift crds and controller functional.

- openshift controller manager
- cluster version operator
- control plane operator(s)\*
- oauth server\+

@spadgett spadgett Feb 5, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@csrwng Are there any lingering issues where the console backend rejects the OAuth server certificate in this deployment?

@csrwng csrwng Feb 7, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spadgett no issue at the moment. Thx!

- vpn server\+

\* - new component

\+ - provisioned by IBM

Once a control plane is running, the Kubernetes API server connects to a VPN server to
which worker nodes also connect. This enables the API server to access the pod and service
network of the worker nodes. This is needed for aggregated API services running on the
worker nodes.

The cluster version operator running on the management cluster connects to the Kubernetes
API server via the service network and starts applying manifests for SLOs that must run
on the worker nodes. These include (among others):

- network (dns, multus)
- ingress
- image registry
- OLM
- samples
- console

The CVO needs to skip manifests that instantiate control plane operators, machine API related
operators, and others that do not apply to this topology. These include:

- cluster-version-operator
- cluster-kube-apiserver-operator
- cluster-kube-controller-manager-operator
- cluster-kube-scheduler-operator
- cluster-openshift-apiserver-operator
- cluster-openshift-controller-manager-operator
- cluster-autoscaler-operator
- cluster-machine-approver
- insights-operator
- machine-config-operator
- cloud-credential-operator
- cluster-authentication-operator

Components/Changes needed for this to work:

#### OpenShift ROKS Toolkit CLI

CLI tool that takes a release image and a configuration file as input and produces a set
of manifests that provision a control plane that runs on an existing Kubernetes cluster.
PKI certificates are not generated by the tool. Manifests of control plane components
expect certificates to have been generated before provisioning and to exist as secrets
and configmaps on the control plane namespace following a predefined contract.

This tool is invoked by IBM Public Cloud provisioning code to when instantiating a new cluster.

#### Control Plane Operator
An operator that:
- Projects global configuration into the target cluster. Watches
and overwrites any changes to global configuration initiated inside the user
cluster.
- Ensures that the Kubernetes Controller Manager CA bundle includes self-signed CAs
generated by the router and the service CA operator.

#### Cluster Version Operator Changes
In phase I the cluster version operator excludes operator manifests that
have an annotation in the form:

`exclude.release.openshift.io/[identifier]=true`

whenever a global variable in the form

`EXCLUDE_MANIFESTS=[identifier]`

is present. For IBM public cloud, the value of \[identifier\] is
`internal-openshift-hosted`

#### Excluded Manifests
Manifests for the components that should be skipped by the CVO whenever
the `EXCLUDE_MANIFESTS=internal-openshift-hosted` environment variable is
present, should include the annotation to exclude them. In most cases, only the
minimum set of manifests to allow skipping the component should be annotated.
However, in the case of the Machine API and Machine Configuration operators,
the CRDs that represent machines, machinesets and autoscalers should also be
skipped. Monitoring alerts for components that do not get installed in the user

@lilic lilic Aug 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part won't be that easy, unless we get provided with a list of it. Also it will not fire if there are no metrics for those components, so don't think its a problem.

@csrwng csrwng Aug 5, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what we addressed with openshift/cluster-monitoring-operator#705
No other alerts related to control plane components have surfaced.

cluster should also be skipped where possible.

#### Console Changes
The console should not report the control plane as being down if no metrics

@s-urbaniak s-urbaniak Aug 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all edge cases for disabling monitoring of control plane components on the worker clusters covered in openshift/cluster-monitoring-operator#705 ?

@csrwng csrwng Aug 5, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far yes

datapoints exist for control plane components in this configuration.

@spadgett spadgett Feb 5, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, this shouldn't be an issue.

cc @rawagner @andybraren

@andybraren andybraren Feb 5, 2020

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe Control Plane components would be shown as "Not available" if that's acceptable (better than "Down").

image

@spadgett spadgett Feb 5, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andybraren We probably need to remove that since we never expect to have control plane metrics. It's misleading to say not available.

@rawagner rawagner Feb 6, 2020 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, created an issue to track this for 4.5 Dashboards https://issues.redhat.com/browse/MGMT-438


### GA

For GA, there is more native support for IBM Public Cloud as a supported
cloud provider in the product. Operators for control plane
components (based on the control plane self-hosted operators) are deployed on
the management cluster.

Components and changes required for this phase:

#### Control Plane Operators
Break up the single Beta control plane operator into different operators that
vendor code from the self-hosted control plane operators in the product. The vendored
code should include config observers that assemble a new configuration for their
respective control plane components. This will ensure that drift in future versions
is kept under control and that a single code base is used to manage control plane
configuration.

@sttts sttts Feb 5, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this beta control plane operator?

@csrwng csrwng Feb 5, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're working on it, will be added to the current hypershift-toolkit repo. For the second phase we will create separate repos for each of the control plane controllers.


#### Cluster Version Operator Changes
The CVO should support the concept of an install profile that allows selecting manifests
based on which profile is in effect. This should make it possible to modify the node
selector of SLOs that are installed on IBM Public Cloud clusters to target worker nodes.
(See separate proposal for installation profiles).

#### Ingress Operator
Add native support for IBM Public Cloud as a cloud provider.

#### Storage Operator
Create storage classes needed for IBM Public Cloud.

#### Image Registry Operator
Support IBM Public Cloud natively by provisioning the appropriate storage (RWO PVC)

#### OAuth Server
Support IBM public cloud as a provider.

### Post-GA

Add support for managed RHCOS nodes.

#### Managed Workers
RHCOS adds support for bootstrapping on IBM Public Cloud. The MCO is added to
the components that get installed on the management cluster. This enables upgrading
of RHCOS nodes using the same mechanisms as in self-hosted OpenShift.

@wking wking Apr 8, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So compute nodes still run machine-config daemons and cluster admins can write MachineConfig entries, create MachineSets, and all that good stuff? They just don't have any objects representing or control over the control-plane machines?


## Design Details

### Test Plan


### Graduation Criteria


### Upgrade / Downgrade Strategy


### Version Skew Strategy