Skip to content

OpenStack Bring Your Own External Connectivity #342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 2 commits into from
Jul 16, 2020
Merged

OpenStack Bring Your Own External Connectivity #342

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a31711d
A feature proposal to allow openstack users installing openshift to c…
May 26, 2020
369a51a
remove dynamic port allocation
Jul 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 130 additions & 0 deletions enhancements/installer/openstack-no-fip-install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
title: openstack-bring-your-own-external-connectivity
authors:
- "@egarcia"
- "@adduarte"
reviewers:
- "@mandre"
- "@pierre"
approvers:
- "@mandre"
- "@pierre"
creation-date: 2020-5-26
last-updated: 2020-5-26
status: implementable
---

# OpenStack BYO External Connectivity

## Release Signoff Checklist

- [x] Enhancement is `implementable`
- [x] Design details are appropriately documented from clear requirements
- [x] Test plan is defined
- [x] Graduation criteria
- [ ] User-facing documentation is created in [openshift/docs]

## Summary

Numerous customers have requested the ability to run the OpenShift on OpenStack IPI installer without using Floating IP addresses.
In the installer, we use floating IPs as a way to set up external connectivity to the OpenShift cluster. Depending on how the OpenStack cluster
is set up, the use of floating IPs may be impossible.

## Motivation

### Goals

- Extend the installer to support provider networks
Copy link
Member

@mandre mandre Jul 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC we won't be able to fully support provider networks until we make the switch to the external cloud provider. That's because provider network likely does not have nova-metadata agent and it's currently a requirement for the in-tree openstack cloud provider.
After switching to the external cloud provider, we'll be able to drop the nova-metadata requirement and use config-drive instead.

Copy link

@jichenjc jichenjc Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I asked same question because the in tree hard code to metadata and don't support config drive

so the questions come to when the external cloud provider will be considered?

Copy link
Member

@mandre mandre Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd like to switch to external cloud provider in openshift 4.7. From what I understand, the in-tree cloud provider will be removed from the k8s code base in the 1.21 cycle, mapping to openshift 4.8 so we'll have to make the switch eventually. And we already have our hands full for 4.6.

- Allow the installer to be run without setting up external connectivity
Copy link
Contributor

@abhinavdahiya abhinavdahiya Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allow the installer to be run without setting up external connectivity

does this mean that endpoints like API, Ingress are only accessible to the internal network and not Internet or is it something else. because if it is the former we should keep in mind how this interacts with

/openshift-install explain installconfig.publish
KIND:     InstallConfig
VERSION:  v1

RESOURCE: <string>
  Publish controls how the user facing endpoints of the cluster like the Kubernetes API, OpenShift routes etc. are exposed. When no strategy is specified, the strategy is "External".

Copy link

@adduarte adduarte Jun 16, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regardless of installation setup, the real requirement is ip access from the host you are adding to the api endpoints. I can think of a half a dozen ways users can accomplished and setup the rquired routing, from the very complicated to the simplest. We usually set the routing up for the user in IP installations while UPI expects the user to take some action on the mater.
technically fip addresses are not required for instllation. They are only required to provide access to the cluster. the reason the user needs to provide it during a IPI installation is because the installer goes the "extra mile" and makes the association.
We are removing the "requirement" that fips be available, I don't think we are preventing the user from setting up any routing they want.

Copy link

@adduarte adduarte Jun 17, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think, after reading more on the customer requirements, and other, this should probably be changed to:
"Allow the installer to be run without the use of FIPs" The phrase "Allow the installer to be run without setting up external connectivity" could be read to mean that the installer will succeed without any external connectivity. But looking at the title the intent is that the user bring their own external connectivity, which funny enough could include using FIPs.
@iamemilio I think the phrase means the installer will NOT setup the external connectivity, but there should be external connecitvity, right? ( you state that below).


### Non-Goals

This enhancement does not intend to provide any guidance or features that go further than allowing customers to deploy without Floating IPs. Operations like attaching loadbalancers to the entrypoints, or any day 2 operations are beyond the scope of this enhancement. It is important to note that while we aim to support installs without external connectivity, we are not supporting installs on networks that are unable to reach the host cluster's OpenStack APIs and services.

Copy link

@adduarte adduarte Jun 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If external connectivity then it will up to the user to setup the appropriate private registries to provide the necessary images to deploy. We can probably assume the user will have to setup following instructions similar to here https://docs.openshift.com/container-platform/4.4/installing/install_config/installing-restricted-networks-preparations.html

## Proposal

In order to implement this feature fully, the following changes must be made:
1. Make `externalNetwork` an optional argument in the install-config when `machinesSubnet` is set
2. Make setting `lbFloatingIP` when no external network is passed enforced as invalid usage

### Design Details

#### Make `externalNetwork` an optional argument in the install-config when `machinesSubnet` is set

Implementing this can be done by changing the validations and the defaults so that a blank external network can be passed to the installer code. We would also disable all floating IP creation in the installer when this is the case.

#### Disable `lbFloatingIP` when no external network is passed

In the installer, the external network is used to provision floating IPs as well as the router. The router no longer gets created when using `machinesSubnet`, so the only changes that would need to be made are changes in the installers defaults and validations that allow the a blank `lbFloatingIP` to be pased. Then we will have to make a conditional block that does not attach the floating IP when `lbFloatingIP` is blank.

### User Stories

#### Provider Networks

As an enterprise OpenStack cluster administrator, I want to use the IPI installer but my cluster does not support Floating IPs. I need a way to install the OpenShift IPI installer on provider networks instead. I may also need to clear IP addresses I use on the provider network through an internal audit.

##### Example Usage

Using the proposed feature, the user would not set `externalNetwork` or `lbFloatingIP`. Then, they will set`machinesSubnet` to the ID of an existing subnet on a provider network that they want to install the cluster onto, and set the `networking.machineNetwork` to the CIDR of that subnet. They will also set custom ports IPs for the `apiVP` and `ingressVIP` to ensure the IP they take is available. Here is an example install config:
Copy link

@adduarte adduarte Jun 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couldn't we dynamically query the cidr using the subnet id? the equivalent of openstack command :
"openstack subnet show -c cidr -f value", then validate the choice of the apivp and ingress vip
This would assure the correct cidr and avoid user error

Copy link
Member

@mandre mandre Jun 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this should be one of the new validation mentioned in the proposal.
We need query the subnet CIDR to check that it matches the user-provided networking.machineNetwork.

Copy link
Author

@iamemilio iamemilio Jun 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already do that: https://github.com/openshift/installer/blob/b9c94af65d10fb4a3223a4552bfc4b088e4232a5/pkg/types/openstack/validation/platform.go#L25-L40

Copy link
Author

@iamemilio iamemilio Jun 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real improvement we should consider is auto-fill, but I think that should be a part of the validation epic


```yaml
apiVersion: v1
baseDomain: example.com
metadata:
name: test-cluster
networking:
machineNetwork:
- cidr: 10.0.1.0/17
platform:
openstack:
cloud: mycloud
computeFlavor: m1.s2.xlarge
machinesSubnet: fa806b2f-abcd-4bce-b9db-124bc64209bf
apiVIP: `10.0.1.19`
ingressVIP: `10.0.1.20`
pullSecret: '{"auths": ...}'
sshKey: ssh-ed25519 AAAA...
```

#### No Public External Access

As an administrator with an on-prem OpenStack cluster, I want to install OpenShift in a way that does not expose API and Ingress endpoints to an external network. Instead, I would prefer to attach my own loadbalancers to the cluster entrypoints post install.


### Test Plan

Unit tests and validations for this feature will be added to the installer to make sure that correct usage is enforced and that this feature does not hinder the usage of other features. To ensure GA readiness, it will be vetted by the QE team as well to make sure that it works with the followin use cases: self-signed certs, customer-provided networks, baremetal workers, scale-out, upgrades, and a restricted-network install.

### Graduation Criteria

This enhancement will follow standard graduation criteria.

#### Dev Preview -> Tech Preview

- Ability to utilize the enhancement end to end
- End user documentation, relative API stability
- Sufficient test coverage
- Gather feedback from users rather than just developers

#### Tech Preview -> GA

- Community Testing
- Sufficient time for feedback
- Upgrade testing from 4.3 clusters utilizing this enhancement to later releases
- Downgrade and scale testing are not relevant to this enhancement
- E2E testing is not necessary for this feature

### Infrastructure Needs

- OpenStack cluster with provider networks
- PSI

### Drawbacks

We have to be very careful about how we implement this. Making features that were previously required optional may confuse users.
We also need to be certian that users will not be able to mistakenly install a cluster that was intended to have external connectivity without it.
In order to guard against this, we intend to improved the documentation and validations.

#### Upgrade/Downgrade strategy

This feature will be released in 4.6, and will not be backported. We will also not plan to support migration of existing clusters to support this feature, since the changes are install time only.