Skip to content

metallb: Update operator, CRD, and BGP stack plans #772

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 1 commit into from
May 25, 2021
Merged

metallb: Update operator, CRD, and BGP stack plans #772

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 56 additions & 17 deletions enhancements/network/on-prem-service-load-balancers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ custom manner. For more information on the existing `IngressIPs` feature:

* https://github.com/openshift/api/blob/master/config/v1/types_network.go#L99
* https://github.com/openshift/openshift-docs/pull/21388
* https://docs.openshift.com/container-platform/4.7/networking/configuring_ingress_cluster_traffic/configuring-externalip.html

### Goals

Expand Down Expand Up @@ -229,35 +230,55 @@ doing all of your testing in a virtual network built on top of it.

#### Operator

This is the first of two alternatives for how we might integrate MetalLB in
OpenShift.

We must also create an operator for MetalLB. We should develop an operator
that is generally useful to the MetalLB community. We should also have an
OpenShift version of this operator for our use.
that is generally useful to the MetalLB community. The upstream operator has
been proposed [here](https://github.com/metallb/metallb/pull/859). We should
also have an OpenShift version of this operator for our use, which would
contain any needed OpenShift specific customizations.

It is assumed that the MetalLB operator would be managed by OLM as an optional
additional component to be installed on on-premise clusters. However, in the
[ROADMAP.md
The MetalLB operator will be managed by OLM as an optional additional component
to be installed on on-premise clusters. However, in the [ROADMAP.md
document](https://github.com/openshift/enhancements/blob/master/ROADMAP.md),
there is an item to "Front the API servers and other master services with
service load balancers". If this functionality is required at install time,
the details on management of this operator may be revisited.
the details on management of this operator may be revisited, since MetalLB may
Copy link

@dcbw dcbw May 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@russellb FYI @aojea has plans to make this not required, at least for Go-based things, by making client-go able to accept multiple API servers and balance/fail-over when required, instead of funneling into a single point of failure like the current keepalived/haproxy metal model or the slow-as-molasses-to-update cloud LB model, both of which we've had tons of problems with.

Copy link
Contributor Author

@russellb russellb May 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! that's good to know.

I think the impact of that would mean it's less likely that MetalLB would ever be a required component.

become a hard requirement in some environments.

There is a start of a
[metallb-operator](https://github.com/cybertron/metallb-operator) available and
a [video demo](https://www.youtube.com/watch?v=WgOZno0D7nw).

#### Alternative Integration: Cloud Controller Manager
#### Configuration Interfaces

MetalLB's current configuration interface is a single `ConfigMap`. We only
want to support CRD based configuration through OpenShift. The ultimate goal
will be to enhance MetalLB with a native CRD based configuration interface.
The design work has begun here:

https://github.com/metallb/metallb/pull/833

Since this is an upstream effort, the completion of this work may take longer
than the timeline for when we want to ship the first piece of MetalLB support
for OpenShift, which will be to support MetalLB in layer2 mode only.

An alternative integration approach would be via a cloud controller manager
(CCM). An example of this is the [packet.net
CCM](https://github.com/packethost/packet-ccm), which ensures MetalLB is
deployed and also configures it properly to work in packet.net’s BGP
environment.
To mitigate this timing risk, we will consider adding a minimal configuration
CRD to the MetalLB operator which only covers the configuration needed for
layer2 mode. This will buy us some additional time to determine whether we will
build a more complete interface within the core MetalLB project, or whether we
will continue to extend the configuration interface provided by the operator.

These integration options must be explored in more detail as part of a more
detailed integration proposal.
The operator will reconcile this minimal layer2-only configuration CRD by
creating a MetalLB ConfigMap.

#### FRR for BGP

We have concerns about the long term viability of the built-in BGP stack in
MetalLB. We have multiple features we'd like to add to the BGP stack and feel
the project is better served by relying on an external BGP implementation. We
are actively working upstream to propose the option of using FRR as the BGP
stack behind MetalLB.

https://github.com/metallb/metallb/pull/832

### Risks and Mitigations

Expand Down Expand Up @@ -375,6 +396,24 @@ These risks to version skew must be mitigated through upstream community
engagement and contributions, as well as informed management of upgrades in the
MetalLB operator.

### Graduation Criteria

This feature is expected to be released as GA (layer2 only as a first release),
and later BGP once the FRR integration and complete CRD based configuration
interface are complete.

#### Dev Preview -> Tech Preview

N/A

#### Tech Preview -> GA

N/A

#### Removing a deprecated feature

N/A

## Implementation History

* (May, 2020) - Technical due diligence and upstream engagement beginning
Expand Down