Support service type Load-Balancer

[pires]

It wasn't clear to me while experiment with this that Type: LoadBalancer is not supported so leaving this here in the hopes of helping someone hitting the same issue.

[shettyg]

We probably don't even log a warning when someone uses "Type: LoadBalancer". I will send in a patch to do proper logging.

[lanoxx]

Ouch, I just spend a couple of hours trying to debug my broken services that I had setup for ingress, until I just found this issue. Is there any plan to fix this soon or is this a larger issue. What is the recommended way to setup ingress with OVN-Kubernetes?

[shettyg]

So, I know that ingress works with ovn-kubernetes because the guys at openshift and vmware have previously confirmed it.

I have never set it up. So, I don't really understand why service type load-balancer is important for ingress to work. My understanding is that

1. Your ingress POD will run with host network.
2. Packet arrives to that host IP + port
3. Ingress POD will NAT it to a Service IP.
4. There is a route in host which says that the service IP is reachable from a OVS interface (the one we setup on each node)
5. Linux pushes the packet to that interface
6. From here, it should just work like a regular service IP that OVN supports.

@dcbw can add more color or correct me.

[lanoxx]

@shettyg There are several ways to setup ingress. With a service of type load balancer Kubernetes will automatically deply an AWS Classic Load Balancer that forwards traffic to a service port. On the service port you could then have an ingress controller listening, such as traefik. Once that is setup, you can create Ingress resources in Kubernetes that will cause the ingress controller to route traffic to the respective application services.

Another way that I tried was to use the aws-alb-ingress-controller, it also registers a classic load balancer but it will forward traffic to a service of type node port and automatically updates the load balancers target groups if nodes change.

I have not yet attempted to run a pod with host network enabled, that would be essentially the same as a service of type node port, but without the indirection that services provide. So I am essentially tying my ingress to a specific node (which could be a problem if that node dies or the pod gets scheduled to a different node). Please, correct me if that statement is wrong.

[shettyg]

There are several ways to setup ingress. With a service of type load balancer Kubernetes will automatically deply an AWS Classic Load Balancer that forwards traffic to a service port. On the service port you could then have an ingress controller listening, such as traefik. Once that is setup, you can create Ingress resources in Kubernetes that will cause the ingress controller to route traffic to the respective application services.

Another way that I tried was to use the aws-alb-ingress-controller, it also registers a classic load balancer but it will forward traffic to a service of type node port and automatically updates the load balancers target groups if nodes change.

I still do not understand how it works. What is a service port? An example? Is it an IP:port? Is traefik running in a pod or host? So the first step is for traffic to reach traefik. I would like to understand that first. And wondering, how is it that OVN prevents it.

I have not yet attempted to run a pod with host network enabled, that would be essentially the same as a service of type node port, but without the indirection that services provide. So I am essentially tying my ingress to a specific node (which could be a problem if that node dies or the pod gets scheduled to a different node). Please, correct me if that statement is wrong.

I think, there is usually 2 pods running in 2 nodes (with specific node selector added when deploying pod). Not sure how the HA is enforced.

[lanoxx]

@shettyg My understanding is that a LoadBalancer service is basically just a service of type NodePort (e.g its an IP:port opened on the host), with the additional benefit that a load balancer is created through the cloud provider API and the load balancer configuration is wired to target traffic to the correct nodes. Since NodePort services appear to be supported I am not sure why LoadBalancer services are not.

Traefik is running in a pod. I configure it to expose its public listening port as a service of type load balancer. Kubernetes will then provision an AWS LoadBalancer that targets the node port of the traefik service.

The flow of traffic is then like this:

                                       +-> Node with Traefik pod    +---> Traefik Pod +---> Service Pod
                                       |           ^
                                       |           |
Inbound Traffic +-> AWS Load Balancer ++           |
                                       |           |
                                       |           +
                                       +-> Node without Traefik pod

[lanoxx]

@shettyg

Here is some more example, if I deploy my traefik ingress controller then I get the following service (kubectl get svc --all-namespaces):

NAMESPACE     NAME                   TYPE           CLUSTER-IP    EXTERNAL-IP                                                                          PORT(S)                                     AGE
kube-system   traefik-external       LoadBalancer   10.96.0.204   af00174b939a011e9ba19022b2e3974d-2088150912.eu-central-1.elb.amazonaws.com           80:30466/TCP,443:32733/TCP,8080:32268/TCP   5m34s
[...]

Next I check on the Linux node if ovs knows about the node ports that the service opens (e.g. withovs-ofctl dump-flows brens5), I am expecting to see ports 30466 and 32268 to appear in the output but they are not listed:

cookie=0x0, duration=1099559.417s, table=0, n_packets=9154376, n_bytes=1185501598, priority=100,ip,in_port="k8s-patch-brens" actions=ct(commit,zone=64000),output:ens5
cookie=0x0, duration=1099559.414s, table=0, n_packets=14101419, n_bytes=6936795811, priority=50,ip,in_port=ens5 actions=ct(table=1,zone=64000)
cookie=0x0, duration=1099559.362s, table=0, n_packets=14, n_bytes=859, priority=100,tcp,in_port=ens5,tp_dst=32114 actions=output:"k8s-patch-brens"
cookie=0x0, duration=1099559.359s, table=0, n_packets=2217452, n_bytes=168722795, priority=100,tcp,in_port=ens5,tp_dst=32306 actions=output:"k8s-patch-brens"
cookie=0x0, duration=1099561.029s, table=0, n_packets=5756529, n_bytes=659258214, priority=0 actions=NORMAL
cookie=0x0, duration=1099559.411s, table=1, n_packets=8400761, n_bytes=3935118050, priority=100,ct_state=+est+trk actions=output:"k8s-patch-brens"
cookie=0x0, duration=1099559.405s, table=1, n_packets=0, n_bytes=0, priority=100,ct_state=+rel+trk actions=output:"k8s-patch-brens"
cookie=0x0, duration=1099559.401s, table=1, n_packets=5701146, n_bytes=3001704113, priority=0 actions=LOCAL

[lanoxx]

@shettyg
I am not too familiar with the OVN code base, but while searching through the code, I found there are several place where logic is guarded explicitly against kapi.ServiceTypeNodePort, such as:

https://github.com/openvswitch/ovn-kubernetes/blob/master/go-controller/pkg/ovn/service.go#L58
https://github.com/openvswitch/ovn-kubernetes/blob/master/go-controller/pkg/cluster/gateway_shared_intf.go#L18
https://github.com/openvswitch/ovn-kubernetes/blob/master/go-controller/pkg/ovn/endpoints.go#L58
https://github.com/openvswitch/ovn-kubernetes/blob/master/go-controller/pkg/ovn/endpoints.go#L90

Given that service type LoadBalancer is essentially just a service type NodePort, with some additional extra bits, I am wondering if it would just be enough to extend the logic to include kapi.ServiceTypeLoadBalancer in those guards.

[shettyg]

Since NodePort services appear to be supported I am not sure why LoadBalancer services are not.

That is because, I simply did not understand what "LoadBalancer" service is - in spite of trying to read about it multiple times.

Given that service type LoadBalancer is essentially just a service type NodePort, with some additional extra bits, I am wondering if it would just be enough to extend the logic to include kapi.ServiceTypeLoadBalancer in those guards.

With service type nodeport, we add an LB entry in OVN database for every OVN gateway (which could mean every node). Looks like for ServiceTypeLoadBalancer, we should only do it for specific nodes?

[shettyg]

Reading, your comment, it looks like I misunderstood. Looks like what you are saying is that, it is the same as NodePort. You can simply expand the checks in code for ServiceTypeNodePort and see whether it solves your issue. If it does, we can look at adding it to code officially.

[girishmg]

@lanoxx I think we already handle what you are looking for. We handle ExternalIPs for a service in handleExternalIPs(). This method adds a LoadBalancer entry (mapping External IP to NodePort IPs) on the first L3 gateway router (the one with IP address 100.64.1.2).

[girishmg]

Here is some more example, if I deploy my traefik ingress controller then I get the following service (kubectl get svc --all-namespaces):

NAMESPACE     NAME                   TYPE           CLUSTER-IP    EXTERNAL-IP                                                                          PORT(S)                                     AGE
kube-system   traefik-external       LoadBalancer   10.96.0.204   af00174b939a011e9ba19022b2e3974d-2088150912.eu-central-1.elb.amazonaws.com           80:30466/TCP,443:32733/TCP,8080:32268/TCP   5m34s
[...]

Next I check on the Linux node if ovs knows about the node ports that the service opens (e.g. withovs-ofctl dump-flows brens5), I am expecting to see ports 30466 and 32268 to appear in the output but they are not listed:

Does running following command on your setup return anything?
ovn-nbctl find logical_router options:lb_force_snat_ip=100.64.1.2

If it does return an entry, then on that L3 gateway router we should have created a load balance entry mapping ExternalIP to NodePort IPs.

Also, AFAIK OVS doesn't support adding DNS names for physical IP. In your case External-IP is af00174b939a011e9ba19022b2e3974d-2088150912.eu-central-1.elb.amazonaws.com.

[lanoxx]

@shettyg

You can simply expand the checks in code for ServiceTypeNodePort and see whether it solves your issue. If it does, we can look at adding it to code officially.

This did work to some degree and partially solved my problem. I searched through the code for all occurences of ServiceTypeNodePort and expanded that with a logical OR for ServiceTypeLoadBalancer. Now I have NodePorts exposed for services for type LoadBalancer and I can access them on my Linux minion. The NodePorts also appear on the Windows machine when I run ovs-ofctl dump-flows "vEthernet (Ethernet 2)", but connecting to the ports only gives me timeouts.

@girishmg I don't quite understand the mechanism you are describing, but I think we are talking about two different things.

Kubernetes supports a special service type called LoadBalancer that works in connection with the cloud provider option of Kubernetes. What it does is to invoke the cloud provider (in my case AWS) and create a native load balancer in the cloud. For AWS a classic load balancer is created. The DNS name for that load balancer is assigned by AWS automatically and Kubernetes only picks up DNS name and displays it in the output of kubectl get svc or kubectl get ingress.

The mechanism you are suggesting seems to be about something that OVN supports by itself. Where can I read more about that?

[shettyg]

This did work to some degree and partially solved my problem. I searched through the code for all occurences of ServiceTypeNodePort and expanded that with a logical OR for ServiceTypeLoadBalancer. Now I have NodePorts exposed for services for type LoadBalancer and I can access them on my Linux minion. The NodePorts also appear on the Windows machine when I run ovs-ofctl dump-flows "vEthernet (Ethernet 2)", but connecting to the ports only gives me timeouts.

Can you access the pod from linux minion's nodeport?

[lanoxx]

Yes. The pod that I am trying to access is a traeffic-ingress-controller. It runs on the Linux minion and I can access it via the Linux minion's NodePort.

Here is an example. The LoadBalancer service exposes a traefik reverse proxy which runs on port 80 via NodePort 30466:

NAMESPACE     NAME                   TYPE           CLUSTER-IP    EXTERNAL-IP                                                                          PORT(S)                                     AGE
kube-system   traefik-external       LoadBalancer   10.96.0.204   af00174b939a011e9ba19022b2e3974d-2088150912.eu-central-1.elb.amazonaws.com           80:30466/TCP,443:32733/TCP,8080:32268/TCP   10d

The traefik reverse proxy forwards requests to the Kubernetes dashboard which runs on the Linux Minion (e.g. 172.33.69.225):

NAMESPACE     NAME                                         READY   STATUS    RESTARTS   AGE   IP            NODE                                             NOMINATED NODE
kube-system   kubernetes-dashboard-6948557947-sfwdv        1/1     Running   1          22d   10.245.0.6    ip-172-33-69-225.eu-central-1.compute.internal   <none>

When I access the NodePort of the Linux Minion (e.g. 172.33.69.225:30466) from my master node (e.g. 172.33.75.8), then I can get a response from the dashboard:

$ curl http://172.33.69.225:30466
<!doctype html> <html ng-app="kubernetesDashboard">
    [...]
</html>

If I try the same from the Windows node (e.g. 172.33.66.23:30466), I get a timeout.

[shettyg]

Is this a problem just for LoadBalancer or does regular NodePort not work from Windows either?

[lanoxx]

I have a NodePort service exposed on the Windows node which also does not work. It is visible in the output of ovs-ofctl dump-flows but it does not return an traffic. Also see issue: #611

[lanoxx]

I think this was fixed by commit: 9b9d146

[danwinship]

AWS-style LoadBalancers were fixed in #738. GCP-style LoadBalancers were fixed in #1126.