Consider supporting Service.Type=LoadBalancer

[SEJeff]

I'd envision it using some magic such as ECMP, but it doesn't much matter tbh how it works:

This project from google is a P.O.C. of doing something similar. This gives us using bare metal a way to have load balanced service ips using dedicated vips.

A less-elegant alternative is the keepalived-cloud-provider project. When doing a single AS per rack topology (like we are), this means you have to pin instances of this per rack, which is painful and not as resilient when standard BGP does a better job.

[murali-reddy]

Its not hard to implement given Kube-router already does similar functionality for external IP's. Make sense to support it for completion.

[murali-reddy]

I will take a shot at it. Adding support for --advertise-loadbalancer-ip is stright forward. How do we do IP address pool management for the services:

• with out taking any dependecy on etcd (ConfigMaps?)
• avoiding concurrency issue while allocating ip for the service
  any suggestions @bzub @SEJeff ?

[SEJeff]

Playing devil's advocate, how does the service ip range get handled? If I remember (without looking), it is managed by the controller manager. Does that just create Endpoint objects serially more or less?

One approach for this is something like Endpoint that is a CRD and a single (1 per cluster) controller to manage it. That doesn't omit the dependency on etcd, but kubernetes can't run without etcd, so why avoid it?

[murali-reddy]

kubernetes can't run without etcd, so why avoid it?

I meant, Kube-Router not taking any dependency on etcd, like what calico does.

[danderson]

As a basic thing, you could just make kube-router program IPVS when it sees a Service with status.loadBalancer.ingress[*].ip . That field is what kube-proxy looks at to program correct routing for LoadBalancer services. The IPVS behavior should be exactly the same as externalIPs, so it should be a pretty trivial change in kube-router.

Once you've done that, kube-router should be compatible (at a basic level) with MetalLB, and also with all cloud providers like GCP, where a platform-specific component handles IP allocation.

And then you can separately decide if you want to implement your own IP allocators, or just say that other services like MetalLB provide that :)

[murali-reddy]

@danderson thanks for your comments. Your input make perfect sense.

We will also revisit metallb integration to make it work with Kube-Router. There are users who wish to use kube-router just as service proxy so there is no reason why kube-router cannot work with metal LB (as BGP is no longer in conflicting point).

[ConnorJC3]

I see that there is support introduced for MetalLB in BGP mode with #354 (and #355). However, could I suggest adding support for the ARP mode as well?

I know that MetalLB's ARP mode has scaling issues - however I am in a scenario where the network is not controlled by me and does not support BGP. Therefore, could a flag be added that just treats LoadBalancer service IPs as if they were an externalIP field?

Obviously this will be less reliable/HA than the BGP method, but in some scenarios it's the only option.

[murali-reddy]

@ccatlett2000 You should be able to use MetalLB ARP mode. kube-router does not advertise service's load balanacer IP's if not expllictly asked to do so i.e) by enabling --advertise-loadbalancer-ip flag.
Kube-router configures IPVS with load balancer IP. Please try and see.

cc @jjo

[jjo]

Indeed WfM, on my raspberry pi2 cluster fwiw.

Linking manifests for both (just note the arm image(s) for kube-router):

• kube-router --advertise-loadbalancer-ips=true -> https://gist.github.com/jjo/8c616aaf795284bb5b85d02143745f63
• metallb w/ARP mode -> https://gist.github.com/jjo/bac1e4cd12c854fc58ca2de084b3b874

[ConnorJC3]

@murali-reddy Ah, I read this blog post and because of it I was under the impression that IPVS proxy wouldn't be created for LoadBalancer services. However, if that isn't the case I will certainly try it out when I set up this cluster today/tomorrow.

[jjo]

@ccatlett2000: you're correct, indeed --advertise-loadbalancer-ips was just recently added ( PR #354 ).

[vladimirtiukhtin]

kube-router 0.2.0-beta.5 works perfect on 1.9.2 bare metal cluster with MetalLB and --advertise-loadbalancer-ips=true flag. All flags I am using are:

--kubeconfig=/etc/kubernetes/kube-router.conf --run-firewall=true --run-router=true --run-service-proxy=true --advertise-loadbalancer-ip

I have stopped using kube-proxy at all

[murali-reddy]

@tiukhtinvladimir thanks for letting us know. We are pretty close to get 0.2.0 release out. Once released will update the documentation on kube-router and metalb compatibility.

/cc @danderson @jjo

[danderson]

Great news! Once 0.2.0 is released, I'll write documentation on the integration for MetalLB, and document which kube-router configurations are supported.

One clarification for me: if you set --advertise-loadbalancer-ips=true, what exactly happens? Does it do both dataplane programming on nodes, and advertising the service over BGP to external peers? Is the latter optional, i.e. if I don't want to peer with external peers, will you still program the node dataplane (i.e. same behavior as kube-proxy) ?

[jjo]

One clarification for me: if you set --advertise-loadbalancer-ips=true, what exactly happens? Does it do both dataplane programming on nodes, and advertising the service over BGP to external peers?
Is the latter optional, i.e. if I don't want to peer with external peers, will you still program the node dataplane (i.e. same behavior as kube-proxy) ?

If --advertise-loadbalancer-ips is set, host node IP/IPVS ("dataplane programming") always happens, then BGP advertising if peers (--peers-... args) are set.

[wmturner]

Is there any update on the documentation for using metallb w/ kube-router? @danderson

[aauren]

@murali-reddy Do you still have interest in supporting loadbalancer service types in kube-router now that metallb seems to be a reasonable alternative?

Maybe the conclusion to this ticket is just some some documentation on partnering kube-router with metallb?

[vladimirtiukhtin]

@aauren a pair of kube-router and metallb works for me, but obviously I would prefer to have all in one, considering all required BGP functionality already implemented in kube-router

[murali-reddy]

@murali-reddy Do you still have interest in supporting loadbalancer service types in kube-router now that metallb seems to be a reasonable alternative?

@aauren Yes. Except for IPAM (assigning the IP for the loadbalancer service from the pool of IP's) all functionality is already there in kube-router. Its just users started using kube-router along with metallb so this issue was not prioritized. However as mentioned having nativley in kube-router means one more less moving piece.

Maybe the conclusion to this ticket is just some some documentation on partnering kube-router with metallb?

We did that already. We have some documentation https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#advertising-ips

But somehow original documentation from #354 is lost

[aauren]

Closed via #1501