Solicit feedback on supporting upstreams with non-routable Pod/VM IPs

[rosskukulinski]

At launch, Gimbal requires that discovered Application VMs (OpenStack) and Pods (Kubernetes) must have routable IPs that can be reached from the Gimbal cluster. While this is sufficient for our initial user and other users with flat IP namespaces, this will likely prohibit other common scenarios.

We should extend Gimbal to support Kubernetes and OpenStack deployments that do not provide routable IPs. This could include clusters that use an overlay network (e.g. Weave or Flannel) or that simply do not provide routable IPs.

One proposed solution would be to configure a GRE tunnel per upstream cluster.

This goal of this issue is to solicit feedback from the community about their deployments and use-cases so that we can design a viable solution.

[ncdc]

One possible option - set up 1+ contour instances in each upstream using hostNetwork: true, then forward from the routing cluster to the upstream cluster using the IP of the node in the upstream cluster where contour/envoy is running.

[alexbrand]

Thanks @ncdc - this is one approach I was thinking about. I think of it as "tiered ingress", where gimbal ingress proxies send traffic to backend cluster ingress proxies. The only detail here is that the discovery components would have to sync endpoints with the node IPs instead of the pod IPs.

[ncdc]

The only detail here is that the discovery components would have to sync endpoints with the node IPs instead of the pod IPs.

Exactly. I'd be happy to donate my demo code that does this :-)

[stevesloka]

I wrote up a design doc here to further discuss the idea of "edge" proxies: https://docs.google.com/document/d/1kCPeX3QDeK2s7PX91prkFSPufIb17EQQCZrBE2BUM2w/edit?usp=sharing

[jcrowthe]

Hey, I am also very interested in this work for an alternative to routable pod IP's. Do we have an update?