Support for ExternalName's used for cross-namespace Services

[r4j4h]

Hi there,

Summary

ExternalName services do not get endpoints created, and as far as I can tell the Ingress controller depends exclusively on the Endpoints created from Services, so ExternalName services cannot currently be used to reach a Service across namespaces using the Nginx Ingress Controller.

Background

Today we tried including a surrogate Service in the local namespace referencing a service in another namespace via an ExternalName pointed to its in-cluster DNS:

kind: Service
apiVersion: v1
metadata:
  name: other-service
  namespace: current-namespace
spec:
  type: ExternalName
  externalName: other-service.other-namespace.svc.cluster.local
  ports:
  - port: 443

When resolving that service within a pod, the appropriate Pod IP from the other namespace is returned.

When nginx comes to handle the service, however, it logs "no endpoints found" for the service and NewUpstreamWithDefaultServer() marks the upstream as 127.0.0.1:8181. When accessing the path through nginx, an error is given.

This is to be expected, as externalName Services do not get endpoints created, but the Ingress controller depends exclusively on the Endpoints created from Services. This latter part was surprising to me.

Proposal

This issue is to bring up this point and see what thoughts are around the idea of having the nginx Ingress Controller behave differently when no Endpoints can be found to support this case.

My thought is that there are two locations we could try to resolve the name using DNS before falling back to the current behavior:

• kubernetes-ingress/nginx-controller/nginx/configurator.go

  Lines 580 to 602 in 8aca003

  	func (cnf *Configurator) createUpstream(ingEx *IngressEx, name string, backend *extensions.IngressBackend, namespace string, stickyCookie string, lbMethod string) Upstream {
  	var ups Upstream
  	if cnf.isPlus() {
  	ups = Upstream{Name: name, StickyCookie: stickyCookie}
  	} else {
  	ups = NewUpstreamWithDefaultServer(name)
  	}
  	endps, exists := ingEx.Endpoints[backend.ServiceName+backend.ServicePort.String()]
  	if exists {
  	var upsServers []UpstreamServer
  	for _, endp := range endps {
  	addressport := strings.Split(endp, ":")
  	upsServers = append(upsServers, UpstreamServer{addressport[0], addressport[1]})
  	}
  	if len(upsServers) > 0 {
  	ups.UpstreamServers = upsServers
  	}
  	}
  	ups.LBMethod = lbMethod
  	return ups
  	}

• kubernetes-ingress/nginx-controller/nginx/nginx.go

  Lines 135 to 140 in 8aca003

  	func NewUpstreamWithDefaultServer(name string) Upstream {
  	return Upstream{
  	Name: name,
  	UpstreamServers: []UpstreamServer{UpstreamServer{Address: "127.0.0.1", Port: "8181"}},
  	}
  	}

A similar discussion on the contrib nginx Ingress controller has led me to think modifying the proxy_pass directive directly is more correct by convention for pointing to DNS. If so, then the changes needed would affect here (and in the respective Nginx Plus-variant template)

kubernetes-ingress/nginx-controller/nginx/templates/nginx.ingress.tmpl

Lines 92 to 96 in 8aca003

	{{if $location.SSL}}
	proxy_pass https://{{$location.Upstream.Name}}{{$location.Rewrite}};
	{{else}}
	proxy_pass http://{{$location.Upstream.Name}}{{$location.Rewrite}};
	{{end}}

For reference on ExternalName Services, see kubernetes/website#5822.

P.S. While researching this I learned about the newly added Mergeable Ingress Types support which may allow working around this issue without using ExternalName Services. I will update this with my findings after trying it.

[pleshakov]

@r4j4h ExternalName services is a valid use case that we don't currently support.

I think a simple implementation would involve changing getEndpointsForIngressBackend method: if a service has an external name, instead of trying to fetch its endpoints, which it doesn't have, add an endpoint manually in the form of DNS-name:port. In the end, for such a service, Configurator will generate an upstream with one server:

upstream some-service {
  server DNS-name:port;
}

Caveats:

• When NGINX is reloaded, NGINX will try to resolve that DNS name using the system DNS resolver (/etc/resolv.conf file). If NGINX fails to resolve the name, it will fail to reload.
• Once the name is resolved, it won't be updated until the next reload, even if the record expires based on its TTL value.

A different approach would involve resolving a DNS name inside a location (note that the resolver must be defined as well -- http://nginx.org/en/docs/http/ngx_http_core_module.html?#resolver):

location / {
        set $backend_servers backends.example.com;
        proxy_pass http://$backend_servers:8080;
    }

However, it has a drawback that you cannot configure things like http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive.

More info about DNS in NGINX -- https://www.nginx.com/blog/dns-service-discovery-nginx-plus/

Mergeable Ingress Types allows splitting an Ingress resource into multiple Ingress resources across a single or multiple namespaces.

[r4j4h]

That is a beautiful response @pleshakov! I appreciate the summary of caveats with those approaches. Mergeable Ingress Types sounds like the best way forward for our use case.

Regarding the status of this Issue, I am not sure if I should close it with this comment or not. I want to leave the choice of keeping it open to you as it sounds like one one hand we could establish support for ExternalName Services even with the drawbacks and that may be desirable, but on the other hand because Mergeable Ingress Types offer the benefits without the drawbacks, and even obviates the need for the ExternalName Service itself, pointing everyone to them instead may be cleaner and result in a smoother, more full-featured experience for everyone.

[pleshakov]

thanks. I think it is better to keep the issue open.
Also, there is another use case for ExternalName services -- routing traffic to the services outside of the Kubernetes cluster. That use cases cannot be covered by the Mergeable Ingress Types.

[Poseiden]

Does your ingress resource and svc in different namespace?

[calicoderco]

If I need to point some ingress to an AWS S3 bucket how do I do that? I have tried pointing it to an externalname service like the author of this issue did, but that led me to this issue. Is there any known, easy workaround?

[pleshakov]

@calicoderco external services are not supported in this Ingress Controller. However, we'll be adding this features early next year.

[sombralibre]

I've overcome the issue with this workaround:

kind: Service
apiVersion: v1
metadata:
  name: s3-downtime
spec:
  type: ExternalName
  externalName: downtime-xxx-xxx.s3-website-xxxx-1.amazonaws.com
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: downtime-foo-com
  annotations:
    kubernetes.io/ingress.class: "ingress-public"
    nginx.org/redirect-to-https: "True"
    nginx.org/location-snippets: |
      location / {
        proxy_set_header Host downtime-xxx-xxx.s3-website-xxxx-1.amazonaws.com;
        proxy_pass http://s3-downtime:80;
      }
spec:
  rules:
  - host: downtime.foo.com
    http:
      paths:
      - path: /
        backend:
          serviceName: s3-downtime
          servicePort: 80

It works as I expected, however will be fine to have a more native way to implement configurations like this.

[flexchar]

@sombralibre Can you elaborate on your answer?

I am specifically trying to set up Ingress to route to services in separate namespaces. I have everything (pod+service) working in a namespace and I create main ingress + service of type ExternalName, however ingress complains that:
service "default/my-service" is type "ExternalName", expected "NodePort" or "LoadBalancer";

I really trying to wrap my head around but feels as if I keep beating the dead horse. I have multiple applications that needs similar resources, for ex. Laravel and Magento uses redis. Namespaces solves it perfectly and keeps it organized. But then I wanna use single and simple Ingress to serve them over magento.example.com and laravel.example.com etc.