From f6645447d039af8e995e55033645b8282f3ae4c0 Mon Sep 17 00:00:00 2001 From: longwuyuan Date: Sun, 21 Jul 2024 16:53:40 +0530 Subject: [PATCH] added real-client-ip faq --- docs/faq.md | 79 ++++++++++++++++++- docs/user-guide/miscellaneous.md | 2 +- docs/user-guide/retaining-client-ipaddress.md | 44 ----------- 3 files changed, 78 insertions(+), 47 deletions(-) delete mode 100644 docs/user-guide/retaining-client-ipaddress.md diff --git a/docs/faq.md b/docs/faq.md index bd1d267183..020474d5c7 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,7 +1,9 @@ # FAQ -## How can I easily install multiple instances of the ingress-nginx controller in the same cluster? +## Multiple controller in one cluster + +Question - How can I easily install multiple instances of the ingress-nginx controller in the same cluster? You can install them in different namespaces. @@ -61,7 +63,80 @@ helm install ingress-nginx-2 ingress-nginx/ingress-nginx \ ## Retaining Client IPAddress -Please read [Retain Client IPAddress Guide here](./user-guide/retaining-client-ipaddress.md). +Question - How to obtain the real-client-ipaddress ? + +The goto solution for retaining the real-client IPaddress is to enable PROXY protocol. + +Enabling PROXY protocol has to be done on both, the Ingress NGINX controller, as well as the L4 load balancer, in front of the controller. + +The real-client IP address is lost by default, when traffic is forwarded over the network. But enabling PROXY protocol ensures that the connection details are retained and hence the real-client IP address doesn't get lost. + +Enabling proxy-protocol on the controller is documented [here](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-proxy-protocol) . + +For enabling proxy-protocol on the LoadBalancer, please refer to the documentation of your infrastructure provider because that is where the LB is provisioned. + +Some more info available [here](https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#source-ip-address) + +Some more info on proxy-protocol is [here](https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#proxy-protocol) + +### client-ipaddress on single-node cluster + +Single node clusters are created for dev & test uses with tools like "kind" or "minikube". A trick to simulate a real use network with these clusters (kind or minikube) is to install Metallb and configure the ipaddress of the kind container or the minikube vm/container, as the starting and ending of the pool for Metallb in L2 mode. Then the host ip becomes a real client ipaddress, for curl requests sent from the host. + +After installing ingress-nginx controller on a kind or a minikube cluster with helm, you can configure it for real-client-ip with a simple change to the service that ingress-nginx controller creates. The service object of --type LoadBalancer has a field service.spec.externalTrafficPolicy. If you set the value of this field to "Local" then the real-ipaddress of a client is visible to the controller. + +``` +% kubectl explain service.spec.externalTrafficPolicy +KIND: Service +VERSION: v1 + +FIELD: externalTrafficPolicy + +DESCRIPTION: + externalTrafficPolicy describes how nodes distribute service traffic they + receive on one of the Service's "externally-facing" addresses (NodePorts, + ExternalIPs, and LoadBalancer IPs). If set to "Local", the proxy will + configure the service in a way that assumes that external load balancers + will take care of balancing the service traffic between nodes, and so each + node will deliver traffic only to the node-local endpoints of the service, + without masquerading the client source IP. (Traffic mistakenly sent to a + node with no endpoints will be dropped.) The default value, "Cluster", uses + the standard behavior of routing to all endpoints evenly (possibly modified + by topology and other features). Note that traffic sent to an External IP or + LoadBalancer IP from within the cluster will always get "Cluster" semantics, + but clients sending to a NodePort from within the cluster may need to take + traffic policy into account when picking a node. + + Possible enum values: + - `"Cluster"` routes traffic to all endpoints. + - `"Local"` preserves the source IP of the traffic by routing only to + endpoints on the same node as the traffic was received on (dropping the + traffic if there are no local endpoints). +``` + +### client-ipaddress L7 + +The solution is to get the real client IPaddress from the ["X-Forward-For" HTTP header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For) + +Example : If your application pod behind Ingress NGINX controller, uses the NGINX webserver and the reverseproxy inside it, then you can do the following to preserve the remote client IP. + +- First you need to make sure that the X-Forwarded-For header reaches the backend pod. This is done by using a Ingress NGINX conftroller ConfigMap key. Its documented [here](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers) + +- Next, edit `nginx.conf` file inside your app pod, to contain the directives shown below: + +``` +set_real_ip_from 0.0.0.0/0; # Trust all IPs (use your VPC CIDR block in production) +real_ip_header X-Forwarded-For; +real_ip_recursive on; + +log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" ' + '"$http_user_agent" ' + 'host=$host x-forwarded-for=$http_x_forwarded_for'; + +access_log /var/log/nginx/access.log main; + +``` ## Kubernetes v1.22 Migration diff --git a/docs/user-guide/miscellaneous.md b/docs/user-guide/miscellaneous.md index e7d1ac59c4..80a38db518 100644 --- a/docs/user-guide/miscellaneous.md +++ b/docs/user-guide/miscellaneous.md @@ -17,7 +17,7 @@ By default NGINX path type is Prefix to not break existing definitions ## Proxy Protocol -If you are using a L4 proxy to forward the traffic to the NGINX pods and terminate HTTP/HTTPS there, you will lose the remote endpoint's IP address. To prevent this you could use the [Proxy Protocol](http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt) for forwarding traffic, this will send the connection details before forwarding the actual TCP connection itself. +If you are using a L4 proxy to forward the traffic to the Ingress NGINX pods and terminate HTTP/HTTPS there, you will lose the remote endpoint's IP address. To prevent this you could use the [PROXY Protocol](http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt) for forwarding traffic, this will send the connection details before forwarding the actual TCP connection itself. Amongst others [ELBs in AWS](http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html) and [HAProxy](http://www.haproxy.org/) support Proxy Protocol. diff --git a/docs/user-guide/retaining-client-ipaddress.md b/docs/user-guide/retaining-client-ipaddress.md deleted file mode 100644 index 237bd3004b..0000000000 --- a/docs/user-guide/retaining-client-ipaddress.md +++ /dev/null @@ -1,44 +0,0 @@ - -## Retaining Client IPAddress - -Please read this https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#source-ip-address , to get details of retaining the client IPAddress. - -### Using proxy-protocol - -Please read this https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#proxy-protocol , to use proxy-protocol for retaining client IPAddress - - -### Using the K8S spec service.spec.externalTrafficPolicy - -``` -% kubectl explain service.spec.externalTrafficPolicy -KIND: Service -VERSION: v1 - -FIELD: externalTrafficPolicy - -DESCRIPTION: - externalTrafficPolicy describes how nodes distribute service traffic they - receive on one of the Service's "externally-facing" addresses (NodePorts, - ExternalIPs, and LoadBalancer IPs). If set to "Local", the proxy will - configure the service in a way that assumes that external load balancers - will take care of balancing the service traffic between nodes, and so each - node will deliver traffic only to the node-local endpoints of the service, - without masquerading the client source IP. (Traffic mistakenly sent to a - node with no endpoints will be dropped.) The default value, "Cluster", uses - the standard behavior of routing to all endpoints evenly (possibly modified - by topology and other features). Note that traffic sent to an External IP or - LoadBalancer IP from within the cluster will always get "Cluster" semantics, - but clients sending to a NodePort from within the cluster may need to take - traffic policy into account when picking a node. - - Possible enum values: - - `"Cluster"` routes traffic to all endpoints. - - `"Local"` preserves the source IP of the traffic by routing only to - endpoints on the same node as the traffic was received on (dropping the - traffic if there are no local endpoints). - -``` - - -- Setting the field `externalTrafficPolicy`, in the ingress-controller service, to a value of `Local` retains the client's ipaddress, within the scope explained above