Skip to content

[WIP] reword/improve ingress documentation #3173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jschaul wants to merge 2 commits into from
Closed

[WIP] reword/improve ingress documentation #3173

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6c45a7a
reword/improve ingress documentation based on PR feedback
jschaul Mar 22, 2023
a892887
...
jschaul Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/4-docs/ingress-traffic
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Document the new ingress-nginx-controller, current recommendations, and adjust older documentation on the topic
8 changes: 0 additions & 8 deletions charts/ingress-nginx-controller/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,6 @@ ingress-nginx:
# upstream services. Comes with tradeoff considerations, see
# documentation on "ingress" on docs.wire.com
externalTrafficPolicy: Local
nodePorts:
# If you set service.type = NodePort, then the nginx controller instance
# is exposed on ports 31773 (https) and 31772 (http) on the node on
# which it runs. You should add a port-forwarding rule on the node or on
# the loadbalancer that forwards ports 443 and 80 to these respective
# ports.
https: 31773
http: 31772
config:
# NOTE: These are some sane defaults (compliant to TR-02102-2), you may
# want to overrride them on your own installation For TR-02102-2 see
Expand Down
24 changes: 18 additions & 6 deletions docs/src/how-to/install/ingress.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,19 @@ To inspect default TLS settings, see [defaults in the latest code](https://githu

## Installing on bare-metal without dynamic load balancer support

In case you cannot create a `kind: service` of `type: LoadBalancer`, then you
can fall back to manually ensure traffic reaches your installation:
```{note}
Most managed K8s clusters have support for LoadBalancers. Kubernetes clusters that are manually set up can also support this, by using a provider/environment-specific [CCM (Cloud Controller Manager)](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) (see hcloud and digitalocean for examples).

In case you're provisioning on bare metal, without any hardware load balancer support in front,
you might be using MetalLB, which supports BGP or Failover [L2 ARP announcements](https://metallb.universe.tf/configuration/_advanced_l2_configuration/).
Copy link
Contributor

@flokli flokli Mar 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the ticket description, we probably shouldn't be linking to any of this repositories, but rather to https://kubernetes.github.io/ingress-nginx/deploy/baremetal/. We can still mention these as examples, but that should be it.

Copy link
Contributor

@flokli flokli Mar 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this paragraph describing MetalLB specifically? That's all included in https://kubernetes.github.io/ingress-nginx/deploy/baremetal/.


Please read [baremetal deployment documentation](https://kubernetes.github.io/ingress-nginx/deploy/baremetal/)

The choice of CCM highly depends on the environment you choose to deploy wire-server in.
```

In case you cannot change the type of kubernetes cluster you run, and therefore cannot create a `kind: service` of `type: LoadBalancer`, then you
can fall back to NodePorts and manually ensure traffic reaches your installation:

```yaml
# helmfile.yaml
Expand All @@ -61,6 +72,9 @@ ingress-nginx:
kind: DaemonSet
service:
type: NodePort
nodePorts:
https: 31773
http: 31772
```

Then, on each of your kubernetes worker nodes, two ports are exposed: ports
Expand All @@ -71,8 +85,6 @@ forwards ports 443 and 80 to these respective ports. Any traffic hitting the htt

Downsides of this approach: The NodePort approach always requires manual configuration of some external load balancer/firewall to round-robin between node IPs and is error-prone. It's also a bit annoying to have to decide on some global ports that may not be used otherwise.

Most managed K8s clusters have support for LoadBalancers, you can also get this for your own clusters in hcloud etc. It's even possible to do it for pure bare metal, without any "load balancer hardware", by using BGP or some leadership election over who's announcing the "load balancer ip" via ARP (https://metallb.universe.tf/configuration/_advanced_l2_configuration/).

### Using NodePort (not the default) with externalTrafficPolicy=Local (the default)

Normally, NodePort will listen to traffic on all nodes, and uses kube-proxy
Expand All @@ -95,8 +107,8 @@ redirect them to the correct service, whilst maintaining the source ip
address. The ingress controller is sort of taking over the role of what
kube-proxy was doing before.
More information:
- https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typenodeport
- https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
- [source-ip](https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typenodeport)
- [ingress and baremetal](https://kubernetes.github.io/ingress-nginx/deploy/baremetal/)

There are also downsides to setting `externalTrafficPolicy: Local`, please look at the [following blog post](https://www.asykim.com/blog/deep-dive-into-kubernetes-external-traffic-policies), which very clearly explains the upsides and
downsides of this setting