|
| 1 | + |
| 2 | +# Private IP Only Cluster |
| 3 | + |
| 4 | +Below details about a private IP only cluster, set up with [hetzner-k3s][hk], are provided. |
| 5 | + |
| 6 | +We describe a setup, where the cluster is 'behind' a proxy node, with a pub ip, reachable via ssh. Alternatively, you could add a vpn on top. |
| 7 | + |
| 8 | +```mermaid |
| 9 | +flowchart LR |
| 10 | + A[World] --> B[Bastion Proxy<br/>IP pub<br/>Opt.LoadBalancer] |
| 11 | + B --priv net--> M1[Master 1<br/>...<br/>Master 3] |
| 12 | + B --priv net--> w1[Worker 1<br/>...<br/>Worker n] |
| 13 | + B --priv net--> a1[Autoscaled 1<br/>...<br/>Autoscaled n] |
| 14 | +``` |
| 15 | +Why: |
| 16 | + |
| 17 | +1. want a cost effective cluster, but |
| 18 | +1. don't want to ever have to recover a broken k8s. So: 3 masters. |
| 19 | +1. => Workloads on (cheap) masters - but with **autoscaled** add/delete workers if required. |
| 20 | + |
| 21 | +IPs: Priv IPs are for free -> Only 1 pub IP (on a bastion outside the k8s cluster, which runs trivially restorable services w/o k8s). Also more secure, only this to shield. |
| 22 | + |
| 23 | +💡 [This repo](https://github.com/axgkl/pyhk3) provides a set of python functions, to automate that setup. |
| 24 | + |
| 25 | +💡 The node may be super slim, resource/cost wise. Distribution: We tested with ubuntu/amd64, but any other _should_ work as well. |
| 26 | + |
| 27 | +## Features |
| 28 | + |
| 29 | +- Creates a private network, using the hetzner ui, say on 10.1.0.0/16, named `ten-1`. |
| 30 | +- Creates a bastion node, with a pub ip and membership within that network. Usually that first node gets `.2` assigned, i.e. `10.1.0.2` in our example. |
| 31 | +- Secures the node but keeps allowing outgoing traffic to the internet and ssh access from your local machine, possibly from a jump host. |
| 32 | +- Kicks off the installation of the cluster, using [hetzner-k3s][hk], with a config, which creates a private IP only cluster. |
| 33 | +- Installs tools |
| 34 | + |
| 35 | + |
| 36 | +### Tools Installed |
| 37 | + |
| 38 | +- [hetzner-k3s][hk] |
| 39 | +- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) |
| 40 | +- [helm](https://helm.sh/docs/intro/install/) |
| 41 | + |
| 42 | +💡 The setup function in this repo use [binenv][binenv] to install kubectl and helm, since fast, distri independent and w/o the need for a big package mgmt. framework. |
| 43 | + |
| 44 | + |
| 45 | + |
| 46 | +#### Notes on the Post Create Config |
| 47 | + |
| 48 | +We add this post create commands hetzner-k3s: |
| 49 | + |
| 50 | +```yaml |
| 51 | +post_create_commands: |
| 52 | + - echo "Started" > /.status # just to debug |
| 53 | + - timedatectl set-timezone Europe/Berlin |
| 54 | + - echo 'ecdsa-sha2-nistp256 AAAAE2V....= root@citest-proxy' >> /root/.ssh/authorized_keys |
| 55 | + - echo 'ecdsa-sha2-nistp256 AAAAE2V....= gk@axgk' >> /root/.ssh/authorized_keys |
| 56 | + - echo "root:$(head -c 50 /dev/urandom | base64)" | chpasswd |
| 57 | + - mkdir -p /etc/network/interfaces.d |
| 58 | + - iface="$(ip -o -4 addr list | grep " 10.1." | cut -d " " -f 2)" |
| 59 | + - | |
| 60 | + cat > /etc/network/interfaces.d/$iface <<EOF |
| 61 | + auto $iface |
| 62 | + iface $iface inet dhcp |
| 63 | + post-up ip route add default via 10.1.0.1 |
| 64 | + post-up ip route add 169.254.169.254 via 172.31.1.1 |
| 65 | + EOF |
| 66 | + - rm -f /etc/resolv.conf |
| 67 | + - | |
| 68 | + cat > /etc/resolv.conf <<EOF |
| 69 | + nameserver 185.12.64.1 |
| 70 | + nameserver 185.12.64.2 |
| 71 | + edns edns0 trust-ad |
| 72 | + search . |
| 73 | + EOF |
| 74 | + - ip route add 169.254.0.0/16 via 172.31.1.1 |
| 75 | + - ip route add default via 10.1.0.1 |
| 76 | + - echo "Done" > /.status # just to debug |
| 77 | +``` |
| 78 | +
|
| 79 | +
|
| 80 | +These commands are basically run as cloud init, after a node is created. |
| 81 | +
|
| 82 | +Since we use the 'classic' way of configuring the network, using /etc/network/interfaces.d, on ubuntu we needed to add the `ifupdown` package. |
| 83 | + |
| 84 | +- Added ssh pub keys, which should be allowed to log in to the k3s nodes. |
| 85 | +- Run `chpasswd`, in order to avoid any mails from hetzner, reminding you to change the password, after autoscaled nodes are created. |
| 86 | +- Find the interface name of the one interface, which is in the private network (distri dependent) |
| 87 | +- Create the interface config file, with the default route to the gateway of the private network, which is our bastion node. This has to be the `.1` address, not the priv ip of the bastion node! |
| 88 | +- Also add a route to hetzner's api server on 169.254.169.254, which seems to be always via 172.31.1.1 |
| 89 | +- Lastly we configure hetzner's DNS servers and configure the above routes via cli commands, avoiding the need for a reboot. |
| 90 | + |
| 91 | +With such a config, the hetzner-k3s setup should run through, creating a private IP only cluster. |
| 92 | + |
| 93 | +[hk]: https://github.com/vitobotta/hetzner-k3s |
| 94 | +[binenv]: https://github.com/devops-works/binenv |
| 95 | + |
| 96 | + |
| 97 | + |
0 commit comments