Skip to content

Commit e2f20ac

Browse files
garutilorenzogarutilorenzo
authored
Merge pull request garutilorenzo#7 from garutilorenzo/l4_public_lb
Replace the public L7 load balancer with a L4 load balancer
2 parents de6d9e9 + abd5ab2 commit e2f20ac

16 files changed

+900
-206
lines changed

‎README.md

+28-73
Original file line numberDiff line numberDiff line change
@@ -170,69 +170,6 @@ rerun this command to reinitialize your working directory. If you forget, other
170170
commands will detect it and remind you to do so if necessary.
171171
```
172172

173-
#### Generate sel signed SSL certificate for the public LB (L7)
174-
175-
**NOTE** If you already own a valid certificate skip this step and set the correct values for the variables: PATH_TO_PUBLIC_LB_CERT and PATH_TO_PUBLIC_LB_KEY
176-
177-
We need to generate the certificates (sel signed) for our public load balancer (Layer 7). To do this we need *openssl*, open a terminal and follow this step:
178-
179-
Generate the key:
180-
181-
```
182-
openssl genrsa 2048 > privatekey.pem
183-
Generating RSA private key, 2048 bit long modulus (2 primes)
184-
.......+++++
185-
...............+++++
186-
e is 65537 (0x010001)
187-
```
188-
189-
Generate the a new certificate request:
190-
191-
```
192-
openssl req -new -key privatekey.pem -out csr.pem
193-
You are about to be asked to enter information that will be incorporated
194-
into your certificate request.
195-
What you are about to enter is what is called a Distinguished Name or a DN.
196-
There are quite a few fields but you can leave some blank
197-
For some fields there will be a default value,
198-
If you enter '.', the field will be left blank.
199-
-----
200-
Country Name (2 letter code) [AU]:IT
201-
State or Province Name (full name) [Some-State]:Italy
202-
Locality Name (eg, city) []:Brescia
203-
Organization Name (eg, company) [Internet Widgits Pty Ltd]:GL Ltd
204-
Organizational Unit Name (eg, section) []:IT
205-
Common Name (e.g. server FQDN or YOUR name) []:testlb.domainexample.com
206-
Email Address []:[email protected]
207-
208-
Please enter the following 'extra' attributes
209-
to be sent with your certificate request
210-
A challenge password []:
211-
An optional company name []:
212-
```
213-
214-
Generate the public CRT:
215-
216-
```
217-
openssl x509 -req -days 365 -in csr.pem -signkey privatekey.pem -out public.crt
218-
Signature ok
219-
subject=C = IT, ST = Italy, L = Brescia, O = GL Ltd, OU = IT, CN = testlb.domainexample.com, emailAddress = [email protected]
220-
Getting Private key
221-
```
222-
223-
This is the final result:
224-
225-
```
226-
ls
227-
228-
csr.pem privatekey.pem public.crt
229-
```
230-
231-
Now set the variables:
232-
233-
* PATH_TO_PUBLIC_LB_CERT: ~/full_path/public.crt
234-
* PATH_TO_PUBLIC_LB_KEY: ~/full_path/privatekey.pem
235-
236173
### Oracle provider setup
237174

238175
In the *example/* directory of this repo you need to create a terraform.tfvars file, the file will look like:
@@ -266,8 +203,6 @@ Once you have created the terraform.tfvars file edit the main.tf file (always in
266203
| `k3s_token` | `yes` | The token of your K3s cluster. [How to](#generate-random-token) generate a random token |
267204
| `my_public_ip_cidr` | `yes` | your public ip in cidr format (Example: 195.102.xxx.xxx/32) |
268205
| `environment` | `yes` | Current work environment (Example: staging/dev/prod). This value is used for tag all the deployed resources |
269-
| `PATH_TO_PUBLIC_LB_CERT` | `yes` | Path to the public LB certificate. See [how to](#generate-sel-signed-ssl-certificate-for-the-public-lb-l7) generate the certificate |
270-
| `PATH_TO_PUBLIC_LB_KEY` | `yes` | Path to the public LB key. See [how to](#generate-sel-signed-ssl-certificate-for-the-public-lb-l7) generate the key |
271206
| `compute_shape` | `no` | Compute shape to use. Default VM.Standard.A1.Flex. **NOTE** Is mandatory to use this compute shape for provision 4 always free VMs |
272207
| `os_image_id` | `no` | Image id to use. Default image: Canonical-Ubuntu-20.04-aarch64-2022.01.18-0. See [how](#how-to-list-all-the-os-images) to list all available OS images |
273208
| `oci_core_vcn_dns_label` | `no` | VCN DNS label. Default: defaultvcn |
@@ -287,12 +222,17 @@ Once you have created the terraform.tfvars file edit the main.tf file (always in
287222
| `k3s_server_pool_size` | `no` | Number of k3s servers deployed. Default 2 |
288223
| `k3s_worker_pool_size` | `no` | Number of k3s workers deployed. Default 2 |
289224
| `install_nginx_ingress` | `no` | Boolean value, install kubernetes nginx ingress controller instead of Traefik. Default: true. For more information see [Nginx ingress controller](#nginx-ingress-controller) |
225+
nginx_ingress_controller_http_nodeport
226+
| `nginx_ingress_controller_http_nodeport` | `30080` | NodePort where nginx ingress will listen for http traffic |
227+
| `nginx_ingress_controller_https_nodeport` | `30443` | NodePort where nginx ingress will listen for https traffic |
290228
| `install_longhorn` | `no` | Boolean value, install longhorn "Cloud native distributed block storage for Kubernetes". Default: true |
291229
| `longhorn_release` | `no` | Longhorn release. Default: v1.2.3 |
230+
| `install_certmanager` | `no` | Boolean value, install [cert manager](https://cert-manager.io/) "Cloud native certificate management". Default: true |
231+
| `longhorn_release` | `no` | Cert manager release. Default: v1.8.2 |
232+
| `certmanager_email_address` | `no` | Email address used for signing https certificates. Defaul: [email protected] |
292233
| `unique_tag_key` | `no` | Unique tag name used for tagging all the deployed resources. Default: k3s-provisioner |
293234
| `unique_tag_value` | `no` | Unique value used with unique_tag_key. Default: https://github.com/garutilorenzo/k3s-oci-cluster |
294235
| `PATH_TO_PUBLIC_KEY` | `no` | Path to your public ssh key (Default: "~/.ssh/id_rsa.pub) |
295-
| `PATH_TO_PRIVATE_KEY` | `no` | Path to your private ssh key (Default: "~/.ssh/id_rsa) |
296236

297237
#### Generate random token
298238

@@ -399,7 +339,7 @@ This setup will automatically install [longhorn](https://longhorn.io/). Longhorn
399339

400340
In this environment [Nginx ingress controller](https://kubernetes.github.io/ingress-nginx/) is used instead of the standard [Traefik](https://traefik.io/) ingress controller.
401341

402-
The installation is the [bare metal](https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal-clusters) installation, the ingress controller then is exposed via a LoadBalancer Service.
342+
The installation is the [bare metal](https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal-clusters) installation, the ingress controller then is exposed via a NodePort Service.
403343

404344
```yaml
405345
---
@@ -418,25 +358,26 @@ spec:
418358
port: 80
419359
protocol: TCP
420360
targetPort: 80
361+
nodePort: ${nginx_ingress_controller_http_nodeport} # default to 30080
421362
- name: https
422363
port: 443
423364
protocol: TCP
424-
targetPort: 80
425-
type: LoadBalancer
365+
targetPort: 443
366+
nodePort: ${nginx_ingress_controller_https_nodeport} # default to 30443
367+
type: NodePort
426368
```
427369
428-
To properly configure all the Forwarded HTTP Headers (L7 Headers) this parameters are added to che ConfigMap:
370+
To get the real ip address of the clients using a public L4 load balancer we need to use the proxy protocol feature of nginx ingress controller:
429371
430372
```yaml
431373
---
432374
apiVersion: v1
433375
data:
434376
allow-snippet-annotations: "true"
435-
use-forwarded-headers: "true"
436-
compute-full-forwarded-for: "true"
437377
enable-real-ip: "true"
438-
forwarded-for-header: "X-Forwarded-For"
439378
proxy-real-ip-cidr: "0.0.0.0/0"
379+
proxy-body-size: "20m"
380+
use-proxy-protocol: "true"
440381
kind: ConfigMap
441382
metadata:
442383
labels:
@@ -451,6 +392,20 @@ metadata:
451392
namespace: ingress-nginx
452393
```
453394
395+
**NOTE** to use nginx ingress controller with the proxy protocol enabled, an external nginx instance is used as proxy. Nginx will be installed on each worker node and the configuation of nginx will:
396+
397+
* listen in proxy protocol mode
398+
* forward the traffic from port 80 to nginx_ingress_controller_http_nodeport (default to 30080) on any server of the cluster
399+
* forward the traffic from port 443 to nginx_ingress_controller_https_nodeport (default to 30443) on any server of the cluster
400+
401+
This is the final result:
402+
403+
Client -> Public L4 LB -> nginx proxy (with proxy protocol enabled) -> nginx ingress (with proxy protocol enabled) -> k3s service -> pod(s)
404+
405+
### Cert-manager
406+
407+
[cert-manager](https://cert-manager.io/docs/) is used to issue certificates from a variety of supported source. To use cert-manager take a look at [nginx-ingress-cert-manager.yml](deployments/nginx/nginx-ingress-cert-manager.yml) and [nginx-configmap-cert-manager.yml](deployments/nginx/nginx-configmap-cert-manager.yml) example. To use cert-manager and get the certificate you **need** set on your DNS configuration the public ip address of the load balancer.
408+
454409
## Deploy
455410
456411
We are now ready to deploy our infrastructure. First we ask terraform to plan the execution with:

‎data.tf

+25-2
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,22 @@ data "template_cloudinit_config" "k3s_server_tpl" {
44

55
part {
66
content_type = "text/x-shellscript"
7-
content = templatefile("${path.module}/files/k3s-install-server.sh", { k3s_token = var.k3s_token, is_k3s_server = true, install_nginx_ingress = var.install_nginx_ingress, compartment_ocid = var.compartment_ocid, availability_domain = var.availability_domain, k3s_url = local.k3s_int_lb_dns_name, k3s_tls_san = local.k3s_int_lb_dns_name, install_longhorn = var.install_longhorn, longhorn_release = var.longhorn_release })
7+
content = templatefile("${path.module}/files/k3s-install-server.sh", {
8+
k3s_token = var.k3s_token,
9+
is_k3s_server = true,
10+
install_nginx_ingress = var.install_nginx_ingress,
11+
install_certmanager = var.install_certmanager,
12+
certmanager_release = var.certmanager_release,
13+
certmanager_email_address = var.certmanager_email_address,
14+
compartment_ocid = var.compartment_ocid,
15+
availability_domain = var.availability_domain,
16+
k3s_url = oci_load_balancer_load_balancer.k3s_load_balancer.ip_addresses[0],
17+
k3s_tls_san = oci_load_balancer_load_balancer.k3s_load_balancer.ip_addresses[0],
18+
install_longhorn = var.install_longhorn,
19+
longhorn_release = var.longhorn_release,
20+
nginx_ingress_controller_http_nodeport = var.nginx_ingress_controller_http_nodeport,
21+
nginx_ingress_controller_https_nodeport = var.nginx_ingress_controller_https_nodeport,
22+
})
823
}
924
}
1025

@@ -14,7 +29,15 @@ data "template_cloudinit_config" "k3s_worker_tpl" {
1429

1530
part {
1631
content_type = "text/x-shellscript"
17-
content = templatefile("${path.module}/files/k3s-install-agent.sh", { k3s_token = var.k3s_token, is_k3s_server = false, k3s_url = local.k3s_int_lb_dns_name })
32+
content = templatefile("${path.module}/files/k3s-install-agent.sh", {
33+
k3s_token = var.k3s_token,
34+
is_k3s_server = false,
35+
k3s_url = oci_load_balancer_load_balancer.k3s_load_balancer.ip_addresses[0],
36+
http_lb_port = var.http_lb_port,
37+
https_lb_port = var.https_lb_port,
38+
nginx_ingress_controller_http_nodeport = var.nginx_ingress_controller_http_nodeport,
39+
nginx_ingress_controller_https_nodeport = var.nginx_ingress_controller_https_nodeport,
40+
})
1841
}
1942
}
2043

0 commit comments

Comments
 (0)