|
| 1 | +# Proposal to split containers |
| 2 | + |
| 3 | +* All the NGINX files should live on one container |
| 4 | + * No file other than NGINX files should exist on this container |
| 5 | + * This includes not mounting the service account |
| 6 | +* All the controller files should live on a different container |
| 7 | + * Controller container should have bare minimum to work (just go program) |
| 8 | + * ServiceAccount should be mounted just on controller |
| 9 | + |
| 10 | +* Inside nginx container, there should be a really small http listener just able |
| 11 | +to start, stop and reload NGINX |
| 12 | + |
| 13 | +## Roadmap (what needs to be done) |
| 14 | +* Map what needs to be done to mount the SA just on controller container |
| 15 | +* Map all the required files for NGINX to work |
| 16 | +* Map all the required network calls between controller and NGINX |
| 17 | + * eg.: Dynamic lua reconfiguration |
| 18 | +* Map problematic features that will need attention |
| 19 | + * SSLPassthrough today happens on controller process and needs to happen on NGINX |
| 20 | + |
| 21 | +### Ports and endpoints on NGINX container |
| 22 | +* Public HTTP/HTTPs port - 80 and 443 |
| 23 | +* Lua configuration port - 10246 (HTTP) and 10247 (Stream) |
| 24 | +* 3333 (temp) - Dataplane controller http server |
| 25 | + * /reload - (POST) Reloads the configuration. |
| 26 | + * "config" argument is the location of temporary file that should be used / moved to nginx.conf |
| 27 | + * /test - (POST) Test the configuration of a given file location |
| 28 | + * "config" argument is the location of temporary file that should be tested |
| 29 | + |
| 30 | +### Mounting empty SA on controller container |
| 31 | + |
| 32 | +```yaml |
| 33 | +kind: Pod |
| 34 | +apiVersion: v1 |
| 35 | +metadata: |
| 36 | + name: test |
| 37 | +spec: |
| 38 | + containers: |
| 39 | + - name: nginx |
| 40 | + image: nginx:latest |
| 41 | + ports: |
| 42 | + - containerPort: 80 |
| 43 | + - name: othernginx |
| 44 | + image: alpine:latest |
| 45 | + command: ["/bin/sh"] |
| 46 | + args: ["-c", "while true; do date; sleep 3; done"] |
| 47 | + volumeMounts: |
| 48 | + - mountPath: /var/run/secrets/kubernetes.io/serviceaccount |
| 49 | + name: emptysecret |
| 50 | + volumes: |
| 51 | + - name: emptysecret |
| 52 | + emptyDir: |
| 53 | + sizeLimit: 1Mi |
| 54 | +``` |
| 55 | +
|
| 56 | +### Mapped folders on NGINX configuration |
| 57 | +**WARNING** We need to be aware of inter mount containers and inode problems. If we |
| 58 | +mount a file instead of a directory, it may take time to reflect the file value on |
| 59 | +the target container |
| 60 | +
|
| 61 | +* "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;;"; - Lua scripts |
| 62 | +* "/var/log/nginx" - NGINX logs |
| 63 | +* "/tmp/nginx (nginx.pid)" - NGINX pid directory / file, fcgi socket, etc |
| 64 | +* " /etc/nginx/geoip" - GeoIP database directory - OK - /etc/ingress-controller/geoip |
| 65 | +* /etc/nginx/mime.types - Mime types |
| 66 | +* /etc/ingress-controller/ssl - SSL directory (fake cert, auth cert) |
| 67 | +* /etc/ingress-controller/auth - Authentication files |
| 68 | +* /etc/nginx/modsecurity - Modsecurity configuration |
| 69 | +* /etc/nginx/owasp-modsecurity-crs - Modsecurity rules |
| 70 | +* /etc/nginx/tickets.key - SSL tickets - OK - /etc/ingress-controller/tickets.key |
| 71 | +* /etc/nginx/opentelemetry.toml - OTEL config - OK - /etc/ingress-controller/telemetry |
| 72 | +* /etc/nginx/opentracing.json - Opentracing config - OK - /etc/ingress-controller/telemetry |
| 73 | +* /etc/nginx/modules - NGINX modules |
| 74 | +* /etc/nginx/fastcgi_params (maybe) - fcgi params |
| 75 | +* /etc/nginx/template - Template, may be used by controller only |
| 76 | +
|
| 77 | +##### List of modules |
| 78 | +``` |
| 79 | +ngx_http_auth_digest_module.so ngx_http_modsecurity_module.so |
| 80 | +ngx_http_brotli_filter_module.so ngx_http_opentracing_module.so |
| 81 | +ngx_http_brotli_static_module.so ngx_stream_geoip2_module.so |
| 82 | +ngx_http_geoip2_module.so |
| 83 | +``` |
| 84 | + |
| 85 | +##### List of files that may be removed |
| 86 | +``` |
| 87 | +-rw-r--r-- 1 www-data www-data 1077 Jun 23 19:44 fastcgi.conf |
| 88 | +-rw-r--r-- 1 www-data www-data 1077 Jun 23 19:44 fastcgi.conf.default |
| 89 | +-rw-r--r-- 1 www-data www-data 1007 Jun 23 19:44 fastcgi_params |
| 90 | +-rw-r--r-- 1 www-data www-data 1007 Jun 23 19:44 fastcgi_params.default |
| 91 | +drwxr-xr-x 2 www-data www-data 4096 Jun 23 19:34 geoip |
| 92 | +-rw-r--r-- 1 www-data www-data 2837 Jun 23 19:44 koi-utf |
| 93 | +-rw-r--r-- 1 www-data www-data 2223 Jun 23 19:44 koi-win |
| 94 | +drwxr-xr-x 6 www-data www-data 4096 Sep 19 14:13 lua |
| 95 | +-rw-r--r-- 1 www-data www-data 5349 Jun 23 19:44 mime.types |
| 96 | +-rw-r--r-- 1 www-data www-data 5349 Jun 23 19:44 mime.types.default |
| 97 | +drwxr-xr-x 2 www-data www-data 4096 Jun 23 19:44 modsecurity |
| 98 | +drwxr-xr-x 2 www-data www-data 4096 Jun 23 19:44 modules |
| 99 | +-rw-r--r-- 1 www-data www-data 18275 Oct 1 21:28 nginx.conf |
| 100 | +-rw-r--r-- 1 www-data www-data 2656 Jun 23 19:44 nginx.conf.default |
| 101 | +-rwx------ 1 www-data www-data 420 Oct 1 21:28 opentelemetry.toml |
| 102 | +-rw-r--r-- 1 www-data www-data 2 Oct 1 21:28 opentracing.json |
| 103 | +drwxr-xr-x 7 www-data www-data 4096 Jun 23 19:44 owasp-modsecurity-crs |
| 104 | +-rw-r--r-- 1 www-data www-data 636 Jun 23 19:44 scgi_params |
| 105 | +-rw-r--r-- 1 www-data www-data 636 Jun 23 19:44 scgi_params.default |
| 106 | +drwxr-xr-x 2 www-data www-data 4096 Sep 19 14:13 template |
| 107 | +-rw-r--r-- 1 www-data www-data 664 Jun 23 19:44 uwsgi_params |
| 108 | +-rw-r--r-- 1 www-data www-data 664 Jun 23 19:44 uwsgi_params.default |
| 109 | +-rw-r--r-- 1 www-data www-data 3610 Jun 23 19:44 win-utf |
| 110 | +``` |
0 commit comments