Add reconfig tests setup and results (#1116)

* Add reconfig tests setup and results * Review feedback * Add TOC, change bash -> console in commands
nginxinc · Oct 11, 2023 · 704f8a8 · 704f8a8
1 parent b3fbf98
commit 704f8a8
Show file tree

Hide file tree

Showing 10 changed files with 415 additions and 0 deletions.
diff --git a/tests/reconfig/results/v1.0.0.md b/tests/reconfig/results/v1.0.0.md
@@ -0,0 +1,61 @@
+# Reconfiguration testing Results
+
+<!-- TOC -->
+- [Reconfiguration testing Results](#reconfiguration-testing-results)
+  - [Test environment](#test-environment)
+  - [Results Table](#results-table)
+  - [NumResources -\> Total Resources](#numresources---total-resources)
+  - [Observations](#observations)
+<!-- TOC -->
+
+## Test environment
+
+GKE cluster:
+
+- Node count: 3
+- Instance Type: e2-medium
+- k8s version: 1.27.4-gke.900
+- Zone: europe-west2-b
+- Total vCPUs: 6
+- Total RAM: 12GB
+- Max pods per node: 110
+
+NGF deployment:
+
+- NGF version: edge - git commit 72b6c6ef8915c697626eeab88fdb6a3ce15b8da0
+- NGINX Version: 1.25.2
+
+## Results Table
+
+| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) |
+| ----------- | ------------ | -------------------- | ------------------------ | ------------- | -------------------------- |
+| 1           | 30           | 5                    | 5                        | 2             | 166                        |
+| 1           | 150          | 7                    | 7                        | 2             | 353                        |
+| 2           | 30           | 21                   | <1                       | 30            | 142                        |
+| 2           | 150          | 123                  | <1                       | 46            | 190                        |
+| 3           | 30           | <1                   | <1                       | 93            | 137                        |
+| 3           | 150          | 1                    | 1                        | 453           | 127                        |
+
+## NumResources -> Total Resources
+| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Total Resources |
+| ------------ | -------- | ------- | --------------- | ---------- | ---------------- | -------------------- | ---------- | --------------- |
+| x            | 1        | 1       | 1               | x+1        | 2x               | 2x                   | 3x         | <total>         |
+| 30           | 1        | 1       | 1               | 31         | 60               | 60                   | 90         | 244             |
+| 150          | 1        | 1       | 1               | 151        | 300              | 300                  | 450        | 1204            |
+
+## Observations
+
+1. We are reloading after reconciling a ReferenceGrant even when there is no Gateway. This is because we treat every
+   upsert/delete of a ReferenceGrant as a change. This means we will regenerate NGINX config every time a ReferenceGrant
+   is created, updated (generation must change), or deleted, even if it does not apply to the accepted Gateway.
+
+   Issue filed: https://github.com/nginxinc/nginx-gateway-fabric/issues/1124
+
+2. We are reloading after reconciling a HTTPRoute even when there is no accepted Gateway and no config being generated.
+
+   Issue filed: https://github.com/nginxinc/nginx-gateway-fabric/issues/1123
+
+3. All reloads were in the <500ms bucket. A slight increase in the reload time based on number of configured resources
+   resulting in NGINX configuration changes was observed.
+
+4. No errors (NGF or NGINX) were observed in any test run.
diff --git a/tests/reconfig/scripts/cafe-routes.yaml b/tests/reconfig/scripts/cafe-routes.yaml
@@ -0,0 +1,57 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+  name: cafe-tls-redirect
+spec:
+  parentRefs:
+  - name: gateway.networking.k8s.io/v1beta1
+    namespace: default
+    sectionName: http
+  hostnames:
+  - "cafe.example.com"
+  rules:
+  - filters:
+    - type: RequestRedirect
+      requestRedirect:
+        scheme: https
+        port: 443
+---
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+  name: coffee
+spec:
+  parentRefs:
+  - name: gateway
+    namespace: default
+    sectionName: https
+  hostnames:
+  - "cafe.example.com"
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /coffee
+    backendRefs:
+    - name: coffee
+      port: 80
+---
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+  name: tea
+spec:
+  parentRefs:
+  - name: gateway
+    sectionName: https
+    namespace: default
+  hostnames:
+  - "cafe.example.com"
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /tea
+    backendRefs:
+    - name: tea
+      port: 80
diff --git a/tests/reconfig/scripts/cafe.yaml b/tests/reconfig/scripts/cafe.yaml
@@ -0,0 +1,65 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: coffee
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: coffee
+  template:
+    metadata:
+      labels:
+        app: coffee
+    spec:
+      containers:
+      - name: coffee
+        image: nginxdemos/nginx-hello:plain-text
+        ports:
+        - containerPort: 8080
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: coffee
+spec:
+  ports:
+  - port: 80
+    targetPort: 8080
+    protocol: TCP
+    name: http
+  selector:
+    app: coffee
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: tea
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: tea
+  template:
+    metadata:
+      labels:
+        app: tea
+    spec:
+      containers:
+      - name: tea
+        image: nginxdemos/nginx-hello:plain-text
+        ports:
+        - containerPort: 8080
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: tea
+spec:
+  ports:
+  - port: 80
+    targetPort: 8080
+    protocol: TCP
+    name: http
+  selector:
+    app: tea
diff --git a/tests/reconfig/scripts/certificate-ns-and-cafe-secret.yaml b/tests/reconfig/scripts/certificate-ns-and-cafe-secret.yaml
@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: certificate
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cafe-secret
+  namespace: certificate
+type: kubernetes.io/tls
+data:
+  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNzakNDQVpvQ0NRQzdCdVdXdWRtRkNEQU5CZ2txaGtpRzl3MEJBUXNGQURBYk1Sa3dGd1lEVlFRRERCQmoKWVdabExtVjRZVzF3YkdVdVkyOXRNQjRYRFRJeU1EY3hOREl4TlRJek9Wb1hEVEl6TURjeE5ESXhOVEl6T1ZvdwpHekVaTUJjR0ExVUVBd3dRWTJGbVpTNWxlR0Z0Y0d4bExtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTHFZMnRHNFc5aStFYzJhdnV4Q2prb2tnUUx1ek10U1Rnc1RNaEhuK3ZRUmxIam8KVzFLRnMvQVdlS25UUStyTWVKVWNseis4M3QwRGtyRThwUisxR2NKSE50WlNMb0NEYUlRN0Nhck5nY1daS0o4Qgo1WDNnVS9YeVJHZjI2c1REd2xzU3NkSEQ1U2U3K2Vab3NPcTdHTVF3K25HR2NVZ0VtL1Q1UEMvY05PWE0zZWxGClRPL051MStoMzROVG9BbDNQdTF2QlpMcDNQVERtQ0thaEROV0NWbUJQUWpNNFI4VERsbFhhMHQ5Z1o1MTRSRzUKWHlZWTNtdzZpUzIrR1dYVXllMjFuWVV4UEhZbDV4RHY0c0FXaGRXbElweHlZQlNCRURjczN6QlI2bFF1OWkxZAp0R1k4dGJ3blVmcUVUR3NZdWxzc05qcU95V1VEcFdJelhibHhJZVVDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBcjkrZWJ0U1dzSnhLTGtLZlRkek1ISFhOd2Y5ZXFVbHNtTXZmMGdBdWVKTUpUR215dG1iWjlpbXQKL2RnWlpYVE9hTElHUG9oZ3BpS0l5eVVRZVdGQ2F0NHRxWkNPVWRhbUloOGk0Q1h6QVJYVHNvcUNOenNNLzZMRQphM25XbFZyS2lmZHYrWkxyRi8vblc0VVNvOEoxaCtQeDljY0tpRDZZU0RVUERDRGh1RUtFWXcvbHpoUDJVOXNmCnl6cEJKVGQ4enFyM3paTjNGWWlITmgzYlRhQS82di9jU2lyamNTK1EwQXg4RWpzQzYxRjRVMTc4QzdWNWRCKzQKcmtPTy9QNlA0UFlWNTRZZHMvRjE2WkZJTHFBNENCYnExRExuYWRxamxyN3NPbzl2ZzNnWFNMYXBVVkdtZ2todAp6VlZPWG1mU0Z4OS90MDBHUi95bUdPbERJbWlXMGc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
+  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzZtTnJSdUZ2WXZoSE4KbXI3c1FvNUtKSUVDN3N6TFVrNExFeklSNS9yMEVaUjQ2RnRTaGJQd0ZuaXAwMFBxekhpVkhKYy92TjdkQTVLeApQS1VmdFJuQ1J6YldVaTZBZzJpRU93bXF6WUhGbVNpZkFlVjk0RlAxOGtSbjl1ckV3OEpiRXJIUncrVW51L25tCmFMRHF1eGpFTVBweGhuRklCSnYwK1R3djNEVGx6TjNwUlV6dnpidGZvZCtEVTZBSmR6N3Rid1dTNmR6MHc1Z2kKbW9RelZnbFpnVDBJek9FZkV3NVpWMnRMZllHZWRlRVJ1VjhtR041c09va3R2aGxsMU1udHRaMkZNVHgySmVjUQo3K0xBRm9YVnBTS2NjbUFVZ1JBM0xOOHdVZXBVTHZZdFhiUm1QTFc4SjFINmhFeHJHTHBiTERZNmpzbGxBNlZpCk0xMjVjU0hsQWdNQkFBRUNnZ0VBQnpaRE50bmVTdWxGdk9HZlFYaHRFWGFKdWZoSzJBenRVVVpEcUNlRUxvekQKWlV6dHdxbkNRNlJLczUyandWNTN4cU9kUU94bTNMbjNvSHdNa2NZcEliWW82MjJ2dUczYnkwaVEzaFlsVHVMVgpqQmZCcS9UUXFlL2NMdngvSkczQWhFNmJxdFRjZFlXeGFmTmY2eUtpR1dzZk11WVVXTWs4MGVJVUxuRmZaZ1pOCklYNTlSOHlqdE9CVm9Sa3hjYTVoMW1ZTDFsSlJNM3ZqVHNHTHFybmpOTjNBdWZ3ZGRpK1VDbGZVL2l0K1EvZkUKV216aFFoTlRpNVFkRWJLVStOTnYvNnYvb2JvandNb25HVVBCdEFTUE05cmxFemIralQ1WHdWQjgvLzRGY3VoSwoyVzNpcjhtNHVlQ1JHSVlrbGxlLzhuQmZ0eVhiVkNocVRyZFBlaGlPM1FLQmdRRGlrR3JTOTc3cjg3Y1JPOCtQClpoeXltNXo4NVIzTHVVbFNTazJiOTI1QlhvakpZL2RRZDVTdFVsSWE4OUZKZnNWc1JRcEhHaTFCYzBMaTY1YjIKazR0cE5xcVFoUmZ1UVh0UG9GYXRuQzlPRnJVTXJXbDVJN0ZFejZnNkNQMVBXMEg5d2hPemFKZUdpZVpNYjlYTQoybDdSSFZOcC9jTDlYbmhNMnN0Q1lua2Iwd0tCZ1FEUzF4K0crakEyUVNtRVFWNXA1RnRONGcyamsyZEFjMEhNClRIQ2tTazFDRjhkR0Z2UWtsWm5ZbUt0dXFYeXNtekJGcnZKdmt2eUhqbUNYYTducXlpajBEdDZtODViN3BGcVAKQWxtajdtbXI3Z1pUeG1ZMXBhRWFLMXY4SDNINGtRNVl3MWdrTWRybVJHcVAvaTBGaDVpaGtSZS9DOUtGTFVkSQpDcnJjTzhkUVp3S0JnSHA1MzRXVWNCMVZibzFlYStIMUxXWlFRUmxsTWlwRFM2TzBqeWZWSmtFb1BZSEJESnp2ClIrdzZLREJ4eFoyWmJsZ05LblV0YlhHSVFZd3lGelhNcFB5SGxNVHpiZkJhYmJLcDFyR2JVT2RCMXpXM09PRkgKcmppb21TUm1YNmxhaDk0SjRHU0lFZ0drNGw1SHhxZ3JGRDZ2UDd4NGRjUktJWFpLZ0w2dVJSSUpBb0dCQU1CVApaL2p5WStRNTBLdEtEZHUrYU9ORW4zaGxUN3hrNXRKN3NBek5rbWdGMU10RXlQUk9Xd1pQVGFJbWpRbk9qbHdpCldCZ2JGcXg0M2ZlQ1Z4ZXJ6V3ZEM0txaWJVbWpCTkNMTGtYeGh3ZEVteFQwVit2NzZGYzgwaTNNYVdSNnZZR08KditwVVovL0F6UXdJcWZ6dlVmV2ZxdStrMHlhVXhQOGNlcFBIRyt0bEFvR0FmQUtVVWhqeFU0Ym5vVzVwVUhKegpwWWZXZXZ5TW54NWZyT2VsSmRmNzlvNGMvMHhVSjh1eFBFWDFkRmNrZW96dHNpaVFTNkN6MENRY09XVWxtSkRwCnVrdERvVzM3VmNSQU1BVjY3NlgxQVZlM0UwNm5aL2g2Tkd4Z28rT042Q3pwL0lkMkJPUm9IMFAxa2RjY1NLT3kKMUtFZlNnb1B0c1N1eEpBZXdUZmxDMXc9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
diff --git a/tests/reconfig/scripts/create-resources-gw-last.sh b/tests/reconfig/scripts/create-resources-gw-last.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+num_namespaces=$1
+
+# Create namespaces
+for ((i=1; i<=$num_namespaces; i++)); do
+    namespace_name="namespace$i"
+    kubectl create namespace "$namespace_name"
+done
+
+# Create single instance resources
+kubectl create -f certificate-ns-and-cafe-secret.yaml
+kubectl create -f reference-grant.yaml
+
+# Create backend service and apps
+for ((i=1; i<=$num_namespaces; i++)); do
+    namespace_name="namespace$i"
+    sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe.yaml | kubectl apply -n "$namespace_name" -f -
+done
+
+# Create routes
+for ((i=1; i<=$num_namespaces; i++)); do
+    namespace_name="namespace$i"
+    sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe-routes.yaml | kubectl apply -n "$namespace_name" -f -
+done
+
+# Wait for apps to be ready
+sleep 60
+
+# Create Gateway
+kubectl create -f gateway.yaml
diff --git a/tests/reconfig/scripts/create-resources-routes-last.sh b/tests/reconfig/scripts/create-resources-routes-last.sh
@@ -0,0 +1,29 @@
+#!/bin/bash
+
+num_namespaces=$1
+
+# Create namespaces
+for ((i=1; i<=$num_namespaces; i++)); do
+    namespace_name="namespace$i"
+    kubectl create namespace "$namespace_name"
+done
+
+# Create backend service and apps
+for ((i=1; i<=$num_namespaces; i++)); do
+    namespace_name="namespace$i"
+    sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe.yaml | kubectl apply -n "$namespace_name" -f -
+done
+
+# Wait for apps to be ready
+sleep 60
+
+# Create single instance resources
+kubectl create -f certificate-ns-and-cafe-secret.yaml
+kubectl create -f reference-grant.yaml
+kubectl create -f gateway.yaml
+
+# Create routes
+for ((i=1; i<=$num_namespaces; i++)); do
+    namespace_name="namespace$i"
+    sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe-routes.yaml | kubectl apply -n "$namespace_name" -f -
+done
diff --git a/tests/reconfig/scripts/delete-multiple.sh b/tests/reconfig/scripts/delete-multiple.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+num_namespaces=$1
+
+# Delete namespaces
+for ((i=1; i<=$num_namespaces; i++)); do
+    namespace_name="namespace$i"
+    kubectl delete namespace "$namespace_name"
+done
+
+# Delete single instance resources
+kubectl delete -f gateway.yaml
+kubectl delete -f reference-grant.yaml
+kubectl delete -f certificate-ns-and-cafe-secret.yaml
diff --git a/tests/reconfig/scripts/gateway.yaml b/tests/reconfig/scripts/gateway.yaml
@@ -0,0 +1,25 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: Gateway
+metadata:
+  name: gateway
+spec:
+  gatewayClassName: nginx
+  listeners:
+  - name: http
+    port: 80
+    protocol: HTTP
+    allowedRoutes:
+      namespaces:
+        from: "All"
+  - name: https
+    port: 443
+    protocol: HTTPS
+    allowedRoutes:
+      namespaces:
+        from: "All"
+    tls:
+      mode: Terminate
+      certificateRefs:
+      - kind: Secret
+        name: cafe-secret
+        namespace: certificate
diff --git a/tests/reconfig/scripts/reference-grant.yaml b/tests/reconfig/scripts/reference-grant.yaml
@@ -0,0 +1,14 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: ReferenceGrant
+metadata:
+  name: access-to-cafe-secret
+  namespace: certificate
+spec:
+  to:
+  - group: ""
+    kind: Secret
+    name: cafe-secret # if you omit this name, then Gateways in default ns can access all Secrets in the certificate ns
+  from:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    namespace: default