From b46f54f75a7bbf431b05b7c92ba5727c88f8409d Mon Sep 17 00:00:00 2001
From: Michael Pleshakov <m.pleshakov@f5.com>
Date: Mon, 9 Oct 2023 15:50:39 -0400
Subject: [PATCH 1/7] Add zero downtime upgrade test plan and results

Problem:
- We don't know if it is possible to upgrade NGF from a previous
version without downtime.

Solution:
- Prepare a test plan to test for zero-downtime upgrades of NGF.
- Test NFG and share the results (TO-DO).

SOLVES -- https://github.com/nginxinc/nginx-gateway-fabric/issues/950
---
 tests/zero-downtime-upgrades/README.md | 85 ++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)
 create mode 100644 tests/zero-downtime-upgrades/README.md

diff --git a/tests/zero-downtime-upgrades/README.md b/tests/zero-downtime-upgrades/README.md
new file mode 100644
index 000000000..c21a64988
--- /dev/null
+++ b/tests/zero-downtime-upgrades/README.md
@@ -0,0 +1,85 @@
+# Zero-Downtime Upgrades
+
+This document describes a test plan for testing zero-downtime upgrades of NGF.
+
+*Zero-downtime upgrades* means that during an NGF upgrade clients don't experience any
+interruptions to the traffic they send to applications exposed via NGF.
+
+## Goals
+
+- Ensure that upgrading NFG doesn't lead to any loss of traffic flowing through the data plane.
+- Ensure that after an upgrade, NGF can process changes to resources.
+
+## Non-Goals
+
+During an upgrade, Kubernetes will shut down existing NGF pods by sending a SIGTERM. If the pod doesn't terminate in 30
+seconds (the default period) , Kubernetes will send a SIGKILL.
+
+When proxying Websocket or any long-lived connections, NGINX will not terminate until
+that connection is closed by either the client or the backend. This means that unless all those connections are closed
+by clients/backends before or during an upgrade (which is highly unlikely), NGINX will not terminate, which means
+Kubernetes will kill NGINX. As a result, the clients will see the connections abruptly closed and thus experience
+downtime.
+
+As a result, we *will not* use any long-live connections in this test, because NGF cannot support zero-downtime upgrades
+in this case.
+
+## Test Environment
+
+- A Kubernetes cluster with 3 nodes on GKE
+    - Node: e2-medium (2 vCPU, 4GB memory)
+    - Enabled GKE logging.
+- Tester VMs:
+    - Configuration:
+        - Debian
+        - Install packages: wrk
+    - Location - same zone as the Kubernetes cluster.
+    - First VM - for HTTP traffic
+    - Second VM - for sending HTTPs traffic
+- NGF
+    - Deployment with 2 replicas
+    - Exposed via a Service with type LoadBalancer, private IP
+    - Gateway, two listeners - HTTP and HTTPs
+    - Two apps:
+        - Coffee - 3 replicas
+        - Tea - 3 replicas
+    - Two HTTPRoutes
+        - Coffee (HTTP)
+        - Tea (HTTPS)
+
+## Steps
+
+### Start
+
+- Create a cluster
+- Deploy NGF, previous latest stable version with 2 replicas.
+- Expose NGF via Service Load Balancer, internal (only accessible within the Google Cloud region)
+- Deploy backend apps
+- Configure Gateway
+- Expose apps via HTTPRoutes
+- Check statuses of the Gateway and HTTPRoutes
+- Start sending traffic using wrk from tester VMs: HTTP and HTTPs.
+- Check that NGINX access logs look good (all responses 200)
+- Check that there are no errors in NGINX errors logs and NGF error logs.
+
+### Upgrade
+
+- Upgrade to the new version build from main branch. For example, `kubectl apply -f` the latest manifests.
+- Check that the new pods are running and the old one are removed.
+
+## After Upgrade
+
+- Update the Gateway resource - add one listener -- and make sure NGF processed the update (it updates the status of
+  the Gateway resource accordingly).
+
+### Analyze
+
+- Stop wrk and save the output
+- Check old pods logs (in Google Monitoring)
+    - NGINX Access logs - no errors
+    - NGINX Error logs - no errors
+    - NGF logs - no errors
+- Check New pods (in Google Monitoring)
+    - NGINX Access logs - no errors
+    - NGINX Error logs - no errors
+    - NGF logs - no errors

From 7121d7fc0b1b02290a150beb9fe34afabd97b1c3 Mon Sep 17 00:00:00 2001
From: Michael Pleshakov <m.pleshakov@f5.com>
Date: Fri, 13 Oct 2023 19:38:40 -0400
Subject: [PATCH 2/7] Update the draft plan and add test results

---
 tests/zero-downtime-upgrades/README.md        |  85 ---
 .../manifests/cafe-routes.yaml                |  37 ++
 .../manifests/cafe-secret.yaml                |   8 +
 .../manifests/cafe.yaml                       |  65 ++
 .../manifests/gateway-updated.yaml            |  24 +
 .../manifests/gateway.yaml                    |  20 +
 tests/zero-downtime-upgrades/requests-plot.gp |  21 +
 .../results/1.0.0/1.0.0.md                    | 231 +++++++
 .../results/1.0.0/http.csv                    | 600 ++++++++++++++++++
 .../results/1.0.0/http.png                    | Bin 0 -> 6548 bytes
 .../results/1.0.0/https.csv                   | 600 ++++++++++++++++++
 .../results/1.0.0/https.png                   | Bin 0 -> 6635 bytes
 .../zero-downtime-upgrades.md                 | 252 ++++++++
 13 files changed, 1858 insertions(+), 85 deletions(-)
 delete mode 100644 tests/zero-downtime-upgrades/README.md
 create mode 100644 tests/zero-downtime-upgrades/manifests/cafe-routes.yaml
 create mode 100644 tests/zero-downtime-upgrades/manifests/cafe-secret.yaml
 create mode 100644 tests/zero-downtime-upgrades/manifests/cafe.yaml
 create mode 100644 tests/zero-downtime-upgrades/manifests/gateway-updated.yaml
 create mode 100644 tests/zero-downtime-upgrades/manifests/gateway.yaml
 create mode 100644 tests/zero-downtime-upgrades/requests-plot.gp
 create mode 100644 tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
 create mode 100644 tests/zero-downtime-upgrades/results/1.0.0/http.csv
 create mode 100644 tests/zero-downtime-upgrades/results/1.0.0/http.png
 create mode 100644 tests/zero-downtime-upgrades/results/1.0.0/https.csv
 create mode 100644 tests/zero-downtime-upgrades/results/1.0.0/https.png
 create mode 100644 tests/zero-downtime-upgrades/zero-downtime-upgrades.md

diff --git a/tests/zero-downtime-upgrades/README.md b/tests/zero-downtime-upgrades/README.md
deleted file mode 100644
index c21a64988..000000000
--- a/tests/zero-downtime-upgrades/README.md
+++ /dev/null
@@ -1,85 +0,0 @@
-# Zero-Downtime Upgrades
-
-This document describes a test plan for testing zero-downtime upgrades of NGF.
-
-*Zero-downtime upgrades* means that during an NGF upgrade clients don't experience any
-interruptions to the traffic they send to applications exposed via NGF.
-
-## Goals
-
-- Ensure that upgrading NFG doesn't lead to any loss of traffic flowing through the data plane.
-- Ensure that after an upgrade, NGF can process changes to resources.
-
-## Non-Goals
-
-During an upgrade, Kubernetes will shut down existing NGF pods by sending a SIGTERM. If the pod doesn't terminate in 30
-seconds (the default period) , Kubernetes will send a SIGKILL.
-
-When proxying Websocket or any long-lived connections, NGINX will not terminate until
-that connection is closed by either the client or the backend. This means that unless all those connections are closed
-by clients/backends before or during an upgrade (which is highly unlikely), NGINX will not terminate, which means
-Kubernetes will kill NGINX. As a result, the clients will see the connections abruptly closed and thus experience
-downtime.
-
-As a result, we *will not* use any long-live connections in this test, because NGF cannot support zero-downtime upgrades
-in this case.
-
-## Test Environment
-
-- A Kubernetes cluster with 3 nodes on GKE
-    - Node: e2-medium (2 vCPU, 4GB memory)
-    - Enabled GKE logging.
-- Tester VMs:
-    - Configuration:
-        - Debian
-        - Install packages: wrk
-    - Location - same zone as the Kubernetes cluster.
-    - First VM - for HTTP traffic
-    - Second VM - for sending HTTPs traffic
-- NGF
-    - Deployment with 2 replicas
-    - Exposed via a Service with type LoadBalancer, private IP
-    - Gateway, two listeners - HTTP and HTTPs
-    - Two apps:
-        - Coffee - 3 replicas
-        - Tea - 3 replicas
-    - Two HTTPRoutes
-        - Coffee (HTTP)
-        - Tea (HTTPS)
-
-## Steps
-
-### Start
-
-- Create a cluster
-- Deploy NGF, previous latest stable version with 2 replicas.
-- Expose NGF via Service Load Balancer, internal (only accessible within the Google Cloud region)
-- Deploy backend apps
-- Configure Gateway
-- Expose apps via HTTPRoutes
-- Check statuses of the Gateway and HTTPRoutes
-- Start sending traffic using wrk from tester VMs: HTTP and HTTPs.
-- Check that NGINX access logs look good (all responses 200)
-- Check that there are no errors in NGINX errors logs and NGF error logs.
-
-### Upgrade
-
-- Upgrade to the new version build from main branch. For example, `kubectl apply -f` the latest manifests.
-- Check that the new pods are running and the old one are removed.
-
-## After Upgrade
-
-- Update the Gateway resource - add one listener -- and make sure NGF processed the update (it updates the status of
-  the Gateway resource accordingly).
-
-### Analyze
-
-- Stop wrk and save the output
-- Check old pods logs (in Google Monitoring)
-    - NGINX Access logs - no errors
-    - NGINX Error logs - no errors
-    - NGF logs - no errors
-- Check New pods (in Google Monitoring)
-    - NGINX Access logs - no errors
-    - NGINX Error logs - no errors
-    - NGF logs - no errors
diff --git a/tests/zero-downtime-upgrades/manifests/cafe-routes.yaml b/tests/zero-downtime-upgrades/manifests/cafe-routes.yaml
new file mode 100644
index 000000000..e679756d6
--- /dev/null
+++ b/tests/zero-downtime-upgrades/manifests/cafe-routes.yaml
@@ -0,0 +1,37 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+  name: coffee
+spec:
+  parentRefs:
+  - name: gateway
+    sectionName: http
+  hostnames:
+  - "cafe.example.com"
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /coffee
+    backendRefs:
+    - name: coffee
+      port: 80
+---
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+  name: tea
+spec:
+  parentRefs:
+  - name: gateway
+    sectionName: https
+  hostnames:
+  - "cafe.example.com"
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /tea
+    backendRefs:
+    - name: tea
+      port: 80
diff --git a/tests/zero-downtime-upgrades/manifests/cafe-secret.yaml b/tests/zero-downtime-upgrades/manifests/cafe-secret.yaml
new file mode 100644
index 000000000..4510460bb
--- /dev/null
+++ b/tests/zero-downtime-upgrades/manifests/cafe-secret.yaml
@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cafe-secret
+type: kubernetes.io/tls
+data:
+  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNzakNDQVpvQ0NRQzdCdVdXdWRtRkNEQU5CZ2txaGtpRzl3MEJBUXNGQURBYk1Sa3dGd1lEVlFRRERCQmoKWVdabExtVjRZVzF3YkdVdVkyOXRNQjRYRFRJeU1EY3hOREl4TlRJek9Wb1hEVEl6TURjeE5ESXhOVEl6T1ZvdwpHekVaTUJjR0ExVUVBd3dRWTJGbVpTNWxlR0Z0Y0d4bExtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTHFZMnRHNFc5aStFYzJhdnV4Q2prb2tnUUx1ek10U1Rnc1RNaEhuK3ZRUmxIam8KVzFLRnMvQVdlS25UUStyTWVKVWNseis4M3QwRGtyRThwUisxR2NKSE50WlNMb0NEYUlRN0Nhck5nY1daS0o4Qgo1WDNnVS9YeVJHZjI2c1REd2xzU3NkSEQ1U2U3K2Vab3NPcTdHTVF3K25HR2NVZ0VtL1Q1UEMvY05PWE0zZWxGClRPL051MStoMzROVG9BbDNQdTF2QlpMcDNQVERtQ0thaEROV0NWbUJQUWpNNFI4VERsbFhhMHQ5Z1o1MTRSRzUKWHlZWTNtdzZpUzIrR1dYVXllMjFuWVV4UEhZbDV4RHY0c0FXaGRXbElweHlZQlNCRURjczN6QlI2bFF1OWkxZAp0R1k4dGJ3blVmcUVUR3NZdWxzc05qcU95V1VEcFdJelhibHhJZVVDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBcjkrZWJ0U1dzSnhLTGtLZlRkek1ISFhOd2Y5ZXFVbHNtTXZmMGdBdWVKTUpUR215dG1iWjlpbXQKL2RnWlpYVE9hTElHUG9oZ3BpS0l5eVVRZVdGQ2F0NHRxWkNPVWRhbUloOGk0Q1h6QVJYVHNvcUNOenNNLzZMRQphM25XbFZyS2lmZHYrWkxyRi8vblc0VVNvOEoxaCtQeDljY0tpRDZZU0RVUERDRGh1RUtFWXcvbHpoUDJVOXNmCnl6cEJKVGQ4enFyM3paTjNGWWlITmgzYlRhQS82di9jU2lyamNTK1EwQXg4RWpzQzYxRjRVMTc4QzdWNWRCKzQKcmtPTy9QNlA0UFlWNTRZZHMvRjE2WkZJTHFBNENCYnExRExuYWRxamxyN3NPbzl2ZzNnWFNMYXBVVkdtZ2todAp6VlZPWG1mU0Z4OS90MDBHUi95bUdPbERJbWlXMGc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
+  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzZtTnJSdUZ2WXZoSE4KbXI3c1FvNUtKSUVDN3N6TFVrNExFeklSNS9yMEVaUjQ2RnRTaGJQd0ZuaXAwMFBxekhpVkhKYy92TjdkQTVLeApQS1VmdFJuQ1J6YldVaTZBZzJpRU93bXF6WUhGbVNpZkFlVjk0RlAxOGtSbjl1ckV3OEpiRXJIUncrVW51L25tCmFMRHF1eGpFTVBweGhuRklCSnYwK1R3djNEVGx6TjNwUlV6dnpidGZvZCtEVTZBSmR6N3Rid1dTNmR6MHc1Z2kKbW9RelZnbFpnVDBJek9FZkV3NVpWMnRMZllHZWRlRVJ1VjhtR041c09va3R2aGxsMU1udHRaMkZNVHgySmVjUQo3K0xBRm9YVnBTS2NjbUFVZ1JBM0xOOHdVZXBVTHZZdFhiUm1QTFc4SjFINmhFeHJHTHBiTERZNmpzbGxBNlZpCk0xMjVjU0hsQWdNQkFBRUNnZ0VBQnpaRE50bmVTdWxGdk9HZlFYaHRFWGFKdWZoSzJBenRVVVpEcUNlRUxvekQKWlV6dHdxbkNRNlJLczUyandWNTN4cU9kUU94bTNMbjNvSHdNa2NZcEliWW82MjJ2dUczYnkwaVEzaFlsVHVMVgpqQmZCcS9UUXFlL2NMdngvSkczQWhFNmJxdFRjZFlXeGFmTmY2eUtpR1dzZk11WVVXTWs4MGVJVUxuRmZaZ1pOCklYNTlSOHlqdE9CVm9Sa3hjYTVoMW1ZTDFsSlJNM3ZqVHNHTHFybmpOTjNBdWZ3ZGRpK1VDbGZVL2l0K1EvZkUKV216aFFoTlRpNVFkRWJLVStOTnYvNnYvb2JvandNb25HVVBCdEFTUE05cmxFemIralQ1WHdWQjgvLzRGY3VoSwoyVzNpcjhtNHVlQ1JHSVlrbGxlLzhuQmZ0eVhiVkNocVRyZFBlaGlPM1FLQmdRRGlrR3JTOTc3cjg3Y1JPOCtQClpoeXltNXo4NVIzTHVVbFNTazJiOTI1QlhvakpZL2RRZDVTdFVsSWE4OUZKZnNWc1JRcEhHaTFCYzBMaTY1YjIKazR0cE5xcVFoUmZ1UVh0UG9GYXRuQzlPRnJVTXJXbDVJN0ZFejZnNkNQMVBXMEg5d2hPemFKZUdpZVpNYjlYTQoybDdSSFZOcC9jTDlYbmhNMnN0Q1lua2Iwd0tCZ1FEUzF4K0crakEyUVNtRVFWNXA1RnRONGcyamsyZEFjMEhNClRIQ2tTazFDRjhkR0Z2UWtsWm5ZbUt0dXFYeXNtekJGcnZKdmt2eUhqbUNYYTducXlpajBEdDZtODViN3BGcVAKQWxtajdtbXI3Z1pUeG1ZMXBhRWFLMXY4SDNINGtRNVl3MWdrTWRybVJHcVAvaTBGaDVpaGtSZS9DOUtGTFVkSQpDcnJjTzhkUVp3S0JnSHA1MzRXVWNCMVZibzFlYStIMUxXWlFRUmxsTWlwRFM2TzBqeWZWSmtFb1BZSEJESnp2ClIrdzZLREJ4eFoyWmJsZ05LblV0YlhHSVFZd3lGelhNcFB5SGxNVHpiZkJhYmJLcDFyR2JVT2RCMXpXM09PRkgKcmppb21TUm1YNmxhaDk0SjRHU0lFZ0drNGw1SHhxZ3JGRDZ2UDd4NGRjUktJWFpLZ0w2dVJSSUpBb0dCQU1CVApaL2p5WStRNTBLdEtEZHUrYU9ORW4zaGxUN3hrNXRKN3NBek5rbWdGMU10RXlQUk9Xd1pQVGFJbWpRbk9qbHdpCldCZ2JGcXg0M2ZlQ1Z4ZXJ6V3ZEM0txaWJVbWpCTkNMTGtYeGh3ZEVteFQwVit2NzZGYzgwaTNNYVdSNnZZR08KditwVVovL0F6UXdJcWZ6dlVmV2ZxdStrMHlhVXhQOGNlcFBIRyt0bEFvR0FmQUtVVWhqeFU0Ym5vVzVwVUhKegpwWWZXZXZ5TW54NWZyT2VsSmRmNzlvNGMvMHhVSjh1eFBFWDFkRmNrZW96dHNpaVFTNkN6MENRY09XVWxtSkRwCnVrdERvVzM3VmNSQU1BVjY3NlgxQVZlM0UwNm5aL2g2Tkd4Z28rT042Q3pwL0lkMkJPUm9IMFAxa2RjY1NLT3kKMUtFZlNnb1B0c1N1eEpBZXdUZmxDMXc9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
diff --git a/tests/zero-downtime-upgrades/manifests/cafe.yaml b/tests/zero-downtime-upgrades/manifests/cafe.yaml
new file mode 100644
index 000000000..9c1a83548
--- /dev/null
+++ b/tests/zero-downtime-upgrades/manifests/cafe.yaml
@@ -0,0 +1,65 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: coffee
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: coffee
+  template:
+    metadata:
+      labels:
+        app: coffee
+    spec:
+      containers:
+      - name: coffee
+        image: nginxdemos/nginx-hello:plain-text
+        ports:
+        - containerPort: 8080
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: coffee
+spec:
+  ports:
+  - port: 80
+    targetPort: 8080
+    protocol: TCP
+    name: http
+  selector:
+    app: coffee
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: tea
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: tea
+  template:
+    metadata:
+      labels:
+        app: tea
+    spec:
+      containers:
+      - name: tea
+        image: nginxdemos/nginx-hello:plain-text
+        ports:
+        - containerPort: 8080
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: tea
+spec:
+  ports:
+  - port: 80
+    targetPort: 8080
+    protocol: TCP
+    name: http
+  selector:
+    app: tea
diff --git a/tests/zero-downtime-upgrades/manifests/gateway-updated.yaml b/tests/zero-downtime-upgrades/manifests/gateway-updated.yaml
new file mode 100644
index 000000000..c501ce839
--- /dev/null
+++ b/tests/zero-downtime-upgrades/manifests/gateway-updated.yaml
@@ -0,0 +1,24 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: Gateway
+metadata:
+  name: gateway
+spec:
+  gatewayClassName: nginx
+  listeners:
+  - name: http
+    port: 80
+    protocol: HTTP
+    hostname: "*.example.com"
+  - name: http-new
+    port: 80
+    protocol: HTTP
+    hostname: "*.example.org"
+  - name: https
+    port: 443
+    protocol: HTTPS
+    hostname: "*.example.com"
+    tls:
+      mode: Terminate
+      certificateRefs:
+      - kind: Secret
+        name: cafe-secret
diff --git a/tests/zero-downtime-upgrades/manifests/gateway.yaml b/tests/zero-downtime-upgrades/manifests/gateway.yaml
new file mode 100644
index 000000000..593d17e49
--- /dev/null
+++ b/tests/zero-downtime-upgrades/manifests/gateway.yaml
@@ -0,0 +1,20 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: Gateway
+metadata:
+  name: gateway
+spec:
+  gatewayClassName: nginx
+  listeners:
+  - name: http
+    port: 80
+    protocol: HTTP
+    hostname: "*.example.com"
+  - name: https
+    port: 443
+    protocol: HTTPS
+    hostname: "*.example.com"
+    tls:
+      mode: Terminate
+      certificateRefs:
+      - kind: Secret
+        name: cafe-secret
diff --git a/tests/zero-downtime-upgrades/requests-plot.gp b/tests/zero-downtime-upgrades/requests-plot.gp
new file mode 100644
index 000000000..b08003d70
--- /dev/null
+++ b/tests/zero-downtime-upgrades/requests-plot.gp
@@ -0,0 +1,21 @@
+set terminal png size 800,600
+set output "graph.png"
+set title "Request Outcomes Over Time"
+
+set xdata time
+set timefmt "%Y-%m-%d %H:%M:%S"
+set datafile separator ","
+
+# Y-axis settings
+set yrange [-0.5:1.5]   # Provide some padding around 0 and 1 for better visualization
+set ytics ("Failed" 0, "Success" 1)
+set grid ytics           # Gridlines for Y
+
+# Define the palette: 0 for red (Failure) and 1 for green (Success)
+set palette defined (0 "red", 1 "green")
+
+# Hide the colorbox
+unset colorbox
+
+# Plotting data
+plot "results.csv" using 1:2:2 with points palette pointtype 7 pointsize 1.5 title "Request Status"
diff --git a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
new file mode 100644
index 000000000..4bc9e4cc9
--- /dev/null
+++ b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
@@ -0,0 +1,231 @@
+# Results
+
+<!-- TOC -->
+- [Results](#results)
+  - [Versions](#versions)
+  - [Start](#start)
+  - [Upgrades](#upgrades)
+    - [Analyze](#analyze)
+      - [Tester VMs](#tester-vms)
+      - [Old pods](#old-pods)
+      - [New pods](#new-pods)
+  - [Opened Issues](#opened-issues)
+  - [Future Improvements](#future-improvements)
+<!-- TOC -->
+
+## Versions
+
+Kubernetes:
+
+```text
+Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3-gke.100", GitCommit:"6466b51b762a5c49ae3fb6c2c7233ffe1c96e48c", GitTreeState:"clean", BuildDate:"2023-06-23T09:27:28Z", GoVersion:"go1.20.5 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
+```
+
+Old NGF version:
+
+```text
+"version":"0.6.0"
+"commit":"803e6d612a9574362bda28868d4410943ffaf66a"
+"date":"2023-08-31T16:32:37Z"
+```
+
+New NGF version:
+
+```text
+version: "edge"
+commit: "5324908e6e1145bec5f2f0ab80b312a809ad1744"
+date: "2023-10-13T18:29:23Z"
+```
+
+## Start
+
+Deployed Pods:
+
+```text
+NAME                             READY   STATUS    RESTARTS   AGE   IP            NODE                                       NOMINATED NODE   READINESS GATES
+nginx-gateway-55cb958549-nbdqz   2/2     Running   0          11s   10.112.7.8    gke-michael-2-default-pool-18ad0f59-19nb   <none>           <none>
+nginx-gateway-55cb958549-nj95k   2/2     Running   0          11s   10.112.6.25   gke-michael-2-default-pool-18ad0f59-zfw4   <none>           <none>
+```
+
+Logs check:
+
+- nginx-gateway-55cb958549-nbdqz
+  - NGINX logs - no errors
+  - NGF logs - 2 errors like below with failure to update status of a resource.
+
+      ```json
+      {
+        "name": "nginx",
+        "ts": "2023-10-13T21:50:29Z",
+        "logger": "statusUpdater",
+        "error": "Operation cannot be fulfilled on gatewayclasses.gateway.networking.k8s.io \"nginx\": the object has been modified; please apply your changes to the latest version and try again",
+        "kind": "GatewayClass",
+        "namespace": "",
+        "msg": "Failed to update status",
+        "stacktrace": "github.com/nginxinc/nginx-kubernetes-gateway/internal/framework/status.(*updaterImpl).update\n\t/home/runner/work/nginx-kubernetes-gateway/nginx-kubernetes-gateway/internal/framework/status/updater.go:166\ngithub.com/nginxinc/nginx-kubernetes-gateway/internal/framework/status.(*updaterImpl).Update\n\t/home/runner/work/nginx-kubernetes-gateway/nginx-kubernetes-gateway/internal/framework/status/updater.go:90\ngithub.com/nginxinc/nginx-kubernetes-gateway/internal/mode/static.(*eventHandlerImpl).HandleEventBatch\n\t/home/runner/work/nginx-kubernetes-gateway/nginx-kubernetes-gateway/internal/mode/static/handler.go:101\ngithub.com/nginxinc/nginx-kubernetes-gateway/internal/framework/events.(*EventLoop).Start.func1.1\n\t/home/runner/work/nginx-kubernetes-gateway/nginx-kubernetes-gateway/internal/framework/events/loop.go:68",
+        "level": "error"
+      }
+      ```
+
+      Such errors are expected in 0.6.0 because we're running two replicas and leader election wasn't implemented yet.
+      As a result, we see conflicts when both replicas try to update statuses.
+- nginx-gateway-55cb958549-nj95k
+  - NGINX logs - no error
+  - NGF logs - 6 errors related to status updates.
+
+## Upgrades
+
+New pods:
+
+```text
+nginx-gateway-578b49bc58-hmx5x   2/2     Running   0          11s   10.112.1.9    gke-michael-2-default-pool-18ad0f59-7dqr   <none>           <none>
+nginx-gateway-578b49bc58-r4ckb   2/2     Running   0          17s   10.112.5.26   gke-michael-2-default-pool-18ad0f59-l0cq   <none>           <none>
+```
+
+Note: the new pods were scheduled on different from the old pods nodes, as we wanted.
+
+Check that one of the NGF pods became the leader:
+
+```text
+I1013 22:02:32.226414       7 leaderelection.go:250] attempting to acquire leader lease nginx-gateway/nginx-gateway-leader-election...
+I1013 22:02:32.248857       7 leaderelection.go:260] successfully acquired lease nginx-gateway/nginx-gateway-leader-election
+{"level":"info","ts":"2023-10-13T22:02:32Z","logger":"leaderElector","msg":"Started leading"}
+```
+
+Pod nginx-gateway-578b49bc58-r4ckb is the leader.
+
+Gateway status has been updated with the new listener.
+
+### Analyze
+
+#### Tester VMs
+
+Tester 1 wrk output:
+
+```text
+Running 1m test @ http://cafe.example.com/coffee
+  2 threads and 100 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency    29.23ms   39.24ms   1.15s    95.23%
+    Req/Sec     1.47k   765.16     4.46k    82.25%
+  Latency Distribution
+     50%   21.66ms
+     75%   36.03ms
+     90%   56.05ms
+     99%  113.86ms
+  175177 requests in 1.00m, 62.59MB read
+  Socket errors: connect 0, read 25, write 102, timeout 0
+Requests/sec:   2918.55
+Transfer/sec:      1.04MB
+```
+
+There are socket errors, but no timeout or connect errors.
+
+Tester 1 graph:
+
+![http.png](http.png)
+
+As we can see, there is period when curl failed to send requests
+to the tea app.
+
+Tester 2 wrk output:
+
+```text
+Running 1m test @ https://cafe.example.com/tea
+  2 threads and 100 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency    27.59ms   20.92ms 276.16ms   75.13%
+    Req/Sec     1.40k   717.86     3.85k    80.83%
+  Latency Distribution
+     50%   22.65ms
+     75%   36.58ms
+     90%   55.70ms
+     99%   96.40ms
+  166933 requests in 1.00m, 58.48MB read
+  Socket errors: connect 85, read 43, write 0, timeout 0
+Requests/sec:   2780.49
+Transfer/sec:      0.97MB
+```
+
+There socket errors including 85 connect errors.
+
+Tester 2 graph:
+
+![https.png](https.png)
+
+As we can see, there is period where curl failed to send requests
+to the coffee app.
+
+#### Old pods
+
+- nginx-gateway-55cb958549-nbdqz
+  - NFG handled a panic before exiting:
+
+      ```text
+      INFO 2023-10-13T22:02:33.321381968Z [resource.labels.containerName: nginx-gateway] {"level":"info", "msg":"Stopping and waiting for caches", "ts":"2023-10-13T22:02:33Z"}
+      INFO 2023-10-13T22:02:33.321821473Z [resource.labels.containerName: nginx-gateway] {"level":"info", "msg":"Stopping and waiting for webhooks", "ts":"2023-10-13T22:02:33Z"}
+      INFO 2023-10-13T22:02:33.322852013Z [resource.labels.containerName: nginx-gateway] {"level":"info", "msg":"Stopping and waiting for HTTP servers", "ts":"2023-10-13T22:02:33Z"}
+      ERROR 2023-10-13T22:02:33.323409193Z [resource.labels.containerName: nginx-gateway] [controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
+      ERROR 2023-10-13T22:02:33.323442347Z [resource.labels.containerName: nginx-gateway] Detected at:
+      ERROR 2023-10-13T22:02:33.323451863Z [resource.labels.containerName: nginx-gateway] > goroutine 67 [running]:
+      ERROR 2023-10-13T22:02:33.323457043Z [resource.labels.containerName: nginx-gateway] > runtime/debug.Stack()
+      ERROR 2023-10-13T22:02:33.323462448Z [resource.labels.containerName: nginx-gateway] > $GOROOT/src/runtime/debug/stack.go:24 +0x5e
+      ERROR 2023-10-13T22:02:33.323467879Z [resource.labels.containerName: nginx-gateway] > sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
+      ERROR 2023-10-13T22:02:33.323472975Z [resource.labels.containerName: nginx-gateway] > pkg/mod/sigs.k8s.io/controller-runtime@v0.16.0/pkg/log/log.go:60 +0xcd
+      ERROR 2023-10-13T22:02:33.323478263Z [resource.labels.containerName: nginx-gateway] > sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc0003d6dc0, 0xc000288418?)
+      ERROR 2023-10-13T22:02:33.323483535Z [resource.labels.containerName: nginx-gateway] > pkg/mod/sigs.k8s.io/controller-runtime@v0.16.0/pkg/log/deleg.go:111 +0x32
+      ERROR 2023-10-13T22:02:33.323488655Z [resource.labels.containerName: nginx-gateway] > github.com/go-logr/logr.Logger.Enabled(...)
+      ERROR 2023-10-13T22:02:33.323494496Z [resource.labels.containerName: nginx-gateway] > pkg/mod/github.com/go-logr/logr@v1.2.4/logr.go:261
+      ERROR 2023-10-13T22:02:33.323499859Z [resource.labels.containerName: nginx-gateway] > github.com/go-logr/logr.Logger.Info({{0x1b58a48?, 0xc0003d6dc0?}, 0x1b55578?}, {0x19139af, 0x35}, {0x0, 0x0, 0x0})
+      ERROR 2023-10-13T22:02:33.323504827Z [resource.labels.containerName: nginx-gateway] > pkg/mod/github.com/go-logr/logr@v1.2.4/logr.go:274 +0x72
+      ERROR 2023-10-13T22:02:33.323509715Z [resource.labels.containerName: nginx-gateway] > sigs.k8s.io/controller-runtime/pkg/metrics/server.(*defaultServer).Start.func1()
+      ERROR 2023-10-13T22:02:33.323514683Z [resource.labels.containerName: nginx-gateway] > pkg/mod/sigs.k8s.io/controller-runtime@v0.16.0/pkg/metrics/server/server.go:231 +0x74
+      ERROR 2023-10-13T22:02:33.323519441Z [resource.labels.containerName: nginx-gateway] > created by sigs.k8s.io/controller-runtime/pkg/metrics/server.(*defaultServer).Start in goroutine 65
+      ERROR 2023-10-13T22:02:33.323524696Z [resource.labels.containerName: nginx-gateway] > pkg/mod/sigs.k8s.io/controller-runtime@v0.16.0/pkg/metrics/server/server.go:229 +0x825
+      INFO 2023-10-13T22:02:33.325625043Z [resource.labels.containerName: nginx-gateway] {"level":"info", "msg":"Wait completed, proceeding to shutdown the manager", "ts":"2023-10-13T22:02:33Z"}
+      ```
+
+      This error is specific to 0.6.0 release, graceful shutdown of the used edge version (tried separately) doesn't
+      lead to such an error.
+  - NGINX
+    - Access logs - all responses are 200.
+    - Error logs - no errors or warnings.
+- nginx-gateway-55cb958549-nj95k
+  - NGF - same error.
+  - NGINX
+    - Access logs - only 200 responses.
+    - Error logs - no errors or warnings.
+
+#### New pods
+
+- nginx-gateway-578b49bc58-hmx5x
+  - NGF - no errors
+  - NGINX
+    - Access logs - 46 responses like below:
+
+          ```text
+          INFO 2023-10-13T22:03:17.618556029Z [resource.labels.containerName: nginx] 10.128.0.9 - - [13/Oct/2023:22:03:17 +0000] "GET /coffee HTTP/1.1" 499 0 "-" "-"
+          . . .
+          INFO 2023-10-13T22:03:22.747023411Z [resource.labels.containerName: nginx] 10.128.15.241 - - [13/Oct/2023:22:03:22 +0000] "GET /tea HTTP/1.1" 499 0 "-" "-"
+          ```
+
+          Meaning clients closed connection (499 status code).
+          Those requests belong to wrk (curl requests have `curl` user agent in the logs).
+          All requests for coffee are during the same time. Same for tea.
+          That's probably how wrk closed connections before it exited after 60s.
+    - Error logs - No errors or warnings.
+- nginx-gateway-578b49bc58-r4ckb
+  - NGF - no errors.
+  - NGINX
+    - Access logs - 35 responses similar to the first pod. Same conclusion as above.
+    - Error logs - No errors or warnings.
+
+## Opened Issues
+
+- Traffic loss during an upgrade. (to be opened soon)
+
+## Future Improvements
+
+- Use helm for upgrade, to catch any helm-related bugs preventing an upgrade. In this test,
+  we didn't use helm because 0.6.0 release does not allow you to configure the number of replicas
+  and pod affinity.
diff --git a/tests/zero-downtime-upgrades/results/1.0.0/http.csv b/tests/zero-downtime-upgrades/results/1.0.0/http.csv
new file mode 100644
index 000000000..30d50c4c5
--- /dev/null
+++ b/tests/zero-downtime-upgrades/results/1.0.0/http.csv
@@ -0,0 +1,600 @@
+2023-10-13 22:02:16.401267754+00:00,1
+2023-10-13 22:02:16.517072322+00:00,1
+2023-10-13 22:02:16.630980807+00:00,1
+2023-10-13 22:02:16.746924746+00:00,1
+2023-10-13 22:02:16.864835368+00:00,1
+2023-10-13 22:02:16.981624780+00:00,1
+2023-10-13 22:02:17.099343751+00:00,1
+2023-10-13 22:02:17.215674226+00:00,1
+2023-10-13 22:02:17.332868389+00:00,1
+2023-10-13 22:02:17.449274400+00:00,1
+2023-10-13 22:02:17.565786594+00:00,1
+2023-10-13 22:02:17.704246902+00:00,1
+2023-10-13 22:02:17.845398819+00:00,1
+2023-10-13 22:02:17.981309833+00:00,1
+2023-10-13 22:02:18.145217228+00:00,1
+2023-10-13 22:02:18.278931776+00:00,1
+2023-10-13 22:02:18.417801725+00:00,1
+2023-10-13 22:02:18.548661822+00:00,1
+2023-10-13 22:02:18.699792112+00:00,1
+2023-10-13 22:02:18.835705391+00:00,1
+2023-10-13 22:02:18.974460460+00:00,1
+2023-10-13 22:02:19.116605364+00:00,1
+2023-10-13 22:02:19.254361903+00:00,1
+2023-10-13 22:02:19.391363458+00:00,1
+2023-10-13 22:02:19.534092239+00:00,1
+2023-10-13 22:02:19.673546496+00:00,1
+2023-10-13 22:02:19.810678816+00:00,1
+2023-10-13 22:02:19.945477351+00:00,1
+2023-10-13 22:02:20.082527062+00:00,1
+2023-10-13 22:02:20.216756932+00:00,1
+2023-10-13 22:02:20.355692658+00:00,1
+2023-10-13 22:02:20.493617689+00:00,1
+2023-10-13 22:02:20.618808138+00:00,1
+2023-10-13 22:02:20.752063198+00:00,1
+2023-10-13 22:02:20.909011702+00:00,1
+2023-10-13 22:02:21.047813748+00:00,1
+2023-10-13 22:02:21.181231455+00:00,1
+2023-10-13 22:02:21.347212639+00:00,1
+2023-10-13 22:02:21.477722536+00:00,1
+2023-10-13 22:02:21.609348537+00:00,1
+2023-10-13 22:02:21.744690966+00:00,1
+2023-10-13 22:02:21.887596612+00:00,1
+2023-10-13 22:02:22.023796658+00:00,1
+2023-10-13 22:02:22.164814593+00:00,1
+2023-10-13 22:02:22.300710442+00:00,1
+2023-10-13 22:02:22.441841371+00:00,1
+2023-10-13 22:02:22.579973798+00:00,1
+2023-10-13 22:02:22.717057907+00:00,1
+2023-10-13 22:02:22.866475270+00:00,1
+2023-10-13 22:02:23.023339741+00:00,1
+2023-10-13 22:02:23.185375712+00:00,1
+2023-10-13 22:02:23.340773178+00:00,1
+2023-10-13 22:02:23.517860180+00:00,1
+2023-10-13 22:02:23.691238114+00:00,1
+2023-10-13 22:02:23.874298745+00:00,1
+2023-10-13 22:02:24.033022752+00:00,1
+2023-10-13 22:02:24.191817349+00:00,1
+2023-10-13 22:02:24.339531648+00:00,1
+2023-10-13 22:02:24.491559332+00:00,1
+2023-10-13 22:02:24.659885507+00:00,1
+2023-10-13 22:02:24.817504273+00:00,1
+2023-10-13 22:02:24.970862241+00:00,1
+2023-10-13 22:02:25.132621091+00:00,1
+2023-10-13 22:02:25.298381903+00:00,1
+2023-10-13 22:02:25.454528214+00:00,1
+2023-10-13 22:02:25.666781921+00:00,1
+2023-10-13 22:02:25.819514974+00:00,1
+2023-10-13 22:02:25.972053208+00:00,1
+2023-10-13 22:02:26.128448810+00:00,1
+2023-10-13 22:02:26.291277182+00:00,1
+2023-10-13 22:02:26.457262866+00:00,1
+2023-10-13 22:02:26.621374575+00:00,1
+2023-10-13 22:02:26.782638599+00:00,1
+2023-10-13 22:02:26.924320813+00:00,1
+2023-10-13 22:02:27.084548447+00:00,1
+2023-10-13 22:02:27.245900192+00:00,1
+2023-10-13 22:02:27.390848169+00:00,1
+2023-10-13 22:02:27.550779399+00:00,1
+2023-10-13 22:02:27.709282648+00:00,1
+2023-10-13 22:02:27.846770454+00:00,1
+2023-10-13 22:02:27.986144605+00:00,1
+2023-10-13 22:02:28.142165944+00:00,1
+2023-10-13 22:02:28.324334787+00:00,1
+2023-10-13 22:02:28.509048145+00:00,1
+2023-10-13 22:02:28.668261018+00:00,1
+2023-10-13 22:02:28.820619095+00:00,1
+2023-10-13 22:02:29.001766630+00:00,1
+2023-10-13 22:02:29.170908421+00:00,1
+2023-10-13 22:02:29.342051527+00:00,1
+2023-10-13 22:02:29.489414563+00:00,1
+2023-10-13 22:02:29.648566894+00:00,1
+2023-10-13 22:02:29.811374505+00:00,1
+2023-10-13 22:02:29.984628642+00:00,1
+2023-10-13 22:02:30.151872620+00:00,1
+2023-10-13 22:02:30.312363421+00:00,1
+2023-10-13 22:02:30.470920586+00:00,1
+2023-10-13 22:02:30.639996237+00:00,1
+2023-10-13 22:02:30.796254353+00:00,1
+2023-10-13 22:02:30.992835066+00:00,1
+2023-10-13 22:02:31.177160462+00:00,1
+2023-10-13 22:02:31.348744188+00:00,1
+2023-10-13 22:02:31.514879298+00:00,1
+2023-10-13 22:02:31.669582967+00:00,1
+2023-10-13 22:02:31.831035610+00:00,1
+2023-10-13 22:02:31.989780701+00:00,1
+2023-10-13 22:02:32.162517034+00:00,1
+2023-10-13 22:02:32.326925680+00:00,1
+2023-10-13 22:02:32.499661198+00:00,1
+2023-10-13 22:02:32.649564862+00:00,1
+2023-10-13 22:02:32.843295959+00:00,1
+2023-10-13 22:02:32.969887599+00:00,1
+2023-10-13 22:02:33.126156503+00:00,1
+2023-10-13 22:02:33.287255016+00:00,1
+2023-10-13 22:02:33.457019900+00:00,1
+2023-10-13 22:02:34.675234820+00:00,0
+2023-10-13 22:02:36.690478761+00:00,1
+2023-10-13 22:02:36.875470301+00:00,1
+2023-10-13 22:02:37.085451651+00:00,1
+2023-10-13 22:02:37.261998248+00:00,1
+2023-10-13 22:02:37.442375520+00:00,0
+2023-10-13 22:02:39.458650544+00:00,0
+2023-10-13 22:02:41.473866940+00:00,0
+2023-10-13 22:02:43.487966100+00:00,0
+2023-10-13 22:02:45.502372654+00:00,0
+2023-10-13 22:02:47.516837206+00:00,0
+2023-10-13 22:02:49.530364255+00:00,0
+2023-10-13 22:02:51.544570033+00:00,0
+2023-10-13 22:02:53.558568756+00:00,1
+2023-10-13 22:02:53.698340492+00:00,1
+2023-10-13 22:02:53.868853750+00:00,0
+2023-10-13 22:02:55.883441757+00:00,0
+2023-10-13 22:02:57.898610598+00:00,0
+2023-10-13 22:02:59.914635782+00:00,0
+2023-10-13 22:03:01.930556357+00:00,0
+2023-10-13 22:03:03.946743117+00:00,1
+2023-10-13 22:03:04.086644904+00:00,1
+2023-10-13 22:03:04.217027306+00:00,1
+2023-10-13 22:03:04.342423437+00:00,1
+2023-10-13 22:03:04.513518511+00:00,1
+2023-10-13 22:03:04.636036685+00:00,1
+2023-10-13 22:03:04.761123289+00:00,1
+2023-10-13 22:03:04.912162539+00:00,1
+2023-10-13 22:03:05.106096044+00:00,1
+2023-10-13 22:03:05.232308411+00:00,1
+2023-10-13 22:03:05.354976094+00:00,1
+2023-10-13 22:03:05.475825973+00:00,1
+2023-10-13 22:03:05.649688247+00:00,1
+2023-10-13 22:03:05.770811466+00:00,1
+2023-10-13 22:03:05.922898479+00:00,1
+2023-10-13 22:03:06.073930857+00:00,1
+2023-10-13 22:03:06.224405148+00:00,1
+2023-10-13 22:03:06.369697004+00:00,1
+2023-10-13 22:03:06.492797511+00:00,1
+2023-10-13 22:03:06.653686514+00:00,1
+2023-10-13 22:03:06.814933922+00:00,1
+2023-10-13 22:03:06.964783765+00:00,1
+2023-10-13 22:03:07.118853001+00:00,1
+2023-10-13 22:03:07.262523221+00:00,1
+2023-10-13 22:03:07.383893775+00:00,1
+2023-10-13 22:03:07.531555807+00:00,1
+2023-10-13 22:03:07.649608349+00:00,1
+2023-10-13 22:03:07.801743244+00:00,1
+2023-10-13 22:03:07.938675507+00:00,1
+2023-10-13 22:03:08.057765190+00:00,1
+2023-10-13 22:03:08.178800499+00:00,1
+2023-10-13 22:03:08.309831929+00:00,1
+2023-10-13 22:03:08.458225092+00:00,1
+2023-10-13 22:03:08.595358403+00:00,1
+2023-10-13 22:03:08.741849374+00:00,1
+2023-10-13 22:03:08.914195300+00:00,1
+2023-10-13 22:03:09.050413306+00:00,1
+2023-10-13 22:03:09.185885297+00:00,1
+2023-10-13 22:03:09.348828973+00:00,1
+2023-10-13 22:03:09.473147151+00:00,1
+2023-10-13 22:03:09.608633229+00:00,1
+2023-10-13 22:03:09.740745006+00:00,1
+2023-10-13 22:03:09.873536292+00:00,1
+2023-10-13 22:03:09.993061361+00:00,1
+2023-10-13 22:03:10.118795471+00:00,1
+2023-10-13 22:03:10.256107626+00:00,1
+2023-10-13 22:03:10.387025485+00:00,1
+2023-10-13 22:03:10.519129775+00:00,1
+2023-10-13 22:03:10.727938970+00:00,1
+2023-10-13 22:03:10.964045946+00:00,1
+2023-10-13 22:03:11.104379761+00:00,1
+2023-10-13 22:03:11.255892441+00:00,1
+2023-10-13 22:03:11.382794529+00:00,1
+2023-10-13 22:03:11.522340463+00:00,1
+2023-10-13 22:03:11.659804048+00:00,1
+2023-10-13 22:03:11.791812891+00:00,1
+2023-10-13 22:03:11.931213335+00:00,1
+2023-10-13 22:03:12.056934577+00:00,1
+2023-10-13 22:03:12.215684050+00:00,1
+2023-10-13 22:03:12.363564242+00:00,1
+2023-10-13 22:03:12.502069973+00:00,1
+2023-10-13 22:03:12.641964201+00:00,1
+2023-10-13 22:03:12.803468796+00:00,1
+2023-10-13 22:03:12.939324324+00:00,1
+2023-10-13 22:03:13.105831517+00:00,1
+2023-10-13 22:03:13.262822099+00:00,1
+2023-10-13 22:03:13.396586107+00:00,1
+2023-10-13 22:03:13.527261475+00:00,1
+2023-10-13 22:03:13.668955579+00:00,1
+2023-10-13 22:03:13.812379073+00:00,1
+2023-10-13 22:03:13.953277964+00:00,1
+2023-10-13 22:03:14.144827802+00:00,1
+2023-10-13 22:03:14.378206687+00:00,1
+2023-10-13 22:03:14.513231935+00:00,1
+2023-10-13 22:03:14.639906755+00:00,1
+2023-10-13 22:03:14.787302797+00:00,1
+2023-10-13 22:03:14.929731038+00:00,1
+2023-10-13 22:03:15.081391365+00:00,1
+2023-10-13 22:03:15.233099182+00:00,1
+2023-10-13 22:03:15.375624567+00:00,1
+2023-10-13 22:03:15.569605419+00:00,1
+2023-10-13 22:03:15.741449851+00:00,1
+2023-10-13 22:03:15.937712820+00:00,1
+2023-10-13 22:03:16.074272503+00:00,1
+2023-10-13 22:03:16.218844393+00:00,1
+2023-10-13 22:03:16.354464203+00:00,1
+2023-10-13 22:03:16.497887921+00:00,1
+2023-10-13 22:03:16.646961409+00:00,1
+2023-10-13 22:03:16.805857945+00:00,1
+2023-10-13 22:03:16.964259876+00:00,1
+2023-10-13 22:03:17.114069451+00:00,1
+2023-10-13 22:03:17.260543437+00:00,1
+2023-10-13 22:03:17.410165017+00:00,1
+2023-10-13 22:03:17.548306648+00:00,1
+2023-10-13 22:03:17.686913059+00:00,1
+2023-10-13 22:03:17.819833338+00:00,1
+2023-10-13 22:03:17.939432599+00:00,1
+2023-10-13 22:03:18.063856984+00:00,1
+2023-10-13 22:03:18.186225002+00:00,1
+2023-10-13 22:03:18.311172535+00:00,1
+2023-10-13 22:03:18.453862667+00:00,1
+2023-10-13 22:03:18.575840579+00:00,1
+2023-10-13 22:03:18.700464978+00:00,1
+2023-10-13 22:03:18.830411559+00:00,1
+2023-10-13 22:03:18.956539606+00:00,1
+2023-10-13 22:03:19.092666874+00:00,1
+2023-10-13 22:03:19.221199267+00:00,1
+2023-10-13 22:03:19.351408377+00:00,1
+2023-10-13 22:03:19.477805768+00:00,1
+2023-10-13 22:03:19.603699312+00:00,1
+2023-10-13 22:03:19.732808383+00:00,1
+2023-10-13 22:03:19.858914693+00:00,1
+2023-10-13 22:03:19.980187466+00:00,1
+2023-10-13 22:03:20.109461663+00:00,1
+2023-10-13 22:03:20.249326315+00:00,1
+2023-10-13 22:03:20.375360148+00:00,1
+2023-10-13 22:03:20.497820553+00:00,1
+2023-10-13 22:03:20.621289053+00:00,1
+2023-10-13 22:03:20.742800655+00:00,1
+2023-10-13 22:03:20.873062897+00:00,1
+2023-10-13 22:03:21.007924433+00:00,1
+2023-10-13 22:03:21.140099734+00:00,1
+2023-10-13 22:03:21.266642490+00:00,1
+2023-10-13 22:03:21.393285879+00:00,1
+2023-10-13 22:03:21.513091009+00:00,1
+2023-10-13 22:03:21.641472202+00:00,1
+2023-10-13 22:03:21.767076415+00:00,1
+2023-10-13 22:03:21.911954305+00:00,1
+2023-10-13 22:03:22.040241808+00:00,1
+2023-10-13 22:03:22.174208424+00:00,1
+2023-10-13 22:03:22.295018103+00:00,1
+2023-10-13 22:03:22.465931820+00:00,1
+2023-10-13 22:03:22.592940782+00:00,1
+2023-10-13 22:03:22.721731511+00:00,1
+2023-10-13 22:03:22.848523342+00:00,1
+2023-10-13 22:03:22.965041242+00:00,1
+2023-10-13 22:03:23.081484265+00:00,1
+2023-10-13 22:03:23.199838887+00:00,1
+2023-10-13 22:03:23.315903415+00:00,1
+2023-10-13 22:03:23.432998590+00:00,1
+2023-10-13 22:03:23.549290736+00:00,1
+2023-10-13 22:03:23.666451937+00:00,1
+2023-10-13 22:03:23.783512469+00:00,1
+2023-10-13 22:03:23.900062337+00:00,1
+2023-10-13 22:03:24.016975750+00:00,1
+2023-10-13 22:03:24.134122669+00:00,1
+2023-10-13 22:03:24.251166980+00:00,1
+2023-10-13 22:03:24.367795273+00:00,1
+2023-10-13 22:03:24.484597151+00:00,1
+2023-10-13 22:03:24.600395352+00:00,1
+2023-10-13 22:03:24.716741566+00:00,1
+2023-10-13 22:03:24.833061371+00:00,1
+2023-10-13 22:03:24.949702306+00:00,1
+2023-10-13 22:03:25.065614363+00:00,1
+2023-10-13 22:03:25.182807197+00:00,1
+2023-10-13 22:03:25.299959894+00:00,1
+2023-10-13 22:03:25.420025301+00:00,1
+2023-10-13 22:03:25.538973088+00:00,1
+2023-10-13 22:03:25.655301686+00:00,1
+2023-10-13 22:03:25.772412671+00:00,1
+2023-10-13 22:03:25.889647814+00:00,1
+2023-10-13 22:03:26.006769246+00:00,1
+2023-10-13 22:03:26.124124578+00:00,1
+2023-10-13 22:03:26.240372166+00:00,1
+2023-10-13 22:03:26.356742609+00:00,1
+2023-10-13 22:03:26.473973209+00:00,1
+2023-10-13 22:03:26.590731969+00:00,1
+2023-10-13 22:03:26.707930673+00:00,1
+2023-10-13 22:03:26.824644560+00:00,1
+2023-10-13 22:03:26.941088293+00:00,1
+2023-10-13 22:03:27.058076025+00:00,1
+2023-10-13 22:03:27.174029492+00:00,1
+2023-10-13 22:03:27.290701184+00:00,1
+2023-10-13 22:03:27.407839184+00:00,1
+2023-10-13 22:03:27.524239718+00:00,1
+2023-10-13 22:03:27.641195050+00:00,1
+2023-10-13 22:03:27.760419381+00:00,1
+2023-10-13 22:03:27.877599436+00:00,1
+2023-10-13 22:03:27.993948763+00:00,1
+2023-10-13 22:03:28.110887333+00:00,1
+2023-10-13 22:03:28.227752397+00:00,1
+2023-10-13 22:03:28.344167278+00:00,1
+2023-10-13 22:03:28.460750869+00:00,1
+2023-10-13 22:03:28.578341685+00:00,1
+2023-10-13 22:03:28.694956575+00:00,1
+2023-10-13 22:03:28.811983845+00:00,1
+2023-10-13 22:03:28.928898015+00:00,1
+2023-10-13 22:03:29.045882114+00:00,1
+2023-10-13 22:03:29.162295662+00:00,1
+2023-10-13 22:03:29.280598889+00:00,1
+2023-10-13 22:03:29.398526455+00:00,1
+2023-10-13 22:03:29.514498638+00:00,1
+2023-10-13 22:03:29.631428088+00:00,1
+2023-10-13 22:03:29.748490625+00:00,1
+2023-10-13 22:03:29.864491856+00:00,1
+2023-10-13 22:03:29.981544037+00:00,1
+2023-10-13 22:03:30.100156070+00:00,1
+2023-10-13 22:03:30.216825443+00:00,1
+2023-10-13 22:03:30.332970597+00:00,1
+2023-10-13 22:03:30.449452949+00:00,1
+2023-10-13 22:03:30.566133743+00:00,1
+2023-10-13 22:03:30.682419377+00:00,1
+2023-10-13 22:03:30.799550992+00:00,1
+2023-10-13 22:03:30.916057435+00:00,1
+2023-10-13 22:03:31.033456469+00:00,1
+2023-10-13 22:03:31.149942289+00:00,1
+2023-10-13 22:03:31.266591655+00:00,1
+2023-10-13 22:03:31.383622595+00:00,1
+2023-10-13 22:03:31.499844231+00:00,1
+2023-10-13 22:03:31.616232982+00:00,1
+2023-10-13 22:03:31.733519544+00:00,1
+2023-10-13 22:03:31.849843448+00:00,1
+2023-10-13 22:03:31.967277483+00:00,1
+2023-10-13 22:03:32.083312271+00:00,1
+2023-10-13 22:03:32.200161935+00:00,1
+2023-10-13 22:03:32.317035405+00:00,1
+2023-10-13 22:03:32.433863288+00:00,1
+2023-10-13 22:03:32.550372393+00:00,1
+2023-10-13 22:03:32.666806674+00:00,1
+2023-10-13 22:03:32.783835430+00:00,1
+2023-10-13 22:03:32.900507542+00:00,1
+2023-10-13 22:03:33.016851251+00:00,1
+2023-10-13 22:03:33.132950737+00:00,1
+2023-10-13 22:03:33.249277868+00:00,1
+2023-10-13 22:03:33.366254065+00:00,1
+2023-10-13 22:03:33.482444809+00:00,1
+2023-10-13 22:03:33.598638010+00:00,1
+2023-10-13 22:03:33.714924729+00:00,1
+2023-10-13 22:03:33.832095110+00:00,1
+2023-10-13 22:03:33.949655578+00:00,1
+2023-10-13 22:03:34.066460488+00:00,1
+2023-10-13 22:03:34.182882666+00:00,1
+2023-10-13 22:03:34.298995234+00:00,1
+2023-10-13 22:03:34.415613834+00:00,1
+2023-10-13 22:03:34.531904058+00:00,1
+2023-10-13 22:03:34.647668996+00:00,1
+2023-10-13 22:03:34.764135195+00:00,1
+2023-10-13 22:03:34.880264878+00:00,1
+2023-10-13 22:03:34.996519731+00:00,1
+2023-10-13 22:03:35.112868652+00:00,1
+2023-10-13 22:03:35.228814924+00:00,1
+2023-10-13 22:03:35.344849894+00:00,1
+2023-10-13 22:03:35.461895101+00:00,1
+2023-10-13 22:03:35.577794017+00:00,1
+2023-10-13 22:03:35.693502568+00:00,1
+2023-10-13 22:03:35.809818188+00:00,1
+2023-10-13 22:03:35.925565705+00:00,1
+2023-10-13 22:03:36.041577955+00:00,1
+2023-10-13 22:03:36.157562405+00:00,1
+2023-10-13 22:03:36.273611687+00:00,1
+2023-10-13 22:03:36.389839888+00:00,1
+2023-10-13 22:03:36.506451912+00:00,1
+2023-10-13 22:03:36.622311793+00:00,1
+2023-10-13 22:03:36.738495245+00:00,1
+2023-10-13 22:03:36.855095310+00:00,1
+2023-10-13 22:03:36.971341835+00:00,1
+2023-10-13 22:03:37.087735763+00:00,1
+2023-10-13 22:03:37.203878798+00:00,1
+2023-10-13 22:03:37.319836438+00:00,1
+2023-10-13 22:03:37.435822799+00:00,1
+2023-10-13 22:03:37.552114326+00:00,1
+2023-10-13 22:03:37.668341929+00:00,1
+2023-10-13 22:03:37.785602301+00:00,1
+2023-10-13 22:03:37.902336468+00:00,1
+2023-10-13 22:03:38.018729012+00:00,1
+2023-10-13 22:03:38.135622404+00:00,1
+2023-10-13 22:03:38.252226479+00:00,1
+2023-10-13 22:03:38.368606362+00:00,1
+2023-10-13 22:03:38.485760253+00:00,1
+2023-10-13 22:03:38.601651841+00:00,1
+2023-10-13 22:03:38.718590077+00:00,1
+2023-10-13 22:03:38.835363940+00:00,1
+2023-10-13 22:03:38.952154931+00:00,1
+2023-10-13 22:03:39.068445383+00:00,1
+2023-10-13 22:03:39.185385667+00:00,1
+2023-10-13 22:03:39.300959581+00:00,1
+2023-10-13 22:03:39.417610737+00:00,1
+2023-10-13 22:03:39.533556011+00:00,1
+2023-10-13 22:03:39.649516250+00:00,1
+2023-10-13 22:03:39.765381369+00:00,1
+2023-10-13 22:03:39.882010410+00:00,1
+2023-10-13 22:03:39.998764045+00:00,1
+2023-10-13 22:03:40.114780059+00:00,1
+2023-10-13 22:03:40.230825292+00:00,1
+2023-10-13 22:03:40.347509169+00:00,1
+2023-10-13 22:03:40.463819742+00:00,1
+2023-10-13 22:03:40.579911691+00:00,1
+2023-10-13 22:03:40.695877615+00:00,1
+2023-10-13 22:03:40.811768605+00:00,1
+2023-10-13 22:03:40.928281302+00:00,1
+2023-10-13 22:03:41.045033479+00:00,1
+2023-10-13 22:03:41.161046106+00:00,1
+2023-10-13 22:03:41.277745277+00:00,1
+2023-10-13 22:03:41.393933707+00:00,1
+2023-10-13 22:03:41.510114180+00:00,1
+2023-10-13 22:03:41.626022414+00:00,1
+2023-10-13 22:03:41.742935880+00:00,1
+2023-10-13 22:03:41.859937458+00:00,1
+2023-10-13 22:03:41.975765066+00:00,1
+2023-10-13 22:03:42.092355947+00:00,1
+2023-10-13 22:03:42.209383000+00:00,1
+2023-10-13 22:03:42.326092155+00:00,1
+2023-10-13 22:03:42.442088974+00:00,1
+2023-10-13 22:03:42.558464691+00:00,1
+2023-10-13 22:03:42.674280248+00:00,1
+2023-10-13 22:03:42.791379253+00:00,1
+2023-10-13 22:03:42.908070489+00:00,1
+2023-10-13 22:03:43.026073412+00:00,1
+2023-10-13 22:03:43.143387681+00:00,1
+2023-10-13 22:03:43.260332007+00:00,1
+2023-10-13 22:03:43.377631840+00:00,1
+2023-10-13 22:03:43.494211447+00:00,1
+2023-10-13 22:03:43.611301480+00:00,1
+2023-10-13 22:03:43.727961656+00:00,1
+2023-10-13 22:03:43.844835615+00:00,1
+2023-10-13 22:03:43.961273953+00:00,1
+2023-10-13 22:03:44.077695217+00:00,1
+2023-10-13 22:03:44.193889980+00:00,1
+2023-10-13 22:03:44.310945488+00:00,1
+2023-10-13 22:03:44.428692399+00:00,1
+2023-10-13 22:03:44.544730304+00:00,1
+2023-10-13 22:03:44.660930278+00:00,1
+2023-10-13 22:03:44.778067251+00:00,1
+2023-10-13 22:03:44.894214519+00:00,1
+2023-10-13 22:03:45.010329940+00:00,1
+2023-10-13 22:03:45.126184596+00:00,1
+2023-10-13 22:03:45.241934435+00:00,1
+2023-10-13 22:03:45.359568640+00:00,1
+2023-10-13 22:03:45.476953775+00:00,1
+2023-10-13 22:03:45.593264520+00:00,1
+2023-10-13 22:03:45.709174385+00:00,1
+2023-10-13 22:03:45.826104338+00:00,1
+2023-10-13 22:03:45.943084678+00:00,1
+2023-10-13 22:03:46.059576773+00:00,1
+2023-10-13 22:03:46.175956114+00:00,1
+2023-10-13 22:03:46.292341533+00:00,1
+2023-10-13 22:03:46.408506480+00:00,1
+2023-10-13 22:03:46.524077853+00:00,1
+2023-10-13 22:03:46.639982135+00:00,1
+2023-10-13 22:03:46.756035890+00:00,1
+2023-10-13 22:03:46.873474618+00:00,1
+2023-10-13 22:03:46.989418992+00:00,1
+2023-10-13 22:03:47.105524005+00:00,1
+2023-10-13 22:03:47.221085049+00:00,1
+2023-10-13 22:03:47.337085134+00:00,1
+2023-10-13 22:03:47.452806824+00:00,1
+2023-10-13 22:03:47.569265473+00:00,1
+2023-10-13 22:03:47.685746684+00:00,1
+2023-10-13 22:03:47.802134561+00:00,1
+2023-10-13 22:03:47.918292076+00:00,1
+2023-10-13 22:03:48.034881960+00:00,1
+2023-10-13 22:03:48.151203813+00:00,1
+2023-10-13 22:03:48.267733709+00:00,1
+2023-10-13 22:03:48.384411934+00:00,1
+2023-10-13 22:03:48.500953909+00:00,1
+2023-10-13 22:03:48.617464042+00:00,1
+2023-10-13 22:03:48.733106306+00:00,1
+2023-10-13 22:03:48.849462605+00:00,1
+2023-10-13 22:03:48.966071240+00:00,1
+2023-10-13 22:03:49.082465195+00:00,1
+2023-10-13 22:03:49.197986828+00:00,1
+2023-10-13 22:03:49.313859552+00:00,1
+2023-10-13 22:03:49.430306867+00:00,1
+2023-10-13 22:03:49.547776785+00:00,1
+2023-10-13 22:03:49.663773918+00:00,1
+2023-10-13 22:03:49.779911657+00:00,1
+2023-10-13 22:03:49.895967206+00:00,1
+2023-10-13 22:03:50.012298967+00:00,1
+2023-10-13 22:03:50.130216593+00:00,1
+2023-10-13 22:03:50.247070556+00:00,1
+2023-10-13 22:03:50.363302052+00:00,1
+2023-10-13 22:03:50.479769887+00:00,1
+2023-10-13 22:03:50.596167860+00:00,1
+2023-10-13 22:03:50.712540010+00:00,1
+2023-10-13 22:03:50.829929144+00:00,1
+2023-10-13 22:03:50.945875310+00:00,1
+2023-10-13 22:03:51.061940002+00:00,1
+2023-10-13 22:03:51.178657450+00:00,1
+2023-10-13 22:03:51.294643419+00:00,1
+2023-10-13 22:03:51.410575223+00:00,1
+2023-10-13 22:03:51.526553594+00:00,1
+2023-10-13 22:03:51.643412049+00:00,1
+2023-10-13 22:03:51.759089152+00:00,1
+2023-10-13 22:03:51.875257397+00:00,1
+2023-10-13 22:03:51.991136009+00:00,1
+2023-10-13 22:03:52.107694196+00:00,1
+2023-10-13 22:03:52.223874536+00:00,1
+2023-10-13 22:03:52.340870362+00:00,1
+2023-10-13 22:03:52.456915567+00:00,1
+2023-10-13 22:03:52.572992006+00:00,1
+2023-10-13 22:03:52.689119749+00:00,1
+2023-10-13 22:03:52.805867281+00:00,1
+2023-10-13 22:03:52.921557120+00:00,1
+2023-10-13 22:03:53.038327880+00:00,1
+2023-10-13 22:03:53.154521521+00:00,1
+2023-10-13 22:03:53.270567345+00:00,1
+2023-10-13 22:03:53.387600861+00:00,1
+2023-10-13 22:03:53.504755014+00:00,1
+2023-10-13 22:03:53.620734064+00:00,1
+2023-10-13 22:03:53.736949981+00:00,1
+2023-10-13 22:03:53.855054496+00:00,1
+2023-10-13 22:03:53.971330261+00:00,1
+2023-10-13 22:03:54.087573799+00:00,1
+2023-10-13 22:03:54.204598724+00:00,1
+2023-10-13 22:03:54.320647952+00:00,1
+2023-10-13 22:03:54.436971449+00:00,1
+2023-10-13 22:03:54.553515908+00:00,1
+2023-10-13 22:03:54.669796528+00:00,1
+2023-10-13 22:03:54.786006060+00:00,1
+2023-10-13 22:03:54.901954444+00:00,1
+2023-10-13 22:03:55.017903480+00:00,1
+2023-10-13 22:03:55.134342837+00:00,1
+2023-10-13 22:03:55.250825934+00:00,1
+2023-10-13 22:03:55.366937071+00:00,1
+2023-10-13 22:03:55.483427642+00:00,1
+2023-10-13 22:03:55.599551801+00:00,1
+2023-10-13 22:03:55.715816011+00:00,1
+2023-10-13 22:03:55.833075311+00:00,1
+2023-10-13 22:03:55.950062560+00:00,1
+2023-10-13 22:03:56.066246570+00:00,1
+2023-10-13 22:03:56.182844419+00:00,1
+2023-10-13 22:03:56.299472552+00:00,1
+2023-10-13 22:03:56.415931672+00:00,1
+2023-10-13 22:03:56.532222612+00:00,1
+2023-10-13 22:03:56.648741586+00:00,1
+2023-10-13 22:03:56.766476002+00:00,1
+2023-10-13 22:03:56.884079768+00:00,1
+2023-10-13 22:03:57.000162046+00:00,1
+2023-10-13 22:03:57.116598453+00:00,1
+2023-10-13 22:03:57.232377346+00:00,1
+2023-10-13 22:03:57.347984635+00:00,1
+2023-10-13 22:03:57.464166860+00:00,1
+2023-10-13 22:03:57.580052154+00:00,1
+2023-10-13 22:03:57.696762158+00:00,1
+2023-10-13 22:03:57.813942527+00:00,1
+2023-10-13 22:03:57.930403715+00:00,1
+2023-10-13 22:03:58.047200508+00:00,1
+2023-10-13 22:03:58.163766199+00:00,1
+2023-10-13 22:03:58.279836028+00:00,1
+2023-10-13 22:03:58.396179231+00:00,1
+2023-10-13 22:03:58.511771293+00:00,1
+2023-10-13 22:03:58.627676352+00:00,1
+2023-10-13 22:03:58.743988103+00:00,1
+2023-10-13 22:03:58.860144131+00:00,1
+2023-10-13 22:03:58.976318672+00:00,1
+2023-10-13 22:03:59.092089305+00:00,1
+2023-10-13 22:03:59.208609380+00:00,1
+2023-10-13 22:03:59.324888205+00:00,1
+2023-10-13 22:03:59.441815787+00:00,1
+2023-10-13 22:03:59.558670020+00:00,1
+2023-10-13 22:03:59.674762764+00:00,1
+2023-10-13 22:03:59.793043614+00:00,1
+2023-10-13 22:03:59.910217777+00:00,1
+2023-10-13 22:04:00.027179784+00:00,1
+2023-10-13 22:04:00.144635362+00:00,1
+2023-10-13 22:04:00.262162419+00:00,1
+2023-10-13 22:04:00.379458299+00:00,1
+2023-10-13 22:04:00.495714913+00:00,1
+2023-10-13 22:04:00.611902177+00:00,1
+2023-10-13 22:04:00.728601484+00:00,1
+2023-10-13 22:04:00.845492153+00:00,1
+2023-10-13 22:04:00.961872709+00:00,1
+2023-10-13 22:04:01.078414449+00:00,1
+2023-10-13 22:04:01.194583271+00:00,1
+2023-10-13 22:04:01.310817545+00:00,1
+2023-10-13 22:04:01.427235330+00:00,1
diff --git a/tests/zero-downtime-upgrades/results/1.0.0/http.png b/tests/zero-downtime-upgrades/results/1.0.0/http.png
new file mode 100644
index 0000000000000000000000000000000000000000..8139920d3f75fac17e25cdecdeea1cd8dd8508ce
GIT binary patch
literal 6548
zcmeHLX;f3mw%(1>+9J?S(4q{^HngCiAVy4_v0H`~5EKDHL768>7?S{Putl3_iy%Qj
zL{Mf35J;Hp2tj5+K!%V;49FB_Vu0k_gRZ{!y><V*fA5cL)j6lmu2Z|d@7ueo&Z^%{
zjZe$2Qd$K7vS$oWoC8><1|Vg%av5sLdhT-!HGt_^^OFdIpq8Yhq@_nC02!$S1O$Rg
z9MsgnQRhCSQVqzsN;0zdGC2|eM+St9kdU{})YZ|hxWAjxeo9J8IyyQ=Mn+CfP9zfP
z(W6InI=!;8vcA5a&*!5<OlWIU1=Or}0}~b#e<O&18-^6U%n=BX4nRkPtLUAyHFB$F
zq~+)L$c-DlP$&Qm5sf3T1QCJ8H^}-w5t8WynT(u-d9u&Y!@!AwJ7o;lVv5Y{Y)=wd
z4S5EFFu?<8<tA~drZ@{%E%JIGuU}8bp@mGoTuHu+keBe|kR-HRMv}ac%1Q*~gH$#!
z5+;y|q@sqG$jgSx++YNkgs7F5m!s&!Qn0uT=A}XOEeKb_>?)X%fhk#-l!Ni_VQejo
zu7lwXFr)y3KLB3|K5T;b%Fw3*JT>5Mg`REDwF5eLLdR}sQ-`-2(6R@bwV-J)G-yM;
z4%8li8a?0~gsMaE3Il%`KqVF`a8Py(N{<8cBov<ph9MOG3<bYHz7gb_K#nP7or6p>
zNVkCJmXKxz&n`lWH6+_Wk{u*ohQ|&Ne+A+kA@&-?T!&~Ec;E{6-GS-}QC@J@2k!X7
zZ9lk0fCzuM83?}xL0AZoNkAe)NEihD27x!hKLQB1!0$Hr-T|Mx;1va)RB*o!t`EQ^
z8m`B{wODYBgDdgi@E9&9f?X2WB!hJdTzm#rX<+#rEYiU&6V7FUX%3j=f>A#FQUE^}
zf*}J=7sE*=9501qWq_*yY$X`{1(;WGs0t2pK(7W4)PhbuXg9##CeUhzJuRT|7S!8d
zcL(h3gdJV5tp~PpL5&A0eW3gvHhlmkKKw8U3PZ4A7}kx#+A;Wk9ONcJb_!%>VAU+F
z6vDS6ke-KSiy$Rl$~6N$*#b0!zw<J@;0qx4YU%nWT0?FVfWo0OCytu?$B%KpyH9&3
zb>i3Nooc4bU$&GYM#a)8%a0uo-DzjRTW8NmUGZWt_$FmOF`2sk%>Dl^DV)*Ta2I3-
z-^EnVNDVsD98TL4l^!A*{(fGyJK6^6*rsu{_f<&A_`EJbPj!+yLMwZgZkKQ7to;);
znV4cu-&u6Q-okHWjeJR7r**aT*0SbeT%2S)TeFU-?2##_IDDKN_ryp1Y^XO|zlSx#
znbEplowc9u?K<YxJh_H_i+H`;^C^#4e#kA`C9+5<$Ic_9q;uc3UoEFs$kUvSn;h}W
zF9<FNdbB9Cu1V3LWp@i@v5S8TnDfD*XNBz%)rI2ju+GUR^-Y3o<Q`U5N%f5iMdj2T
zMP}b@OSwxI``kgd)Qq_I!|gFNWx1W~xwI8R+9QJ)y0lMGxbv-o*E1?tKlL4+2|VoZ
z;F`=;CgZRqI8lr0gtsrj4fo5WOmavF^+^WTYq2xBsx3=PuiKoh$r6PtMx$R%E1x}{
zVni{?qip>nM<{HYJm*)0OAX3!>5x61Cf&ldQA;y{YMGD=91JD=v^Cz6QQqT*QL$?b
z)9me!-$SK;?mxfHuw!#=hcb4T!6hZ)?ZmQIaROz#Mq!kdXl({Ylj_F6uD@epMC#74
zu6Q@v_lB+DByYQ4GP<KZcm;h+%gzA#a}C7irYN<x$(GI)FWq%Ym&MMXPb@owm1Tx3
z*SAyaC?e){va1!kt;=wY46Q6ZP3qjl(}2L6N6J>b-jY=AFYeV7v~x;ZllVbYZ&m|e
zsw96lvOz^ol}+vpx77Nc7xM9FD<@67%itt~*MzN7IwE%dBZd;hVvbd-4zW6|KZx`K
zTazZ&&`t8@$L27XG^rX(Shw{1FRZ^YEA@zCNV^(36G4?|+;ZXB8HC7dUmC7XT;v2T
z@FdTtJ(3R}xFNx8S@MvkwZ(}SJ`+V#s42{{^O!4Uw4w;8oxXqCvSW6%C~JN{B&T@m
zzw>ze5R(4DUox8?+Y{+CSl%Spzg;xHtMbg$$911_$79!@y`Mvpir=nD?U4PaMMhzz
z(>p@S;%$50y%y=PUaqIgobr!0Y9G1c0n{|Lpm=NMRMPpd<d6DVhYS`cNQL4k1e<p>
zYx&7HzP7%BpKM5YN?+y<#a*S6p3biS*e8Q!xPn5N5937<;TONVojh1NFf-9MiSQQu
zuOPMV2OppDh<$vDrH>hZ?M@1^PFVFkI%#)cqpPtzuip5kA2Tn*Wbg?s@l+5|BE(1}
zT;?&{zsF?vJ#(S7o^P`I)vtA<@wK!!w!Sk5v!eEBEPwm6>3V~~9cdBR>%s^{GrEpM
zkU?az>ZG-?$O5X{mL(M7+3F^Gx@}&kS~O%nj4qDriN&mto8*6^<hU1dHbr=z@9LD&
zcYhw6h=_Hg)Q<T#4E1r^=vX%mJ-Lw=j}&SOo(u_Cm%cP)%ugt}TP8HF<W;;H#<Qh1
zdLDGY?ky2^FRuS(d5*q8a7V>1Rg(RVYrlEU=NzTz>GaE{$?j=Hh%{dK$}wJIPlsoV
z^xU{&%@?h0E~!N=J9$rPiO!EpyuVhz>hMnzBzQVnTRN7FulzN|GKd+?n#w(hw^(*-
zYgtcgR_3r|+(US8H;FegEJ+|a=q1+oZ<{T4_Z25gx42S%@k0n5n{DxhbQv3)Mvr$5
zTA5K(L$8C}PFV&9x2ncc>^$n+uw}Tod;{A<d#FWj%w+de8E#(%x14m<MyPj#E|c*z
zI8dNMYmQdo&74l{i|O9WHPNE7^PafiZT8uz`9))*6swUrLNq7*RKR5NXvFTs1lEdO
zHy(5J#7~?*@!Ok(GQAra3#p^>a;;K73OQNhg9umk;Ta3rNN2pg`vVgb?v?41mVTO~
zdH<r%$l$^ORb@X5eQPzYx6Nj)8;&0uY9>@Z_}kyU&l#0t*h)k_JvQCE*|YLUNq$8%
z>4>guMbGV$BYtxm@DaxibxQw8_fS2WY*6m*oF>#_hYQBvAJ-xHkxUvJ%XUqFGBofI
z2cwLuhsWy#6KNmi6)bt-Z?9d^RA`Pvkuy01zR&&#@*@N8kscjdk$q3o2L{rt(49cl
z)~b-}RjhAgrOI#B5!T$up8J%F)kZ;3hKLfg%n*x*ZC=~oSvYd<^1|cG=W1?BtfSJA
zkOzH=X7db{G?@`p&;ga$##vO*6-$C@cPH`2CulRXUjk<%b7Fq4sm9S?Q=o9|`je}^
zR=?`-H3_~>hOaBZ|6xV+KdnLFCmC>awW3IS-p<mV8a4$qNewl6dWC+p4!Ih-y4)#r
zu_R=+Nki+%K2z1{YU!4Wmh?Eu`y0!LhVwHcI&4|PUf&esBs1A|k8OyO_v@C;9`!sD
zcEiu2Uct~(*M>39dtAUTA93lG5={^0zbZMzT3oE!_3iMVy~3R#qr{FJN~_eyhjkNw
zxmYRLoUv9TN#gyyiGFyIDAHv0+tX#!EAw;aZ|2kXP<^ny#8UGToUS}-_72@|(XG1O
zNhU?(N@;*-r-5)xjv#)DB}wX;#oZ0rjat|o{~NxDmxqKv`XeN`Tk6Gp_=zC?O6|;V
zqtxW(KD$`$nuP9~t!vmqHBD;HL~o20R}dUJljyL4OB_$LBHo_e(OZmTrXoro+(I{d
zWYoIwcw<5}dHJ_1_FdFZIwA617*4dbouSQm;+!2rZ*P@e`bj^x^r!r_`c;Sjha`|c
zB^u2gHxzAPmv4I}v%HMOJ3NXx$HX<W78+0!KjtSYMi_$sVb|*p>^H{uPbXGJgm6+W
z-wzhA`+K{0XsO3PEuI(D);3kxD*Wd(NTS6|RUgT&$jzxuA2KEs7+t_eauyG}5&hV?
z4wf+%++HPoP|r~_U28&zoL&8StQ?JV@i4wFH`=V^ha$XsOprAuy3V_b`Qp-QYlCQd
zgO=c(_Nc_X{I~@n!a#Y>%+meM`L-X=ly$2<jaQkT&K0%EvhfRZ6Tj8XRp-+B=bu;%
zc(Mgu`K;jjX!~iaNRG#nTSa!eG2h|c@QX`FO3;yleH;m*SPk|7&8{cU{++y;=v$UO
zfnUCR?l{>;k4w0tZ7?usPtUNX+@<_ptUcOWoZWSIn&C*GliFBpr#uCM+s5;m@wHXC
zg=PUI42L#z+dI~J(UJBeX;THFDrIZR2144Da-7RN%S4CQx3Bj7KCCRpmU}p}_yFoG
zwGrF?klP@!AgSnzx7DKzGv8@PS6wkHajWfm@skRz_I>D!!|p_H&O`eu?u+1vbQMBe
z21~M{J<pJ{8`X3FPnLS}b(2%IxtRVgb{;XlVh_u;uC;DKHR5g_#h#8STCJgbmM_X}
zTT(SpeXt*Wb3^H)PP@$$<TV7we2jTRa2j{wzHn=8=N01ShZ&n9+9C*nV|VQd)w9A6
zc7CXeAsrQ^<3ttwodc2#<IkKEYWZJ66{z0C$x5iD+h;6Mb~g(+l39$29``Dz1xg;e
zi>35c@uf^Zt6q_~6z9}WNxZ6|SmQvTU+l57Y~9o{E0bbXDz-uB152pBgxcFW)hCa2
zz)Zh;(IE&`FwstJdYOf<BsE}cB&;g8zwIvj4+8Rud097mlV*01O)im8N2?wgT*BoR
zP#CD|HfF9#;Rbl<aoiu(&ayDv{k$)pR7t-dAZ5_)%&+q?bz+|jW16aT2hWqp%Yj*Y
zJx(D9$-1ymBcS|fJ4q?X)*MH}VP${@(>2WpjV<52k>eg)tHE0ERv)C<xp~&*z89F6
zvq#za;;C(PjOuf)N>O52MbBQhy=SN86OC*)G6v|cHQ3C;#Et~wMURD8Vb_|gtxKtr
z$8g}9qwVuodZ$tMQ}?0*A~&6;%7@fr7NkBF<nX(nEbulxmhWy@>-j_bC^I>uBbeoP
zZ)!hj6CsB1kPu;%H>DoLWmwnQYoz(?`%D-OLfwOm)TvrQ`&oX0$jGc@GoGZ*>zg3D
zF8)=n6y?~kuhzftNKkzfJ2itwQBD{d?NCQ`GVa8PiLWy71g3LfPcftO3S+gk{ukE!
z+-A)+&9dWb-C39l=k^(N#w_{V@xM=fdxx<vD@=I@dYt;vqh&aA()dL7vCCoq0%yn$
A?EnA(

literal 0
HcmV?d00001

diff --git a/tests/zero-downtime-upgrades/results/1.0.0/https.csv b/tests/zero-downtime-upgrades/results/1.0.0/https.csv
new file mode 100644
index 000000000..d3d325af9
--- /dev/null
+++ b/tests/zero-downtime-upgrades/results/1.0.0/https.csv
@@ -0,0 +1,600 @@
+2023-10-13 22:02:21.184746897+00:00,1
+2023-10-13 22:02:21.321018409+00:00,1
+2023-10-13 22:02:21.473356383+00:00,1
+2023-10-13 22:02:21.618478189+00:00,1
+2023-10-13 22:02:21.771118689+00:00,1
+2023-10-13 22:02:21.909057170+00:00,1
+2023-10-13 22:02:22.056445856+00:00,1
+2023-10-13 22:02:22.199365527+00:00,1
+2023-10-13 22:02:22.341174617+00:00,1
+2023-10-13 22:02:22.488659579+00:00,1
+2023-10-13 22:02:22.623583884+00:00,1
+2023-10-13 22:02:22.783559452+00:00,1
+2023-10-13 22:02:23.247941403+00:00,1
+2023-10-13 22:02:23.449314208+00:00,1
+2023-10-13 22:02:23.648557144+00:00,1
+2023-10-13 22:02:23.830022963+00:00,1
+2023-10-13 22:02:24.001528570+00:00,1
+2023-10-13 22:02:24.176782155+00:00,1
+2023-10-13 22:02:24.341924256+00:00,1
+2023-10-13 22:02:24.516710161+00:00,1
+2023-10-13 22:02:24.694309545+00:00,1
+2023-10-13 22:02:24.848809768+00:00,1
+2023-10-13 22:02:25.024128032+00:00,1
+2023-10-13 22:02:25.197032209+00:00,1
+2023-10-13 22:02:25.360705800+00:00,1
+2023-10-13 22:02:25.560468918+00:00,1
+2023-10-13 22:02:25.699232500+00:00,1
+2023-10-13 22:02:25.869833961+00:00,1
+2023-10-13 22:02:26.030750900+00:00,1
+2023-10-13 22:02:26.190952121+00:00,1
+2023-10-13 22:02:26.349695412+00:00,1
+2023-10-13 22:02:26.533647429+00:00,1
+2023-10-13 22:02:26.702319807+00:00,1
+2023-10-13 22:02:26.859648509+00:00,1
+2023-10-13 22:02:27.031556383+00:00,1
+2023-10-13 22:02:27.205249688+00:00,1
+2023-10-13 22:02:27.429731083+00:00,1
+2023-10-13 22:02:27.590639914+00:00,1
+2023-10-13 22:02:27.752230249+00:00,1
+2023-10-13 22:02:27.913914373+00:00,1
+2023-10-13 22:02:28.077544773+00:00,1
+2023-10-13 22:02:28.258100979+00:00,1
+2023-10-13 22:02:28.436154903+00:00,1
+2023-10-13 22:02:28.654440090+00:00,1
+2023-10-13 22:02:28.864383162+00:00,1
+2023-10-13 22:02:29.067389152+00:00,1
+2023-10-13 22:02:29.239499861+00:00,1
+2023-10-13 22:02:29.421341342+00:00,1
+2023-10-13 22:02:29.602479864+00:00,1
+2023-10-13 22:02:29.783842823+00:00,1
+2023-10-13 22:02:29.963148280+00:00,1
+2023-10-13 22:02:30.141098567+00:00,1
+2023-10-13 22:02:30.323270853+00:00,1
+2023-10-13 22:02:30.495656934+00:00,1
+2023-10-13 22:02:30.662688592+00:00,1
+2023-10-13 22:02:30.823020267+00:00,1
+2023-10-13 22:02:30.971605765+00:00,1
+2023-10-13 22:02:31.143829904+00:00,1
+2023-10-13 22:02:31.314985868+00:00,1
+2023-10-13 22:02:31.479212878+00:00,1
+2023-10-13 22:02:31.658325390+00:00,1
+2023-10-13 22:02:31.830740019+00:00,1
+2023-10-13 22:02:32.020335188+00:00,1
+2023-10-13 22:02:32.193569460+00:00,1
+2023-10-13 22:02:32.359159731+00:00,1
+2023-10-13 22:02:32.555939904+00:00,1
+2023-10-13 22:02:32.731576715+00:00,1
+2023-10-13 22:02:32.967358996+00:00,1
+2023-10-13 22:02:33.132393412+00:00,1
+2023-10-13 22:02:33.278403463+00:00,1
+2023-10-13 22:02:33.515517951+00:00,1
+2023-10-13 22:02:34.612214898+00:00,1
+2023-10-13 22:02:34.832235632+00:00,0
+2023-10-13 22:02:36.844854793+00:00,0
+2023-10-13 22:02:38.858253565+00:00,1
+2023-10-13 22:02:39.116925896+00:00,1
+2023-10-13 22:02:39.376472340+00:00,1
+2023-10-13 22:02:39.605836738+00:00,1
+2023-10-13 22:02:39.845415990+00:00,1
+2023-10-13 22:02:39.977734785+00:00,1
+2023-10-13 22:02:40.197046617+00:00,1
+2023-10-13 22:02:40.327529487+00:00,0
+2023-10-13 22:02:42.340998912+00:00,0
+2023-10-13 22:02:44.353836817+00:00,0
+2023-10-13 22:02:46.367588786+00:00,1
+2023-10-13 22:02:46.546992726+00:00,1
+2023-10-13 22:02:46.727381515+00:00,0
+2023-10-13 22:02:48.740696970+00:00,1
+2023-10-13 22:02:48.895745026+00:00,1
+2023-10-13 22:02:49.052907066+00:00,0
+2023-10-13 22:02:51.066548872+00:00,0
+2023-10-13 22:02:53.079979232+00:00,0
+2023-10-13 22:02:55.092748920+00:00,1
+2023-10-13 22:02:55.236745718+00:00,1
+2023-10-13 22:02:55.391354005+00:00,0
+2023-10-13 22:02:57.405043392+00:00,1
+2023-10-13 22:02:57.560904388+00:00,1
+2023-10-13 22:02:57.730005156+00:00,0
+2023-10-13 22:02:59.743943141+00:00,1
+2023-10-13 22:02:59.905522153+00:00,1
+2023-10-13 22:03:00.140684152+00:00,0
+2023-10-13 22:03:02.154923237+00:00,0
+2023-10-13 22:03:04.168064131+00:00,1
+2023-10-13 22:03:04.368490224+00:00,1
+2023-10-13 22:03:04.515998568+00:00,1
+2023-10-13 22:03:04.675432444+00:00,1
+2023-10-13 22:03:04.829843549+00:00,1
+2023-10-13 22:03:05.000999137+00:00,1
+2023-10-13 22:03:05.138150000+00:00,1
+2023-10-13 22:03:05.282571881+00:00,1
+2023-10-13 22:03:05.514418060+00:00,1
+2023-10-13 22:03:05.691124427+00:00,1
+2023-10-13 22:03:05.824810660+00:00,1
+2023-10-13 22:03:05.961293047+00:00,1
+2023-10-13 22:03:06.124908281+00:00,1
+2023-10-13 22:03:06.262319094+00:00,1
+2023-10-13 22:03:06.420013107+00:00,1
+2023-10-13 22:03:06.560183780+00:00,1
+2023-10-13 22:03:06.712086178+00:00,1
+2023-10-13 22:03:06.893794189+00:00,1
+2023-10-13 22:03:07.030311107+00:00,1
+2023-10-13 22:03:07.187538335+00:00,1
+2023-10-13 22:03:07.328538221+00:00,1
+2023-10-13 22:03:07.502678651+00:00,1
+2023-10-13 22:03:07.639543804+00:00,1
+2023-10-13 22:03:07.894908828+00:00,1
+2023-10-13 22:03:08.049108938+00:00,1
+2023-10-13 22:03:08.196170478+00:00,1
+2023-10-13 22:03:08.331443309+00:00,1
+2023-10-13 22:03:08.498290314+00:00,1
+2023-10-13 22:03:08.642470073+00:00,1
+2023-10-13 22:03:08.785764395+00:00,1
+2023-10-13 22:03:08.988147273+00:00,1
+2023-10-13 22:03:09.135589989+00:00,1
+2023-10-13 22:03:09.294885645+00:00,1
+2023-10-13 22:03:09.485762197+00:00,1
+2023-10-13 22:03:09.644422496+00:00,1
+2023-10-13 22:03:09.802960450+00:00,1
+2023-10-13 22:03:09.958349345+00:00,1
+2023-10-13 22:03:10.103732233+00:00,1
+2023-10-13 22:03:10.255537211+00:00,1
+2023-10-13 22:03:10.405563231+00:00,1
+2023-10-13 22:03:10.548097428+00:00,1
+2023-10-13 22:03:10.687484117+00:00,1
+2023-10-13 22:03:10.861448642+00:00,1
+2023-10-13 22:03:11.093915383+00:00,1
+2023-10-13 22:03:11.240768624+00:00,1
+2023-10-13 22:03:11.433579944+00:00,1
+2023-10-13 22:03:11.590825099+00:00,1
+2023-10-13 22:03:11.748392062+00:00,1
+2023-10-13 22:03:11.888859534+00:00,1
+2023-10-13 22:03:12.057412302+00:00,1
+2023-10-13 22:03:12.231518360+00:00,1
+2023-10-13 22:03:12.379240802+00:00,1
+2023-10-13 22:03:12.554040438+00:00,1
+2023-10-13 22:03:12.708757283+00:00,1
+2023-10-13 22:03:12.861118824+00:00,1
+2023-10-13 22:03:13.014821634+00:00,1
+2023-10-13 22:03:13.163860572+00:00,1
+2023-10-13 22:03:13.328263331+00:00,1
+2023-10-13 22:03:13.490174397+00:00,1
+2023-10-13 22:03:13.672847352+00:00,1
+2023-10-13 22:03:13.824437581+00:00,1
+2023-10-13 22:03:13.977985945+00:00,1
+2023-10-13 22:03:14.148721084+00:00,1
+2023-10-13 22:03:14.311583813+00:00,1
+2023-10-13 22:03:14.500698731+00:00,1
+2023-10-13 22:03:14.673873939+00:00,1
+2023-10-13 22:03:14.827990817+00:00,1
+2023-10-13 22:03:15.004268657+00:00,1
+2023-10-13 22:03:15.173179612+00:00,1
+2023-10-13 22:03:15.341352715+00:00,1
+2023-10-13 22:03:15.538957295+00:00,1
+2023-10-13 22:03:15.769897596+00:00,1
+2023-10-13 22:03:15.936417114+00:00,1
+2023-10-13 22:03:16.090601874+00:00,1
+2023-10-13 22:03:16.246442442+00:00,1
+2023-10-13 22:03:16.399814768+00:00,1
+2023-10-13 22:03:16.559172721+00:00,1
+2023-10-13 22:03:16.721650102+00:00,1
+2023-10-13 22:03:16.900749928+00:00,1
+2023-10-13 22:03:17.080719522+00:00,1
+2023-10-13 22:03:17.235721890+00:00,1
+2023-10-13 22:03:17.389349208+00:00,1
+2023-10-13 22:03:17.553924960+00:00,1
+2023-10-13 22:03:17.710821667+00:00,1
+2023-10-13 22:03:17.847019727+00:00,1
+2023-10-13 22:03:17.992103737+00:00,1
+2023-10-13 22:03:18.138492614+00:00,1
+2023-10-13 22:03:18.291335701+00:00,1
+2023-10-13 22:03:18.437371142+00:00,1
+2023-10-13 22:03:18.596761474+00:00,1
+2023-10-13 22:03:18.738099885+00:00,1
+2023-10-13 22:03:18.890859503+00:00,1
+2023-10-13 22:03:19.051352761+00:00,1
+2023-10-13 22:03:19.330118198+00:00,1
+2023-10-13 22:03:19.476385026+00:00,1
+2023-10-13 22:03:19.622299101+00:00,1
+2023-10-13 22:03:19.760411303+00:00,1
+2023-10-13 22:03:19.907537864+00:00,1
+2023-10-13 22:03:20.056134902+00:00,1
+2023-10-13 22:03:20.210620707+00:00,1
+2023-10-13 22:03:20.361761456+00:00,1
+2023-10-13 22:03:20.539346268+00:00,1
+2023-10-13 22:03:20.725937595+00:00,1
+2023-10-13 22:03:20.878024760+00:00,1
+2023-10-13 22:03:21.032180831+00:00,1
+2023-10-13 22:03:21.179398800+00:00,1
+2023-10-13 22:03:21.378733694+00:00,1
+2023-10-13 22:03:21.537125619+00:00,1
+2023-10-13 22:03:21.681733271+00:00,1
+2023-10-13 22:03:21.855376371+00:00,1
+2023-10-13 22:03:22.016395738+00:00,1
+2023-10-13 22:03:22.171516431+00:00,1
+2023-10-13 22:03:22.332843962+00:00,1
+2023-10-13 22:03:22.477753877+00:00,1
+2023-10-13 22:03:22.640172434+00:00,1
+2023-10-13 22:03:22.800917356+00:00,1
+2023-10-13 22:03:22.927471855+00:00,1
+2023-10-13 22:03:23.054446810+00:00,1
+2023-10-13 22:03:23.180440859+00:00,1
+2023-10-13 22:03:23.306729472+00:00,1
+2023-10-13 22:03:23.432471730+00:00,1
+2023-10-13 22:03:23.557531417+00:00,1
+2023-10-13 22:03:23.682844271+00:00,1
+2023-10-13 22:03:23.808228311+00:00,1
+2023-10-13 22:03:23.933413103+00:00,1
+2023-10-13 22:03:24.060089889+00:00,1
+2023-10-13 22:03:24.186096479+00:00,1
+2023-10-13 22:03:24.311751378+00:00,1
+2023-10-13 22:03:24.438273108+00:00,1
+2023-10-13 22:03:24.562942653+00:00,1
+2023-10-13 22:03:24.688334504+00:00,1
+2023-10-13 22:03:24.813601032+00:00,1
+2023-10-13 22:03:24.939191024+00:00,1
+2023-10-13 22:03:25.066001005+00:00,1
+2023-10-13 22:03:25.191794544+00:00,1
+2023-10-13 22:03:25.317555806+00:00,1
+2023-10-13 22:03:25.443014062+00:00,1
+2023-10-13 22:03:25.568996301+00:00,1
+2023-10-13 22:03:25.693843222+00:00,1
+2023-10-13 22:03:25.819077857+00:00,1
+2023-10-13 22:03:25.944255482+00:00,1
+2023-10-13 22:03:26.069860301+00:00,1
+2023-10-13 22:03:26.195206586+00:00,1
+2023-10-13 22:03:26.320708168+00:00,1
+2023-10-13 22:03:26.445864356+00:00,1
+2023-10-13 22:03:26.571360947+00:00,1
+2023-10-13 22:03:26.696993272+00:00,1
+2023-10-13 22:03:26.822843277+00:00,1
+2023-10-13 22:03:26.948105804+00:00,1
+2023-10-13 22:03:27.074222048+00:00,1
+2023-10-13 22:03:27.200836106+00:00,1
+2023-10-13 22:03:27.326507022+00:00,1
+2023-10-13 22:03:27.453984935+00:00,1
+2023-10-13 22:03:27.579510888+00:00,1
+2023-10-13 22:03:27.705810908+00:00,1
+2023-10-13 22:03:27.831862842+00:00,1
+2023-10-13 22:03:27.958677149+00:00,1
+2023-10-13 22:03:28.085014723+00:00,1
+2023-10-13 22:03:28.210928111+00:00,1
+2023-10-13 22:03:28.336356168+00:00,1
+2023-10-13 22:03:28.462845287+00:00,1
+2023-10-13 22:03:28.588022489+00:00,1
+2023-10-13 22:03:28.713435090+00:00,1
+2023-10-13 22:03:28.838468334+00:00,1
+2023-10-13 22:03:28.964409535+00:00,1
+2023-10-13 22:03:29.090500951+00:00,1
+2023-10-13 22:03:29.216773668+00:00,1
+2023-10-13 22:03:29.345170353+00:00,1
+2023-10-13 22:03:29.472479296+00:00,1
+2023-10-13 22:03:29.598190129+00:00,1
+2023-10-13 22:03:29.723601715+00:00,1
+2023-10-13 22:03:29.849303259+00:00,1
+2023-10-13 22:03:29.975077412+00:00,1
+2023-10-13 22:03:30.100871329+00:00,1
+2023-10-13 22:03:30.226893691+00:00,1
+2023-10-13 22:03:30.353811712+00:00,1
+2023-10-13 22:03:30.479637357+00:00,1
+2023-10-13 22:03:30.605871520+00:00,1
+2023-10-13 22:03:30.731724572+00:00,1
+2023-10-13 22:03:30.857599391+00:00,1
+2023-10-13 22:03:30.986195643+00:00,1
+2023-10-13 22:03:31.111690720+00:00,1
+2023-10-13 22:03:31.237934863+00:00,1
+2023-10-13 22:03:31.363664737+00:00,1
+2023-10-13 22:03:31.489270408+00:00,1
+2023-10-13 22:03:31.614315623+00:00,1
+2023-10-13 22:03:31.740360178+00:00,1
+2023-10-13 22:03:31.865647978+00:00,1
+2023-10-13 22:03:31.991881294+00:00,1
+2023-10-13 22:03:32.118396537+00:00,1
+2023-10-13 22:03:32.244784118+00:00,1
+2023-10-13 22:03:32.371284559+00:00,1
+2023-10-13 22:03:32.496896076+00:00,1
+2023-10-13 22:03:32.622722972+00:00,1
+2023-10-13 22:03:32.749058113+00:00,1
+2023-10-13 22:03:32.874796261+00:00,1
+2023-10-13 22:03:33.001212029+00:00,1
+2023-10-13 22:03:33.128091804+00:00,1
+2023-10-13 22:03:33.254025803+00:00,1
+2023-10-13 22:03:33.380958564+00:00,1
+2023-10-13 22:03:33.507331074+00:00,1
+2023-10-13 22:03:33.633181012+00:00,1
+2023-10-13 22:03:33.759237531+00:00,1
+2023-10-13 22:03:33.885138244+00:00,1
+2023-10-13 22:03:34.011286027+00:00,1
+2023-10-13 22:03:34.137464757+00:00,1
+2023-10-13 22:03:34.264108545+00:00,1
+2023-10-13 22:03:34.390151085+00:00,1
+2023-10-13 22:03:34.515588318+00:00,1
+2023-10-13 22:03:34.640905288+00:00,1
+2023-10-13 22:03:34.767523990+00:00,1
+2023-10-13 22:03:34.893348093+00:00,1
+2023-10-13 22:03:35.018298848+00:00,1
+2023-10-13 22:03:35.143626453+00:00,1
+2023-10-13 22:03:35.269147212+00:00,1
+2023-10-13 22:03:35.395335652+00:00,1
+2023-10-13 22:03:35.522013969+00:00,1
+2023-10-13 22:03:35.647476639+00:00,1
+2023-10-13 22:03:35.773788150+00:00,1
+2023-10-13 22:03:35.899985894+00:00,1
+2023-10-13 22:03:36.025999128+00:00,1
+2023-10-13 22:03:36.152239457+00:00,1
+2023-10-13 22:03:36.278883076+00:00,1
+2023-10-13 22:03:36.404200128+00:00,1
+2023-10-13 22:03:36.530293514+00:00,1
+2023-10-13 22:03:36.656274387+00:00,1
+2023-10-13 22:03:36.782832738+00:00,1
+2023-10-13 22:03:36.910135545+00:00,1
+2023-10-13 22:03:37.037349808+00:00,1
+2023-10-13 22:03:37.163531530+00:00,1
+2023-10-13 22:03:37.289375238+00:00,1
+2023-10-13 22:03:37.415074483+00:00,1
+2023-10-13 22:03:37.540987600+00:00,1
+2023-10-13 22:03:37.670588582+00:00,1
+2023-10-13 22:03:37.797794715+00:00,1
+2023-10-13 22:03:37.925146720+00:00,1
+2023-10-13 22:03:38.052538411+00:00,1
+2023-10-13 22:03:38.179353009+00:00,1
+2023-10-13 22:03:38.304895048+00:00,1
+2023-10-13 22:03:38.431824426+00:00,1
+2023-10-13 22:03:38.557477675+00:00,1
+2023-10-13 22:03:38.683236091+00:00,1
+2023-10-13 22:03:38.809219031+00:00,1
+2023-10-13 22:03:38.935361215+00:00,1
+2023-10-13 22:03:39.061013978+00:00,1
+2023-10-13 22:03:39.189715906+00:00,1
+2023-10-13 22:03:39.315059840+00:00,1
+2023-10-13 22:03:39.440467902+00:00,1
+2023-10-13 22:03:39.566152307+00:00,1
+2023-10-13 22:03:39.691794987+00:00,1
+2023-10-13 22:03:39.817394938+00:00,1
+2023-10-13 22:03:39.942823649+00:00,1
+2023-10-13 22:03:40.068342463+00:00,1
+2023-10-13 22:03:40.194810370+00:00,1
+2023-10-13 22:03:40.321474950+00:00,1
+2023-10-13 22:03:40.447593119+00:00,1
+2023-10-13 22:03:40.573549615+00:00,1
+2023-10-13 22:03:40.700106680+00:00,1
+2023-10-13 22:03:40.825399021+00:00,1
+2023-10-13 22:03:40.951492731+00:00,1
+2023-10-13 22:03:41.077344823+00:00,1
+2023-10-13 22:03:41.204127028+00:00,1
+2023-10-13 22:03:41.330876963+00:00,1
+2023-10-13 22:03:41.458188932+00:00,1
+2023-10-13 22:03:41.583307993+00:00,1
+2023-10-13 22:03:41.708774003+00:00,1
+2023-10-13 22:03:41.834829472+00:00,1
+2023-10-13 22:03:41.960376908+00:00,1
+2023-10-13 22:03:42.086597614+00:00,1
+2023-10-13 22:03:42.213266027+00:00,1
+2023-10-13 22:03:42.339780201+00:00,1
+2023-10-13 22:03:42.465226448+00:00,1
+2023-10-13 22:03:42.592146913+00:00,1
+2023-10-13 22:03:42.718290216+00:00,1
+2023-10-13 22:03:42.844434306+00:00,1
+2023-10-13 22:03:42.970025403+00:00,1
+2023-10-13 22:03:43.095523287+00:00,1
+2023-10-13 22:03:43.221643234+00:00,1
+2023-10-13 22:03:43.348293223+00:00,1
+2023-10-13 22:03:43.473238431+00:00,1
+2023-10-13 22:03:43.598741220+00:00,1
+2023-10-13 22:03:43.724359786+00:00,1
+2023-10-13 22:03:43.849851247+00:00,1
+2023-10-13 22:03:43.975638203+00:00,1
+2023-10-13 22:03:44.102819287+00:00,1
+2023-10-13 22:03:44.229276570+00:00,1
+2023-10-13 22:03:44.355155975+00:00,1
+2023-10-13 22:03:44.480609295+00:00,1
+2023-10-13 22:03:44.606510677+00:00,1
+2023-10-13 22:03:44.732658474+00:00,1
+2023-10-13 22:03:44.859118543+00:00,1
+2023-10-13 22:03:44.984516591+00:00,1
+2023-10-13 22:03:45.111036725+00:00,1
+2023-10-13 22:03:45.237404515+00:00,1
+2023-10-13 22:03:45.364013101+00:00,1
+2023-10-13 22:03:45.489636499+00:00,1
+2023-10-13 22:03:45.615944874+00:00,1
+2023-10-13 22:03:45.742087549+00:00,1
+2023-10-13 22:03:45.868123221+00:00,1
+2023-10-13 22:03:45.994346788+00:00,1
+2023-10-13 22:03:46.120035205+00:00,1
+2023-10-13 22:03:46.247058017+00:00,1
+2023-10-13 22:03:46.372879326+00:00,1
+2023-10-13 22:03:46.498807334+00:00,1
+2023-10-13 22:03:46.624060841+00:00,1
+2023-10-13 22:03:46.750128969+00:00,1
+2023-10-13 22:03:46.875172369+00:00,1
+2023-10-13 22:03:47.000034688+00:00,1
+2023-10-13 22:03:47.125549180+00:00,1
+2023-10-13 22:03:47.251405775+00:00,1
+2023-10-13 22:03:47.378191602+00:00,1
+2023-10-13 22:03:47.504645200+00:00,1
+2023-10-13 22:03:47.630185041+00:00,1
+2023-10-13 22:03:47.756675094+00:00,1
+2023-10-13 22:03:47.882434486+00:00,1
+2023-10-13 22:03:48.008818622+00:00,1
+2023-10-13 22:03:48.135178776+00:00,1
+2023-10-13 22:03:48.261731058+00:00,1
+2023-10-13 22:03:48.387778594+00:00,1
+2023-10-13 22:03:48.513575028+00:00,1
+2023-10-13 22:03:48.640050474+00:00,1
+2023-10-13 22:03:48.766553962+00:00,1
+2023-10-13 22:03:48.893483253+00:00,1
+2023-10-13 22:03:49.019796233+00:00,1
+2023-10-13 22:03:49.145677227+00:00,1
+2023-10-13 22:03:49.272381583+00:00,1
+2023-10-13 22:03:49.398971755+00:00,1
+2023-10-13 22:03:49.525568747+00:00,1
+2023-10-13 22:03:49.650808505+00:00,1
+2023-10-13 22:03:49.776563923+00:00,1
+2023-10-13 22:03:49.901979058+00:00,1
+2023-10-13 22:03:50.028294352+00:00,1
+2023-10-13 22:03:50.154876969+00:00,1
+2023-10-13 22:03:50.281268950+00:00,1
+2023-10-13 22:03:50.406846724+00:00,1
+2023-10-13 22:03:50.532655764+00:00,1
+2023-10-13 22:03:50.658744143+00:00,1
+2023-10-13 22:03:50.785262166+00:00,1
+2023-10-13 22:03:50.912049870+00:00,1
+2023-10-13 22:03:51.038166452+00:00,1
+2023-10-13 22:03:51.164059408+00:00,1
+2023-10-13 22:03:51.290635259+00:00,1
+2023-10-13 22:03:51.417351696+00:00,1
+2023-10-13 22:03:51.542959567+00:00,1
+2023-10-13 22:03:51.668343319+00:00,1
+2023-10-13 22:03:51.795359700+00:00,1
+2023-10-13 22:03:51.921152661+00:00,1
+2023-10-13 22:03:52.047479089+00:00,1
+2023-10-13 22:03:52.172926412+00:00,1
+2023-10-13 22:03:52.300312703+00:00,1
+2023-10-13 22:03:52.426321196+00:00,1
+2023-10-13 22:03:52.552341292+00:00,1
+2023-10-13 22:03:52.677936084+00:00,1
+2023-10-13 22:03:52.804042020+00:00,1
+2023-10-13 22:03:52.930248905+00:00,1
+2023-10-13 22:03:53.057335660+00:00,1
+2023-10-13 22:03:53.183320437+00:00,1
+2023-10-13 22:03:53.309338509+00:00,1
+2023-10-13 22:03:53.435014276+00:00,1
+2023-10-13 22:03:53.560604124+00:00,1
+2023-10-13 22:03:53.686574463+00:00,1
+2023-10-13 22:03:53.812535025+00:00,1
+2023-10-13 22:03:53.938405240+00:00,1
+2023-10-13 22:03:54.064568079+00:00,1
+2023-10-13 22:03:54.190613293+00:00,1
+2023-10-13 22:03:54.316616056+00:00,1
+2023-10-13 22:03:54.442955798+00:00,1
+2023-10-13 22:03:54.569594773+00:00,1
+2023-10-13 22:03:54.695332678+00:00,1
+2023-10-13 22:03:54.822210720+00:00,1
+2023-10-13 22:03:54.948382723+00:00,1
+2023-10-13 22:03:55.074529220+00:00,1
+2023-10-13 22:03:55.200114534+00:00,1
+2023-10-13 22:03:55.325439665+00:00,1
+2023-10-13 22:03:55.452038443+00:00,1
+2023-10-13 22:03:55.578621070+00:00,1
+2023-10-13 22:03:55.704748729+00:00,1
+2023-10-13 22:03:55.831554327+00:00,1
+2023-10-13 22:03:55.958390504+00:00,1
+2023-10-13 22:03:56.084275179+00:00,1
+2023-10-13 22:03:56.210195440+00:00,1
+2023-10-13 22:03:56.335535827+00:00,1
+2023-10-13 22:03:56.461915687+00:00,1
+2023-10-13 22:03:56.587977285+00:00,1
+2023-10-13 22:03:56.713792201+00:00,1
+2023-10-13 22:03:56.840539063+00:00,1
+2023-10-13 22:03:56.966465662+00:00,1
+2023-10-13 22:03:57.091951741+00:00,1
+2023-10-13 22:03:57.217436928+00:00,1
+2023-10-13 22:03:57.342866825+00:00,1
+2023-10-13 22:03:57.468052049+00:00,1
+2023-10-13 22:03:57.593869229+00:00,1
+2023-10-13 22:03:57.719208598+00:00,1
+2023-10-13 22:03:57.845083280+00:00,1
+2023-10-13 22:03:57.971047376+00:00,1
+2023-10-13 22:03:58.097193658+00:00,1
+2023-10-13 22:03:58.222989758+00:00,1
+2023-10-13 22:03:58.350000820+00:00,1
+2023-10-13 22:03:58.475766940+00:00,1
+2023-10-13 22:03:58.601672447+00:00,1
+2023-10-13 22:03:58.728600818+00:00,1
+2023-10-13 22:03:58.854417322+00:00,1
+2023-10-13 22:03:58.980745469+00:00,1
+2023-10-13 22:03:59.106833290+00:00,1
+2023-10-13 22:03:59.233720489+00:00,1
+2023-10-13 22:03:59.360178853+00:00,1
+2023-10-13 22:03:59.487410524+00:00,1
+2023-10-13 22:03:59.613596429+00:00,1
+2023-10-13 22:03:59.739858166+00:00,1
+2023-10-13 22:03:59.867807339+00:00,1
+2023-10-13 22:03:59.995769672+00:00,1
+2023-10-13 22:04:00.122807924+00:00,1
+2023-10-13 22:04:00.249635657+00:00,1
+2023-10-13 22:04:00.377370213+00:00,1
+2023-10-13 22:04:00.505354184+00:00,1
+2023-10-13 22:04:00.631963593+00:00,1
+2023-10-13 22:04:00.759326289+00:00,1
+2023-10-13 22:04:00.885475500+00:00,1
+2023-10-13 22:04:01.011726212+00:00,1
+2023-10-13 22:04:01.138593645+00:00,1
+2023-10-13 22:04:01.264701597+00:00,1
+2023-10-13 22:04:01.391663256+00:00,1
+2023-10-13 22:04:01.517319519+00:00,1
+2023-10-13 22:04:01.643794099+00:00,1
+2023-10-13 22:04:01.770899844+00:00,1
+2023-10-13 22:04:01.897205141+00:00,1
+2023-10-13 22:04:02.022746814+00:00,1
+2023-10-13 22:04:02.149585467+00:00,1
+2023-10-13 22:04:02.277361011+00:00,1
+2023-10-13 22:04:02.402888731+00:00,1
+2023-10-13 22:04:02.528567513+00:00,1
+2023-10-13 22:04:02.653968340+00:00,1
+2023-10-13 22:04:02.780130882+00:00,1
+2023-10-13 22:04:02.905254791+00:00,1
+2023-10-13 22:04:03.031450472+00:00,1
+2023-10-13 22:04:03.157331714+00:00,1
+2023-10-13 22:04:03.283012960+00:00,1
+2023-10-13 22:04:03.408142074+00:00,1
+2023-10-13 22:04:03.533983519+00:00,1
+2023-10-13 22:04:03.659383284+00:00,1
+2023-10-13 22:04:03.785481277+00:00,1
+2023-10-13 22:04:03.911344442+00:00,1
+2023-10-13 22:04:04.037531119+00:00,1
+2023-10-13 22:04:04.163376253+00:00,1
+2023-10-13 22:04:04.289537811+00:00,1
+2023-10-13 22:04:04.415204432+00:00,1
+2023-10-13 22:04:04.540455133+00:00,1
+2023-10-13 22:04:04.665793253+00:00,1
+2023-10-13 22:04:04.790976219+00:00,1
+2023-10-13 22:04:04.915999797+00:00,1
+2023-10-13 22:04:05.041376820+00:00,1
+2023-10-13 22:04:05.167131764+00:00,1
+2023-10-13 22:04:05.292454316+00:00,1
+2023-10-13 22:04:05.418454850+00:00,1
+2023-10-13 22:04:05.543627154+00:00,1
+2023-10-13 22:04:05.668913116+00:00,1
+2023-10-13 22:04:05.794728173+00:00,1
+2023-10-13 22:04:05.920296922+00:00,1
+2023-10-13 22:04:06.046410970+00:00,1
+2023-10-13 22:04:06.172420721+00:00,1
+2023-10-13 22:04:06.297815534+00:00,1
+2023-10-13 22:04:06.423648485+00:00,1
+2023-10-13 22:04:06.548882484+00:00,1
+2023-10-13 22:04:06.674713493+00:00,1
+2023-10-13 22:04:06.800199209+00:00,1
+2023-10-13 22:04:06.926027124+00:00,1
+2023-10-13 22:04:07.051460481+00:00,1
+2023-10-13 22:04:07.176943458+00:00,1
+2023-10-13 22:04:07.302022033+00:00,1
+2023-10-13 22:04:07.427411183+00:00,1
+2023-10-13 22:04:07.552870102+00:00,1
+2023-10-13 22:04:07.678319667+00:00,1
+2023-10-13 22:04:07.803316587+00:00,1
+2023-10-13 22:04:07.928313346+00:00,1
+2023-10-13 22:04:08.054583330+00:00,1
+2023-10-13 22:04:08.179810126+00:00,1
+2023-10-13 22:04:08.305503651+00:00,1
+2023-10-13 22:04:08.430845156+00:00,1
+2023-10-13 22:04:08.555997687+00:00,1
+2023-10-13 22:04:08.681057721+00:00,1
+2023-10-13 22:04:08.805949388+00:00,1
+2023-10-13 22:04:08.931234726+00:00,1
+2023-10-13 22:04:09.057192063+00:00,1
+2023-10-13 22:04:09.182454515+00:00,1
+2023-10-13 22:04:09.307819813+00:00,1
+2023-10-13 22:04:09.433429287+00:00,1
+2023-10-13 22:04:09.558757121+00:00,1
+2023-10-13 22:04:09.684625362+00:00,1
+2023-10-13 22:04:09.810543933+00:00,1
+2023-10-13 22:04:09.936385044+00:00,1
+2023-10-13 22:04:10.061937995+00:00,1
+2023-10-13 22:04:10.188011628+00:00,1
+2023-10-13 22:04:10.313574755+00:00,1
+2023-10-13 22:04:10.439410563+00:00,1
+2023-10-13 22:04:10.565055898+00:00,1
+2023-10-13 22:04:10.691093863+00:00,1
+2023-10-13 22:04:10.816972211+00:00,1
+2023-10-13 22:04:10.942550245+00:00,1
diff --git a/tests/zero-downtime-upgrades/results/1.0.0/https.png b/tests/zero-downtime-upgrades/results/1.0.0/https.png
new file mode 100644
index 0000000000000000000000000000000000000000..438a39f4d7177cb4dfa73cb63f8d6afdf4d3327d
GIT binary patch
literal 6635
zcmeHLdpOi--~X;w?Jm;0S&Bk!tD;P0<!nfzEvKB4Qz{`0AtR?j%e2`-l5!dkN<xHY
zG!COih@5ktF^|JIn?b|G7{6y`_kEsy-|N~x-v8czx~}_oy07p3`F_5K`~KbE`!`E-
z6N&F+zXJe?)21iS0c=tP5IO(NCP7c}vw#zV4zN6HV~ik(peH*!TllC0h(hWB0T!#y
z0GgV>&}#&#Q-mmkx+vtx#i)1ygD60v5FGOHiL$a_sKGxw1mk37W!2Qw%*@QLUcHLL
z;T}JJTwY#YS6BDp!v`jlDNwO+>{uczvABO?!FJ)F2*RSrAjCH%EEduS0&Q?baL4V9
z->VvL|HpUy&Yk`ODgdqcwpkETM4+t`kxE8zmJ3l)h%u~11x!B-U6}a2hU8rtC$_vy
z%Z^e+o&YO`^#Ho*+4Mw916yxYloUisNkth5T2$1<x~Pjtl#ssxk}YT#k!*jYt_~5{
zgVeQ>vKEkqY+~yh<V|Z`X*gn#jVRXE)(Y71MPPjstck+vX5fAU%ijS<43;EdaVxO5
z!Tk3yw;g7FfN5!%+yP8k7?*?5-7q2#3`L;t1=^1=_!A5$LEi!BQHGBy&~+F(P|$t^
zT93g8HE33cCJkuRgm>CduLE!OpbiZb1E@Ix)kZ)zhDsA4nZm1IpyCXanL()qlvqOX
zIViG*=eF?79`et_lM9gN2szG>?Fxj8ka-C*E<>6-Jo124Pe{2254_>NFC_Xwf<N31
zfZqe*&JDPY1$+p^hQhC55EB7WIKV|gL=1%e3ZbzOf(PtvxN!#pe}{m(;GY0~iQs!5
zydS`|6!1(1k4NC12A4D7QYKs^fNM55=YV4#TzCTK^TGZZ*ggmAA~;tJmL*_O3T9<+
zrUHI>1*Rl0sRUy(7*)fG8Ze*$x(@W-f=)eXzXQ!i&}ahnW>EV8$6Db?JD@t?a2Ke2
z1mzw$&<9Eb@Y5jtNQ1p}P-KAo2<#pOxp9zX!j4Iho`xT0VEY_=KM&j3uyqk6mOzXH
z-z~#8T-dw{qHC~e9YpxTT+`E#s1RiExBjLMfdE_Uh1XXpDqH0Mq?1mcG_(oHn5T=~
z&$*C$((U^9+jbZpsmMb1&3yB8$x<rZ2E{8vVYr?5C8m#KM{?P0%>L8&_x(+4ACNI&
zjxhN@+d4eB!3)jak9EvQ!H?4SaL`25Agjc~uBq}j{Afq8FGhwG(mYRBJy=NEWh2WG
zyJD|r{oHDMdQoYa)8Obqu`1_)3yNKGnbM|oKHbjPN}Vs+G8Q{?o<uHj!g^J{Do!A=
zs*cjPr**r`j8>Db{ybG}jLwl84B}1Q!yZVeQb~8V&U)d>j;I~7ZgaN1r@_KnyRurQ
z#E+rt{26(N<J3$YKl+AE$wscfqZ50AqQyyejScGbmYB}fe>AaKL|zJ7T-&#E7hc{Z
zL$!kc>4}kpM&lEInwev7ReBhKO+-KD@Dz@-g4M{VWYJ?<bw^GPUI~%tLIPM}5!#of
z1bws%88}LUjHw>^h^u1H>bG6feK^u-Mh_8<%*E4NYud4nsr)@diw)(?2{KI_pl<U@
z-f%9ey`4YF!!LywWE@Hy=`Ev?_#JN16<C=j=I*HO*cV0la%;&Z))g9DeTHwKHsx$g
zo6Fc)>89~N#<Rq`dK;HsO>$2sq^>bWlk?A0i}Uk8(WO-gmtIYZ-scDB$5`8TGbSd6
zOM=5_WCqo9%wcT5#mj~*J0y~hzxPfI;KWCUZ$8+bs^s^z#tr1Vi&Kv3H<KcFR)lYU
zteWT;Mj4#H?orM!c2Vr+pP3Z|C~&<;SJz0v<<_{ArL!K6Elc7YXCLw0kR7jJtc6C?
z`guK=g#3iaI?u5lv3h%j(5{6QFRpp}3%y+p>`Tk+jgOOO_{F|G75T2yyv{S@i~B>V
z+1=8)*q3B}@RkfzqMT6Y(Xr`CqmCZ2D?9f3-WTo4JV8!SY&OC!FKL<L3iS9BXvQFa
zHbJ>W>CPFWMnvh#j-bKOr9Jgifr*$Zw`kfRZyD2WQsq+}8vQ#fog=S*IEm#SZHlcl
z_`5tt>|^Ucky79N)8&B<iO)J>e!qmWDPv#zaJ1{n?&O%$OUu}(<a*JiTDOdSuDC6l
zr#GTSV|K3WK1jLLkIf5;&X(k&BzgB9sBYtoY%tvzNlQGL+Z@AeyO^9Qb-%UH&l|XF
zJQU)0>DGmB@8tAVPtJ~T=_EA{<3hl4Y#)W|JO8IBdo}cPH}`M_kE@zU@J;_qy5$p6
z%U#9O)kqt&XP0hu_IdUtv-KQSx(5R64T`@u?(`o1w7jr9RamNu_9<-PcHE_$3R~y1
zvFqzT&U^j>ADezs#49P5N9SI{`)TfuxVw^8-sm3T<M7`yegXoy6aL}H)YlpqMg#L$
zT#Z2k1Bt|oo%}{f&c#tWRiW1%xtpgU$Uq)LhQIFP`akeL)uw%t<XPCIb&9;6^VHbb
z;JRlS{<@G=f=3r;QLnqMx>2%b+@i&GWHGfZqI-YkhU<kp>?Y*S$G2#_#ag@^jG!2p
zqbuh;SNQCfaG5h-3zZVX%p%vesH~p)Oy=##!yOG=d*g0W&c?Y;Oe&t>@myK6`=>>C
zJh3Z3TB;aYaSo~`PDbaLes%4q19zl^rhj?;;)j1HK?$j>pgBjo;@1ORRW-Eci0e+f
zcx9VUCJnQgb`Nd2*eaak144P{yTieBBy%N*r^#31Zzw#%EQRpiE1~1<pW;K_7mgt~
zdGu{HzYB_)WQxrV9BFyc%U+G#VfSuo3vV&j>jAg5{%v?sR`Ilw%2jOj`}Wb4Cx_O4
z;wsdupiqfq&$d)mvAlyb!F?a8u`8b$D;>TLwsgtC&e+or5-b{L-lt*8E(J8EuOD)8
zXv=X8U>jDiEQx4k$u);ESPf!%%7J66Z>K4Avv;d`R!({cRrJq2o<3@4(HoTfL$c2B
zxW{61jX+(C<fw#Sh&yf9OpcD0xLBTM1~+mC!u5AjavwlF?To$tAfdfs=Cx26<_~4B
zQ;jKCRt>91^!#?eazA=OasH-r2CBk%{GBNxJ09+K`&{@{?Ulyd@%H6ns&6tn*=!|&
zMDSTE4|2z%<0Id{y{yR#oSxH}OUO?o<K%5s35GdZ6d$j+J{12{mLW&(p+dc!i;H6Z
zjKZCj&mMi5hcLC93zROuknYfGZuGdk>}00laFjG=7AajqYsqmfTUQ)jh$G!hcdlYT
zG)C-fYd)*?3jQMZEQ->IXu6Z?+1&8(V|Erj0=upMv^zkWf1ekT6~=ciGIEe|$0_xv
z8uw)BZ+RU)mn25dmv`r~t=4dlx@tI;PLEFAWc+;SY{N`}&;{52^mnts--uzg$QpNd
z<o%5g5j{!?3iUgOMUo7J%WRU!$6qX^4t{z4;)gFu@MSW5Sqc6RE8^((Vwz0N#L_LJ
z>l@xrpX={34sG?2Hr3%ri=AZ5*h`zTQHK)I8mM8)a{F5)lrOp_;K%r9`jrM`j|=W}
z=Ur`zzP3`f2+iA79u*!Gy*};aT#cK(LVcp*NFunrPBL~<(7#n-`&#aBBJtVKN=J>h
z*BzrFJA#uUZo}!hCOHJ@eIIwmo{0GaxK5wDm;aGZbd*RH?Lwt){ANq}d2lpPY`mRt
z<_8lQHbZ;Bf4aGMKHna5<-Z2@w=3ey91rk1MGkJwClF96s@q;h_(y+q@6OzJN-Xd2
zC}(Uno<k#|hdE0Gv__^MC1SQIviepkkzw|N)-<XxMF><t6AhYK`#Ncww>X7w)Na?l
z*&<^~**x=JWM|us+RdHzpXaQLg3->3^s4h!EB*`4MO?Z?#7BXUBUa&-pwL_NaJP~t
zr}KB6WB`Y)v%b$+7vYE`xrlz*;C|WN|36ka%w(%btf2f|zauDkyF_F3X%yW#opWS^
z4m>XvA7Rpra!|x#c=UhQ%vm*?D04bWqRHd4Y}o&s*mwk0^K|XTR{#6qEh0%F8!Kgr
z|EbM6xXxa8t5@_`Bf-H^*y`%kjC*<`HJJDOK;hL-rMcv1sd0zzd9PWVZ^7f`8q<8M
z+|OIf7>B*Wn^1ernd_b^`2qSqqyS})F*^&UCsHm)6nE4~>2{NRy0pFdxt{!VNXB!Y
zU^9m-oPKTSz{FxH{~uaPsI<FOa+3SRNJHtn>t>xUeu0<!HG6%AB&ccEaGDDK)Y4bh
zT5`l-<w-%oZqvqg`couC0DoKvUoM2-v=Y%gS~|3wTxj;PT_U%TdOQK2$>Dkql@e(L
zAvE)PuE{cM<kZWn8a_LNgCcE6!m)iOL&cAq3*7|7k7M{YE-UCNx0X`+OhJZt6*`4r
znM!3ir%iulVec`dD^Co&bl#cyl6=(XCT+?pBD~V=xQPIaC@0@@m)dK=T;GGc)c>%6
zrgt<YmEVr(KjuRURi31>&NBBuu;p?`)+K-9(noi#^|@yHeeyqV9-bbj-e}GDlE;WU
z%J)g7PUcBUV(;`w%F#>R`!OLl8e1Lp1lD=y2k82=*xYL<&T4MiAek|P`Z4o1E)rF0
z0)o9`MZO3J6B1(HzVdWPSJrN~>$G}0{sw8>sz#ou`$DUN9@w?AIMVjgJYmM4`p_*`
zLIA7vC#<#)-^Tp!V2|jXwLLp5{=NrqMv!Pxp&X{JxnA*>X`1uTb)mT?P{q2I#ja?C
z@v8!&Ki~zl281D!C3-1);2qUca{28mB!LA?8y86-$s<};J>7kSlZ6T93*~N;=^EnR
zym%FYjPA|KmgQ*)$GG|9y6@7q{t-$G8p+IhwSJ_5N0|2$M1ovUeRQtPCi3m)Cv9$n
z)h#_+N^-W^$I@+8T*~j_)h7|c%ZWvm7CHRv<h6HV1JA2#h4FhTGo35+TV#a*V3*|3
zu@sH1Tz}=pzd+@K{glyeoT~$FYa8#~I5|^gtz&LAk!c!|j?2U2cdyS6a=lTM!=#$l
zuvzgkul~V;YVAQxkInE7;)d_=bmc4S)el3}Rk3PKPu;NjOFzc7-&Jj#@$f=xbLa^L
zPpLBzRo;a^5HM!=WxELSjfl$Xvg)wPJ$6~{CA=VSeIHH+_CI-DV)P=tSx)FV>V}3H
zGrYgKAVVVTND9F}{BiJFU9E0r%gm>$?Wu*izVebfr{wvLQ9abqRGh+f^*JV0VkmQ~
z#$a$uY4@8#K^mGfefrCV6Gi`Kr5lp|R4^f#Hf?cwkAC^z2g!K}Ym#wpJrCCcP%45G
PYdCFee)7eMi!uKKYl4f;

literal 0
HcmV?d00001

diff --git a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
new file mode 100644
index 000000000..6bf695673
--- /dev/null
+++ b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
@@ -0,0 +1,252 @@
+# Zero-Downtime Upgrades
+
+This document describes a test plan for testing zero-downtime upgrades of NGF.
+
+*Zero-downtime upgrades* means that during an NGF upgrade clients don't experience any
+interruptions to the traffic they send to applications exposed via NGF.
+
+<!-- TOC -->
+
+- [Zero-Downtime Upgrades](#zero-downtime-upgrades)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+  - [Test Environment](#test-environment)
+  - [Steps](#steps)
+    - [Start](#start)
+    - [Upgrade](#upgrade)
+    - [After Upgrade](#after-upgrade)
+    - [Analyze](#analyze)
+  - [Results](#results)
+  - [Appendix](#appendix)
+    - [Pod Affinity](#pod-affinity)
+    - [Converting Curl Output to a Graph](#converting-curl-output-to-a-graph)
+
+<!-- TOC -->
+
+## Goals
+
+- Ensure that upgrading NFG doesn't lead to any loss of traffic flowing through the data plane.
+- Ensure that after an upgrade, NGF can process changes to resources.
+- Detect if any special instructions will be required to provide to users to perform
+  an upgrade.
+
+## Non-Goals
+
+During an upgrade, Kubernetes will shut down existing NGF pods by sending a SIGTERM. If the pod doesn't terminate in 30
+seconds (the default period) , Kubernetes will send a SIGKILL.
+
+When proxying Websocket or any long-lived connections, NGINX will not terminate until
+that connection is closed by either the client or the backend. This means that unless all those connections are closed
+by clients/backends before or during an upgrade (which is highly unlikely), NGINX will not terminate, which means
+Kubernetes will kill NGINX. As a result, the clients will see the connections abruptly closed and thus experience
+downtime.
+
+As a result, we *will not* use any long-live connections in this test, because NGF cannot support zero-downtime upgrades
+in this case.
+
+## Test Environment
+
+- A Kubernetes cluster with 10 nodes on GKE
+  - Node: e2-medium (2 vCPU, 4GB memory)
+  - Enabled GKE logging.
+- Tester VMs on Google Cloud Platform:
+  - Configuration:
+    - Debian
+    - Install packages: wrk, curl, gnuplot
+  - Location - same zone as the Kubernetes cluster.
+  - First VM for HTTP traffic
+  - Second VM - for sending HTTPs traffic
+- NGF
+  - Deployment with 2 replicas scheduled on different nodes.
+  - Exposed via a Service with type LoadBalancer, private IP
+  - Gateway, two listeners - HTTP and HTTPs
+  - Two backends:
+    - Coffee - 3 replicas
+    - Tea - 3 replicas
+  - Two HTTPRoutes
+    - Coffee (HTTP)
+    - Tea (HTTPS)
+
+Notes:
+
+- For sending traffic, we will use both wrk and curl.
+  - *wrk* will generate a lot of traffic continuously, and it will have a high chance of catching of any
+      (however small) periods of downtime.
+  - *curl* will generate 1 request every 0.1s. While it might not catch small periods of downtime, it will
+      give us timeline of failed request for big periods of downtime, which wrk doesn't do.
+- We use pod anti-affinity to tell Kubernetes to schedule NGF pods on different nodes. We also use a 10 node cluster so
+  that the chance of Kubernetes scheduling new pods on the same
+  nodes is minimal. Scheduling new pods on different nodes will help better catch
+  any interdependencies with an external load balancer (typically the node of a new pod will be added
+  to the pool in the load balancer, and the node of an old one will be removed).
+
+## Steps
+
+### Start
+
+1. Create a cluster.
+2. Deploy a previous latest stable version with 2 replicas with added [anti-affinity](#pod-affinity).
+3. Expose NGF via a Service Load Balancer, internal (only accessible within the Google Cloud region) by adding
+   `networking.gke.io/load-balancer-type: "Internal"` annotation to the Service.
+4. Deploy backend apps:
+
+    ```console
+    kubectl apply -f manifests/cafe.yaml
+    ```
+
+5. Configure Gateway:
+
+    ```console
+    kubectl apply -f manifests/cafe-secret.yaml
+    kubectl apply -f manifests/gateway.yaml
+    ```
+
+6. Expose apps via HTTPRoutes
+
+    ```console
+    kubectl apply -f manifests/cafe-routes.yaml
+    ```
+
+7. Check statuses of the Gateway and HTTPRoutes for errors.
+8. In Google Monitoring, check NGF and NGINX error logs for errors.
+9. In Tester VMs, update `/etc/hosts` to have an entry with the External IP of the NGF Service (`10.128.0.10` in this
+   case):
+
+   ```text
+   10.128.0.10 cafe.example.com
+   ```
+
+### Upgrade
+
+1. Follow the [upgrade instructions](/docs/installation.md#upgrade-nginx-gateway-fabric-from-manifests) to:
+    1. Upgrade Gateway API version to the one that matches the supported version of new release.
+    2. Upgrade NGF CRDs.
+2. Start sending traffic using wrk from tester VMs for 1 minute:
+    - Tester VM 1:
+        - wrk:
+
+          ```console
+          wrk -t2 -c100 -d60s --latency --timeout 2s  http://cafe.example.com/coffee
+          ```
+
+        - curl:
+
+          ```console
+          for i in `seq 1 600`; do printf  "\nRequest $i\n" && date --rfc-3339=ns && curl -sS --connect-timeout 2 http://cafe.example.com/coffee 2>&1  && sleep 0.1s; done > results.txt
+          ```
+
+    - Tester VM 2:
+        - wrk:
+
+          ```console
+          wrk -t2 -c100 -d60s --latency --timeout 2s  https://cafe.example.com/tea
+          ```
+
+        - curl:
+
+          ```console
+          for i in `seq 1 600`; do printf  "\nRequest $i\n" && date --rfc-3339=ns && curl -k -sS --connect-timeout 2 https://cafe.example.com/tea 2>&1  && sleep 0.1s; done > results.txt
+          ```
+
+3. **Immediately** upgrade NFG manifests by
+   following [upgrade instructions](/docs/installation.md#upgrade-nginx-gateway-fabric-from-manifests).
+   > Don't forget to modify the manifests to have 2 replicas and pod affinity.
+4. Ensure the new pods are running and the old ones terminate.
+
+### After Upgrade
+
+1. Update the Gateway resource by adding one new listener `http-new`:
+
+    ```console
+    kubectl apply -f manifests/gateway-updated.yaml
+    ```
+
+2. Check that at NGF has a leader elected among the new pods:
+
+    ```console
+    kubectl -n nginx-gateway logs <nkg-pod> | grep leader
+    ```
+
+3. Ensure the status of the Gateway resource includes the new listener.
+
+### Analyze
+
+- Tester VMs:
+  - Analyze the output of wrk commands for errors and latencies.
+  - Create graphs from curl output (see [instructions](#converting-curl-output-to-a-graph) in Appendix) and check for
+      any failures on them.
+- Check the old pods logs in Google Monitoring
+  - NGINX Access logs - we expect only 200 responses.
+      Google Monitoring query:
+
+      ```text
+      severity=INFO
+      "GET" "HTTP/1.1" -"200"
+      ```
+
+  - NGINX Error logs - we expect no errors or warnings
+      Google Monitoring query:
+
+      ```text
+      severity=ERROR
+      SEARCH("`[warn]`") OR SEARCH("`[error]`")
+      ```
+
+  - NGF logs - we expect no errors
+  - Specifically look at the NFG logs before it exited, to make sure all components shutdown correctly.
+- Check the new pods (in Google Monitoring)
+  - NGINX Access logs - only 200 responses.
+  - NGINX Error logs - no errors or warnings.
+  - NGF logs - no errors
+
+## Results
+
+- [1.0.0](results/1.0.0/1.0.0.md)
+
+## Appendix
+
+### Pod Affinity
+
+- To ensure Kubernetes doesn't schedule NFG pods on the same nodes, use an anti-affinity rule:
+
+    ```yaml
+        spec:
+          affinity:
+            podAntiAffinity:
+              requiredDuringSchedulingIgnoredDuringExecution:
+              - topologyKey: kubernetes.io/hostname
+                labelSelector:
+                  matchLabels:
+                    app.kubernetes.io/name: nginx-gateway
+    ```
+
+### Converting Curl Output to a Graph
+
+The output of a curl command is saved in `results.txt`. To convert it into a graph,
+go through the following steps:
+
+1. Convert the output into a csv file:
+
+    ```console
+    awk '
+    /Request [0-9]+/ {
+        getline
+        datetime = $0
+        getline
+        if ($1 == "curl:") {
+            print datetime ",0"  # Failed
+        } else {
+            print datetime ",1"  # Success
+        }
+    }' results.txt > results.csv
+    ```
+
+2. Plot a graph using the csv file:
+
+    ```console
+    gnuplot requests-plot.gp
+    ```
+
+   As a result, gnuplot will create `graph.png` with a graph.
+3. Download the resulting `graph.png` to you local machine.
+4. Also download `results.csv`.

From 59d3d066698322f3e1e77137ed8177a16a0a2e7b Mon Sep 17 00:00:00 2001
From: Michael Pleshakov <m.pleshakov@f5.com>
Date: Mon, 16 Oct 2023 13:08:21 -0400
Subject: [PATCH 3/7] NFG -> NFG

---
 tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md    | 2 +-
 tests/zero-downtime-upgrades/zero-downtime-upgrades.md | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
index 4bc9e4cc9..95ad8d7d0 100644
--- a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
+++ b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
@@ -159,7 +159,7 @@ to the coffee app.
 #### Old pods
 
 - nginx-gateway-55cb958549-nbdqz
-  - NFG handled a panic before exiting:
+  - NGG handled a panic before exiting:
 
       ```text
       INFO 2023-10-13T22:02:33.321381968Z [resource.labels.containerName: nginx-gateway] {"level":"info", "msg":"Stopping and waiting for caches", "ts":"2023-10-13T22:02:33Z"}
diff --git a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
index 6bf695673..f4b62f12d 100644
--- a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
+++ b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
@@ -25,7 +25,7 @@ interruptions to the traffic they send to applications exposed via NGF.
 
 ## Goals
 
-- Ensure that upgrading NFG doesn't lead to any loss of traffic flowing through the data plane.
+- Ensure that upgrading NGF doesn't lead to any loss of traffic flowing through the data plane.
 - Ensure that after an upgrade, NGF can process changes to resources.
 - Detect if any special instructions will be required to provide to users to perform
   an upgrade.
@@ -148,7 +148,7 @@ Notes:
           for i in `seq 1 600`; do printf  "\nRequest $i\n" && date --rfc-3339=ns && curl -k -sS --connect-timeout 2 https://cafe.example.com/tea 2>&1  && sleep 0.1s; done > results.txt
           ```
 
-3. **Immediately** upgrade NFG manifests by
+3. **Immediately** upgrade NGF manifests by
    following [upgrade instructions](/docs/installation.md#upgrade-nginx-gateway-fabric-from-manifests).
    > Don't forget to modify the manifests to have 2 replicas and pod affinity.
 4. Ensure the new pods are running and the old ones terminate.
@@ -193,7 +193,7 @@ Notes:
       ```
 
   - NGF logs - we expect no errors
-  - Specifically look at the NFG logs before it exited, to make sure all components shutdown correctly.
+  - Specifically look at the NGF logs before it exited, to make sure all components shutdown correctly.
 - Check the new pods (in Google Monitoring)
   - NGINX Access logs - only 200 responses.
   - NGINX Error logs - no errors or warnings.
@@ -207,7 +207,7 @@ Notes:
 
 ### Pod Affinity
 
-- To ensure Kubernetes doesn't schedule NFG pods on the same nodes, use an anti-affinity rule:
+- To ensure Kubernetes doesn't schedule NGF pods on the same nodes, use an anti-affinity rule:
 
     ```yaml
         spec:

From cd798d65dbf8c2112454f3dbd5115e1ed53c54b8 Mon Sep 17 00:00:00 2001
From: Michael Pleshakov <m.pleshakov@f5.com>
Date: Mon, 16 Oct 2023 13:13:02 -0400
Subject: [PATCH 4/7] pod -> Pod

---
 .../results/1.0.0/1.0.0.md                    | 18 +++++++--------
 .../zero-downtime-upgrades.md                 | 22 +++++++++----------
 2 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
index 95ad8d7d0..69d010786 100644
--- a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
+++ b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
@@ -7,8 +7,8 @@
   - [Upgrades](#upgrades)
     - [Analyze](#analyze)
       - [Tester VMs](#tester-vms)
-      - [Old pods](#old-pods)
-      - [New pods](#new-pods)
+      - [Old Pods](#old-pods)
+      - [New Pods](#new-pods)
   - [Opened Issues](#opened-issues)
   - [Future Improvements](#future-improvements)
 <!-- TOC -->
@@ -75,16 +75,16 @@ Logs check:
 
 ## Upgrades
 
-New pods:
+New Pods:
 
 ```text
 nginx-gateway-578b49bc58-hmx5x   2/2     Running   0          11s   10.112.1.9    gke-michael-2-default-pool-18ad0f59-7dqr   <none>           <none>
 nginx-gateway-578b49bc58-r4ckb   2/2     Running   0          17s   10.112.5.26   gke-michael-2-default-pool-18ad0f59-l0cq   <none>           <none>
 ```
 
-Note: the new pods were scheduled on different from the old pods nodes, as we wanted.
+Note: the new Pods were scheduled on different from the old Pods nodes, as we wanted.
 
-Check that one of the NGF pods became the leader:
+Check that one of the NGF Pods became the leader:
 
 ```text
 I1013 22:02:32.226414       7 leaderelection.go:250] attempting to acquire leader lease nginx-gateway/nginx-gateway-leader-election...
@@ -156,7 +156,7 @@ Tester 2 graph:
 As we can see, there is period where curl failed to send requests
 to the coffee app.
 
-#### Old pods
+#### Old Pods
 
 - nginx-gateway-55cb958549-nbdqz
   - NGG handled a panic before exiting:
@@ -196,7 +196,7 @@ to the coffee app.
     - Access logs - only 200 responses.
     - Error logs - no errors or warnings.
 
-#### New pods
+#### New Pods
 
 - nginx-gateway-578b49bc58-hmx5x
   - NGF - no errors
@@ -217,7 +217,7 @@ to the coffee app.
 - nginx-gateway-578b49bc58-r4ckb
   - NGF - no errors.
   - NGINX
-    - Access logs - 35 responses similar to the first pod. Same conclusion as above.
+    - Access logs - 35 responses similar to the first Pod. Same conclusion as above.
     - Error logs - No errors or warnings.
 
 ## Opened Issues
@@ -228,4 +228,4 @@ to the coffee app.
 
 - Use helm for upgrade, to catch any helm-related bugs preventing an upgrade. In this test,
   we didn't use helm because 0.6.0 release does not allow you to configure the number of replicas
-  and pod affinity.
+  and Pod affinity.
diff --git a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
index f4b62f12d..e2edd9afa 100644
--- a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
+++ b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
@@ -32,7 +32,7 @@ interruptions to the traffic they send to applications exposed via NGF.
 
 ## Non-Goals
 
-During an upgrade, Kubernetes will shut down existing NGF pods by sending a SIGTERM. If the pod doesn't terminate in 30
+During an upgrade, Kubernetes will shut down existing NGF Pods by sending a SIGTERM. If the Pod doesn't terminate in 30
 seconds (the default period) , Kubernetes will send a SIGKILL.
 
 When proxying Websocket or any long-lived connections, NGINX will not terminate until
@@ -74,10 +74,10 @@ Notes:
       (however small) periods of downtime.
   - *curl* will generate 1 request every 0.1s. While it might not catch small periods of downtime, it will
       give us timeline of failed request for big periods of downtime, which wrk doesn't do.
-- We use pod anti-affinity to tell Kubernetes to schedule NGF pods on different nodes. We also use a 10 node cluster so
-  that the chance of Kubernetes scheduling new pods on the same
-  nodes is minimal. Scheduling new pods on different nodes will help better catch
-  any interdependencies with an external load balancer (typically the node of a new pod will be added
+- We use Pod anti-affinity to tell Kubernetes to schedule NGF Pods on different nodes. We also use a 10 node cluster so
+  that the chance of Kubernetes scheduling new Pods on the same
+  nodes is minimal. Scheduling new Pods on different nodes will help better catch
+  any interdependencies with an external load balancer (typically the node of a new Pod will be added
   to the pool in the load balancer, and the node of an old one will be removed).
 
 ## Steps
@@ -150,8 +150,8 @@ Notes:
 
 3. **Immediately** upgrade NGF manifests by
    following [upgrade instructions](/docs/installation.md#upgrade-nginx-gateway-fabric-from-manifests).
-   > Don't forget to modify the manifests to have 2 replicas and pod affinity.
-4. Ensure the new pods are running and the old ones terminate.
+   > Don't forget to modify the manifests to have 2 replicas and Pod affinity.
+4. Ensure the new Pods are running and the old ones terminate.
 
 ### After Upgrade
 
@@ -161,7 +161,7 @@ Notes:
     kubectl apply -f manifests/gateway-updated.yaml
     ```
 
-2. Check that at NGF has a leader elected among the new pods:
+2. Check that at NGF has a leader elected among the new Pods:
 
     ```console
     kubectl -n nginx-gateway logs <nkg-pod> | grep leader
@@ -175,7 +175,7 @@ Notes:
   - Analyze the output of wrk commands for errors and latencies.
   - Create graphs from curl output (see [instructions](#converting-curl-output-to-a-graph) in Appendix) and check for
       any failures on them.
-- Check the old pods logs in Google Monitoring
+- Check the old Pods logs in Google Monitoring
   - NGINX Access logs - we expect only 200 responses.
       Google Monitoring query:
 
@@ -194,7 +194,7 @@ Notes:
 
   - NGF logs - we expect no errors
   - Specifically look at the NGF logs before it exited, to make sure all components shutdown correctly.
-- Check the new pods (in Google Monitoring)
+- Check the new Pods (in Google Monitoring)
   - NGINX Access logs - only 200 responses.
   - NGINX Error logs - no errors or warnings.
   - NGF logs - no errors
@@ -207,7 +207,7 @@ Notes:
 
 ### Pod Affinity
 
-- To ensure Kubernetes doesn't schedule NGF pods on the same nodes, use an anti-affinity rule:
+- To ensure Kubernetes doesn't schedule NGF Pods on the same nodes, use an anti-affinity rule:
 
     ```yaml
         spec:

From 00567b57fa2f6d6ab10a08d01ab685c5caad8399 Mon Sep 17 00:00:00 2001
From: Michael Pleshakov <m.pleshakov@f5.com>
Date: Mon, 16 Oct 2023 13:16:57 -0400
Subject: [PATCH 5/7] Improve leader election check

---
 tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md    | 6 +++---
 tests/zero-downtime-upgrades/zero-downtime-upgrades.md | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
index 69d010786..b974e755f 100644
--- a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
+++ b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
@@ -87,9 +87,9 @@ Note: the new Pods were scheduled on different from the old Pods nodes, as we wa
 Check that one of the NGF Pods became the leader:
 
 ```text
-I1013 22:02:32.226414       7 leaderelection.go:250] attempting to acquire leader lease nginx-gateway/nginx-gateway-leader-election...
-I1013 22:02:32.248857       7 leaderelection.go:260] successfully acquired lease nginx-gateway/nginx-gateway-leader-election
-{"level":"info","ts":"2023-10-13T22:02:32Z","logger":"leaderElector","msg":"Started leading"}
+kubectl -n nginx-gateway get lease
+NAME                            HOLDER                           AGE
+nginx-gateway-leader-election   nginx-gateway-578b49bc58-r4ckb   1m 
 ```
 
 Pod nginx-gateway-578b49bc58-r4ckb is the leader.
diff --git a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
index e2edd9afa..06c7bada3 100644
--- a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
+++ b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md
@@ -164,7 +164,7 @@ Notes:
 2. Check that at NGF has a leader elected among the new Pods:
 
     ```console
-    kubectl -n nginx-gateway logs <nkg-pod> | grep leader
+    kubectl -n nginx-gateway get lease
     ```
 
 3. Ensure the status of the Gateway resource includes the new listener.

From b475b35e0ec59ccbf7f20007738150fe6612e973 Mon Sep 17 00:00:00 2001
From: Michael Pleshakov <m.pleshakov@f5.com>
Date: Mon, 16 Oct 2023 13:18:20 -0400
Subject: [PATCH 6/7] Add a link to the opened issue

---
 tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
index b974e755f..d9a736d8d 100644
--- a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
+++ b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
@@ -222,7 +222,7 @@ to the coffee app.
 
 ## Opened Issues
 
-- Traffic loss during an upgrade. (to be opened soon)
+- Clients experience downtime during NGF upgrade -- https://github.com/nginxinc/nginx-gateway-fabric/issues/1143
 
 ## Future Improvements
 

From 314d4c90d91f20c8887a95b31df73f7009f28749 Mon Sep 17 00:00:00 2001
From: Michael Pleshakov <m.pleshakov@f5.com>
Date: Mon, 16 Oct 2023 13:21:49 -0400
Subject: [PATCH 7/7] Fix linting

---
 tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
index d9a736d8d..f61124703 100644
--- a/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
+++ b/tests/zero-downtime-upgrades/results/1.0.0/1.0.0.md
@@ -89,7 +89,7 @@ Check that one of the NGF Pods became the leader:
 ```text
 kubectl -n nginx-gateway get lease
 NAME                            HOLDER                           AGE
-nginx-gateway-leader-election   nginx-gateway-578b49bc58-r4ckb   1m 
+nginx-gateway-leader-election   nginx-gateway-578b49bc58-r4ckb   1m
 ```
 
 Pod nginx-gateway-578b49bc58-r4ckb is the leader.