Add configurable delayed termination instructions (#1171)

* Add configurable delayed termination instructions
nginxinc · Oct 23, 2023 · bb95dea · bb95dea
1 parent 725ff40
commit bb95dea
Show file tree

Hide file tree

Showing 2 changed files with 147 additions and 0 deletions.
diff --git a/deploy/helm-chart/README.md b/deploy/helm-chart/README.md
@@ -1,5 +1,24 @@
 # NGINX Gateway Fabric Helm Chart
 
+- [NGINX Gateway Fabric Helm Chart](#nginx-gateway-fabric-helm-chart)
+  - [Introduction](#introduction)
+  - [Prerequisites](#prerequisites)
+    - [Installing the Gateway API resources](#installing-the-gateway-api-resources)
+  - [Installing the Chart](#installing-the-chart)
+    - [Installing the Chart from the OCI Registry](#installing-the-chart-from-the-oci-registry)
+    - [Installing the Chart via Sources](#installing-the-chart-via-sources)
+      - [Pulling the Chart](#pulling-the-chart)
+      - [Installing the Chart](#installing-the-chart-1)
+  - [Upgrading the Chart](#upgrading-the-chart)
+    - [Upgrading the Gateway Resources](#upgrading-the-gateway-resources)
+    - [Upgrading the CRDs](#upgrading-the-crds)
+    - [Upgrading the Chart from the OCI Registry](#upgrading-the-chart-from-the-oci-registry)
+    - [Upgrading the Chart from the Sources](#upgrading-the-chart-from-the-sources)
+    - [Configure Delayed Termination for Zero Downtime Upgrades](#configure-delayed-termination-for-zero-downtime-upgrades)
+  - [Uninstalling the Chart](#uninstalling-the-chart)
+    - [Uninstalling the Gateway Resources](#uninstalling-the-gateway-resources)
+  - [Configuration](#configuration)
+
 ## Introduction
 
 This chart deploys the NGINX Gateway Fabric in your Kubernetes cluster.
@@ -59,6 +78,10 @@ helm install my-release . --create-namespace --wait -n nginx-gateway
 
 ## Upgrading the Chart
 
+> **Note**
+> See [below](#configure-delayed-termination-for-zero-downtime-upgrades) for instructions on how to configure delayed
+> termination if required for zero downtime upgrades in your environment.
+
 ### Upgrading the Gateway Resources
 
 Before you upgrade a release, ensure the Gateway API resources are the correct version as supported by the NGINX
@@ -105,6 +128,55 @@ the release `my-release`, run:
 helm upgrade my-release . -n nginx-gateway
 ```
 
+### Configure Delayed Termination for Zero Downtime Upgrades
+
+To achieve zero downtime upgrades (meaning clients will not see any interruption in traffic while a rolling upgrade is
+being performed on NGF), you may need to configure delayed termination on the NGF Pod, depending on your environment.
+
+> **Note**
+> When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed
+> by either the client or the backend. This means that unless all those connections are closed by clients/backends
+> before or during an upgrade, NGINX will not terminate, which means Kubernetes will kill NGINX. As a result, the
+> clients will see the connections abruptly closed and thus experience downtime.
+
+1. Add `lifecycle` to both the nginx and the nginx-gateway container definition. To do so, update your `values.yaml`
+   file to include the following (update the `sleep` values to what is required in your environment):
+
+   ```yaml
+    nginxGateway:
+    <...>
+    lifecycle:
+        preStop:
+        exec:
+            command:
+            - /usr/bin/gateway
+            - sleep
+            - --duration=40s # This flag is optional, the default is 30s
+
+    nginx:
+    <...>
+    lifecycle:
+        preStop:
+        exec:
+            command:
+            - /bin/sleep
+            - "40"
+   ```
+
+2. Ensure the `terminationGracePeriodSeconds` matches or exceeds the `sleep` value from the `preStopHook` (the default
+   is 30). This is to ensure Kubernetes does not terminate the Pod before the `preStopHook` is complete. To do so,
+   update your `values.yaml` file to include the following (update the value to what is required in your environment):
+
+   ```yaml
+   terminationGracePeriodSeconds: 50
+   ```
+
+> **Note**
+> More information on container lifecycle hooks can be found
+> [here](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) and a detailed
+> description of Pod termination behavior can be found in
+> [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/Pods/Pod-lifecycle/#Pod-termination).
+
 ## Uninstalling the Chart
 
 To uninstall/delete the release `my-release`:

diff --git a/docs/installation.md b/docs/installation.md
@@ -2,6 +2,23 @@
 
 This guide walks you through how to install NGINX Gateway Fabric on a generic Kubernetes cluster.
 
+- [Installation](#installation)
+  - [Prerequisites](#prerequisites)
+  - [Deploy NGINX Gateway Fabric using Helm](#deploy-nginx-gateway-fabric-using-helm)
+  - [Deploy NGINX Gateway Fabric from Manifests](#deploy-nginx-gateway-fabric-from-manifests)
+  - [Expose NGINX Gateway Fabric](#expose-nginx-gateway-fabric)
+    - [Create a NodePort Service](#create-a-nodeport-service)
+    - [Create a LoadBalancer Service](#create-a-loadbalancer-service)
+  - [Upgrading NGINX Gateway Fabric](#upgrading-nginx-gateway-fabric)
+    - [Upgrade NGINX Gateway Fabric from Manifests](#upgrade-nginx-gateway-fabric-from-manifests)
+    - [Upgrade NGINX Gateway Fabric using Helm](#upgrade-nginx-gateway-fabric-using-helm)
+    - [Configure Delayed Termination for Zero Downtime Upgrades](#configure-delayed-termination-for-zero-downtime-upgrades)
+      - [Configure Delayed Termination Using Manifests](#configure-delayed-termination-using-manifests)
+      - [Configure Delayed Termination Using Helm](#configure-delayed-termination-using-helm)
+  - [Uninstalling NGINX Gateway Fabric](#uninstalling-nginx-gateway-fabric)
+    - [Uninstall NGINX Gateway Fabric from Manifests](#uninstall-nginx-gateway-fabric-from-manifests)
+    - [Uninstall NGINX Gateway Fabric using Helm](#uninstall-nginx-gateway-fabric-using-helm)
+
 ## Prerequisites
 
 - [kubectl](https://kubernetes.io/docs/tasks/tools/)
@@ -131,6 +148,10 @@ Create a Service with type `LoadBalancer` using the appropriate manifest for you
 
 ## Upgrading NGINX Gateway Fabric
 
+> **Note**
+> See [below](#configure-delayed-termination-for-zero-downtime-upgrades) for instructions on how to configure delayed
+> termination if required for zero downtime upgrades in your environment.
+
 ### Upgrade NGINX Gateway Fabric from Manifests
 
 1. Upgrade the Gateway Resources
@@ -168,6 +189,60 @@ Create a Service with type `LoadBalancer` using the appropriate manifest for you
 To upgrade NGINX Gateway Fabric when the deployment method is Helm, please follow the instructions
 [here](/deploy/helm-chart/README.md#upgrading-the-chart).
 
+### Configure Delayed Termination for Zero Downtime Upgrades
+
+To achieve zero downtime upgrades (meaning clients will not see any interruption in traffic while a rolling upgrade is
+being performed on NGF), you may need to configure delayed termination on the NGF Pod, depending on your environment.
+
+> **Note**
+> When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed
+> by either the client or the backend. This means that unless all those connections are closed by clients/backends
+> before or during an upgrade, NGINX will not terminate, which means Kubernetes will kill NGINX. As a result, the
+> clients will see the connections abruptly closed and thus experience downtime.
+
+#### Configure Delayed Termination Using Manifests
+
+Edit the `deploy/manifests/nginx-gateway.yaml` to include the following:
+
+1. Add `lifecycle` prestop hooks to both the nginx and the nginx-gateway container definitions:
+
+   ```yaml
+   <...>
+   name: nginx-gateway
+   <...>
+   lifecycle:
+   preStop:
+      exec:
+      command:
+      - /usr/bin/gateway
+      - sleep
+      - --duration=40s # This flag is optional, the default is 30s
+   <...>
+   name: nginx
+   <...>
+   lifecycle:
+   preStop:
+      exec:
+      command:
+      - /bin/sleep
+      - "40"
+   <...>
+   ```
+
+2. Ensure the `terminationGracePeriodSeconds` matches or exceeds the `sleep` value from the `preStopHook` (the default
+   is 30). This is to ensure Kubernetes does not terminate the Pod before the `preStopHook` is complete.
+
+> **Note**
+> More information on container lifecycle hooks can be found
+> [here](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) and a detailed
+> description of Pod termination behavior can be found in
+> [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/Pods/Pod-lifecycle/#Pod-termination).
+
+#### Configure Delayed Termination Using Helm
+
+To configure delayed termination on the NGF Pod when the deployment method is Helm, please follow the instructions
+[here](/deploy/helm-chart/README.md#configure-delayed-termination-for-zero-downtime-upgrades).
+
 ## Uninstalling NGINX Gateway Fabric
 
 ### Uninstall NGINX Gateway Fabric from Manifests