From 5cb480e405ea436fccde23bc29e3d37a42d7b7de Mon Sep 17 00:00:00 2001 From: Ciara Stacke Date: Fri, 20 Oct 2023 16:25:50 +0100 Subject: [PATCH 1/5] Add configurable delayed termination instructions --- deploy/helm-chart/README.md | 66 +++++++++++++++++++++++++++++++++++++ docs/installation.md | 65 ++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+) diff --git a/deploy/helm-chart/README.md b/deploy/helm-chart/README.md index b428944bf..d03bbb3b4 100644 --- a/deploy/helm-chart/README.md +++ b/deploy/helm-chart/README.md @@ -1,5 +1,24 @@ # NGINX Gateway Fabric Helm Chart +- [NGINX Gateway Fabric Helm Chart](#nginx-gateway-fabric-helm-chart) + - [Introduction](#introduction) + - [Prerequisites](#prerequisites) + - [Installing the Gateway API resources](#installing-the-gateway-api-resources) + - [Installing the Chart](#installing-the-chart) + - [Installing the Chart from the OCI Registry](#installing-the-chart-from-the-oci-registry) + - [Installing the Chart via Sources](#installing-the-chart-via-sources) + - [Pulling the Chart](#pulling-the-chart) + - [Installing the Chart](#installing-the-chart-1) + - [Upgrading the Chart](#upgrading-the-chart) + - [Upgrading the Gateway Resources](#upgrading-the-gateway-resources) + - [Upgrading the CRDs](#upgrading-the-crds) + - [Upgrading the Chart from the OCI Registry](#upgrading-the-chart-from-the-oci-registry) + - [Upgrading the Chart from the Sources](#upgrading-the-chart-from-the-sources) + - [Configure Delayed Termination for Zero Downtime Upgrades](#configure-delayed-termination-for-zero-downtime-upgrades) + - [Uninstalling the Chart](#uninstalling-the-chart) + - [Uninstalling the Gateway Resources](#uninstalling-the-gateway-resources) + - [Configuration](#configuration) + ## Introduction This chart deploys the NGINX Gateway Fabric in your Kubernetes cluster. @@ -105,6 +124,53 @@ the release `my-release`, run: helm upgrade my-release . -n nginx-gateway ``` +### Configure Delayed Termination for Zero Downtime Upgrades + +To achieve zero downtime upgrades (meaning clients will not see any interruption in traffic while a rolling upgrade is +being performed on NGF), you may need to configure delayed termination on the NGF pod, depending on your environment. + +> Note: When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed +> by either the client or the backend. This means that unless all those connections are closed by clients/backends +> before or during an upgrade, NGINX will not terminate, which means Kubernetes will kill NGINX. As a result, the +> clients will see the connections abruptly closed and thus experience downtime. + +1. Add `lifecycle` to both the nginx and the nginx-gateway container definition. To do so, update your `values.yaml` + file to include the following (update the `sleep` values to what is required in your environment): + + ```yaml + nginxGateway: + <...> + lifecycle: + preStop: + exec: + command: + - /usr/bin/gateway + - sleep + - --duration=40s # This flag is optional, the default is 30s + + nginx: + <...> + lifecycle: + preStop: + exec: + command: + - /bin/sleep + - "40" + ``` + +2. Ensure the `terminationGracePeriodSeconds` matches or exceeds the `sleep` value from the `preStopHook` (the default + is 30). This is to ensure Kubernetes does not terminate the pod before the `preStopHook` is complete. To do so, + update your `values.yaml` file to include the following (update the value to what is required in your environment): + + ```yaml + terminationGracePeriodSeconds: 50 + ``` + +> Note: More information on container lifecycle hooks can be found +> [here](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) and a detailed +> description of Pod termination behavior can be found in +> [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination). + ## Uninstalling the Chart To uninstall/delete the release `my-release`: diff --git a/docs/installation.md b/docs/installation.md index 6cb3c8254..079b04493 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -2,6 +2,23 @@ This guide walks you through how to install NGINX Gateway Fabric on a generic Kubernetes cluster. +- [Installation](#installation) + - [Prerequisites](#prerequisites) + - [Deploy NGINX Gateway Fabric using Helm](#deploy-nginx-gateway-fabric-using-helm) + - [Deploy NGINX Gateway Fabric from Manifests](#deploy-nginx-gateway-fabric-from-manifests) + - [Expose NGINX Gateway Fabric](#expose-nginx-gateway-fabric) + - [Create a NodePort Service](#create-a-nodeport-service) + - [Create a LoadBalancer Service](#create-a-loadbalancer-service) + - [Upgrading NGINX Gateway Fabric](#upgrading-nginx-gateway-fabric) + - [Upgrade NGINX Gateway Fabric from Manifests](#upgrade-nginx-gateway-fabric-from-manifests) + - [Upgrade NGINX Gateway Fabric using Helm](#upgrade-nginx-gateway-fabric-using-helm) + - [Configure Delayed Termination for Zero Downtime Upgrades](#configure-delayed-termination-for-zero-downtime-upgrades) + - [Configure Delayed Termination Using Manifests](#configure-delayed-termination-using-manifests) + - [Configure Delayed Termination Using Helm](#configure-delayed-termination-using-helm) + - [Uninstalling NGINX Gateway Fabric](#uninstalling-nginx-gateway-fabric) + - [Uninstall NGINX Gateway Fabric from Manifests](#uninstall-nginx-gateway-fabric-from-manifests) + - [Uninstall NGINX Gateway Fabric using Helm](#uninstall-nginx-gateway-fabric-using-helm) + ## Prerequisites - [kubectl](https://kubernetes.io/docs/tasks/tools/) @@ -168,6 +185,54 @@ Create a Service with type `LoadBalancer` using the appropriate manifest for you To upgrade NGINX Gateway Fabric when the deployment method is Helm, please follow the instructions [here](/deploy/helm-chart/README.md#upgrading-the-chart). +### Configure Delayed Termination for Zero Downtime Upgrades + +To achieve zero downtime upgrades (meaning clients will not see any interruption in traffic while a rolling upgrade is +being performed on NGF), you may need to configure delayed termination on the NGF pod, depending on your environment. + +> Note: When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed +> by either the client or the backend. This means that unless all those connections are closed by clients/backends +> before or during an upgrade, NGINX will not terminate, which means Kubernetes will kill NGINX. As a result, the +> clients will see the connections abruptly closed and thus experience downtime. + +#### Configure Delayed Termination Using Manifests + +Edit the `deploy/manifests/nginx-gateway.yaml` to include the following: + +1. Add `lifecycle` to both the nginx and the nginx-gateway container definition: + + ```yaml + <...> + lifecycle: + preStop: + exec: + command: + - /usr/bin/gateway + - sleep + - --duration=40s # This flag is optional, the default is 30s + <...> + lifecycle: + preStop: + exec: + command: + - /bin/sleep + - "40" + <...> + ``` + +2. Ensure the `terminationGracePeriodSeconds` matches or exceeds the `sleep` value from the `preStopHook` (the default + is 30). This is to ensure Kubernetes does not terminate the pod before the `preStopHook` is complete. + +> Note: More information on container lifecycle hooks can be found +> [here](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) and a detailed +> description of Pod termination behavior can be found in +> [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination). + +#### Configure Delayed Termination Using Helm + +To to configure delayed termination on the NGF pod when the deployment method is Helm, please follow the instructions +[here](/deploy/helm-chart/README.md#configure-delayed-termination-for-zero-downtime-upgrades). + ## Uninstalling NGINX Gateway Fabric ### Uninstall NGINX Gateway Fabric from Manifests From accb5a39784754ec9b9925f52f3141751ca5afda Mon Sep 17 00:00:00 2001 From: Ciara Stacke Date: Mon, 23 Oct 2023 14:56:20 +0100 Subject: [PATCH 2/5] First round review feedback --- deploy/helm-chart/README.md | 6 ++++-- docs/installation.md | 12 +++++++++--- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/deploy/helm-chart/README.md b/deploy/helm-chart/README.md index d03bbb3b4..abd39b60e 100644 --- a/deploy/helm-chart/README.md +++ b/deploy/helm-chart/README.md @@ -129,7 +129,8 @@ helm upgrade my-release . -n nginx-gateway To achieve zero downtime upgrades (meaning clients will not see any interruption in traffic while a rolling upgrade is being performed on NGF), you may need to configure delayed termination on the NGF pod, depending on your environment. -> Note: When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed +> **Note** +> When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed > by either the client or the backend. This means that unless all those connections are closed by clients/backends > before or during an upgrade, NGINX will not terminate, which means Kubernetes will kill NGINX. As a result, the > clients will see the connections abruptly closed and thus experience downtime. @@ -166,7 +167,8 @@ being performed on NGF), you may need to configure delayed termination on the NG terminationGracePeriodSeconds: 50 ``` -> Note: More information on container lifecycle hooks can be found +> **Note** +> More information on container lifecycle hooks can be found > [here](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) and a detailed > description of Pod termination behavior can be found in > [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination). diff --git a/docs/installation.md b/docs/installation.md index 079b04493..d8d70a4d8 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -190,7 +190,8 @@ To upgrade NGINX Gateway Fabric when the deployment method is Helm, please follo To achieve zero downtime upgrades (meaning clients will not see any interruption in traffic while a rolling upgrade is being performed on NGF), you may need to configure delayed termination on the NGF pod, depending on your environment. -> Note: When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed +> **Note** +> When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed > by either the client or the backend. This means that unless all those connections are closed by clients/backends > before or during an upgrade, NGINX will not terminate, which means Kubernetes will kill NGINX. As a result, the > clients will see the connections abruptly closed and thus experience downtime. @@ -199,10 +200,12 @@ being performed on NGF), you may need to configure delayed termination on the NG Edit the `deploy/manifests/nginx-gateway.yaml` to include the following: -1. Add `lifecycle` to both the nginx and the nginx-gateway container definition: +1. Add `lifecycle` prestop hooks to both the nginx and the nginx-gateway container definitions: ```yaml <...> + name: nginx-gateway + <...> lifecycle: preStop: exec: @@ -211,6 +214,8 @@ Edit the `deploy/manifests/nginx-gateway.yaml` to include the following: - sleep - --duration=40s # This flag is optional, the default is 30s <...> + name: nginx + <...> lifecycle: preStop: exec: @@ -223,7 +228,8 @@ Edit the `deploy/manifests/nginx-gateway.yaml` to include the following: 2. Ensure the `terminationGracePeriodSeconds` matches or exceeds the `sleep` value from the `preStopHook` (the default is 30). This is to ensure Kubernetes does not terminate the pod before the `preStopHook` is complete. -> Note: More information on container lifecycle hooks can be found +> **Note** +> More information on container lifecycle hooks can be found > [here](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) and a detailed > description of Pod termination behavior can be found in > [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination). From 4b0e7894be9e84d0edd4772b78531a67f01bc6e0 Mon Sep 17 00:00:00 2001 From: Ciara Stacke Date: Mon, 23 Oct 2023 15:38:00 +0100 Subject: [PATCH 3/5] Fix typo --- docs/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation.md b/docs/installation.md index d8d70a4d8..b61479a10 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -236,7 +236,7 @@ Edit the `deploy/manifests/nginx-gateway.yaml` to include the following: #### Configure Delayed Termination Using Helm -To to configure delayed termination on the NGF pod when the deployment method is Helm, please follow the instructions +To configure delayed termination on the NGF pod when the deployment method is Helm, please follow the instructions [here](/deploy/helm-chart/README.md#configure-delayed-termination-for-zero-downtime-upgrades). ## Uninstalling NGINX Gateway Fabric From db6201fa8aa56aa43cd345d232df0279ee769aa7 Mon Sep 17 00:00:00 2001 From: Ciara Stacke Date: Mon, 23 Oct 2023 17:06:51 +0100 Subject: [PATCH 4/5] More review feedback --- deploy/helm-chart/README.md | 6 +++--- docs/installation.md | 12 ++++++++---- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/deploy/helm-chart/README.md b/deploy/helm-chart/README.md index abd39b60e..cf651d78e 100644 --- a/deploy/helm-chart/README.md +++ b/deploy/helm-chart/README.md @@ -127,7 +127,7 @@ helm upgrade my-release . -n nginx-gateway ### Configure Delayed Termination for Zero Downtime Upgrades To achieve zero downtime upgrades (meaning clients will not see any interruption in traffic while a rolling upgrade is -being performed on NGF), you may need to configure delayed termination on the NGF pod, depending on your environment. +being performed on NGF), you may need to configure delayed termination on the NGF Pod, depending on your environment. > **Note** > When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed @@ -160,7 +160,7 @@ being performed on NGF), you may need to configure delayed termination on the NG ``` 2. Ensure the `terminationGracePeriodSeconds` matches or exceeds the `sleep` value from the `preStopHook` (the default - is 30). This is to ensure Kubernetes does not terminate the pod before the `preStopHook` is complete. To do so, + is 30). This is to ensure Kubernetes does not terminate the Pod before the `preStopHook` is complete. To do so, update your `values.yaml` file to include the following (update the value to what is required in your environment): ```yaml @@ -171,7 +171,7 @@ being performed on NGF), you may need to configure delayed termination on the NG > More information on container lifecycle hooks can be found > [here](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) and a detailed > description of Pod termination behavior can be found in -> [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination). +> [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/Pods/Pod-lifecycle/#Pod-termination). ## Uninstalling the Chart diff --git a/docs/installation.md b/docs/installation.md index b61479a10..9ce6a0fde 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -148,6 +148,10 @@ Create a Service with type `LoadBalancer` using the appropriate manifest for you ## Upgrading NGINX Gateway Fabric +> **Note** +> See [below](#configure-delayed-termination-for-zero-downtime-upgrades) for instructions on how to configure delayed +> termination if required for zero downtime upgrades in your environment. + ### Upgrade NGINX Gateway Fabric from Manifests 1. Upgrade the Gateway Resources @@ -188,7 +192,7 @@ To upgrade NGINX Gateway Fabric when the deployment method is Helm, please follo ### Configure Delayed Termination for Zero Downtime Upgrades To achieve zero downtime upgrades (meaning clients will not see any interruption in traffic while a rolling upgrade is -being performed on NGF), you may need to configure delayed termination on the NGF pod, depending on your environment. +being performed on NGF), you may need to configure delayed termination on the NGF Pod, depending on your environment. > **Note** > When proxying Websocket or any long-lived connections, NGINX will not terminate until that connection is closed @@ -226,17 +230,17 @@ Edit the `deploy/manifests/nginx-gateway.yaml` to include the following: ``` 2. Ensure the `terminationGracePeriodSeconds` matches or exceeds the `sleep` value from the `preStopHook` (the default - is 30). This is to ensure Kubernetes does not terminate the pod before the `preStopHook` is complete. + is 30). This is to ensure Kubernetes does not terminate the Pod before the `preStopHook` is complete. > **Note** > More information on container lifecycle hooks can be found > [here](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) and a detailed > description of Pod termination behavior can be found in -> [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination). +> [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/Pods/Pod-lifecycle/#Pod-termination). #### Configure Delayed Termination Using Helm -To configure delayed termination on the NGF pod when the deployment method is Helm, please follow the instructions +To configure delayed termination on the NGF Pod when the deployment method is Helm, please follow the instructions [here](/deploy/helm-chart/README.md#configure-delayed-termination-for-zero-downtime-upgrades). ## Uninstalling NGINX Gateway Fabric From fd0b47be8affd3003f209e3f8cc57bec0f4b29de Mon Sep 17 00:00:00 2001 From: Ciara Stacke Date: Mon, 23 Oct 2023 17:06:51 +0100 Subject: [PATCH 5/5] More review feedback --- deploy/helm-chart/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/deploy/helm-chart/README.md b/deploy/helm-chart/README.md index cf651d78e..b6324958f 100644 --- a/deploy/helm-chart/README.md +++ b/deploy/helm-chart/README.md @@ -78,6 +78,10 @@ helm install my-release . --create-namespace --wait -n nginx-gateway ## Upgrading the Chart +> **Note** +> See [below](#configure-delayed-termination-for-zero-downtime-upgrades) for instructions on how to configure delayed +> termination if required for zero downtime upgrades in your environment. + ### Upgrading the Gateway Resources Before you upgrade a release, ensure the Gateway API resources are the correct version as supported by the NGINX