Skip to content

Ensure that users do not enable auto-upgrades in K8s guides #1185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 1, 2025
Merged

Ensure that users do not enable auto-upgrades in K8s guides #1185

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a68d83a
Ensure that users do not enable auto-upgrades in K8s guides
JakeSCahill Jun 30, 2025
aae036f
Apply suggestions from code review
JakeSCahill Jun 30, 2025
d83628e
Update modules/deploy/pages/deployment-option/self-hosted/kubernetes/…
JakeSCahill Jun 30, 2025
1271781
Update modules/deploy/pages/deployment-option/self-hosted/kubernetes/…
JakeSCahill Jun 30, 2025
3e5de54
Update requirements.adoc
JakeSCahill Jul 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions modules/deploy/pages/deployment-option/self-hosted/kubernetes/aks-guide.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,16 @@ Deploy a secure Redpanda cluster and Redpanda Console in Azure Kubernetes Servic

== Prerequisites

Before you begin, you must have the following:

* You must satisfy the prerequisites listed in the https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-cli#prerequisites[AKS quickstart^]
* Satisfy the prerequisites listed in the https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-cli#prerequisites[AKS quickstart^]
to get access to the Azure CLI.
* https://kubernetes.io/docs/tasks/tools/[`kubectl`^]. Minimum required Kubernetes version: {supported-kubernetes-version}.
* Install https://kubernetes.io/docs/tasks/tools/[`kubectl`^]. Minimum required Kubernetes version: {supported-kubernetes-version}.
+
[,bash]
----
kubectl version --short --client
----

* https://helm.sh/docs/intro/install/[Helm^]. Minimum required Helm version: {supported-helm-version}
* Install https://helm.sh/docs/intro/install/[Helm^]. Minimum required Helm version: {supported-helm-version}
+
[,bash]
----
Expand All @@ -38,8 +36,6 @@ In this step, you create an AKS cluster with three nodes on https://learn.micros
- 2 cores per worker node, which is a requirement for production.
- Local NVMe disks, which is recommended for best performance.

NOTE: The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see xref:deploy:deployment-option/self-hosted/kubernetes/k-requirements.adoc#number-of-workers[Number of workers].

. Create a resource group for Redpanda:
+
[,bash]
Expand All @@ -56,10 +52,15 @@ az aks create -g redpandaResourceGroup -n <cluster-name> \
--generate-ssh-keys \
--enable-node-public-ip \
--node-vm-size Standard_L8s_v3 \
--disable-file-driver
--disable-file-driver \
--node-os-upgrade-channel None <1>
----
+
TIP: For all available options, see the https://learn.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-create[AKS documentation^].
<1> Set the https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-node-os-image[OS upgrade channel^] to `None` to prevent AKS from automatically rebooting or upgrading nodes.
+
For more details, see the xref:deploy:deployment-option/self-hosted/kubernetes/k-requirements.adoc#node-update[requirements and recommendations] for deploying Redpanda in Kubernetes.

For all available options, see the https://learn.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-create[AKS documentation^].

include::deploy:partial$kubernetes/guides/create-storageclass.adoc[leveloffset=+2]

Expand Down
18 changes: 11 additions & 7 deletions modules/deploy/pages/deployment-option/self-hosted/kubernetes/eks-guide.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Then, use `rpk` both as an internal client and an external client to interact wi

== Prerequisites

Before you begin, you must have the following prerequisites.
Before you begin, you must meet the following prerequisites.

=== IAM user

Expand Down Expand Up @@ -252,8 +252,6 @@ In this step, you create an EKS cluster with three nodes on https://aws.amazon.c
- 2 cores per worker node, which is a requirement for production.
- Local NVMe disks, which is recommended for best performance.

NOTE: The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see xref:deploy:deployment-option/self-hosted/kubernetes/k-requirements.adoc#number-of-workers[Number of workers].

. Create an EKS cluster and give it a unique name. If your account is configured with OIDC, add the `--with-oidc` flag to the `create cluster` command.
+
[,bash,lines=4-6]
Expand All @@ -266,16 +264,22 @@ eksctl create cluster \
--external-dns-access
----
+
[TIP]
[IMPORTANT]
====
To see all options:
Do not enable https://docs.aws.amazon.com/eks/latest/userguide/automode.html[auto mode^] (`--enable-auto-mode`) on Amazon EKS clusters running Redpanda.

Auto mode can trigger automatic reboots or node upgrades that disrupt Redpanda brokers, risking data loss or cluster instability. Redpanda requires manual control over node lifecycle events.

For more details, see the xref:deploy:deployment-option/self-hosted/kubernetes/k-requirements.adoc#node-updates[requirements and recommendations] for deploying Redpanda in Kubernetes.
====
+
To see all options:
+
```bash
eksctl create cluster --help
```

+
Or, for help creating an EKS cluster, see the https://eksctl.io/usage/creating-and-managing-clusters/[Creating and managing clusters^] in the `eksctl` documentation.
====

. Make sure that your local `kubeconfig` file points to your EKS cluster:
+
Expand Down
18 changes: 10 additions & 8 deletions modules/deploy/pages/deployment-option/self-hosted/kubernetes/gke-guide.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,15 @@ Deploy a secure Redpanda cluster and Redpanda Console in Google Kubernetes Engin

== Prerequisites

Before you begin, you must have the following:

* Complete the 'Before you begin' steps and the 'Launch Cloud Shell' steps of the https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#before-you-begin[GKE quickstart^]. Cloud Shell comes preinstalled with the Google Cloud CLI, the `kubectl` command-line tool, and the Helm package manager.
* https://kubernetes.io/docs/tasks/tools/[`kubectl`^]. Minimum required Kubernetes version: {supported-kubernetes-version}.
* Ensure https://kubernetes.io/docs/tasks/tools/[`kubectl`^] is installed. Minimum required Kubernetes version: {supported-kubernetes-version}.
+
[,bash]
----
kubectl version --short --client
----

* https://helm.sh/docs/intro/install/[Helm^]. Minimum required Helm version: {supported-helm-version}
* Ensure https://helm.sh/docs/intro/install/[Helm^] is installed. Minimum required Helm version: {supported-helm-version}
+
[,bash]
----
Expand All @@ -37,8 +35,6 @@ In this step, you create a GKE cluster with three nodes on https://cloud.google.
- 2 cores per worker node, which is a requirement for production.
- Local NVMe disks, which is recommended for best performance.

NOTE: The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see xref:deploy:deployment-option/self-hosted/kubernetes/k-requirements.adoc#number-of-workers[Number of workers].

Create a GKE cluster. Replace the `<region>` placeholder with your own region.

[,bash]
Expand All @@ -50,12 +46,18 @@ gcloud container clusters create <cluster-name> \
--region=<region>
----

[TIP]
[IMPORTANT]
====
Do not enable https://docs.aws.amazon.com/eks/latest/userguide/automode.html[node auto-upgrades^] (`--enable-autoupgrade`) on Google GKE clusters running Redpanda.

Node auto-upgrades can trigger automatic reboots or node upgrades that disrupt Redpanda brokers, risking data loss or cluster instability. Redpanda requires manual control over node lifecycle events.

For more details, see the xref:deploy:deployment-option/self-hosted/kubernetes/k-requirements.adoc#node-updates[requirements and recommendations] for deploying Redpanda in Kubernetes.
====

To see all options that you can specify when creating a cluster, see the https://cloud.google.com/sdk/gcloud/reference/container/clusters/create[Cloud SDK reference^].

Or, for help creating a GKE cluster, see the https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#create_cluster[GKE documentation^].
====

include::deploy:partial$kubernetes/guides/create-storageclass.adoc[leveloffset=+2]

Expand Down
16 changes: 4 additions & 12 deletions modules/deploy/partials/requirements.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,25 +52,17 @@ ifndef::env-kubernetes[]
endif::[]

[[node-updates]]
== Node maintenance and operating system upgrades
== Prevent automatic node upgrades in managed Kubernetes clusters

Ensure that node and operating system (OS) upgrades are manually managed when running Redpanda in production. Manual control avoids unplanned reboots or replacements that disrupt Redpanda brokers, causing service downtime, data loss, or quorum instability.

=== Limitations of automatic updates
Common issues with automatic node upgrades include:

Redpanda is stateful. Redpanda brokers manage partition data and leadership, making them sensitive to disruptions. Proper handling during maintenance is required to:

- Avoid data loss, especially for nodes with ephemeral or local storage.
- Ensure smooth leadership transitions by decommissioning brokers before removing a node.
- Minimize service downtime by upgrading nodes one at a time during planned maintenance windows.

However, automatic update mechanisms provided by cloud platforms may not meet Redpanda's stateful requirements. Common issues include:

- Hard timeouts for graceful shutdowns that may not allow Redpanda brokers enough time to complete decommissioning or leadership transitions.
- Hard timeouts for graceful shutdowns that do not allow Redpanda brokers enough time to complete decommissioning or leadership transitions.
- Replacements or reboots without ensuring data has been safely migrated or replicated, risking data loss.
- Parallel upgrades across multiple nodes, which can disrupt quorum or reduce cluster availability.

*Recommendations*:
*Requirements*:
Copy link
Contributor

@micheleRP micheleRP Jul 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not important, but I think the 3 links below look a little odd with just "Azure documentation." Suggest: "See the Azure documentation." for each link


- Disable automatic node maintenance or upgrades.
ifdef::env-kubernetes[]
Expand Down