From f9a41771bc59812e80bb27bd4933391ae0b63263 Mon Sep 17 00:00:00 2001 From: Stephanie Baum Date: Wed, 29 Jul 2026 22:14:25 -0700 Subject: [PATCH] docs(self-managed): correct Cassandra upgrade guidance Signed-off-by: Stephanie Baum --- .../release-notes/0.6.0-to-0.6.1-upgrade.md | 72 ++++++++++++++----- 1 file changed, 55 insertions(+), 17 deletions(-) diff --git a/docs/user/release-notes/0.6.0-to-0.6.1-upgrade.md b/docs/user/release-notes/0.6.0-to-0.6.1-upgrade.md index 6540b61b74..d0f06d0717 100644 --- a/docs/user/release-notes/0.6.0-to-0.6.1-upgrade.md +++ b/docs/user/release-notes/0.6.0-to-0.6.1-upgrade.md @@ -29,6 +29,20 @@ downloaded the 0.6.1 stack, and that you run `make` from the extracted `nvcf-self-managed-stack/` directory with your environment set. + +For an existing data-bearing cluster, do not run an unscoped 0.6.1 stack +install before completing the Cassandra steps for your chosen migration method. +This includes `make install`, `helmfile sync`, and equivalent sync commands. An +unscoped sync attempts to patch the existing Cassandra StatefulSet, fails on its +immutable fields, leaves the Cassandra Helm release in a failed state, and stops +the later Helmfile stages. + +If an unscoped sync already failed, do not uninstall Cassandra or delete its +PersistentVolumeClaims. Continue with the selector-scoped Cassandra install in +your chosen migration method. That install replaces the failed release. Verify +the migration before running the full stack install. + + ```bash export HELMFILE_ENV= # for example eks-cds-qa ``` @@ -48,6 +62,14 @@ available settings can change between releases. Preserve the existing registry, storage, endpoint, credential, and secret values unless this procedure explicitly instructs you to change them. + +Do not change the Cassandra superuser password or application role passwords +during this migration. Preserve the Cassandra and OpenBao credential values +from the 0.6.0 deployment. The 0.6.1 Cassandra chart does not create the old +`cassandra-system/cassandra` Secret, so do not rely on that Secret after the +upgrade. Rotate Cassandra credentials in a separate maintenance procedure. + + Confirm that both files exist before continuing: ```bash @@ -249,12 +271,11 @@ succeed: FROM system_schema.keyspaces; ``` - The 0.6.1 migrations add the `event_ledger` keyspace. Update its replication, - and repeat this operation for any other application keyspace that does not - list both datacenters: + For each application keyspace that does not list both datacenters, update its + replication: ```sql - ALTER KEYSPACE event_ledger WITH replication = { + ALTER KEYSPACE WITH replication = { 'class': 'NetworkTopologyStrategy', '': , '': @@ -275,16 +296,17 @@ succeed: 7. Confirm the migration state is clean: ```sql - SELECT version, dirty FROM schema_migrations.api_keys_api; - SELECT version, dirty FROM schema_migrations.ess_api; - SELECT version, dirty FROM schema_migrations.event_ledger; - SELECT version, dirty FROM schema_migrations.nvcf_api; - SELECT version, dirty FROM schema_migrations.sis_api; + SELECT table_name + FROM system_schema.tables + WHERE keyspace_name = 'schema_migrations'; + + SELECT version, dirty FROM schema_migrations.; ``` - Each query must return one record with `dirty` set to `false`. This confirms - that the cluster can receive later schema migrations. Do not move clients or - remove the old datacenter while a migration is missing or dirty. + Run the second query for each table returned by the first query. Each query + must return one record with `dirty` set to `false`. This confirms that the + cluster can receive later schema migrations. Do not move clients or remove + the old datacenter while a migration is missing or dirty. 8. Move clients to the new datacenter. Point application traffic at the new datacenter using `LOCAL_QUORUM` against the new datacenter name, and confirm @@ -345,8 +367,18 @@ backup and restore, which do not depend on on-disk configuration compatibility. make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=cassandra ``` - For a multi-node cluster, delete and recreate one pod at a time and wait for - each to become ready before the next. + For a multi-node cluster, the StatefulSet controller recreates `cassandra-0` + and then automatically replaces the remaining pods in reverse ordinal order, + one at a time. Do not delete the remaining pods manually. In another + terminal, monitor the automatic rollout: + + ```bash + kubectl -n cassandra-system rollout status statefulset/cassandra \ + --timeout=30m + ``` + + If the rollout stalls, inspect the pod status, events, and Cassandra ring + before taking manual action. 3. Confirm the new pod comes up on the existing data: @@ -388,11 +420,17 @@ row count matches what you recorded before the migration, and confirm the control plane is healthy: ```bash -kubectl get pods -A | grep -vE 'Running|Completed|READY' +kubectl get pods -A --no-headers | awk \ + '$4 != "Running" && $4 != "Completed" { print } + $4 == "Running" { + split($3, ready, "/") + if (ready[1] != ready[2]) print + }' ``` -This command should print nothing. Any output is a pod that is not yet -`Running` or `Completed`. +This command should print nothing. It prints pods that are not `Running` or +`Completed`, and `Running` pods whose ready-container count does not match their +total-container count. Finally, query each table in the `schema_migrations` keyspace as described in the pre-migration check. Every record must have `dirty` set to `false` before