Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 55 additions & 17 deletions docs/user/release-notes/0.6.0-to-0.6.1-upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,20 @@ downloaded the 0.6.1 stack, and that you run `make` from the extracted
`nvcf-self-managed-stack/` directory with your environment set.
</Info>

<Warning>
For an existing data-bearing cluster, do not run an unscoped 0.6.1 stack
install before completing the Cassandra steps for your chosen migration method.
This includes `make install`, `helmfile sync`, and equivalent sync commands. An
unscoped sync attempts to patch the existing Cassandra StatefulSet, fails on its
immutable fields, leaves the Cassandra Helm release in a failed state, and stops
the later Helmfile stages.

If an unscoped sync already failed, do not uninstall Cassandra or delete its
PersistentVolumeClaims. Continue with the selector-scoped Cassandra install in
your chosen migration method. That install replaces the failed release. Verify
the migration before running the full stack install.
</Warning>

```bash
export HELMFILE_ENV=<your-environment> # for example eks-cds-qa
```
Expand All @@ -48,6 +62,14 @@ available settings can change between releases. Preserve the existing registry,
storage, endpoint, credential, and secret values unless this procedure
explicitly instructs you to change them.

<Warning>
Do not change the Cassandra superuser password or application role passwords
during this migration. Preserve the Cassandra and OpenBao credential values
from the 0.6.0 deployment. The 0.6.1 Cassandra chart does not create the old
`cassandra-system/cassandra` Secret, so do not rely on that Secret after the
upgrade. Rotate Cassandra credentials in a separate maintenance procedure.
</Warning>

Confirm that both files exist before continuing:

```bash
Expand Down Expand Up @@ -249,12 +271,11 @@ succeed:
FROM system_schema.keyspaces;
```

The 0.6.1 migrations add the `event_ledger` keyspace. Update its replication,
and repeat this operation for any other application keyspace that does not
list both datacenters:
For each application keyspace that does not list both datacenters, update its
replication:

```sql
ALTER KEYSPACE event_ledger WITH replication = {
ALTER KEYSPACE <keyspace> WITH replication = {
'class': 'NetworkTopologyStrategy',
'<old-datacenter>': <rf>,
'<new-datacenter>': <rf>
Expand All @@ -275,16 +296,17 @@ succeed:
7. Confirm the migration state is clean:

```sql
SELECT version, dirty FROM schema_migrations.api_keys_api;
SELECT version, dirty FROM schema_migrations.ess_api;
SELECT version, dirty FROM schema_migrations.event_ledger;
SELECT version, dirty FROM schema_migrations.nvcf_api;
SELECT version, dirty FROM schema_migrations.sis_api;
SELECT table_name
FROM system_schema.tables
WHERE keyspace_name = 'schema_migrations';

SELECT version, dirty FROM schema_migrations.<table-name>;
```

Each query must return one record with `dirty` set to `false`. This confirms
that the cluster can receive later schema migrations. Do not move clients or
remove the old datacenter while a migration is missing or dirty.
Run the second query for each table returned by the first query. Each query
must return one record with `dirty` set to `false`. This confirms that the
cluster can receive later schema migrations. Do not move clients or remove
the old datacenter while a migration is missing or dirty.

8. Move clients to the new datacenter. Point application traffic at the new
datacenter using `LOCAL_QUORUM` against the new datacenter name, and confirm
Expand Down Expand Up @@ -345,8 +367,18 @@ backup and restore, which do not depend on on-disk configuration compatibility.
make install HELMFILE_ENV="$HELMFILE_ENV" HELMFILE_SELECTOR=name=cassandra
```

For a multi-node cluster, delete and recreate one pod at a time and wait for
each to become ready before the next.
For a multi-node cluster, the StatefulSet controller recreates `cassandra-0`
and then automatically replaces the remaining pods in reverse ordinal order,
one at a time. Do not delete the remaining pods manually. In another
terminal, monitor the automatic rollout:

```bash
kubectl -n cassandra-system rollout status statefulset/cassandra \
--timeout=30m
```

If the rollout stalls, inspect the pod status, events, and Cassandra ring
before taking manual action.
Comment thread
sbaum1994 marked this conversation as resolved.

3. Confirm the new pod comes up on the existing data:

Expand Down Expand Up @@ -388,11 +420,17 @@ row count matches what you recorded before the migration, and confirm the
control plane is healthy:

```bash
kubectl get pods -A | grep -vE 'Running|Completed|READY'
kubectl get pods -A --no-headers | awk \
'$4 != "Running" && $4 != "Completed" { print }
$4 == "Running" {
split($3, ready, "/")
if (ready[1] != ready[2]) print
}'
```

This command should print nothing. Any output is a pod that is not yet
`Running` or `Completed`.
This command should print nothing. It prints pods that are not `Running` or
`Completed`, and `Running` pods whose ready-container count does not match their
total-container count.

Finally, query each table in the `schema_migrations` keyspace as described in
the pre-migration check. Every record must have `dirty` set to `false` before
Expand Down
Loading