[Release-1.22] Secrets-encryption rotation

[dereknola]

Backport of #4372
Documentation can be found at https://rancher.com/docs/k3s/latest/en/security/secrets_encryption/
Signed-off-by: Derek Nola [email protected]

Proposed Changes

Introduces new feature: secrets-encryption keys rotation and enable/disable of encryption. The is all controlled via a new sub command k3s secrets-encrypt.

NAME:
   k3s secrets-encrypt - Control secrets encryption and keys rotation

USAGE:
   k3s secrets-encrypt command [command options] [arguments...]

COMMANDS:
   status     Print current status of secrets encryption
   enable    Enable secrets encryption
   disable   Disable secrets encryption
   prepare   Prepare for encryption keys rotation
   rotate     Rotate secrets encryption keys
   reencrypt  Reencrypt all data with new encryption key

Types of Changes

New Feature
New Integration Test (covers single node)

Verification

Formal documentation to follow:
You can run k3s secrets-encrypt status at any time to see the current stage/state of encryption
For a single node cluster:

• Start server with k3s server --secrets-encryption
• Create a secret to check

kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
kubectl describe secret secret1 -n default

• Call k3s secrets-encrypt prepare
• Call k3s secrets-encrypt status, note, the addition of a new key
• Kill k3s server and restart with same arguments
• Call k3s secrets-encrypt rotate
• Call k3s secrets-encrypt status, note the keys swap
• Kill k3s server, restart with same arguments
• Call k3s secrets-encrypt reencrypt
• Check that secret is still encrypted

kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
kubectl describe secret secret1 -n default

For external DB:

• Start up a external DB (I used mySQL for this)

• Start up 3 K3s servers joined to mySQL
  k3s server --secrets-encryption --datastore-endpoint "mysql://root:mysql@tcp(192.168.1.200:3306)/k3s"

• Select ONE server (S1 going forward) to perform the rotate on (doing any stage on any server is supported)

• Run k3s secrets-encrypt prepare on S1

• Restart S1 with same arguments

• Once S1 is back up, restart the other two servers

• Call k3s secrets-encrypt status on S1 and S2, info should be the same on both

• Run k3s secrets-encrypt rotate on S1

• Restart S1

• Once S1 is back up, restart the other two servers

• Call k3s secrets-encrypt status on S1 and S2, info should be the same on both

• Run k3s secrets-encrypt reencrypt on S1

• Restart S1

• Once S1 is back up, restart the other two servers

• Call k3s secrets-encrypt status on S1 and S2, info should be the same on both

Linked Issues

#4602

User-Facing Change

A CLI subcommand to control secrets encryption. Supports key rotation and enabled/disable of secrets encryption.

Further Comments

• Documentation to follow in next PR.
• This feature works on all 3 DB types (single node, HA with embedded etcd, HA with external DB)