[cass-operator] Unable to remove the additional labels on the seed service through CR

[tylergu]

What happened?
We found it impossible to remove the additional labels by unsetting the key-value pairs under the field spec.additionalServiceConfig.seedService.additionalLabels. We first added a label labelKey: labelValue under the additionalLabels field, and cass-operator correctly adds the label to the seed service. However, later we wanted to remove the labelKey: labelValue label from the seed service by deleting it from the CR. However, by doing so, cass-operator does not remove the additionalLabel from the seed service.

Did you expect to see something different?
We expected that the CR represents the desired state of the application, based on the declarative model of Kubernetes. So we expected that when remove the label from the spec.additionalServiceConfig.seedService.additionalLabels field, cass-operator would automatically remove this label from the seed service.

How to reproduce it (as minimally and precisely as possible):

1. Deploy cass-operator and the server-storage storage class

2. Deploy a basic CassandraDB using the example CR:
   kubectl apply -f example.yaml

   example.yaml

   apiVersion: cassandra.datastax.com/v1beta1
   kind: CassandraDatacenter
   metadata:
     name: cassandra-datacenter
   spec:
     clusterName: cluster1
     config:
       cassandra-yaml:
         authenticator: org.apache.cassandra.auth.PasswordAuthenticator
         authorizer: org.apache.cassandra.auth.CassandraAuthorizer
         role_manager: org.apache.cassandra.auth.CassandraRoleManager
       jvm-options:
         initial_heap_size: 800M
         max_heap_size: 800M
     managementApiAuth:
       insecure: {}
     serverType: cassandra
     serverVersion: 3.11.7
     size: 1
     storageConfig:
       cassandraDataVolumeClaimSpec:
         accessModes:
         - ReadWriteOnce
         resources:
           requests:
             storage: 3Gi
         storageClassName: server-storage

3. Add additional labels to seed service by applying the cr1.yaml

   cr1.yaml

   apiVersion: cassandra.datastax.com/v1beta1
   kind: CassandraDatacenter
   metadata:
     name: cassandra-datacenter
   spec:
     additionalServiceConfig:
       seedService:
         additionalLabels:
           labelKey: labelValue
     clusterName: cluster1
     config:
       cassandra-yaml:
         authenticator: org.apache.cassandra.auth.PasswordAuthenticator
         authorizer: org.apache.cassandra.auth.CassandraAuthorizer
         role_manager: org.apache.cassandra.auth.CassandraRoleManager
       jvm-options:
         initial_heap_size: 800M
         max_heap_size: 800M
     managementApiAuth:
       insecure: {}
     serverType: cassandra
     serverVersion: 3.11.7
     size: 1
     storageConfig:
       cassandraDataVolumeClaimSpec:
         accessModes:
         - ReadWriteOnce
         resources:
           requests:
             storage: 3Gi
         storageClassName: server-storage

4. Observe that the seed service now has the additional label labelKey: labelValue

5. Delete the additionalLabels by applying the example CR again

6. Observe that the seed service still has the label labelKey: labelValue

Environment

• Cass Operator version:

  docker.io/k8ssandra/cass-operator:v1.10.3

• Kubernetes version information:

  Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:11:29Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
  

• Kubernetes cluster kind:

minikube start --vm-driver=docker --cpus 4 --memory 4096 --kubernetes-version v1.21.0

Anything else we need to know?:
The root cause is inside the https://github.com/k8ssandra/cass-operator/blob/973cea89cad9d407fcac0f2ed388e98fdae3bc2a/pkg/reconciliation/reconcile_services.go#L103 where the labels in the desired service are merged into the existing service label map. The comment states that the reason of the merge is because cass-operator does not want to mess up the labels that are added by other sources.
A possible fix is to keep track of the labels that are added by the user using the spec.additionalServiceConfig.seedService.additionalLabels field, so that when users delete some labels from the field, cass-operator can delete these from the service.

[tylergu]

There is a similar but separate bug. But the wrong behavior is on the annotations instead of labels. The root cause of these two bugs are at different lines in the code.
https://github.com/k8ssandra/cass-operator/blob/973cea89cad9d407fcac0f2ed388e98fdae3bc2a/pkg/reconciliation/reconcile_services.go#L103
https://github.com/k8ssandra/cass-operator/blob/973cea89cad9d407fcac0f2ed388e98fdae3bc2a/pkg/reconciliation/reconcile_services.go#L104

[tylergu]

Issued: k8ssandra/cass-operator#330

[tylergu]

Confirmed