Deployment of new version of application with OpenShift extension fails with LabelSelectorRequirement(nil)}: field is immutable

[yrodiere]

Describe the bug

Hey @iocanel I noticed something odd lately when deploying with the OpenShift extension... Opening as a bug since I'm not quite sure, but feel free to requalify.

It seems that the deployment fails because a selector includes the app version (the git SHA in this case), and selectors are immutable:

Error: [ERROR] 	[error]: Build step io.quarkus.kubernetes.deployment.KubernetesDeployer#deploy threw an exception: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: ***/apis/apps/v1/namespaces/prod-quarkus-github-lottery/deployments/quarkus-github-lottery. Message: Deployment.apps "quarkus-github-lottery" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"quarkus-github-lottery", "app.kubernetes.io/version":"7df7e9a331fb67fdfe434ec56b49a3f52a944674"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.selector, message=Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"quarkus-github-lottery", "app.kubernetes.io/version":"7df7e9a331fb67fdfe434ec56b49a3f52a944674"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable, reason=FieldValueInvalid, additionalProperties={})], group=apps, kind=Deployment, name=quarkus-github-lottery, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Deployment.apps "quarkus-github-lottery" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"quarkus-github-lottery", "app.kubernetes.io/version":"7df7e9a331fb67fdfe434ec56b49a3f52a944674"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

I don't think I personally changed anything in this area in my app, so... could it be related to the switch from DeploymentConfig to Deployment?

More importantly, should this be addressed/documented in the extension, since it's apparently the default behavior, and it used to work fine?

See https://github.com/quarkusio/quarkus-github-lottery/actions/runs/8142539385/job/22252314385

Expected behavior

Deployment should erase the previous version of the application -- at least it used to.

Actual behavior

Deployment fails:

Error: [ERROR] 	[error]: Build step io.quarkus.kubernetes.deployment.KubernetesDeployer#deploy threw an exception: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: ***/apis/apps/v1/namespaces/prod-quarkus-github-lottery/deployments/quarkus-github-lottery. Message: Deployment.apps "quarkus-github-lottery" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"quarkus-github-lottery", "app.kubernetes.io/version":"7df7e9a331fb67fdfe434ec56b49a3f52a944674"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.selector, message=Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"quarkus-github-lottery", "app.kubernetes.io/version":"7df7e9a331fb67fdfe434ec56b49a3f52a944674"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable, reason=FieldValueInvalid, additionalProperties={})], group=apps, kind=Deployment, name=quarkus-github-lottery, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Deployment.apps "quarkus-github-lottery" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"quarkus-github-lottery", "app.kubernetes.io/version":"7df7e9a331fb67fdfe434ec56b49a3f52a944674"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

How to Reproduce?

Deploy a new version of an application without removing the previous version one.

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

3.8.1

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift)

[iocanel]

This might be related to a recent change that removes some selectors that are not needed.
You could possibly work around this by deleting the previous resource.

Meanwhile, I'll try to have a look and see if there is something else we can / need to do.

[yrodiere]

You could possibly work around this by deleting the previous resource.

Right. But just to be clear, I'd need to do this on every deployment, because the problem occurs again on the next deployment. It's not just some old, stale resource definition causing trouble.

Meanwhile, I'll try to have a look and see if there is something else we can / need to do.

Thanks.

[iocanel]

I managed to reproduce the issue by changing the version of the application which ends up being used in the LabelSelector of Deployments.

This behavior is controlled by: https://quarkus.io/guides/all-config#quarkus-kubernetes_quarkus-openshift-add-version-to-label-selectors (so you can deal with the issue by setting it to false).

Apparently, DeploymentConfig treated selectors a bit differently (AFAIR, it used its own Selector implementation) and that's why you haven't encountered the issue before.

One option is to improve the error message, the other is to detect such conflicts and delete. I am a bit skeptic about the potential side effects of the later.

[yrodiere]

Thanks, I'll try setting quarkus.openshift.add-version-to-label-selectors to false.

Another option would be to have quarkus.openshift.add-version-to-label-selectors default to false, since re-deployment cannot work with that being set to true?