Add Redhat Openshift support

[hi-im-aren]

Q	A
Bug fix?	no
New feature?	yes
API breaks?	no
Deprecations?	no
Issues	#795
License	Apache 2.0

What's in this PR?

Add openshift support by:

• adding finalizer rbac
• changing default webhook port from protected 443 to 9443

Checklist

• Implementation tested
• User guide and development docs updated (if needed)
  • Add openshift installation docs koperator-docs#89

ToDo

• Kafka broker image is updated (We will customize the broker images in a later step because it seems to work fine with the current ones and the modification is a bigger effort because of the init containers.)
  • Added RHOS requirements to the docker image docker-kafka#29
• Envoy ingress external listener fix API extension: Added KafkaCluster.spec.envoyConfig.PodSecurityContext to the API #956

[dobrerazvan]

Yeah, those seem to be the changes required for rbacs to make the operator work (just tried them today). But there are some changes required to the kafka build to not use the root account otherwise the broker attached pvcs cannot be used (permission denied).

[panyuenlau]

lgtm - although I don't have access to OpenShift so can't really play with it myself

[hi-im-aren]

Yeah, those seem to be the changes required for rbacs to make the operator work (just tried them today). But there are some changes required to the kafka build to not use the root account otherwise the broker attached pvcs cannot be used (permission denied).

Yeah I made those changes as well but forgot to create a PR for it in our kafka docker repo, I will add it in a sec.

[pregnor]

Thanks for the reviews, I will take over this PR and fix the commented things.

[tiswanso]

fyi, we're also going to need to set runAsUser and runAsGroup in the container security context to match the nonroot ID used in the docker-kafka build.

without that RHOS picks a random UID for this mapping and it screws up any filesystem access (owner doesn't map).

[pregnor]

I force-with-lease-pushed the branch to clean up redundant commits I manually rebased, I will fix the comments in subsequent pushes, stay tuned.

[pregnor]

fyi, we're also going to need to set runAsUser and runAsGroup in the container security context to match the nonroot ID used in the docker-kafka build.

without that RHOS picks a random UID for this mapping and it screws up any filesystem access (owner doesn't map).

Fixed in c32648a.

Also I will highlight this in the docs, because it is KafkaCluster CR config details the user needs to pay attention to.

I will retest Koperator using banzaicloud/docker-kafka#29 and #912 on K8s, RHOS 4.10 and 4.11 and if everything seems to work fine I will rerequest the reviews so we can get these merged ASAP.

[tiswanso]

for the koperator Dockerfile, I think the distroless base image should change to gcr.io/distroless/static-debian11:nonroot

and possibly need to ensure the deployment of koperator sets the securitycontext:

  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true

[pregnor]

Converted to draft, because there is still stuff to figure out unfortunately, working on it.

[tiswanso]

I think this needs to change to webhook-server as well for the test instructions to work:

koperator/config/base/webhook/service.yaml

Line 10 in a62b459

targetPort: 443

[pregnor]

I think this needs to change to webhook-server as well for the test instructions to work:

koperator/config/base/webhook/service.yaml

Line 10 in a62b459

targetPort: 443

I mean this will always be outdated if the user overconfigures it, but pushed an update in https://github.com/banzaicloud/koperator/compare/5410bdb1a28dbfa881301952ac7a350dad91f437..8c7f6536849a049fc95e8e92b93746776945b3f7.

(Also removed the commit for the user/group specification, because this way it works on RHOS 4.10 and 4.11 and it doesn't with those and we need to figure out the correct combination of configuration for those with the init containers as well which takes more time but we also want to release the basic RHOS support to move forward.)

[pregnor]

For the external listeners with envoy ingress I added the podSecurityContext propagation which is needed to run fine.
With this we should have covered all basic use-cases.

This last one will build on top of an API change that will be merged in #956 , I will update this PR once that is merged and the new API is tagged. That is what breaks the CI now.

[pregnor]

I added the new Koperator API version containing the envoy podSecurityContext, this is the final form of this PR.