Allow users to use well-known ports for client connections to both ingress controllers

[panyuenlau]

Description

Current implementation: when users set this externalListener[x].anyCastPort value, Koperator uses this directly as the corresponding container port for the envoy controller containers and target port for the associated services.

envoy controller

When users use a well-known port (with port number < 1024), Koperator can handle this for envoy controller since the envoy container is run as root, for example:

kubectl exec envoy-external-kafka-6495bf4fd7-9n2b8 -c envoy -n kafka -- whoami                                                                                                                          
root

But this still could be problematic if the users want to use a customized envoy image but do not wish to run it as root.

istioingress controller

When users use a well-known port, Koperator creates a IstioMeshGateway object with using that well-known port directly as the targetPort for one of the service ports, for example:

    - name: tcp-all-broker
      port: 443 # anyCastPort is set to 443
      protocol: TCP
      targetPort: 443

And the resulting istioingress controller container fails to use this well-known port and therefore the kafka brokers can't be reached by external clients.

What is in this PR?

• Introduce a new optional field, ingressControllerTargetPort under ExternalListenerConfig to allow users to specify the container port that they would like to use in the ingress controller. It also extends the validation webhook implementation to prevent users from using well-known port for the istioingress controller's deployment.

  This way, users can use any anyCastPort values for client access while using either of the ingress controllers

• Add checks to the KafkaCluster validation webhook (both create and update) to prevent invalid user configurations on external listeners

Type of Change

• New Feature

Checklist

• I have read the contributing guidelines
• Existing issues have been referenced (where applicable)
• I have verified this change is not present in other open pull requests
• Functionality is documented
• All code style checks pass
• New code contribution is covered by automated tests
• All new and existing tests pass

[panyuenlau]

FYI - looks like named targetPort can't be used here, becasue istio-operator complains about not having an exact value for this field when it reconciles the IstioMeshGateway resource. But I could me missing something

[pregnor]

That's saddening.

[Kuvesz]

LGTM really

[pregnor]

LGTM, take the other PR's comment into account here

[panyuenlau]

The changes in this validation function are not really required for the original purpose of this PR, it is more for
something that we haven't been doing to prevent user error - https://github.com/banzaicloud/koperator/pull/997/files#diff-7161be6912ad21f597382d9e2dcef3eb028c868b8206ac2d1fafa9f2cde317f5R222-R249

[pregnor]

Other than my question on one of the cases, LGTM.

[Kuvesz]

Looks good to me, like the attention to small details like the healthcheck port. Thanks!