Custom cluster domain support for other values than cluster.local

[wrdls]

Report

Keda doesn't seem to properly support custom cluster domains other than cluster.local (e.g. mycompany.local`).

The chart has a configuration setting for this, but a Github search learns me that this is hardcoded in the code base in a few places, e.g.:

• keda/cmd/adapter/main.go

  Line 140 in ee81112

  return fmt.Sprintf("keda-operator.%s.svc.cluster.local:9666", kedautil.GetPodNamespace())

• keda/pkg/certificates/certificate_manager.go

  Line 106 in ee81112

  fmt.Sprintf("%s.%s.svc.cluster.local", service, namespace),

Using CertManager for certs seems to be a workaround for this, but hardcoding cluster.local is simply not valid.

Additionally, validate if the cluster domain is actually needed in every place.
Kubernetes does not depend on FQDNs to resolve services and pods to IP addresses as it will inject custom search domains in /etc/resolv.conf : https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#namespaces-of-services

Expected Behavior

Setting the clusterDomain in the chart leads to correctly auto generated certs.

Actual Behavior

Certificate which is only valid for cluster.local.

Steps to Reproduce the Problem

1. Deploy a Kubernetes cluster with a custom cluster domain.
2. Deploy Keda through the Helm chart with default settings except the clusterDomain.
3. Observe misconfigured certificates being generated.

Logs from KEDA operator

keda-operator-metrics-apiserver-5c57dfdbc4-qhj6w keda-operator-metrics-apiserver 2023-08-03T17:02:34.665892547+02:00 W0803 15:02:34.665790       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "keda-operator.keda.svc.mycompany.local:9666", ServerName: "keda-operator.keda.svc.mycompany.local:9666", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for keda-operator, keda-operator, keda-operator.keda, keda-operator.keda.svc, keda-operator.keda.svc.local, keda-operator.keda.svc.cluster.local, keda-admission-webhooks, keda-admission-webhooks.keda, keda-admission-webhooks.keda.svc, keda-admission-webhooks.keda.svc.local, keda-admission-webhooks.keda.svc.cluster.local, keda-operator-metrics-apiserver, keda-operator-metrics-apiserver.keda, keda-operator-metrics-apiserver.keda.svc, keda-operator-metrics-apiserver.keda.svc.local, keda-operator-metrics-apiserver.keda.svc.cluster.local, not keda-operator.keda.svc.mycompany.local"

KEDA Version

2.11.2

Kubernetes Version

1.25

Platform

Google Cloud

Scaler Details

N/A

Anything else?

No response

[tomkerkhove]

Yeah, let's fix this - Are you willing to contribute this?

[vitordeap]

Up! With the same issue.. It seems an overkill to deploy the certificate manager only to allow this.. It seems that there is a script inside the operator that creates a certificate (when the certificate manager is disabled) that is simply not considering the clusterDomain value. I think this is the file, it seems that the function getDNSNames is hardcoding the domains, but I'm no expert in go :(

[Shrooblord]

Beautiful! I ran into this problem when operating KEDA in a cluster with a custom domain. I had to reinstall the Helm chart using the clusterDomain value, and delete the keda/kedaorg-certs secret to force a regeneration of the certs. (I'd get "cert invalid for this domain" type of errors in keda-operator-metrics-apiserver otherwise, even after a fresh uninstall and reinstall -- secret was left behind from prior installation).

All works now! Thanks for pointing me in the right direction :)