Get source bootstrap servers from k8s configmap in MM2?

[iftachbyaf]

Hello,

I'm working on setting up our MM2 strimzi deployment as dynamically as possible. I want the source bootstrap servers to read from a k8s configmap that we have which contains the bootstrap servers.

Right now we have the bootstrap servers as a string passed to the chart;

spec:
  clusters:
    - alias: {{ get .Values.clusters.source "alias" }}
      tls:
        trustedCertificates: []
      bootstrapServers: {{ get .Values.clusters.source "bootstrapServers" }}
    - alias: {{ get .Values.clusters.target "alias" }}
      tls:
        trustedCertificates: []
      bootstrapServers: {{ get .Values.clusters.target "bootstrapServers" }}

I saw that it is possible to setup externalConfiguration at the spec schema level, which will load a configmap, but I'm not sure how to set this up and then reference this as the value passed to bootstrapServers exactly? Is there an example somewhere on how to do this?

For reference for normal kafka connect I used

spec:
  config:
    config.providers: secrets,configmaps
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    config.providers.configmaps.class: io.strimzi.kafka.KubernetesConfigMapConfigProvider 

and then loaded it from configmap like this
bootstrapServers: "${configmaps:strimzi/{{ .Values.cluster }}:brokers_tls}"

This syntax doesnt seem to match how I'm supposed to do it on mm2?

Thanks!

[scholzj]

I would not expect it to work to specify a config provider-based value directly in the custom resource. But if it works for you in Connect, it should work the same way for you in MM2.

[iftachbyaf]

I tried setting it up on the target;

    - alias: target
      bootstrapServers: b-2...9094
      config:
      config.providers: env,secrets,configmaps
      config.providers.configmaps.class: io.strimzi.kafka.KubernetesConfigMapConfigProvider
      config.providers.env.class: org.apache.kafka.common.config.provider.EnvVarConfigProvider
      config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider

i am just able to reference the configmap/env var name and the key name. but not the value itself... so im wondering if its even possible

[scholzj]

I don't understand what you mean by this i am just able to reference the configmap/env var name and the key name. but not the value itself.

[iftachbyaf]

So lets say i load this configmap;

  externalConfiguration:
    env:
    - name: msk
      valueFrom:
        configMapKeyRef:
          name: msk
          key: brokers_tls

This creates the env var on the mm2 pods. so far so good.
Now I want to reference the value in brokers_tls (its the bootstrap-servers list), from the env var in some other property in the chart. I tried several methods

  clusters:
    - alias: source
      bootstrapServers: ${env:msk}

This returns value bootstrapServers: ${env:msk}, so it basically passes it as is as a string.
I also tried bootstrapServers: {{ index .Values.source_clusters 0 "key_name" }} or bootstrapServers: {{ index .Values.source_clusters 0 "key_name" }}

these return msk / brokers_tls respectively, as strings. So it does access the env config but I'm unable to retrieve the actual value of the env var... Tried bootstrapServers: {{ index .Values.source_clusters 0 "key" }} / bootstrapServers: {{ index .Values.source_clusters 0 "value" }} but these both return empty strings...

Hope that makes sense.

I also tried setting up retrieval directly from configmap using config.providers.configmaps.class: io.strimzi.kafka.KubernetesConfigMapConfigProvider and then accessing the configmap using "${configmaps:namespace/msk:brokers_tls}", but that again just loads it as string bootstrapServers: "${configmaps:namespace/msk:brokers_tls}"

[scholzj]

Well, I guess that suggests you cannot use it there - which as I said would be the expected result for me.

[iftachbyaf]

I see... I will try to do some more attempts to see if I can manage somehow. Thanks Jakub you've been very helpful