catalogSync: Allow auth to external K8s server using a kubeconfig file

[jose-ledesma]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Enabling syncCatalog won't work if there are no Consul Servers deployed in the cluster. From the values.yaml documentation from the Helm Chart:

# This process assumes that a Consul agent is available on the host IP.
# This is done automatically if clients are enabled. If clients are not
# enabled then set the node selection so that it chooses a node with a
# Consul agent.
syncCatalog:

Nonetheless, the sync-catalog-deployment loads the env vars from
consul.consulK8sConsulServerEnvVars which sets CONSUL_ADDRESSES pointing to the Kubernetes Service created when you deploy Consul Servers

Reproduction Steps

Deploy Consul Clients to a Kubernetes Cluster enabling syncCatalog without deploying Consul Servers in the very same cluster

Logs

$ kubectl logs consul-agent-consul-sync-catalog-6f67c4cc8-ldgw5
2023-01-23T12:15:34.951Z [INFO]  consul-server-connection-manager: trying to connect to a Consul server
2023-01-23T12:15:34.953Z [ERROR] consul-server-connection-manager: connection error: error="failed to discover Consul server addresses: failed to resolve DNS name: consul-agent-consul-server.consul.svc: lookup consul-agent-consul-server.consul.svc on xxxxxx:53: no such host"

Expected behavior

syncCatalog is able to work on Kubernetes Clusters without the Consul Server deployd

Environment details

consul-k8s: 1.0.2
values.yaml:

global:
  domain: "xxdomainxx"
  enabled: false
  datacenter: "xxdatacenterxx"
  tls:
    enabled: true
    verify: false
    enableAutoEncrypt: true
    caCert:
      secretName: "consul-ca-cert"
      secretKey: "tls.crt"
  gossipEncryption:
    secretName: "consul-gossip-key"
    secretKey: 'key'

client:
  enabled: true
  exposeGossipPorts: true


  join:
    - 'provider=k8s kubeconfig=/consul/userconfig/kubeconfig-consul/kubeconfig label_selector="app=consul,component=server" namespace=consul'

connectInject:
  enabled: false

syncCatalog:
  enabled: true
  toConsul: false # We don't want K8S services created in Consul
  toK8S: true
  k8sPrefix: consul-

Additional Context

[david-yu]

Hi @jose-ledesma this should work but you'd need to define externalServers. Where are your servers hosted and what version are they? Take a look at our docs here: https://developer.hashicorp.com/consul/docs/k8s/deployment-configurations/servers-outside-kubernetes

[jose-ledesma]

Hi @jose-ledesma this should work but you'd need to define externalServers. Where are your servers hosted and what version are they? Take a look at our docs here: https://developer.hashicorp.com/consul/docs/k8s/deployment-configurations/servers-outside-kubernetes

Sorry, I probably was not clear enough. Consul Servers are deployed to Kubernetes, but not to the same K8s cluster. For clients we have the k8s plugin for joining, but for syncCatalog it seems there is not something similar.

Out of curiosity, what’s the reason this feature needs to connect to Servers directly and can’t use the agents?

[ishustava]

Hey @jose-ledesma

Consul Servers are deployed to Kubernetes, but not to the same K8s cluster. For clients we have the k8s plugin for joining, but for syncCatalog it seems there is not something similar.

You can continue using it but you will need to change it to a go-discover command. So in your externalServers.hosts you need to set it like exec=discover provider=k8s label_selector="app = consul-server"

Out of curiosity, what’s the reason this feature needs to connect to Servers directly and can’t use the agents?

We've found that having client agents on Kubernetes complicates the deployment and networking requirements because
clients need gossip. The removal of clients allows for a simpler deployment where you can talk directly to servers. David also mentioned some of these in #1438

[jose-ledesma]

Thanks a lot for your help! it was not clear to me how to proceed looking at the docs. For the client I'm already using the join with the provider=k8s, but to do so I'm using the extraVolumes value for mounting the secret(a full kubeconfig). I don't see a clear mechanism to make something similar for the syncCatalog. Am I missing something?

[david-yu]

I believe we probably need to provide plumbing within Helm (via the synCatalog stanza) to allow you to pull in a kubeconfig for use with go-discover on the syncCatalog pod. I don't know if we'll be able to get to this but we're happy to review PRs.