kube service automatically registering in k8s deployment with Teleport cluster name

[stevenGravy]

Description

What happened:

When Teleport proxy_service is deployed to K8s it always adds a kube_service entry with the same name as the Teleport cluster. That causes a problem if you dont' want a k8s service available or you want to specify a specific name with labels. If kube_listen_addr is set on the proxy then a kube_service will automatically show up. That same does not occur on a linux machine only deployment.

Ex from a helm deployment:

proxy_service:
  enabled: true
  public_addr: teleport.example.com:3080
  web_listen_addr: 0.0.0.0:3080
  listen_addr:  0.0.0.0:3023
  tunnel_listen_addr:  0.0.0.0:3024
  ssh_public_addr: teleport.example.com:3023
  tunnel_public_addr: teleport.example.com:3024


  #kube_service declaration
  kube_listen_addr: 0.0.0.0:3026
kubernetes_service:
  enabled: true
  kube_cluster_name: test3
  labels:
    examplelabel: examplevalue
  listen_addr: 0.0.0.0:3027

Deployed in EKS this results in two kubernetes_service entries. A test3 and a teleport.example.com k8s service show.

What you expected to happen:

If only the kubernetes_service is specified with no config file it should only have a single kubernetes_service with the name, labels specified. If only the kube_listen_addr is specified then no kubenertes_service should be specified.

Reproduction Steps

As minimally and precisely as possible, describe step-by-step how to reproduce the problem.
1.In the teleport helm chart comment out the proxy k8s section in values.yaml (lines 170)
2.Specify a kubernetes_service

    kubernetes_service:
      enabled: true
      kube_cluster_name: test3
      labels:
        examplelabel: examplevalue
      listen_addr: 0.0.0.0:3027

3.Deploy
4.Check the number of kubernetes_service from tctl get kube_service

Server Details

• Teleport version (run teleport version): 6.0.2
• Server OS (e.g. from /etc/os-release): AWS Linux
• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS EKS

Client Details

• Tsh version (tsh version): 6.0.2
• Computer OS (e.g. Linux, macOS, Windows): Linux
• Installed via (e.g. apt, yum, brew, website download): helm

[awly]

This behavior is intentional, to preserve backwards-compatibility with pre-5.0 deployments that used proxy_service to talk directly to k8s API, without kubernetes_service.
The only safe way I can see to disable it is by adding a new flag under proxy_service.

While it is annoying, does this actually cause issues in practice @stevenGravy ?

[awly]

I think we can change the behavior in proxy_service based on how the k8s port is configured:

• if kubernetes section under proxy_service is used, register a k8s cluster using the name of the teleport cluster
• if kube_listen_addr/kube_public_addr are used instead, don't register the k8s cluster

[nklaassen]

@awly @russjones this change may still be backwards-incompatible if any users or automation rely on the teleport-named cluster existing even with the new proxy_service+kubernetes_service configuration style. Should this issue be pushed to 7.0?

[awly]

As discussed on the call, this can be pushed to 7.0

[nklaassen]

Fixed in #6786

[stefansedich]

@nklaassen @awly is there any way today before 7.0 to set labels on the proxy kube service? today we set labels on the kubernetes_service which has the same cluster name but I see no labels in the UI. I assume we need the kube_listen_addr to be set for kubernetes access to work? when I turn that off labels show but my client cannot connect to kubectl as it is trying to hit 3026.

Just wondering how I can get RBAC working for k8s with 6.2.x?

[awly]

@stefansedich

is there any way today before 7.0 to set labels on the proxy kube service? today we set labels on the kubernetes_service which has the same cluster name but I see no labels in the UI.

Yes, labels should be set at kubernetes_service. Once it's registered, what do you get from tctl get kube_service?

I assume we need the kube_listen_addr to be set for kubernetes access to work? when I turn that off labels show but my client cannot connect to kubectl as it is trying to hit 3026.

Yes, you need to enable the k8s port on the proxy using kube_listen_addr.
Can you show your full teleport.yaml (from all instances)?

[stefansedich]

@stefansedich

is there any way today before 7.0 to set labels on the proxy kube service? today we set labels on the kubernetes_service which has the same cluster name but I see no labels in the UI.

Yes, labels should be set at kubernetes_service. Once it's registered, what do you get from tctl get kube_service?

I assume we need the kube_listen_addr to be set for kubernetes access to work? when I turn that off labels show but my client cannot connect to kubectl as it is trying to hit 3026.

Yes, you need to enable the k8s port on the proxy using kube_listen_addr.
Can you show your full teleport.yaml (from all instances)?

@awly I get multiple entries looking at tcl get kube_service, one with and one without labels:

kind: kube_service
metadata:
  expires: "2021-06-09T20:38:02Z"
  name: 398b134c-77f7-4288-802b-444bfe80e055
spec:
  addr: 127.0.0.1:3027
  hostname: ""
  kube_clusters:
  - name: teleport.test.com
    static_labels:
      env: training
  rotation:
    current_id: ""
    last_rotated: "0001-01-01T00:00:00Z"
    schedule:
      standby: "0001-01-01T00:00:00Z"
      update_clients: "0001-01-01T00:00:00Z"
      update_servers: "0001-01-01T00:00:00Z"
    started: "0001-01-01T00:00:00Z"
  version: 6.2.1
version: v2
---
kind: kube_service
metadata:
  expires: "2021-06-09T20:35:22Z"
  name: 398b134c-77f7-4288-802b-444bfe80e055-proxy_service
spec:
  addr: 127.0.0.1:3026
  hostname: ""
  kube_clusters:
  - name: teleport.test.com
  rotation:
    current_id: ""
    last_rotated: "0001-01-01T00:00:00Z"
    schedule:
      standby: "0001-01-01T00:00:00Z"
      update_clients: "0001-01-01T00:00:00Z"
      update_servers: "0001-01-01T00:00:00Z"
    started: "0001-01-01T00:00:00Z"
  version: 6.2.1
version: v2

In the UI I get a single service but it has no labels.

[awly]

@stefansedich I see, there are two entries registered - one by kubernetes_service (with labels) and one by proxy_service (due to the bug fixed in this issue).

Can you try changing the kube_cluster_name in your kubernetes_service to something different?
You should see 2 clusters on the UI: the one you see now without labels and the new one with labels.

If this is correct, one way to restrict access to the cluster registered by proxy_service is:

• separate proxy_service and kubernetes_service into separate processes/pods
• don't grant k8s API permissions to the proxy_service

This should prevent the proxy from registering its own kubernetes cluster without labels.