Standalone Terminal / Terminal Locking For K8s on Different Teleport Clusters

[deusxanima]

What

Ability to have each terminal tab/window be treated as a standalone environment from a kubernetes context perspective. Some customers require the ability to work in multiple, separate teleport clusters, each with their own kubernetes cluster, which is currently not possible as each new tsh login overrides the previous kube context even if eval $(tsh env) is used.

Why

Customer ask. Having the ability to treat each terminal tab/window as a standalone environment for kube contexts would also help prevent accidental context switching as devs/users could pin kube contexts to a single terminal tab/window without having to log in/out between contexts each time they switched clusters.

Workaround

n/a

[awly]

One way to handle this with tsh env and without additional plumbing by users:

• change tsh kube login to write a standalone kubeconfig file under ~/.tsh/keys/${PROXY}/${USER}-kube/${TELE_CLUSTER}/${KUBE_CLUSTER}-kubeconfig (right next to the k8s cert)
  • this is in addition to modifying ~/.kube/config
  • existing $KUBECONFIG should be handled carefully - if it points at a file under ~/.tsh/keys/..., we shouldn't treat it as the top-level ~/.kube/config
• tsh env should output export KUBECONFIG=~/.tsh/... (same path as above)
• that way, KUBECONFIG is fronzen per-terminal and points to a cluster-specific file

Workaround until we do that:

$ cat tsh_kube_login.sh
#!/bin/bash
set -euo pipefail
export KUBECONFIG=~/.kube/teleport-${1}
tsh kube login ${1} 1>&2
echo export KUBECONFIG=$KUBECONFIG

$ eval $(./tsh_kube_login.sh prod)

$ kubectl ... # runs on the prod cluster

# in another terminal
$ eval $(./tsh_kube_login.sh staging)

$ kubectl ... # runs on the staging cluster

[Joerger]

• existing $KUBECONFIG should be handled carefully - if it points at a file under ~/.tsh/keys/..., we shouldn't treat it as the top-level ~/.kube/config

What should be treated as the top-level ~/.kube/config in this case? It seems risky to default to ~/.kube/config, since the user would expect that the config set under $KUBECONFIG would be the only one touched.

I see two other options:

1. skip procedures that are meant for top-level kubeconfigs, such as adding all kube clusters to the kubeconfig and changing contexts.
   - For example, if someone runs tsh kube login staging while $KUBECONFIG=~/.tsh/.../prod-kubeconfig, It would fail since there is no top-level kubeconfig to edit.
   • They would need to reset their $KUBECONFIG so that a non-profile kubeconfig can be used.
2. allow users to set $KUBECONFIG as a list. the first one will be the top-level kubeconfig, and the second will be the profile specific kubeconfig.
   - tsh env will instead print export KUBECONFIG=$(KUBECONFIG),~/.tsh/....
   • Note that if $KUBECONFIG is unset, it will default to ~/.kube/config.

Edit: I went with the first option which should be much more simple to use and keep track of.