Preserve roles from user cert when forwarding k8s requests

[awly]

When a proxy forwards a k8s request to another proxy in a trusted leaf
cluster, it dynamically generates a new client key/cert to impersonate
the user. This key/cert is presented to the leaf proxy as if a user was
directly making a request to it.

Auth server, when processing the CSR, would load user identity from the
backend. If this user had temporary role grants via workflow API, they
would not be loaded from the backend.

This means that if a user accesses k8s through:

kubectl -1-> root proxy -2-> leaf proxy -3-> k8s

the 2nd hop would drop their temporary role grants.

To fix this, preserve full user identity from the original client cert
presented to root proxy, as encoded in CSR Subject. The auth server
implicitly trusts the CSR Subject that the proxy presents.
This also requires fixing the Subject encoding on the proxy side, which
was inconsistent with how auth server does it.

One possible downside here is: a compromised proxy can mint k8s client certs for
arbitrary users with arbitrary (but existing) roles.

Fixes #3593

[webvictim]

This is great. Please wait for others before merging though.

[russjones]

@awly I don't suppose kubeconfig supports configuration which would allow us to use SNI to solve this problem? If it did, we could just connect the client and target proxy directly instead of going through certificate re-issue.

[awly]

@russjones interesting idea!
I believe kubectl uses the standard crypto/tls under the hood. crypto/tls always sends SNI unless InsecureSkipVerify is set or if ServerName contains an IP.

If we used SNI, it wouldn't work with:

• proxies sharing the same hostname (using different ports)
• root proxies that only have a public IP and not a hostname

which could be fine, but it's an awkward caveat to document

[webvictim]

A lot of people's Teleport PoCs/trials don't use domains set up with proper TLS at all, they're using IP addresses and --insecure. It would almost certainly make that sort of use case harder.

[fspmarshall]

Since this fix relies on updated proxy behavior, version compatibility is a concern. The auth server should fallback to the old strategy if it doesn't receive any roles from the proxy. You can add a // DELETE IN: 5.0 directive to the fallback logic.

One possible downside here is: a compromised proxy can mint k8s client certs for
arbitrary users with arbitrary (but existing) roles.

If a certificate was constructed in part based on access requests, the request IDs are preserved under the teleport-active-requests extension. The extension is currently only used by tsh, but it exists and could technically be used to solve this problem. Doesn't prevent a malicious proxy from deciding to impersonate a different user though, so its an open question whether the extra complexity is worth it. Also, keeping the user's cert as the source of truth is more consistent with how teleport operates everywhere else.

[awly]

A lot of people's Teleport PoCs/trials don't use domains set up with proper TLS at all, they're using IP addresses and --insecure. It would almost certainly make that sort of use case harder.

@webvictim would or wouldn't make it harder? I imagine we want to make trying the product easy but ensure that prod setups use good security practices.

Since this fix relies on updated proxy behavior, version compatibility is a concern. The auth server should fallback to the old strategy if it doesn't receive any roles from the proxy.

@fspmarshall ahh, good catch. I'll add some heuristic in auth server to detect a CSR from older proxy and fall back to manually producing the Subject. Stand by.

[webvictim]

@webvictim would or wouldn't make it harder? I imagine we want to make trying the product easy but ensure that prod setups use good security practices.

I suppose it wouldn't really make it harder. I agree with your overall assessment - an easyish trial but great production setup is exactly we want to do. Trialling the product without using DNS and valid TLS certs is already a tricky experience so it's unlikely to make a lot of difference.

[awly]

@klizhentas @russjones added UsageKubeOnly enforcement as we discussed.