tsh proxy jump fails with TLS routing enabled

[Joerger]

Related to #11271

Tracked in #13340

What happened:

When using tsh ssh -J leaf-proxy, the connection is successfully made through the leaf proxy, as it was fixed in #11282. However some connections made within the tsh ssh call still attempt to connect to the root proxy, such as the IsMFARequired check.

Note that this still works as expected with TLS Routing turned off.

What you expected to happen:

Jump host should be respected across the entire tsh ssh call, such that it works even with the root cluster completely shut down.

Reproduction Steps

As minimally and precisely as possible, describe step-by-step how to reproduce the problem.

1. Have a setup with a root cluster, a leaf cluster and a node in the leaf cluster.
2. Log into the root cluster: tsh login --proxy=root-proxy --auth=local --user=alice.
3. Log into the leaf cluster: tsh login leaf.
4. Try to use proxy jump: tsh ssh -J leaf-proxy root@node-1.
5. Observe it tries to connect via root proxy. For better effect visibility, shut down root proxy before tsh ssh.

Server Details

• Teleport version (run teleport version): v10

Client Details

• Tsh version (tsh version): v10

Debug Logs

With root cluster shut down:

> tsh ssh -J proxy.example.com:5080 remote
...
INFO [CLIENT]    Successful auth with proxy proxy.remote.com:5080. client/api.go:2965
DEBU [CLIENT]    Found clusters: [{"name":"remote.com","lastconnected":"2022-06-15T17:58:22.507969222-07:00","status":"online"}] client/client.go:128
INFO [CLIENT]    Client= connecting to node=remote on cluster remote.com client/client.go:1422
INFO [CLIENT]    Client= jumping to node=remote on cluster remote.com client/client.go:1565
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/home/bjoerger/.tsh/keys/proxy.example.com/dev-x509.pem" valid until "2022-06-16 08:58:08 +0000 UTC". client/keystore.go:306
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/home/bjoerger/.tsh/keys/proxy.example.com/dev-x509.pem" valid until "2022-06-16 08:58:08 +0000 UTC". client/keystore.go:306
DEBU [KEYSTORE]  Reading certificates from path "/home/bjoerger/.tsh/keys/proxy.example.com/dev-ssh/remote.com-cert.pub". client/keystore.go:329
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/home/bjoerger/.tsh/keys/proxy.example.com/dev-x509.pem" valid until "2022-06-16 08:58:08 +0000 UTC". client/keystore.go:306

ERROR REPORT:
Original Error: *trace.ConnectionProblemError connection error: desc = "transport: Error while dialing failed to dial: dial tcp 127.0.0.1:3080: connect: connection refused"
Stack Trace:
	/home/bjoerger/gravitational/teleport/api/client/client.go:1544 github.com/gravitational/teleport/api/client.(*Client).IsMFARequired
	/home/bjoerger/gravitational/teleport/lib/client/client.go:394 github.com/gravitational/teleport/lib/client.(*ProxyClient).IssueUserCertsWithMFA
	/home/bjoerger/gravitational/teleport/lib/client/client.go:2027 github.com/gravitational/teleport/lib/client.(*ProxyClient).sessionSSHCertificate
	/home/bjoerger/gravitational/teleport/lib/client/client.go:1567 github.com/gravitational/teleport/lib/client.(*ProxyClient).PortForwardToNode
	/home/bjoerger/gravitational/teleport/lib/client/client.go:1424 github.com/gravitational/teleport/lib/client.(*ProxyClient).ConnectToNode
	/home/bjoerger/gravitational/teleport/lib/client/api.go:1851 github.com/gravitational/teleport/lib/client.(*TeleportClient).SSH
	/home/bjoerger/gravitational/teleport/tool/tsh/tsh.go:2484 main.onSSH.func1.1
	/home/bjoerger/gravitational/teleport/lib/client/api.go:719 github.com/gravitational/teleport/lib/client.RetryWithRelogin
	/home/bjoerger/gravitational/teleport/tool/tsh/tsh.go:2483 main.onSSH.func1
	/home/bjoerger/gravitational/teleport/tool/tsh/tsh.go:2391 main.retryWithAccessRequest
	/home/bjoerger/gravitational/teleport/tool/tsh/tsh.go:2482 main.onSSH
	/home/bjoerger/gravitational/teleport/tool/tsh/tsh.go:855 main.Run
	/home/bjoerger/gravitational/teleport/tool/tsh/tsh.go:396 main.main
	/home/bjoerger/.tools/go/1.18.2/src/runtime/proc.go:250 runtime.main
	/home/bjoerger/.tools/go/1.18.2/src/runtime/asm_amd64.s:1571 runtime.goexit
User Message: connection error: desc = "transport: Error while dialing failed to dial: dial tcp 127.0.0.1:3080: connect: connection refused"

[Joerger]

@codingllama it looks like the error I'm getting might be caused by your changes in #12475, which causes the MFA check to use the web proxy address set in the client rather than the one passed down.

teleport/lib/client/mfa.go

Line 90 in 972872a

return PromptMFAChallenge(ctx, c, tc.WebProxyAddr, opts)

[codingllama]

After looking into this I think #12475 is a red herring. From the stack traces, the failure stems from api/client.(*Client).IsMFARequired, which means tsh is still trying to decide whether to prompt for MFA or not.

Digging a bit deeper I found that TLS routing / non-TLS routing code paths use different addresses:

• ConnectToAuthServiceThroughALPNSNIProxy uses proxy.teleportClient.WebProxyAddr, aka the "root" cluster in the example above:

  teleport/lib/client/client.go

  Line 1092 in e5c745f

  Addrs: []string{proxy.teleportClient.WebProxyAddr},

• dialAuthServer uses proxy.proxyAddress, aka the "leaf" cluster:

  teleport/lib/client/client.go

  Line 1310 in e5c745f

  localAddr, err := utils.ParseAddr("tcp://" + proxy.proxyAddress)

It looks correct to me to check MFA against the "root" cluster, as that is the cluster where the user is registered (and thus, the cluster where their credentials will be). That said, I don't think ConnectThroughALPN ever respects the jump host.

I'm not sure what the correct approach is here, so pinging @r0mant and @smallinsky.

[smallinsky]

It looks correct to me to check MFA against the "root" cluster, as that is the cluster where the user is registered (and thus, the cluster where their credentials will be)

The TLS routing flow does the MFA check agains leaf cluster in case of JumpHost == left cluster proxy address Otherwise the MFA check should also fail if root cluster is offline. I think that we should align the TLS Routing flow and does the same think like in former flow where TLS routing is disabled.

I have made a fix for that PR13791

[smallinsky]

Fixed in #13791