Teleport connect on mac OS leaks domain sockets to SSH_AUTH_SOCK until ssh and teleport connect freezes

[andrew-h-teleport]

Expected behavior:
Teleport connect (via the tsh daemon) should close socket connections opened on the mac OS system SSH agent ($SSH_AUTH_SOCK) when instances finish work.

Current behavior:
When teleport connect is running, the tsh daemon adds a new socket to $SSH_AUTH_SOCK for every use action in the teleport connect UI, and automatically by background polling that needs no user interaction. Socket connections accumulate until hitting macOS ssh-agent's limit, which I observed to be 248 connections.

Once the connection limit is reached:
ssh-agent stops responding on macOS and connect's resource list will appear to load indefinitely.
External commands using SSH will also hang.
Exiting from teleport connect releases the sockets.

Example of what teleport connect looks like when the limit is reached:

Bug details:

• Teleport version (tsh and connect): 18.10.3
• Recreation steps:
  Open a macOS terminal, monitor open connections to $SSH_AUTH_SOCK:

while true; do echo "[$(date +'%H:%M:%S')] Open sockets: $(lsof "$SSH_AUTH_SOCK" 2>/dev/null | wc -l)"; sleep 5; done

Launch connect
Navigate around teleport connect (switch resource tabs, search servers/databases, switch clusters) for 5–10 minutes.
Observe the connection count until it caps out at 248.
Teleport connect will hang, other direct ssh commands will also hang
Quit teleport connect -> socket count will drop.

[12:03:50] Open connections to $SSH_AUTH_SOCK:      248
[12:03:55] Open connections to $SSH_AUTH_SOCK:      248
## <teleport connect process stopped here> ##
[12:04:00] Open connections to $SSH_AUTH_SOCK:        2
[12:04:05] Open connections to $SSH_AUTH_SOCK:        2

[zmb3]

@Joerger can you take a look at this and let me know what you think? It looks legit to me.

From what I can tell, sshagent.NewSystemAgentClient() opens a connection to the agent and never closes it.

It's a problem for both tsh and Teleport Connect, but doesn't really matter in tsh since the process is short-lived. Connect on the other hand ends up running this code for just about every RPC.

[Joerger]

@Joerger can you take a look at this and let me know what you think? It looks legit to me.

Yeah, I've confirmed it. It looks like we don't really have a concept of a TeleportClient being closed and releasing the ssh-agent conn and other resources, so I'd recommend that we resolve that directly to avoid future issues — there are likely a few other cases where we are leaking goroutines/clients, which of course we wouldn't see as an error like we do with this socket limit.

Alternatively we could quickly fix this by caching the system agent in the teleterm client.Config like we do for the ClientStore and its Hardware Key Agent.