Gracefully handle corrupted private keys

[atburke]

This change updates tsh to gracefully handle corrupted private key files. Previously, if the key was corrupted, every tsh command would fail (including logout), and the only way to fix it would be to manually remove ~/.tsh. Now, if the key is corrupted tsh logout will successfully log out, and any command that calls RetryWithRelogin will also work properly.

Part of #59549.

Changelog: Fixed corrupted private keys breaking tsh

Test Plan

Assume each test case starts logged in with a manually corrupted key (this can be done by running echo "some data" > ~/.tsh/keys/<proxy-addr>/<teleport-user>.

• tsh status reports that the session is no longer valid.
• tsh logout successfully logs out the user.
• A command with RetryWithRelogin (e.g. tsh ls) successfully logs in when the key is corrupted.

[zmb3]

Do we also plan to address the cause of the corruption?

[atburke]

Do we also plan to address the cause of the corruption?

Not in this change. My understanding is that #59549 only requests handling the key once corrupted, as we don't have a way to properly reproduce the corruption.

[zmb3]

Nic suspects the corruption comes from here:

teleport/lib/utils/fs.go

Lines 339 to 352 in 8489b48

	if runtime.GOOS == "windows" {
	// Windows can't unlink the file before overwriting.
	if f != nil {
	for range 3 {
	if err := overwriteFile(f, fi); err != nil {
	break
	}
	}
	}
	// The file should be closed before removing it on Windows.
	closeErr := trace.ConvertSystemError(f.Close())
	removeErr := trace.ConvertSystemError(os.Remove(filePath))
	return trace.NewAggregate(closeErr, removeErr)
	} else {

[atburke]

I agree that the corruption likely happens there, due to os.Remove() failing when there's still an open handle to the file. The trouble is finding that open handle; I haven't been able to find anywhere where we either forget to close or are concurrently operating on the key file that would trigger that scenario.

[nklaassen]

I agree that the corruption likely happens there, due to os.Remove() failing when there's still an open handle to the file. The trouble is finding that open handle; I haven't been able to find anywhere where we either forget to close or are concurrently operating on the key file that would trigger that scenario.

os.Remove will fail even if another process has the file open. It could be Connect or another invocation of tsh or tsh proxy or a tctl command or really any other process