support proxy db tunnel mfa access

[GavinFrazar]

Closes #12538

This PR adds MFA support for tsh proxy db --tunnel - the basic problem with MFA is that the certs are short-lived and expire after 1 minute, which means the local proxy only worked for 1 minute before this PR.

I added middleware to alpnproxy.LocalProxy which is invoked when the proxy starts and then on each new connection to check/renew database certs.

The database certs are kept in memory only, in preparation for approval of RFD 90 #16739.
The main thing I've left out of this PR is adjusting the TTL for database mfa certs, as that will required RFD approval before implementing.
edit: For now, the UX improves from a broken local proxy every minute, to an MFA check every minute. The RFD will significantly improve the UX further, I'm making this PR sooner than RFD approval since the middleware is needed to unblock Teleport Connect team.

@ravicious

edit:

UX Change

Before

local proxy:

[I] [14:33:33] gavin@mac ~/work/teleport  
$ tsh proxy db --port 8080 --tunnel teleport-mysql                                                  (base) 
Tap any security key
Started authenticated tunnel for the MySQL database "teleport-mysql" in cluster "gavin-dev.cloud.gravitational.io" on 127.0.0.1:8080.

Use the following command to connect to the database:
  $ mysql --port 8080 --host localhost --protocol TCP

mysql:

[I] [14:32:36] gavin@mac ~/work/teleport  
$ echo "select 1;" | mysql -h localhost -P 8080 --protocol TCP -u teleport50
1
1
[I] [14:34:36] gavin@mac ~/work/teleport  
$ echo "select 1;" | mysql -h localhost -P 8080 --protocol TCP -u teleport50
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 57

After:

local proxy:

[I] [14:31:26] gavin@mac ~/work/teleport  
$ tsh proxy db --port 8080 --tunnel teleport-mysql
Started authenticated tunnel for the MySQL database "teleport-mysql" in cluster "gavin-dev.cloud.gravitational.io" on 127.0.0.1:8080.

Use the following command to connect to the database:
  $ mysql --port 8080 --host localhost --protocol TCP
MFA is required to access database "teleport-mysql"
Tap any security key
Database certificate renewed: valid until 2022-10-05T21:32:30Z [valid for 1m0s]
MFA is required to access database "teleport-mysql"
Tap any security key
Database certificate renewed: valid until 2022-10-05T21:33:32Z [valid for 1m0s]

mysql:

[I] [14:31:41] gavin@mac ~/work/teleport  
$ echo "select 1;" | mysql -h localhost -P 8080 --protocol TCP -u teleport50
1
1
[I] [14:32:30] gavin@mac ~/work/teleport  
$ echo "select 1;" | mysql -h localhost -P 8080 --protocol TCP -u teleport50
1
1

And local proxy without MFA required (doesn't print a prompt context message before "Tap any security key"):

[I] [14:35:38] gavin@mac ~/work/teleport  
$ tsh proxy db --port 8080 --tunnel teleport-mysql
Started authenticated tunnel for the MySQL database "teleport-mysql" in cluster "gavin-dev.cloud.gravitational.io" on 127.0.0.1:8080.

Use the following command to connect to the database:
  $ mysql --port 8080 --host localhost --protocol TCP
Database certificate renewed: valid until 2022-10-06T07:27:21Z [valid for 9h52m0s]

OTP local proxy works too, but you need to pass an mfa auth preference (this is true for tsh db login as well)

$ tsh proxy db --port 8080 --tunnel teleport-mysql --auth=local --mfa-mode=otp
Started authenticated tunnel for the MySQL database "teleport-mysql" in cluster "gavin-dev.cloud.gravitational.io" on 127.0.0.1:8080.

Use the following command to connect to the database:
  $ mysql --port 8080 --host localhost --protocol TCP
MFA is required to access database "teleport-mysql"
Enter an OTP code from a device:
Database certificate renewed: valid until 2022-10-06T00:16:11Z [valid for 1m0s]
MFA is required to access database "teleport-mysql"
Enter an OTP code from a device:
Database certificate renewed: valid until 2022-10-06T00:17:18Z [valid for 1m0s]

[GavinFrazar]

@greedy52 @smallinsky this is ready I think.

I've addressed your feedback regarding adding some context before prompting the user for MFA and I've made the integration test more robust using a fake clock to mock cert expiration.

[smallinsky]

@GavinFrazar
In case of normal flow without MFA can we skip renewal of database certs if certs was already generated and are valid ?
For instance:
The certs generated by tsh db connect --db-user=marek --db-name=postgres postgres01

➜  tsh git:(gavinfrazar/proxy_db_tunnel_mfa_access) ✗ ./tsh db connect --db-user=marek --db-name=postgres postgres01
psql (14.4, server 13.4)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, bits: 128, compression: off)
Type "help" for help.

can be reused for tsh proxy db though in current flow always renews the certs:

postgres=> ^D\q
➜  tsh git:(gavinfrazar/proxy_db_tunnel_mfa_access) ✗ ./tsh proxy db postgres01 --tunnel  -p 1111
Started authenticated tunnel for the PostgreSQL database "postgres01" in cluster "root" on 127.0.0.1:1111.

Use the following command to connect to the database:
  $ psql postgres://marek@localhost:1111/postgres

Local proxy tunnel requires credentials to access database "postgres01"
Local proxy tunnel credentials successfully renewed

Also the:

``Local proxy tunnel requires credentials to access database "postgres01"` log entry might be a bit confusing for a user. I would keep this more like a debug log because no user action is needed in this case.

In case of MFA flow right now following message is printed:

local proxy tunnel requires credentials to access database "teleport-mysql"
Enter an OTP code from a device:

Could we emphasise in the log entry that local proxy that this is MFA flow ?

@xinding33 Could you take a look at UX changes ?

[GavinFrazar]

@smallinsky

In case of normal flow without MFA can we skip renewal of database certs if certs was already generated and are valid ?

Done.

Could we emphasize in the log entry that local proxy that this is MFA flow ?

Done. See:

MFA is required to access database "teleport-mysql"
Tap any security key
Database certificate renewed: valid until 2022-10-06T18:23:53Z [valid for 1m0s]

[smallinsky]

Linter seems to fail on a = a + 1 rule but otherwise LGTM.

[github-actions]

@GavinFrazar See the table below for backport results.

Branch	Result
branch/v11	Create PR