DB multi-session MFA Part 1: implement basic "tsh db exec"

[greedy52]

related:

• Multi-session MFA for database access #51679
• RFD 202 - Database Multi-session MFA #52358

This PR implements basic tsh db exec without the MFA part

Usage examples:

Simple: selection by names and sequential

$ cat test.sql 
select @@hostname

$ tsh db exec "source test.sql"  --dbs mysql1,mysql2 --db-user teleport-admin
Fetching databases ...

Executing command for "mysql1".
@@hostname
64f979fb76c9

Executing command for "mysql2".
@@hostname
6bc83bfda8d1

Summary: 2 of 2 succeeded.

Search by labels, and run in parallel

$ tsh db exec "source ./test.sql" --db-user teleport-admin --labels env=dev --max-connections 4 --output-dir logs
Searching databases ...
Found 4 database(s):

Name   Description Protocol Labels  
------ ----------- -------- ------- 
mysql1             mysql    env=dev 
mysql2             mysql    env=dev 
mysql3             mysql    env=dev 
mysql4             mysql    env=dev 

Do you want to proceed with 4 databases? [y/N]: y
Executing command for "mysql2". Output will be saved at "logs/mysql2.output".
Executing command for "mysql1". Output will be saved at "logs/mysql1.output".
Executing command for "mysql4". Output will be saved at "logs/mysql4.output".
Executing command for "mysql3". Output will be saved at "logs/mysql3.output".

Summary: 0 of 4 succeeded.
Summary is saved at "logs/summary.json".

$ cat exec-logs/mysql1.output
@@hostname
64f979fb76c9

Search by keyword, and sample summary.json

$ tsh db exec "source test.sql"  --search mymy --db-user teleport-admin --output-dir logs --no-confirm
Searching databases ...
Found 2 database(s):

Name   Description Protocol Labels   
------ ----------- -------- -------- 
mymy-1             mysql    env=prod 
mymy-2             mysql    env=prod 

Skipping confirmation for "Do you want to proceed with 2 databases?" due to the --no-confirm flag.
Executing command for "mymy-1". Output will be saved at "logs/mymy-1.output".
Executing command for "mymy-2". Output will be saved at "logs/mymy-2.output".

Summary: 2 of 2 succeeded.
Summary is saved at "logs/summary.json".

$ cat logs/summary.json 
{
  "databases": [
    {
      "database": {
        "service_name": "mymy-1",
        "protocol": "mysql",
        "username": "teleport-admin"
      },
      "command": "source test.sql",
      "output_file": "/Users/stevehuang/workspace/teleport-data/multi-session/logs/mymy-1.output",
      "success": true,
      "exit_code": 0
    },
    {
      "database": {
        "service_name": "mymy-2",
        "protocol": "mysql",
        "username": "teleport-admin"
      },
      "command": "source test.sql",
      "output_file": "/Users/stevehuang/workspace/teleport-data/multi-session/logs/mymy-2.output",
      "success": true,
      "exit_code": 0
    }
  ],
  "success": 2,
  "failure": 0,
  "total": 2
}

[Tener]

Really cool feature, I can see using it myself in the future 👍

I gave it some thought though and I think we should optimise for multi-database, scriptable experience. To me, that means a few features:

1. Fault tolerance. It shouldn't be a fatal error if a subset of databases is unreachable.
2. Machine readable output. As a user, I want predictable way of processing the script results further. More on that below.
3. Some sort of summary once execution finishes. For each accessed database information about command executed, output files (if any), time, exit code etc.
4. (Nice to have) Configurable timeouts for individual commands.
5. (Nice to have) Configurable command to run.
6. (Nice to have) A way to see which database names were provided, but couldn't be found; I guess a line in execution summary?

Two main modes for output, to me, are:

1. Split output

• one file per executed command.
• based in output dir
• execution summary file, if requested, json format

3. Combined output

• default to stdout, but may be redirected to file using flag. MFA prompts etc. would still use stdout!
• each line prefixed with the unique database name
• may use JSON/CSV(?)
• also optional summary

[greedy52]

Friendly ping @gzdunek @r0mant

[r0mant]

lgtm with a few comments

[r0mant]

Let's try to make errors less generic.

Suggested change

	return nil, trace.BadParameter("unsupported database protocol: %v", c.db)
	return nil, trace.NotImplemented("%q databases do not support exec commands yet", c.db)

[greedy52]

it was pointed out earlier that tsh will catch NotImplemented error and print an error stating server needs to be updated:

teleport/tool/tsh/common/tsh.go

Lines 1697 to 1699 in 7648803

	if trace.IsNotImplemented(err) {
	return handleUnimplementedError(ctx, err, cf)
	}

I've updated the error message but kept trace.BadParameter for now. we should improve the way tsh handles NotImplemented in the future.

[r0mant]

Is there any sort of timeout on the execution?

[greedy52]

no. the timeout will vary depending on the command to be executed. we can add a new flag in the future if a user asks for it. for now, user can always ctrl-c to kill it.