'tsh login' fails to output identity to a file when using the --out parameter

[webvictim]

What happened: Attempting to log into a Teleport cluster and output the identity to a file results in an x509 error.

$ tsh --debug login --proxy=example.gravitational.co --out=output.pem
DEBU [KEYSTORE]  Returning SSH certificate "/home/user/.tsh/keys/example.gravitational.co/[email protected]" valid until "2020-02-11 04:21:21 -0400 AST", TLS certificate "/home/user/.tsh/keys/example.gravitational.co/[email protected]" valid until "2020-02-11 08:21:21 +0000 UTC". client/keystore.go:262
INFO [CLIENT]    no host login given. defaulting to user client/api.go:784
INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/tmp/ssh-5dnqvjP9Nmb2/agent.3359" client/api.go:2049
DEBU [KEYSTORE]  Returning SSH certificate "/home/user/.tsh/keys/example.gravitational.co/[email protected]" valid until "2020-02-11 04:21:21 -0400 AST", TLS certificate "/home/user/.tsh/keys/example.gravitational.co/[email protected]" valid until "2020-02-11 08:21:21 +0000 UTC". client/keystore.go:262
INFO [KEYAGENT]  Loading key for "[email protected]" client/keyagent.go:108
DEBU [CLIENT]    not using loopback pool for remote proxy addr: example.gravitational.co:3080 client/api.go:2010
DEBU [CLIENT]    HTTPS client init(proxyAddr=example.gravitational.co:3080, insecure=false) client/weblogin.go:252
DEBU [CLIENT]    samlLogin start client/api.go:1955
DEBU [CLIENT]    not using loopback pool for remote proxy addr: example.gravitational.co:3080 client/api.go:2010
DEBU [CLIENT]    HTTPS client init(proxyAddr=example.gravitational.co:3080, insecure=false) client/weblogin.go:252
INFO [CLIENT]    Waiting for response at: http://127.0.0.1:34931. client/redirect.go:128
If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:34931/9d55676d-ea89-482e-95c6-a6727a70d8a6
DEBU [CLIENT]    Got response from browser. client/weblogin.go:387
INFO [CLIENT]    Connecting proxy=example.gravitational.co:3023 login='root' method=0 client/api.go:1588
DEBU [KEYAGENT]  Validated host example.gravitational.co:3023. client/keyagent.go:280
INFO [CLIENT]    Successful auth with proxy example.gravitational.co:3023 client/api.go:1579
DEBU [CLIENT]    Client  is connecting to auth server on cluster "example.gravitational.co". client/client.go:469

ERROR REPORT:
Original Error: *trace.ConnectionProblemError x509: certificate signed by unknown authority
Stack Trace:
	/gopath/src/github.com/gravitational/teleport/lib/httplib/httplib.go:110 github.com/gravitational/teleport/lib/httplib.ConvertResponse
	/gopath/src/github.com/gravitational/teleport/lib/auth/clt.go:350 github.com/gravitational/teleport/lib/auth.(*Client).Get
	/gopath/src/github.com/gravitational/teleport/lib/auth/clt.go:540 github.com/gravitational/teleport/lib/auth.(*Client).GetCertAuthorities
	/gopath/src/github.com/gravitational/teleport/lib/client/api.go:1768 github.com/gravitational/teleport/lib/client.(*TeleportClient).GetTrustedCA
	/gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:475 main.onLogin
	/gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:340 main.Run
	/gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:186 main.main
	/opt/go/src/runtime/proc.go:212 runtime.main
	/opt/go/src/runtime/asm_amd64.s:1358 runtime.goexit
User Message: Get https://teleport.cluster.local/v2/authorities/host?load_keys=false: x509: certificate signed by unknown authority

/etc/teleport.yaml:

teleport:
  log:
    output: stderr
    severity: DEBUG
  data_dir: /var/lib/teleport
  storage:
    type: dir

auth_service:
  enabled: true
  license_file: /var/lib/license/license.pem
  authentication:
    type: oidc
  public_addr: example.gravitational.co:3025
  cluster_name: example.gravitational.co

ssh_service:
  enabled: yes
  public_addr: 127.0.0.1:3022
  labels:
    environment: demo
    role: auth
  commands:
  - command:
    - uptime
    - -p
    name: uptime
    period: 1m
  - command:
    - uname
    - -r
    name: kernel
    period: 1h

proxy_service:
  enabled: true
  public_addr: example.gravitational.co:3080
  ssh_public_addr: example.gravitational.co:3023
  web_listen_addr: 0.0.0.0:3080
  listen_addr: 0.0.0.0:3023
  https_key_file: /var/lib/certs/privkey.pem
  https_cert_file: /var/lib/certs/fullchain.pem
  kubernetes:
    enabled: true
    listen_addr: 0.0.0.0:3026
    public_addr: example.gravitational.co:3026

OpenSSL verify (to show that the web-presented cert verifies OK):

$ openssl s_client -connect example.gravitational.co:3080 | grep Verify
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = example.gravitational.co
verify return:1
Verify return code: 0 (ok)

What you expected to happen: The login to complete successfully and the identity file to be written to output.pem

How to reproduce it (as minimally and precisely as possible): Run the above command against a properly-configured Teleport cluster.

Environment:

• Teleport version (use teleport version): Teleport Enterprise v4.2.1git:v4.2.1-0-gb16ee7d7 go1.13.2
• Tsh version (use tsh version): Teleport v4.2.2 git:v4.2.2-0-gb06a05d2 go1.13.2
• OS (e.g. from /etc/os-release): Fedora 30

[fspmarshall]

Did a quick experiment (fe55219) which does technically fix the issue. Looks like this issue is related to how tsh is configuring its client... Because the --out parameter causes tsh to not save certs to its local cache, subsequent API calls are breaking.

[webvictim]

Interesting. If you fancy PRing this and getting it into 4.3 I'm sure it'd be much appreciated.

[fspmarshall]

@webvictim fe55219 is a hack, and not appropriate for inclusion in an official release. It "fixes" the problem by smashing the local tsh cache. Nothing too nasty; just means that the user would need to re-login to their normal account after using tsh login --out. Nevertheless, I think we can do better.