error: EOF

[capripot]

What happened:
After upgrading from teleport 4.0.4 to 4.0.6, the following error happens when I try to remotely connect to my teleport pod. Same behavior is observed with the versions 3.1.11 and 3.2.9.

$ /usr/local/bin/tsh --proxy=teleport.company.com:443 [email protected] --login=root ssh company-services-console-9ffd9558f-qhtjr
error: EOF

Note that this works fine:

$ /usr/local/bin/tsh --proxy=teleport.company.com:443 [email protected] --login=root ls

What you expected to happen:
It should ssh into the instance, showing a bash prompt.

How to reproduce it (as minimally and precisely as possible):
Upgrade to 4.0.6, 3.1.11 or 3.2.9 and execute same command

Environment:

• Teleport server version: Teleport v3.0.1
• Tsh client version: Teleport v4.0.6 git: go1.13
• OS: macOS 10.14.6 (18G95) / installed with brew

Relevant Debug Logs If Applicable

• tsh --debug

INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/private/tmp/com.apple.launchd.vbNivh6bXY/Listeners" client/api.go:1940
DEBU [KEYSTORE]  Returning SSH certificate "/Users/user/.tsh/keys/teleport.company.com/[email protected]" valid until "2019-09-07 02:56:59 -0700 PDT", TLS certificate "/Users/user/.tsh/keys/teleport.company.com/[email protected]" valid until "2019-09-07 09:56:59 +0000 UTC". client/keystore.go:262
INFO [KEYAGENT]  Loading key for "[email protected]" client/keyagent.go:108
INFO [CLIENT]    Connecting proxy=teleport.company.com:3023 login='root' method=0 client/api.go:1483
DEBU [KEYAGENT]  Validated host teleport.company.com:3023. client/keyagent.go:280
INFO [CLIENT]    Successful auth with proxy teleport.company.com:3023 client/api.go:1474
DEBU [CLIENT]    Found clusters: [{"name":"staging","lastconnected":"2019-09-06T22:11:03.892352227Z","status":"online"}] client/client.go:104
INFO [CLIENT]    Client= connecting to node=company-service-console-9ffd9558f-qhtjr:0@default@staging client/client.go:363
DEBU [KEYAGENT]  Validated host company-service-console-9ffd9558f-qhtjr:0@default@staging. client/keyagent.go:280

ERROR REPORT:
Original Error: *errors.errorString EOF
Stack Trace:
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/lib/client/session.go:256 github.com/gravitational/teleport/lib/client.(*NodeSession).allocateTerminal
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/lib/client/session.go:201 github.com/gravitational/teleport/lib/client.(*NodeSession).interactiveSession
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/lib/client/session.go:398 github.com/gravitational/teleport/lib/client.(*NodeSession).runShell
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/lib/client/api.go:1385 github.com/gravitational/teleport/lib/client.(*TeleportClient).runShell
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/lib/client/api.go:915 github.com/gravitational/teleport/lib/client.(*TeleportClient).SSH
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/tool/tsh/tsh.go:714 main.onSSH.func1
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/lib/client/api.go:301 github.com/gravitational/teleport/lib/client.RetryWithRelogin
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/tool/tsh/tsh.go:713 main.onSSH
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/tool/tsh/tsh.go:324 main.Run
	/private/tmp/teleport-20190906-14721-19gxw6d/teleport-4.0.6/src/github.com/gravitational/teleport/tool/tsh/tsh.go:174 main.main
	/usr/local/opt/go/libexec/src/runtime/proc.go:212 runtime.main
	/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1358 runtime.goexit
User Message:

[sebdoido]

Same issue, i had to rollback to v4.0.4

[capripot]

I also tried 3.1.11 and 3.2.9, and it gave me the same error. Seams that only 3.0 head (3.0.6) works, beside of non-head version like 4.0.4.

[fspmarshall]

Pretty sure this is being caused by de66044, which is part of a larger collection of security fixes introduced in #2954. Looks like using a patched version of tsh with an upatched teleport instance is causing some disagreement on the formatting of session IDs.

ping: @russjones

edit: relevant server-side error: unable to update context: invalid session id